View source for Labs/Ubiquity/Parser 2/Localization Tutorial

<small>If you are interested in command localization, please read the wiki pages on [[Ubiquity_0.5_Command_Localization_Tutorial|localizing commands]] and [[Ubiquity_0.5_Making_Commands_Localizable|making commands localizable]]. This entry is for adding your language to Parser 2, so you can use Ubiquity with the grammar of your language.</small>

== Introduction ==

Ubiquity's Parser 2 was written from the ground up with one of its greatest priorities being internationalization... not just making commands localizable, but actually making it so Parser 2 can be easily taught the grammars of other languages. Key to this undertaking is an idea from the [http://en.wikipedia.org/wiki/Principles_and_parameters Principles and Parameters] school of linguistics, that all languages' grammars are made up of the following: (from wikipedia)

* A finite set of fundamental '''principles''' that are common to all languages; e.g., that a sentence must always have a subject, even if it is not overtly pronounced. 
* A finite set of '''parameters''' that determine syntactic variability amongst languages; e.g., a binary parameter that determines whether or not the subject of a sentence must be overtly pronounced. 

Following this idea, we built a flexible universal parser, Parser 2, and pair it with a (often very small) set of individual language settings.

The result of this architecture is that it takes very little code to teach Parser 2 a new language. With a little bit of JavaScript and knowledge of and interest in your own language, you’ll be able to get at least rudimentary Ubiquity functionality in your language. Follow along in this step by step guide and please [http://ubiquity.mozilla.com/trac/ticket/662 submit your (even incomplete) language files].

== Set up your environment ==

If you’re new to Ubiquity core development, you’ll want to first read the [https://wiki.mozilla.org/Labs/Ubiquity/Ubiquity_0.1_Development_Tutorial Ubiquity Development Tutorial] to learn how to get a live copy of the Ubiquity repository using [http://en.wikipedia.org/wiki/Mercurial Mercurial].

As you read along, you may find it beneficial to follow along in some of the more complete language settings files included in Parser 2: [https://ubiquity.mozilla.com/hg/ubiquity-firefox/raw-file/tip/ubiquity/modules/parser/new/en.js English], [https://ubiquity.mozilla.com/hg/ubiquity-firefox/raw-file/tip/ubiquity/modules/parser/new/ja.js Japanese], [https://ubiquity.mozilla.com/hg/ubiquity-firefox/raw-file/tip/ubiquity/modules/parser/new/da.js Danish].

== Writing your language settings ==

=== The structure of the language file ===

Each language in Parser 2 gets its own settings file. You'll need to look up the [http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes ISO 639-1 code for your language]... Here we'll use English (code <code>en</code>) as an example here and the language settings file would then be called <code>en.js</code> and go in the <code>/ubiquity/modules/parser/new/</code> directory of the repository.

Here is the basic template for a Ubiquity Parser 2 language file:

  function makeParser() {
    var en = new Parser('en');
  ...
    return en;
  };

Everything here is wrapped in a factory function called <code>makeParser</code>. This function initializes the new <code>Parser</code> object with the appropriate language code, sets a bunch of parameters (elided above) and returns it. That's it!

Now let's walk through some of the parameters you must set to get your language working. For reference, the properties the language parser object is required to have are: <code>branching</code>, <code>anaphora</code>, and <code>roles</code>.

=== Identifying your branching parameter ===

  en.branching = 'right'; // or 'left'

One of the first things you'll have to set for your parser is '''the <code>branching</code> parameter'''. Ubiquity Parser 2 uses the branching parameter to decide which direction to look for an argument after finding a delimiter or "role marker" (most often, these are [http://en.wikipedia.org/wiki/adposition prepositions or postpositions]). For example, in English "from" is a delimiter for the <code>goal</code> role and its argument is on its right.

<table>
<tr><td>&nbsp;</td><td>&nbsp;</td><td colspan='2'>[[Image:Arrow-right.png]]</td></tr>
<tr><td><b>to</b></td><td>Mary</td><td><b>from</b></td><td>John</td></tr>
</table>

So "John" is a possible argument for the <code>source</code> role, but "Mary" should not be. Ubiquity can figure this out because English has the property <code>en.branching = 'right'</code>.

In Japanese, on the other hand, the argument of a delimiter like から ("from") is found on the left of that delimiter, so <code>en.branching = 'left'</code>.

<table>
<tr><td colspan='2'>[[Image:Arrow-left.png]]
</td><td>&nbsp;</td><td>&nbsp;</td></tr>
<tr><td>メアリー</td><td><b>-から</b></td><td>ジョン</td><td><b>-に</b></td></tr>
<tr><td>Mary</td><td><b>from</b></td><td>John</td><td><b>to</b></td></tr>
</table>

In general, if your language has prepositions, you should use <code>.branching = 'right'</code> and if your language has postpositions, you can use <code>.branching = 'left'</code>.

==== For more info ====

* see [http://en.wikipedia.org/wiki/Branching_%28linguistics%29 branching] on Wikipedia.

=== Defining your roles ===

  en.roles = [
    {role: 'goal', delimiter: 'to'},
    {role: 'source', delimiter: 'from'},
    {role: 'position', delimiter: 'at'},
    {role: 'position', delimiter: 'on'},
    {role: 'alias', delimiter: 'as'},
    {role: 'instrument', delimiter: 'using'},
    {role: 'instrument', delimiter: 'with'}
  ];

The second required property is the inventory of semantic roles and their corresponding delimiters. Each entry has a <code>role</code> from the [[https://wiki.mozilla.org/Labs/Ubiquity/Parser_2/Semantic_Roles|inventory of semantic roles]] and a corresponding delimiter. Note that this mapping can be many-to-many, i.e., each role can have multiple possible delimiters and different roles can have shared delimiters. Try to make sure to cover all of the roles in the [[Labs/Ubiquity/Parser_2/Semantic_Roles|inventory of semantic roles]].

==== For more info ====

* [http://mitcho.com/blog/projects/writing-commands-with-semantic-roles/ Writing commands with semantic roles], the original proposal
* [[Labs/Ubiquity/Parser_2/Semantic_Roles|Semantic Roles in Parser 2]]
* Wikipedia entry on [http://en.wikipedia.org/wiki/Thematic_relation themantic relations]

=== Entering your anaphora ("magic words") ===

  en.anaphora = ["this", "that", "it", "selection", "him", "her", "them"];

The final required property is the <code>anaphora</code> property which takes a list of "magic words". Currently there is no distinction between all the different [http://en.wikipedia.org/wiki/Deixis deictic] [http://en.wikipedia.org/wiki/Anaphora_%28linguistics%29 anaphora] which might refer to different things.

== Register your language ==

Before testing out your new language settings file, you must register that language with the parser. There is a parser resgistry file at <code>ubiquity/modules/parser/new/parser_registry.json</code>. Open it up and add a new line to the JSON object mapping your language code to the native name of your language or locale. For example, if we wanted to add Danish (language code <code>da</code>), we could add the following line:

  da: "Dansk",

== Special cases ==

Some special language features can be handled by overriding the default behavior from <code>Parser</code>. Please note that the exact implementation of a number of these features are still in flux.

=== Languages with no spaces ===

If your language does not delimit arguments (or words, more generally) with spaces, there will be a need to write a custom <code>wordBreaker()</code> method and set <code>usespaces = false</code> and <code>joindelimiter = '</code><code>'</code>. For an example, please take a look at the [https://ubiquity.mozilla.com/hg/ubiquity-firefox/raw-file/tip/ubiquity/modules/parser/new/ja.js Japanese] or [https://ubiquity.mozilla.com/hg/ubiquity-firefox/raw-file/tip/ubiquity/modules/parser/new/zh.js Chinese].

=== Case marking languages ===

In general, the plan for Parser 2 is to not try to attempt to handle strongly case marked languages, and to instead encourage the use of adpositions (prepositions or postpositions) as role markers. For more information, please read [http://mitcho.com/blog/projects/in-case-of-case/ In Case of Case...].

=== Stripping articles ===

Some languages have some delimiters which combine with articles. For example, in French, the preposition "à" combines with the masculine definite article "le" but not "la":

# à + la = à la
# à + le = au

You can add both "à" and "au" as delimiters of the `goal` role, but then you will get feminine arguments back with the determiner (e.g. "la table") while masculine arguments would be parsed without a determiner (e.g. "chat").

# "'''à''' la table" = "'''to''' the table"
# "'''au''' chat" = "'''to the''' cat"

These types of ''portmanteau'ed prepositions'' can be handled through a process of argument normalization. Each language's parser can optionally define a <code>normalizeArgument()</code> method which takes an argument and returns a list of normalized alternates. Normalized arguments are returned in the form of <code>{prefix: ' ', newInput: ' ', suffix: ' '}</code>. For example, if you feed "la table" to the French <code>normalizeArgument()</code>, it ought to return 

  [{prefix: 'la ', newInput: 'table', suffix: ''}]

If there are no possible normalizations, <code>normalizeArgument()</code> should simply return <code>[]</code>. Each alternative returned by <code>normalizeArgument()</code> is substituted into a copy of the possible parses just before nountype detection. The prefixes and suffixes are stored in the argument (as <code>inactivePrefix</code> and <code>inactiveSuffix</code>) so they can be incorporated into the suggestion display.

Here, for example, is how the inactive prefix "l'" is displayed in the parser playpen (described below). This way the user is told that the "l'" prefix is being ignored, and the nountype detection and verb action can act on the argument "English". (In the future, of course, we could teach this nountype to accept the Catalan "anglès".)

[[Image:Catalan_portmanteau.png]]

The easiest way to produce this output is to use the [https://developer.mozilla.org/en/Core_JavaScript_1.5_Reference/Global_Objects/String/match <code>String.match()</code>] method. For example <code>normalizeArgument()</code> code, take a look at the [http://ubiquity.mozilla.com/hg/ubiquity-firefox/file/tip/ubiquity/modules/parser/new/ca.js Catalan] and [http://ubiquity.mozilla.com/hg/ubiquity-firefox/file/12f5d9abf011/ubiquity/modules/parser/new/fr.js French] parser files.

== Test your parser ==

Now you can go into your <code>about:config</code> page and change the value of "extensions.ubiquity.language" to your language code and restart. All the verbs and nountypes at this point will remain the same as in the English version, but it should obey the argument structure (the word order and delimiters) of your language.

[[Image:Ubiquity_Parser_2_Playpen.png|300px]]

You can also test your parser in the Parser 2 Playpen at <code>chrome://ubiquity/content/playpen.html</code>. There's a video explaining [http://vimeo.com/5013787 how you can use the Parser Playpen].

== Conclusion ==

If you run into any trouble, feel free to ask for help on the [http://groups.google.com/group/ubiquity-i18n Ubiquity i18n listhost] or find mitcho on the Ubiquity IRC channel (mitcho @ irc.mozilla.org#ubiquity). Of course, once you're at a good stopping point, please [http://ubiquity.mozilla.com/trac/ticket/662 contribute your language file to Ubiquity].

The next logical step to getting a better Ubiquity experience in your language is to [[Labs/Ubiquity/Ubiquity_0.5_Command_Localization_Tutorial|localize commands]].