MozillaWiki

WebAPI/KeboardIME: Difference between revisions

Page Discussion

WebAPI/KeboardIME (view source)

Revision as of 03:35, 3 November 2014

1,517 bytes removed ,  3 November 2014
Remove the IDL draft
(Remove the IDL draft)
 
(47 intermediate revisions by 8 users not shown)
Line 1: Line 1:
= Virtual Keyboard/IME API =


__TOC__
__TOC__
Line 18: Line 17:
# [https://groups.google.com/forum/?fromgroups#!topic/mozilla.dev.webapi/Vs3-HGv9NNw WebAPI mailing list post]
# [https://groups.google.com/forum/?fromgroups#!topic/mozilla.dev.webapi/Vs3-HGv9NNw WebAPI mailing list post]
# [https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.webapi/A7dIBaR3lpU Extended API mailing list post]
# [https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.webapi/A7dIBaR3lpU Extended API mailing list post]
# [https://groups.google.com/forum/#!msg/mozilla.dev.webapi/nZLOEvEL3Tk/M-nSnP6HZgoJ Polished Keyboard API]


Implementation:
Implementation:
#{{bug|737110}} - Bug 737110 - Virtual Keyboard API
#{{bug|737110}} - Bug 737110 - Virtual Keyboard API
#{{bug|805586}} - [keyboard] keyboard needs a 'hide keyboard' button(main tracking bug)
#{{bug|844716}} - Enable keyboard Apps to get/set selection range of the input field
#{{bug|860546}} - [keyboard] JS changes to a textfield while keyboard is displayed do not get passed to keyboard
#{{bug|861665}} - Allow IME to get notification when text field content is changed
#{{bug|861515}} - Keyboard should be able to modify the text of the input field directly
#{{bug|838308}} - mozKeyboard should require a permission to use
#{{bug|842436}} - Keyboard API: There should only be one keyboard active, and Gecko should block interaction from non-active keyboards


== Features ==
== Features ==
Line 33: Line 26:
The Virtual Keyboard/IME API supports the following features:
The Virtual Keyboard/IME API supports the following features:


* Notifies the VKB app when the focus text field was changing in other
* Notifies the VKB app when the focus text field was changing in other apps
apps
* Allow user to manual hide the keyboard. Check {{bug|737110}}.
* Allow user to manual hide the keyboard. Check {{bug|737110}}.
* The VKB app should be responsive to properties and the state of the  input field (more than HTML5 input type, including current content,  cursor position, x-inputmode {{bug|796544}}).
* The VKB app should be responsive to properties and the state of the  input field (more than HTML5 input type, including current content,  cursor position, x-inputmode {{bug|796544}}).
Line 45: Line 37:
* The return key label of the VKB can be customized.
* The return key label of the VKB can be customized.


== Proposed API ==
== Proposed Manifest of a 3rd-Party IME ==
 
Just like any other apps, keyboard apps register themselves in the same apps registry. We extend the manifest syntax here to describe layout(s) available in a given keyboard app. Gaia will be paring the manifest. There are 3 special fields to distinguish and describe a 3rd-party IME:
* [Line 4] a "role" field with value "keyboard" declares it's an IME app. Homescreen app will ignore some role types when displaying app icons, and "keyboard" is one of them. (see {{bug|892397}})
* [Line 6-8] a "permissions" field that requests "keyboard" permission. All IME apps need this permission for sending input keys and updating the value of a input field.
* [Line 9-30] a "inputs" field specifies supported layouts. Each layout is described in a key-value pair, where the key represents the layout name (will be shown up on Settings app with the app name), and the value describes the detailed information of the layout, including launch path of the layout and supported input types. (See [[#Layout Matching Algorithm]])
** The allowed value in "types" field is a subset of [http://www.whatwg.org/specs/web-apps/current-work/multipage/the-input-element.html#attr-input-type type attribute of input element]: text, search, tel, number, url, email. Other types will be ignored by FxOS Gaia in the initial version because at this point UI for <select> and <input type=date> (called "value selectors") are not open for 3rd-party implementation.
 
{
  "name": "3rd-party Keyboard",
  "description": "3rd-party Keyboard",
  "type": "privilege",
  "role": "input",
  "launch_path": "/settings.html",
  "developer": {
    "name": "developer's name",
    "url": "https://keyboard.example.com"
  },
  "permissions": {
    "input": {}
  },
  "inputs": {
    "en": {
      "launch_path": "/index.html#en",
      "name": "English",
      "description": "English layout",
      "types": ["url", "text"],
      "locales": {
        "en-US": {
          "name": "English",
          "description": "English layout"
        },
        "zh-TW": {
          "name": "英文",
          "description": "英文鍵盤"
        }
      }
    },
    "en-Dvorak": {
      "launch_path": "/index.html#en-Dvorak",
      "name": "English (Dvorak)",
      "description": "Dvorak layout",
      "types": ["url", "text"]
    },
    "es": {
      "launch_path": "/index.html#es",
      "name": "Spanish",
      "description": "Spanish layout",
      "types": ["url", "text"]
    },
    "pt-BR": {
      "launch_path": "/index.html#pt-BR",
      "name": "Portuguese Brazilian",
      "description": "Portuguese Brazilian layout",
      "types": ["url", "text"]
    },
    "pl": {
      "launch_path": "/index.html#pl",
      "name": "Polish",
      "description": "Polish layout",
      "types": ["url", "text"]
    },
    "ca": {
      "launch_path": "/index.html#ca",
      "name": "Catalan",
      "description": "Catalan layout",
      "types": ["url", "text"]
    },
    "cz": {
      "launch_path": "/index.html#cz",
      "name": "Czech",
      "description": "Czech layout",
      "types": ["url", "text"]
    },
    "fr": {
      "launch_path": "/index.html#fr",
      "name": "French",
      "description": "French layout",
      "types": ["url", "text"]
    },
    "de": {
      "launch_path": "/index.html#de",
      "name": "German",
      "description": "German layout",
      "types": ["url", "text"]
    },
    "nb": {
      "launch_path": "/index.html#nb",
      "name": "Norwegian Bokmal",
      "description": "Norwegian Bokmal layout",
      "types": ["url", "text"]
    },
    "sk": {
      "launch_path": "/index.html#sk",
      "name": "Slovak",
      "description": "Slovak layout",
      "types": ["url", "text"]
    },
    "tr-Q": {
      "launch_path": "/index.html#tr-Q",
      "name": "Turkish Q",
      "description": "Turkish Q layout",
      "types": ["url", "text"]
    },
    "tr-F": {
      "launch_path": "/index.html#tr-F",
      "name": "Turkish F",
      "description": "Turkish F layout",
      "types": ["url", "text"]
    },
    "ro": {
      "launch_path": "/index.html#ro",
      "name": "Romanian",
      "description": "Romanian layout",
      "types": ["url", "text"]
    },
    "ru": {
      "launch_path": "/index.html#ru",
      "name": "Russian",
      "description": "Russian layout",
      "types": ["url", "text"]
    },
    "sr-Cyrl": {
      "launch_path": "/index.html#sr-Cyrl",
      "name": "Serbian (Cyrillic)",
      "description": "Serbian (Cyrillic) layout",
      "types": ["url", "text"]
    },
    "sr-Latn": {
      "launch_path": "/index.html#sr-Latn",
      "name": "Serbian (Latin)",
      "description": "Serbian (Latin) layout",
      "types": ["url", "text"]
    },
    "ar": {
      "launch_path": "/index.html#ar",
      "name": "Arabic",
      "description": "Arabic layout",
      "types": ["url", "text"]
    },
    "he": {
      "launch_path": "/index.html#he",
      "name": "Hebrew",
      "description": "Hebrew layout",
      "types": ["url", "text"]
    },
    "hu": {
      "launch_path": "/index.html#hu",
      "name": "Hungarian",
      "description": "Hungarian layout",
      "types": ["url", "text"]
    },
    "el": {
      "launch_path": "/index.html#el",
      "name": "Greek",
      "description": "Greek layout",
      "types": ["url", "text"]
    },
    "zh-Hans-Pinyin": {
      "launch_path": "/index.html#zh-Hans-Pinyin",
      "name": "Pinyin",
      "description": "Pinyin",
      "types": ["url", "text"]
    },
    "number": {
      "launch_path": "/index.html#numberLayout",
      "name": "Number",
      "description": "Number layout",
      "types": ["number"]
    }
  },
  "locales": {
    "en-US": {
      "name": "3rd-party Keyboard",
      "description": "3rd-party Keyboard"
    },
    "zh-TW": {
      "name": "第三方鍵盤",
      "description": "第三方鍵盤"
    }
  },
  "default_locale": "en-US"
}
 
=== Layout Matching Algorithm ===
 
When an input field is focused, if its <code>type</code> attribute is one of the allowed values stated above, it will be used to filter a set of candidate layouts. A candidate layout means it can handle this input type or is possible to let user input all characters that this input field can accept.
For example, if the type of a input is "url", then a layout with "url" or "text" listed in the <code>types</code> of its manifest will be matched. However, if a input field with type "text", then all layouts that support "text" will be matched, but those layouts that only support "url" will not. This is because we believe layouts that can handle "text" could be a fallback for "url" input type, but not vice versa. We also believe "text" could be a fallback for all allowed types stated above.


The input method API is available to web content who intend to implement an input method, or "input source", or "virtual keyboard".
The matching algorithm of keyboard manager in System app is as follows:


partial interface Navigator {
# With the given type, find all layouts claims to support the said type and put it into the list.
    readonly attribute InputMethod inputMethod;
# Next, find layouts claims to support "text" and put it into the list. Layouts do not get duplicated listing even if it supports both types.
};
# Present the user with the choice of the layouts available to handle the input field. The order of presenting list is depend on UX design and/or user preferences in Settings.


interface InputMethod: EventTarget {
== Proposed API ==
    // Input Method Manager contain a few global methods expose to apps
    readonly attribute InputMethodManager mgmt;
    // Fired when the input context changes, include changes from and to null.
    // The new InputContext instance will be available in the event object under |inputcontext| property.
    // When it changes to null it means the app (the user of this API) no longer has the control of the original focused input field.
    // Note that if the app saves the original context, it might get void; implementation decides when to void the input context.
    attribute EventHandler oninputcontextchange;
    // An "input context" is mapped to a text field that the app is allow to mutate.
    // this attribute should be null when there is no text field currently focused.
    readonly attribute InputContext? inputcontext;
};


// Manages the list of IMEs, enables/disables IME and switches to an IME.
'''The API has made available to privileged apps. See the [http://dxr.mozilla.org/mozilla-central/source/dom/webidl/InputMethod.webidl WebIDL] for the current interface.'''
interface InputMethodManager {
    // Ask the OS to show a list of available IMEs for users to switch from.
    // OS should ignore this request if the app is currently not the active one.
    void showAll();
    // Ask the OS to switch away from the current active Keyboard app.
    // OS should ignore this request if the app is currently not the active one.
    void next();
    // Clear the focus of the current input field.
    // The OS might respond with hidden of the virtual keyboard and void the input context.
    // [JJ] What is removeFocus doing here. Everything that interacts with form elements
    //      happens in the inputContext, so this should be there too.
    void removeFocus();
  };


// The input context, which consists of attributes and information of current input field.
[https://wiki.mozilla.org/index.php?title=WebAPI/KeboardIME&oldid=1029753#Proposed_API History of this section]
// It also hosts the methods available to the keyboard app to mutate the input field represented.
// An "input context" gets void when the app is no longer allowed to interact with the text field,
// e.g., the text field does no longer exist, the app is being switched to background, and etc.
// [JJ] I doubt whether we should have 'name', 'type', etc. here. In the manifest we should
//      have entry points where the keyboard specifies which view to load when going into a
//      certain context. Requiring to do this manually will give extra work.
//      The system should guarantee that the right view is rendered based on entry_points in
//      in manifest (e.g. navigate keyboard to #text/en, or something, based on manifest.
// [Tim] I don't think they are exclusive. A keyboard app might choose to load the same page with the same hash
//      for different types but only to deal with the |type| or |inputmode| difference later.
interface InputContext {
    // The tag name of input field, which is enum of "input", "textarea", or "contenteditable"
    DOMString name;
    // The type of the input field, which is enum of text, number, password, url, search, email, and so on.
    // See http://www.whatwg.org/specs/web-apps/current-work/multipage/states-of-the-type-attribute.html#states-of-the-type-attribute
    DOMString type;
    /*
    * The inputmode string, representing the input mode.
    * See http://www.whatwg.org/specs/web-apps/current-work/multipage/association-of-controls-and-forms.html#input-modalities:-the-inputmode-attribute
    */
    DOMString inputmode;
    /*
    * The primary language for the input field.
    * It is the value of HTMLElement.lang.
    * See http://www.whatwg.org/specs/web-apps/current-work/multipage/elements.html#htmlelement
    */
    DOMString lang;
    /*
    * Get the whole text content of the input field.
    * App is encouraged to use getSurroundingText() below to prevent getting to much things clogged in memory.
    * [JJ] Do we have any case where loading the full text is required?
    *      I don't see it in the example code.
    */
    Promise<DOMString> getText();
    // The start and stop position of the selection.
    readonly attribute long selectionStart;
    readonly attribute long selectionEnd;
    /*
      * Set the selection range of the the editable text.
      * Note: This method cannot be used to move the cursor during composition. Calling this
      * method will cancel composition.
      * @param start The beginning of the selected text.
      * @param length The length of the selected text.
      *
      * Note that the start position should be less or equal to the end position.
      * To move the cursor, set the start and end position to the same value.
      *
      * [JJ] I think that this method should return the same info as the selectionchange event
      *      rather than a boolean.
      */
    Promise<boolean> setSelectionRange(long start, long length);
    /* User moves the cursor, changes the selection, or alters the composing text length
      * TODO: dup with the onsurroundingtextchange event/callback below for cursor moment?
      *
      * [JJ] I would merge this with onsurroundingtextchange to have 1 state event.
      *      in the end, every onselectionchange event will also generate a surrounding
      *      text change event.
    attribute EventHandler onselectionchange;
    /*
    * Get the text content around the cursor of the input field.
    * The text length is limited to 100 characters for each back and forth direction.
    * App is encouraged to use this instead of getText() above to prevent getting to much things clogged in memory.
    * TODO: how to return two values in one Promise object? as two arguments to the callback?
    *
    * [JJ] Would be useful to also include other state info (cursor position, selection pos, etc.)
    *      So we have 1 state object that gets returned regardless of operation
    *      (same one as in the change handlers and setSelectionRange)
    */
    Promise<DOMString beforeText, DOMString afterText> getSurroundingText();
    /*
      *
      * Delete text around the cursor.
      * @param offset The offset from the cursor position where deletion starts.
      * @param length The length of text to delete.
      * TODO: maybe updateSurroundingText(DOMString beforeText, DOMString afterText); ?
      * [JJ] Rather do a replaceSurroundingText(long offset, long length, optional DOMString text)
      *      If text is null or empty, it behaves the same
      */
    Promise<boolean> deleteSurroundingText(long offset, long length);
    /*
    * This event is sent when the text around the cursor is changed, due to either text
    * editing or cursor movement.
    *
    * The event handler function is specified as:
    * @param beforeString Text before and including cursor position.
    * @param afterString Text after and excluing cursor position.
    * function(DOMString beforeText, DOMString afterText) {
    * ...
    *  }
    * TODO: Is this a DOM Event or an attribute allows callback attachment? We should align the wording here.
    * TODO2: should this be |oncursorpositionchange|?
    */
    attribute SurroundingTextChangeEventHandler onsurroundingtextchange;
    /*
      * send a character with its key events.
      * TODO: what does Promise object returns?
      * TODO2: what are the modifiers?
      * Alternative: sendKey(KeyboardEvent event), but we will likely waste memory for creating the KeyboardEvent object.
      * [JJ] Why does this return anything?
      */
    Promise<boolean> sendKey(long keyCode, long charCode, long modifiers);
    /*
      * Set current composition. It will start or update composition.
      * @param cursor Position in the text of the cursor.
      * [JJ] Dont really know about this, but what if endComposition is never called
      *      maybe because unhandled exception?
      * [Tim] If that happens the API implementation should automatically ends the composition
      *          session (with event and confirm the current composition). Same apply when the inputContext is lost
      *          during a unfinished composition session.
      */
    Promise<boolean> setComposition(DOMString text, long cursor);
    /*
      * endComposition and actually commit the text.
      * Ending the composition with an empty string will not send any text.
      * (was |commitText(text, offset, length)|)
      *
      * @param text The text
      */
    Promise<boolean> endComposition(DOMString text);
};


=== Use cases for each of the methods ===
=== Use cases for each of the methods ===


* For a simple virtual keyboard action (send a character and key events w/ each user action), use <code>sendKey()</code>. TODO: should we allow backspace key to be sent from the method? If not, how do send these non-printable characters and it's effect with key events?
* For a simple virtual keyboard action (send a character and key events w/ each user action), use <code>sendKey()</code>. TODO: should we allow backspace key to be sent from the method? If not, how do send these non-printable characters and it's effect with key events?
**[yxl] I perfer to allowing non-printable character, such as backspace key, to be sent, if there is no security issue. This would give the IME more flexibility.
* For spellcheck, autocomplete etc, use surrounding text methods.
* For spellcheck, autocomplete etc, use surrounding text methods.
* For cursor moment helper features, use <code>setSelectionRange()</code> and related attributes.
* For cursor moment helper features, use <code>setSelectionRange()</code> and related attributes.
Line 243: Line 262:
       *      Rather rely on entry points in manifest...
       *      Rather rely on entry points in manifest...
       */
       */
 
     if (inputContext.inputmode === 'numeric' || inputContext.type === 'number') {
     if (inputContext.inputmode === 'numeric' || inputContext.type === 'number') {
       ['1', '3', '3', '7'].forEach(function (k) {
       ['1', '3', '3', '7'].forEach(function (k) {
Line 260: Line 279:
   }, 1000);
   }, 1000);
  }
  }
 
  function stopTyping() {
  function stopTyping() {
   clearTimeout(timer);
   clearTimeout(timer);
  }
  }
 
  var im = navigator.inputMethod;
  var im = navigator.inputMethod;
 
  im.addEventListener('inputcontextchange', function contextchanged(evt) {
  im.addEventListener('inputcontextchange', function contextchanged(evt) {
   if (evt.inputcontext) {
   if (evt.inputcontext) {
Line 276: Line 295:
   }
   }
  });
  });
 
  if (im.inputcontext) {
  if (im.inputcontext) {
   // The webpage here is loaded *after* the user has place the focus on the text field,
   // The webpage here is loaded *after* the user has place the focus on the text field,
Line 296: Line 315:


http://developer.chrome.com/trunk/extensions/input.ime.html
http://developer.chrome.com/trunk/extensions/input.ime.html
[[Category:Web APIs]]
Timdream
Confirmed users
478

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/677801...1029831"