Virtual Keyboard/IME API

Introduction

Virtual Keyboard/IME API aims to implement the system IME as a Web App. It will be used in Firefox OS and use cases could be found in the Firefox OS Keyboard UX document(WIP).

The API provides the communication channel between the IME App and the other App that receives user's inputs.

It is very different from the IME API from Google that aims to re-use the system's IME in a web page.

Status

API discussion:

Implementation:

Bug 737110 bug 737110 - Bug 737110 - Virtual Keyboard API
Bug 805586 bug 805586 - [keyboard] keyboard needs a 'hide keyboard' button(main tracking bug)
Bug 844716 bug 844716 - Enable keyboard Apps to get/set selection range of the input field

Bug 860546 bug 860546 - [keyboard] JS changes to a textfield while keyboard is displayed do not get passed to keyboard

Bug 861665 bug 861665 - Allow IME to get notification when text field content is changed
Bug 861515 bug 861515 - Keyboard should be able to modify the text of the input field directly
Bug 838308 bug 838308 - mozKeyboard should require a permission to use
Bug 842436 bug 842436 - Keyboard API: There should only be one keyboard active, and Gecko should block interaction from non-active keyboards

Features

The Virtual Keyboard/IME API supports the following features:

Notifies the VKB app when the focus text field was changing in other

apps

Allow user to manual hide the keyboard. Check bug 737110.
The VKB app should be responsive to properties and the state of the input field (more than HTML5 input type, including current content, cursor position, x-inputmode bug 796544).
Sends trust
The VKB app should be able to send trusted key events such as they are considered by the other apps as user' inputs.
The VKB app should be able to send a character or a string to the current input cursor position.
Keyboard should be able to overwrite the current value of the input field of the input field and set the cursor position.
The VKB app should be able to move the cursor position or select a specified range of text.
The VKB should be able to switch the focus onto the previous/next input field.
The return key label of the VKB can be customized.

Proposed API

partial interface Navigator {
    readonly attribute InputMethodManager mozInputMethodManager;
    readonly attribute InputMethodConnection mozInputMethodConnection;
};

interface InputMethodConnection {
    // User begins or finishes editing a field.
    // If the user change input directly from one input to another, only one event will be dispatched.
    attribute Function oninputmethodstatechange;

    // Whether the user begins editing.
    readonly attribute boolean inputStarted; 

    // User moves the cursor, changes the selection, or alters the composing text length
    attribute Function onselectionchange;

    // Listener to get noftified when the content of the current input field has
    // changed.
    attribute nsIDOMEventListener ontextchange;

    // Send a keyevent to the input field that is currently attached. It exists in the current interface
    void sendKey(in long keyCode, in long charCode, in long modifiers);

   /*
    * Replace text around the beginning of the current selection range of the
    * editable text and clear the composing text.
    *
    * @param text The string to be replaced with. 
    * @param beforeLength The number of characters to be deleted before the
    * beginning of the current selection range. Defaults to 0.
    * @param afterLength The number of characters to be deleted after the
    * beginning of the current selection range. Defaults to 0.
    */
   void replaceSurroundingText(in DOMString text, [optional] in long beforeLength, 
                               [optional] in long afterLength);

    // Set the text value of the current input field. 
    void setText(in DOMString text);

    /*
     * Get a substring of the text content of the current input field.
     * @param start The start index of the substring.
     * @param end The end index of the substring. The character at the end index
     * is not included.
     */
    void getText(in long start, in long end, in Function callback);

    // Length of the content
    readonly attribute long textLength;

    // Set the composing text before the current cursor position.
    // To clear the composing text, set the text paramter to the empty string.
    void setComposingText(in DOMString text);

    // The start and stop position of the selection. 
    readonly attribute long selectionStart;
    readonly attribute long selectionEnd;

    /*
     * Set the selection range of the the editable text.
     *
     * @param start The beginning of the selected text.
     * @param end The end of the selected text.
     *
     * Note that the start position should be less or equal to the end position.
     * To move the cursor, set the start and end position to the same value.
     */
     void setSelectionRange(in long start, in long end);

     // Clear the focus of the current input field and hide the keyboard.
     void removeFocus();

     // Focus the next text field.
     void advanceFocus();

     // Focus the previous text field.
     void rewindFocus();

     // This read only attribute is true if user can advance the focus to the next
     // text field.
     readonly attribute boolean canAdvanceFocus;

     // This readonly attribute is true if user can rewind the focus back to the
     // previous text field.
     readonly attribute boolean canRewindFocus;

     // The input mode string.
     // https://bugzilla.mozilla.org/show_bug.cgi?id=796544
     // It can be one of the following values:
     // "none"
     // "verbatim"    - no capitalization, no word suggestions
     // "latin"       - word suggestions but no capitalization
     // "latin-prose" - word suggestions and capitalization at the start of sentences
     // "latin-name"  - word suggestions and capitalize each word
     // "digits"      - digits(0-9) only.
     readonly attribute DOMString inputMode;
 
     // The type of the input field, including text, number, password, url, tel and email.
     readonly attribute DOMString inputType;
 
     // Get the return key name, which determines the title of the key displayed in the keyboard and action 
     // performed when it is  pressed. The value could be "done", "next", "go", "search", "send" or other 
     // customized label for the return key.
     // If the value is "next", when pressing the enter key, the focus should be move to the next input field.
     readonly attribute DOMString returnKey;
};

Examples

var conn = navigator.mozInputMethodConnection;

// Called when the user starts or finishes editing an input field
conn.oninputmethodstatechange= function(event) {
    if (conn.inputStarted) {
        // Text input should start
    } else {
        // Text input should end
    }
};

// Insert a string at the current cursor position
conn.replaceSurroundingText('Hello world');

// Clear delete 5 characters before the cursor position.
conn.replaceSurroundingText(, 5);

// Get the selected text
conn.getText(conn.selectionStart, conn.selectionEnd, function callback(text) {
    var selectedText = text;
});

// Get notified when the text content has changed.
conn.ontextchange = function() {
  conn.getText(0, conn.textLength, function(text) {
    console.log(text);
  }
}

// Move the cursor position
var position = 10;
conn.setSelectionRange(position, position);

// Hide the keyboard
conn.removeFocus();

// Switch the focus onto the next input field if possible
if (conn.canAdvanceFocus) {
    conn.advanceFocus();
}

WebAPI/KeboardIME