NPAPI:Pepper

Background

Over the past few months a number of us have been discussing some of the issues facing NPAPI with recent developments in browser technology. First among the challenges we have focused on is giving a clear semantics to NPAPI in a browser that implements plugins in a separate process from the renderer/browser itself. The second challenge is integrating plugin rendering with the browser's compositing process, allowing HTML overlays, etc., to work correctly with plugins. A related aspect of this is the difficulty of writing a plugin due to too much OS and browser sensitivity, especially with windowed plugins. A third issue is defining events, especially in a way that is common across operating systems and browsers. A much smaller, but still important issue, is determining which plugins are available to a browser. This page presents a proposal to address these four issues. A number of other issues remain, including how to integrate peripheral devices (webcams, microphones, etc.) into NPAPI. This is deferred for future discussions.

Out of process plugins

The current NPAPI model defines the threading model within one process such that whatever thread invokes NPP_New is the only thread that may invoke NPN_* functions. Several browsers either currently support or are considering moving NPAPI plugins to a separate process from the renderer process (responsible for rendering a page 's content). With these processes distinct, we have the following:

R = renderer process
Tr = thread in R that causes NPP_New to be invoked
P = plugin process
Tp = thread in P that invokes NPP_New in response to the renderer

Thread model for NPN_* calls

We propose that NPN_* calls, with only one exception, can only be made from Tp. This allows us to maintain an important invariant, namely that if Tp is executing any NPP code, then Tr is blocked. This eliminates the possibility of simultaneous JavaScript and plugin activities.

NPN_PluginThreadAsyncCall

The one exception to NPN_* calls is NPN_PluginThreadAsyncCall.

void    NP_LOADDS NPN_PluginThreadAsyncCall(NPP instance,
                                            void (*func) (void *),
                                            void *userData);

The semantics of NPN_PluginThreadAsyncCall remain fundamentally the same, just with respect to P:

• It can be called from any thread in P.
• Invoking it eventually causes an NPP callback on Tp to func, passing userData.

We make a couple of other implementation comments that make this more usable and help avoid possible deadlocks:

• If two callbacks are requested by the same thread, they should be invoked in the same order they were requested.
• The implementation of NPN_PluginThreadAsyncCall may not call func directly, even if called on Tp, because we need to guarantee the state of Tr and this risks deadlock.

We observe that implementing NPN_PluginThreadAsyncCall this way results in additional call/callback/call latency for NPN_* calls. We believe this is reasonable to provide a well-defined interaction, but realize we may need to address the efficiency of this mechanism.

Plugins and rendering

Currently plugins typically display 2D graphics in one of two modes: windowless or windowed. In the current state of affairs windowless plugins are typically given an RGB surface containing the contents of lower layers and they composite themselves, leading to inconsistent alpha handling. Furthermore, basically all access to 3D accelerated graphics are through the latter. Windowed plugins provide a number of challenges to plugin and browser writers. First, windowed plugins are essentially given a handle to an operating system native window, from which all interaction by the plugin is completely platform-specific. Second, native windows make it very difficult for the browser to do CSS overlays, etc. Third, although NPAPI provides the NPP_HandleEvent API, this is not well-defined with respect to native windowing system events.

To these ends we propose, for both 2D and 3D:

• Only windowless plugins should be supported.
• Compositing should be done by outside the plugin itself, which should only produce an RGBA (2D) or texture (3D).
• No native windowing system events should be delivered to plugins.

This proposal is clearer at this moment regarding 2D graphics. We expect we may need to pass some state (2d vs. 3d, hardware vs. raster, etc.) in through other calls and/or with some extensions to the data structures we propose below.

An Example

A video game looks abstractly like

  ...
  while (true) {
    HandleEvents()
    Simulate(gettimeofday())
    Draw()
  }
  ...

We propose that we continue to set graphics attributes through the existing NPN_GetValue/NPN_SetValue interfaces with some new variables TBD.

There are about four APIs that would be involved in supporting this from the browser

NPP_SetWindow and Windowless Plugins

Whereas currently NPP_SetWindow passes both some geometry information and a handle to a native window (for windowed plugins), in our proposal, it only conveys geometry and format information. For example.

typedef struct _NPWindow {
  void*    window;   // UNUSED
  uint32_t x;        // Position of plugin top left corner relative to top left
  uint32_t y;        // of renderer page. Y increases going down.
  uint32_t width;    // maximum window size
  uint32_t height;
  NPRect   clipRect; // UNUSED
  void*    ws_info;  // Pixel format of the raster backing store. A NPRasterWindowInfo struct.
  NPWindowType type; // UNUSED
} NPWindow;

enum NPPixelFormat {
  NP_RGBA_8888,
  NP_RGB_565,
  // TBD: other pixel formats
};

typedef struct _NPRasterWindowInfo {
  NPPixelFormat format;
} NPRasterWindowInfo;

NPError NP_LOADDS NPP_SetWindow(NPP instance,
                                NPWindow* window);

Getting a Device Context to Draw Into

  PARAMS (details remain to be worked out, but at least):
    - rectangle plugin will draw into
    - discard bit (don't bother to initialize the memory in that rectangle)
  2D DC:
    - x, y, width, height
    - visible rectangles[] // serve as hints
    - stride (byte/pixel difference between (x,y) and (x,y+1))
    - x, y for byte zero of the buffer
    - void* buffer // size is as requested by PARAMS regardless of visible rectangles.

  void NPN_GetDeviceContext(NPP instance, PARAMS, NPDeviceContext** context)

• Allows the plugin to request that the renderer create a context to draw into and call the plugin back.
• On return, the renderer informs the plugin where it can draw into (in *context).

Publishing the Context to the Renderer

  void NPN_PublishDeviceContext(NPP instance, NPDeviceContext* context)

• The plugin informs the renderer that it has drawn into context and that the renderer may composite from it.
• After this call the plugin may no longer draw into the context (it may be deallocated, etc.)

Notification a Context was Displayed

Video players, video games, and other timing sensitive plugins typically adjust their simulation to adapt to the speed at which frames are placed on the screen. A call from the renderer indicating when the bits from a context have reached the screen is needed.

  void NPP_DeviceContextDisplayed(NPP instance)

• The renderer informs the plugin that the last published device context was displayed on the screen.
• For performance we will most likely want to be able to call NPN_GetDeviceContext this before the NPP_DeviceContextDisplayed call from the previous NPN_PublishDeviceContext has come.
• This API could also perhaps be done through an NPEvent of the appropriate type.

Optimizations and Legacy APIs

In order to avoid drawing an entire region on the screen when only a small portion was updated, it may be important for the plugin to use existing APIs.

  void NPN_InvalidateRect(NPP instance, NPRect* invalidRect)

• The plugin informs the renderer that it would like to replace the contents of the specified rectangle.

Event Handling

Windowless plugins should receive all their events from NPP_HandleEvent. We believe that standardizing the event types and payloads based on, for example, WebKit, would be sufficient to provide browser and OS independent events. If not, the goal can be accomplished by possibly extending the set of event types or data on the event structures to convey more information.

Plugin Registration

The current process of determining which plugins are available to the browser typically involves loading various shared libraries and querying them for types and extensions addressed by the plugin. We propose that more efficient method would be to add a special section to the shared library for the plugin.

The section, tentatively named .npapidesc, contains a sequence of strings of the form:

mimetype:[extension][,extension]*:description;

This enables browsers to scan the plugins without loading them, which becomes more difficult with out-of-process plugins and sandbox protection.