Firefox OS/1UA Multi-Screen

Introduction

Goal

Our mission is to support "1 UA multi-screen" rendering on FxOS. Try imagine the following scenario, you have a computer set up in duel monitors, launch a Firefox browser on your primary display, launch another Firefox browser(or open a link in new window), and drag that new window to the secondary monitor. Our concept is similar but slightly different, where the baseline of this project is to allow a FxOS device connect and render contents to any non-FxOS display, or device that doesn't support Firefox browser.

The picture shows a FxOS device(at the bottom) has connected to a dummy display(the Acer monitor) with HDMI cord, and to a tablet(at the lower right) with Wifi Display in the same time. Both Marketplace and Dialler are launched from the FxOS device and displayed to the external displays.

Requirements

Connect with external displays
- For wireless connection, we have implemented Wifi Display with Miracast protocol. That is, any device capable of being a Miracast Sink, can be connected with a FxOS device.
- For wired connection, we have implemented the HDMI output. This requires the support of hotplug detection in Hardware Composer.
Open multiple top-level windows on FxOS
- Gecko supports opening multiple top-level windows on browser naturally, while the FxOS supports single top-level window only, thus we are going to bring this ability back for FxOS.
Manage multi-screen in System App

Advantages

Having multiple top-level windows means that we will have multiple DOM trees and layer trees exist in the system. The secondary window should have no difference regarding to functionality then the original one, which means that we should be able to launch WebApps, web pages, and multimedia on this secondary window.
With the case of "1 UA multi-screen" rendering, contents of remote screen are rendered by primary device, but in the resolution of its display.
Open arbitrary numbers of top-level windows and associate each of them to a display device is doable theoretically, but it depends on the support of hardware capability.

Terminology

Top-level window - A Window object without a parent, currently in FxOS, document loaded into the one and only top-level window is shell.html.
UA - In short of User Agent. A user agent is any software that retrieves, renders and facilitates end user interaction with Web content. -- From User Agent Accessibility Guidelines (UAAG) 2.0
Display device/Screen - Refer to devices for presentation in visual, e.g. monitor, TV, projector, tablet, ...etc.
Primary Display - Refer to the display of FxOS mobile device.
Secondary/Remote/External/Virtual Display - Refer to displays connected with primary device.
- To be more specific, HWC has categorized the display types into follows: Primary the default built-in display, External the display connected through HDMI cord, and Virtual as the display connected via Wifi Display or other wireless protocols.
Wifi Display, Miracast -

User Stories

Architecture Designs

Overview

An overview of multi-screen on FxOS

This is an overview structure design from top to down.

Gecko

DisplayDevice is the new introduced object, it simply represents information of a display, such as display's size, dpi, surface format, and more important, some native data such as FrameBufferSurface and ANativeWindow. The default built-in device could be a DisplayDevice with type of primary, and it would be the one and only primary DisplayDevice. Other connected/disconnected DisplayDevice can be added into/removed from our system through interfaces of AddDevice/RemoveDevice in GonkDisplay. Hardware Composer has defined displays connected via HDMI as "external", and displays which make composited output available within the system as "virtual", thus we share the same here.

Although it might be possible to connect multiple displays of the same type in the same time, we have limited the mapping of display type and display device to one to one at this point, any second DisplayDevice with type that already exist in the list asking for connection would result in an error. We'd like to keep the design simple at the early stage and it doesn't sound like a difficult extension anyway. As you can see from the graph above, GonkDisplay manages a list of DisplayDevices. Any component who requests data of a DisplayDevice from GonkDisplay should provide a display type.

Gaia

Design structure of gaia's part.