Project Silk
Overview
The objective of Project Silk is to provide smoother scrolling and animation on B2G. To achieve the goal, it aligns HW Vsync signals among input, content painting and composition modules.
Architecture
VsyncDispatcherHost/Client
VsyncDispatcherHost/Client stands at the key position of Project Silk. The main role of this object is
- Receive HW vsync-event from vsync driver.
- Real time dispatching vsync-event to registered-event listeners.
There is exactly one VsyncDispatcherHostImpl object in chrome process and one VsyncDispatcherClientImpl in each content process which are derived from VsyncDispatcherHost and VsyncDispatherClient.
In chrome process
- VsyncDispatcherHostImpl creates a dedicated thread, named as vsync dispatcher thread, to dispatch vsync-event. Since vsync-event dispatching is very performance sensitive, we create a dedicate thread which only works on event dispatching to prevent transmission flow from being blocked by irrelevant tasks.
- Receive vsync events from PlatformVsyncTimer. We use HWComposer to implement the PlatformVsyncTimer in Gonk. On platforms which have no HW vsync supported, it's fallback to SW vsync driver automatically.
- Export registry functions to clients who want to receive vsync-event notification. Currently, we have three types of clients: refresh driver, compositor and input dispatcher.
- Sent vsync notifications to registered listeners, on the vsync dispatcher thread.
In Content process, VsyncDispatcherClientImpl's job is relatively simpler:
- Been created and initialized on main thread. The reason that we did not create a dedicated thread here is base on the fact: there is only one vsync listener in content side, which is nsRefreshDriver, and it can be kicked on main thread only.
- Export registry/unregistry function to user clients who want to receive vsync-event notification.
- In main thread, sent vsync notification to registered listeners.
vsync-event dispatching sequence
In current design,
VsyncDispatcherHost:
1. notify the main thread to handle input event.
2. wait the input handling task done, then notify the compositor parent to do compose.
3. send vsync event to content process.
4. tick Chrome refresh driver
VsyncDispatcherClient:
1. tick Content refresh driver
Why execute a sync InputDispatcher task on main thread instead of vsync-dispatcher thread in step 1?
The APZ event processing code all currently runs on the main thread. APZ can't process the input at non-main thread until we complete the Bug 930939. It will do the APZ off the main thread(it will update the APZ at VsyncDispatcher thread).
We also need to wait the APZ to update the frame matrix and forward any required input events to the child processes before we notify the compositor and refresh driver. Thus, we will have the most up to date touch coordinates rather than have a race condition where we might process the previous touch event versus the latest touch events when we composite.
Note1: We need to wait the APZC processed the input event in step 1, then do the remaining task. We might have a large delay for context switch.
Note2: If we have Bug 930939, the order will be:
VsyncDispatcherHost:
1. APZ processs input event at VsyncThread.
2. compositor parent to do compose.
3. send vsync event to content process.
4. tick Chrome refresh driver
VsyncDispatcherClient:
1. tick Content refresh driver
Object life cycle
VsyncDispatcherHost/Client is created as a singleton object in each process.
- VsyncDispatcherHost's life cycle in chrome process
- Create in gfxPlatform::Init
- Release in gfxPlatform::Shutdown
- VsyncDispatcherClient's life cycle in content process
- Create in PVsyncEventChild::Create(VsyncEvent protocol creation). Since we define PVsyncEvent as a top level protocol, VsyncDispatcherClient is constructed in the early stage of content process initialization.
- Release in PVsyncEventChild::ActorDestroy(VsyncEvent protocol destroy)
VsyncObserver
VsyncObserver is the base class for all vsync event observer. It will be ticked when vsync event comes. We should implement this function for our vsync-aligned task.
Here is the tick function prototype:
virtual void TickTask(int64_t timestamp, uint64_t frameNumber);
VsyncEventRegistry
This class provide the registering interface for VsyncObserver. After registering, VsyncDispatcher will call the observer's TickTask() for vsync event. The Register() call is one-shot registry. If observer need another tick, it should register again. The Unregister() has a sync flag. All observer should call the sync unregister before they call destructor. Thus, VsyncDispatcher will not tick the observer while destroying.
The VsyncEventRegistry interface:
virtual uint32_t GetObserverNum(); virtual void Register(VsyncObserver *observer); virtual void Unregister(VsyncObserver *observer, bool sync);
VsyncTimerObserver
VsyncTimerObserver is the base class for vsync timer observer. It will be notified by PlatformVsyncTimer for vsync event. Only VsyncDispatcherHostImpl is derived from this class.
This class only has one interface:
virtual void NotifyVsync(int64_t timestamp);
PlatformVsyncTimer
This is the platform depend vsync timer which provides the vsync events for VsyncTimerObserver.
The following lists the platform-dependency interfaces that should be implemented:
virtual void Shutdown(); virtual void Enable(bool aEnable); virtual uint32_t GetVsyncRate();
GonkVsyncTimer(Gonk implementation of PlatformVsyncTimer)
GonkVsyncTimer which is derived from PlatformVsyncTimer contains the gonk specific timer implementation. Basically, it's created in the VsyncDispatcherHost::Startup calling.
HWComposer2D
Here HWComposer2D is responsible to implement GonkVsyncTimer for B2G. Android HWC only supports HW Vsync after android JB. Therefore, we also need to implement a SW Vsync for system. If there is no HW Vsync support, the SW Vsync will be used. During initialization phase, VsyncDispatcher registers a Vsync notification callback into HWComposer2D and this callback will be called for every Vsync events. If there is no Vsync observer, we can disable the Vsync notifications inside HWComposer2D.
SW Vsync timer
SW Vsync timer is a periodic timer which calls it's callback function with 60Hz frequency. Currently, according to the WIP patch in bug 1043822, we use nanosleep() to implement the timer loop and the most parts of this timer are very similar with those in Android's SW Vsync timer. Our SW Vsync provides some APIs, such as
void StartTimer(); void StopTimerAsync(); void StopTimerSync(); double GetRate() const;
In the normal cases, EnableVsyncEvent() will use StartTimer() and StopTimerAsync() to control this timer. StopTimerSync() is called while we want to shut down VsyncDispatcher because VsyncDispatcher should wait until SW Vsync finishes all it's job. Enabling/Disabling the Vsync module is controlled by HwComposer2D right now, and we think users shouldn't worry about which one (HW or SW Vsync) is using, so we have to use a generic API for HW Vsync and SW Vsync. For example, EnableVsync() in HwcComposer2D is the generic function which handles the enable/disable of Vsync event. We can hide the logic of the decision between HW and SW Vsync in this function. In other words, if we are using HW Vsync, EnableVsync() will call eventControl() to handle it; if we are using SW Vsync, EnableVsync() will call the APIs of SW Vsync timer. By this design, GonkDisplayJB() can also use HwcComposer2D::EnableVsync() to turn on/off the Vsync event according to the status of the display.
Power on/off screen
We can't control the HW Vsync event on/off during power off. We will always turn off the Vsync event at power off and force to turn on at power on. That's ok if VsyncDispatcher receive an unexpected Vsync event at power on. After receiving the Vsync event, VsyncDispatcher can just disable the HW Vsync again.
Vsync event protocol
PVsyncEvent, a new top level protocol, is used to pass Vsync events from chrome to content process, and also provides the register/unregister interfaces for content process to receive Vsync events. In PVsyncEventChild side, it is running at the main thread of content process. In PVsyncEventParent side, it is running at the VsyncDispatcher thread of chrome process. Inside chrome, we dispatch the Vsync event to chrome's refresh driver directly without using PVysncEvent protocol(step 4 in above dispatch sequence).
IPC interfaces
child:
NotifyVsyncEvent(VsyncData aVsyncData);
parent:
RegisterVsyncEvent();
UnregisterVsyncEvent();
Porting Issue
In silk, there are only two platform dependent objects, which are platform vsync timer and InputDispatcher.
In a word, if you want to enable project silk on a specific platform, things need to be done are
- Constructing a concrete implementation of PlatformVsyncTimer interface
- An InputDispatcher instance which is inherit from VsyncObserver
- TBD: (object creation)
Testing Plan
Performace
- Test manually
- With automatic script or tool
- Scroll
- Show we get a smooth position with time
- OMTA
- Check b2g always composes done before next vsync
- requestAnimationFrame
- Check we get a better timestamp for animation
Unit test
- VsyncDispatcher
- Registry - vsync listener should be able to receive sync-notification steadily after registry.
- Unregistry - vsync listener should not receive any vync notification after synchronize unregistry from VsyncDispatcher
Integration test
- First paint - have a test case to make sure we do not break first-paint.
Benchmark
Type:
- requestAnimationFrame, WebGL
- OMTA test
- scrollable layer
Project Silk Discussion at Taipei Workweek
Bugs
New/Open
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Resolved
| ID | Priority | Summary | Status | Assigned to | Target milestone | Last change time |
|---|---|---|---|---|---|---|
| 930939 | P1 | Separate the threads that run gecko and that receive input events from the OS | RESOLVED | Kartikaya Gupta (email:kats@mozilla.staktrace.com) | mozilla38 | 2018-10-31T13:12:35Z |
| 980241 | P2 | Vsync-triggered RefreshDriver | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | --- | 2014-06-12T18:07:26Z |
| 982972 | P2 | investigate using RT priority in compositor on gonk | RESOLVED | Mason Chang [Inactive] [:mchang] | --- | 2014-08-15T00:11:44Z |
| 987523 | P2 | Vsync-triggered CompositorParent | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | --- | 2015-01-29T14:26:00Z |
| 987527 | P1 | Register Vsync monitor to HWComposer | RESOLVED | Boris Chiou [:boris] | 2.1 S3 (29aug) | 2018-10-31T13:12:35Z |
| 987529 | P1 | Integrate vsync dispatcher on B2G | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | 2.1 S4 (12sep) | 2018-10-31T13:12:35Z |
| 990832 | P1 | Build a tool to Visualize Frame Uniformity with ScrollGraph Output | RESOLVED | Jinho Hwang [:jeffhwang] | mozilla33 | 2014-08-20T17:42:53Z |
| 991420 | P1 | (silk) IPC Shows 1-2ms latency sending message from b2g to content process and b2g main thread to composite thread | RESOLVED | Mason Chang [Inactive] [:mchang] | 2.0 S4 (20june) | 2015-01-29T01:23:23Z |
| 991483 | -- | Vsync-triggered GeckoInputDispatcher | RESOLVED | Jinho Hwang [:jeffhwang] | --- | 2018-10-31T13:12:35Z |
| 1035076 | P2 | Measure vsync event latency between VsyncDispatcher and other modules in Project Silk | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | 2.1 S5 (26sep) | 2015-01-29T14:49:24Z |
| 1042017 | P3 | Re-Examine the APZ kinetic Scroll Friction Animation | RESOLVED | Mason Chang [Inactive] [:mchang] | --- | 2014-11-11T17:39:43Z |
| 1043822 | P2 | Support SW Vsync and combine it with HW Vsync | RESOLVED | Boris Chiou [:boris] | 2.1 S3 (29aug) | 2018-10-31T13:12:35Z |
| 1048667 | P1 | Implement Vsync dispatch framework | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | 2.1 S2 (15aug) | 2018-10-31T13:12:35Z |
| 1062117 | -- | Support the priority change for VsyncDispatcher | RESOLVED | Boris Chiou [:boris] | --- | 2018-10-31T13:12:35Z |
| 1062331 | -- | Turn on by default touch event smoothing with gfx.frameuniformity.hw-vsync and gfx.touch.resample | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla38 | 2018-10-31T13:12:35Z |
| 1072627 | P1 | Hardware Composer Vsync Events aren't stable | RESOLVED | Mason Chang [Inactive] [:mchang] | --- | 2018-10-31T13:12:35Z |
| 1073545 | -- | Add Vsync Markers to SPS Profiler | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla35 | 2014-10-06T13:48:12Z |
| 1073704 | -- | Input touch resampling is broken | RESOLVED | Michael Wu [:mwu] | mozilla35 | 2018-10-31T13:12:35Z |
| 1078136 | -- | Tune the thread priority for VsyncDispatcher thread | RESOLVED | --- | 2018-10-31T13:12:35Z | |
| 1078152 | P2 | Provide performance label for frame uniformity | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | --- | 2018-03-02T21:34:10Z |
| 1078160 | -- | Provide an automatic testing environment for QA to check the project silk performance | RESOLVED | --- | 2015-03-25T02:43:27Z | |
| 1078168 | P3 | Measure "where is my water" game overall performance with project silk | RESOLVED | --- | 2015-12-14T06:01:05Z | |
| 1078171 | P3 | Measure "cut the rope" game overall performance with project silk | RESOLVED | --- | 2015-12-14T06:00:56Z | |
| 1083530 | -- | Refactor GeckoTouchDispatcher to use mozilla::TimeStamp instead of nsecs_t for vsync times | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla36 | 2018-10-31T13:12:35Z |
| 1092245 | -- | Create a benchmark to measure refresh driver alignment to vsync and detect regressions | RESOLVED | Mason Chang [Inactive] [:mchang] | --- | 2017-01-30T16:53:10Z |
| 1092978 | P1 | Integrate refresh driver for silk | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | mozilla38 | 2015-01-29T01:22:03Z |
| 1094760 | -- | Compositor will sleep for a long time during composing if we align composing and refresh driver tick with vsync | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | --- | 2015-04-08T01:46:04Z |
| 1095242 | -- | Disable Vsync When Idle | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla38 | 2015-02-22T14:59:55Z |
| 1098701 | -- | [Silk] Gtest for project Silk | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla38 | 2015-01-25T05:32:44Z |
| 1101974 | P1 | Create a single VsyncDisptacher per nsBaseWidget | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla37 | 2014-12-19T15:30:03Z |
| 1102200 | -- | Composing time becomes much longer randomly when we unplug the usb cable on flame-kk device | RESOLVED | --- | 2022-05-30T19:26:41Z | |
| 1102631 | -- | Create a software vsync timer | RESOLVED | Mason Chang [Inactive] [:mchang] | --- | 2016-07-26T21:48:15Z |
| 1117870 | -- | Let the CompositorVsyncObserver unobserve vsync after a configurable number of vsyncs have passed | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla37 | 2022-01-03T17:02:05Z |
| 1118530 | -- | [meta] Turn on by default hardware vsync and the Vsync Aligned Compositor on b2g | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla38 | 2015-07-18T14:29:51Z |
| 1123734 | -- | Notify global VsyncSource in hwc | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | mozilla38 | 2018-10-31T13:12:35Z |
| 1123762 | -- | [META] Turn on vsync-aligned refresh driver by default on b2g | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla39 | 2022-06-01T22:16:00Z |
| 1130678 | -- | dom/broadcastchannel/tests/test_broadcastchannel_sharedWorker.html intermittent with vsync refresh driver | RESOLVED | Mason Chang [Inactive] [:mchang] | mozilla38 | 2022-06-01T22:14:38Z |
| 1139253 | -- | Reuse the same thread when we start/stop vsync event for software vsync timer | RESOLVED | Jerry Shih[:jerry] (UTC+8) (inactive) | 2.2 S8 (20mar) | 2015-03-10T02:04:26Z |
38 Total; 0 Open (0%); 38 Resolved (100%); 0 Verified (0%);
FAQ
- Platform supporting plan
- We are going to support Firefox OS first and extend this framework to other platforms after.
- Can Silk improve rendering performance?
- No
- What version of Android is required?
- We need Kit-Kat or greater. Jelly Bean and ICS are NOT supported.