Firefox/Shutdown Decoders

Overview

Suspending Video Decoders for Firefox is a feature that can reduce CPU & memory usage when a video runs in browser background. In the prototype (Phase 0), we tried to shutdown video decoder while a video element is invisible. The mechanism can save the CPU & Memory usage but it's hard for Firefox to resume video after user goes back to previous web page which contains videos since video decoders has lost necessary information. Currently (Phase 1), we devoted ourselves to implementing this feature on 1) video without audio track and 2) low resolution video (480 P), and we added a new mechanism. The new mechanism is going to switch to another video decoder (we called blank decoder) while video element is invisible and continuously send white screen (empty data) to video player. In our target release (Phase 2), our goal is to enable this feature on all videos.

The following is a step-by-step description of shutdown decoder working flow.

1. In the beginning of the decode framework, streaming or media file is going to be sent to demuxer if Firefox detects a media element.
2. Demuxer helps to separate a combined signal, e.g., a streaming can be separated into audio data and video data by demuxer. After that, audio and video data will be sent to audio and video decoder separately.
3. In data decoding period, audio decoder will works as normal because user may be listening to the music. But, Firefox uses a blank video decoder to replace current video decoder If Firefox detects a invisible video element.

Key Documents

• Code: https://hg.mozilla.org/integration/mozilla-inbound/file/tip/dom/media
• Issues: Meta Bug - Bug 1276556 [META] Tracking enable of background tab video decoder suspend
• Product/UX Trello Board: NA

Other Resources

• Etherpad: Constructing...

Install

Stable Release

Stay tuned.

Developer Release

The developer release is updated each time code is committed to Mozilla-central in HG.

Reporting Issues

If you find a bug or have a suggestion, please submit it on Bugzilla: https://goo.gl/coRinl

Planning

Goals

• Help users to reduce device resource usage (CPU & Memory)
• Finish tasks across videos
• No impact in current user flow

Feature Plan

• Phase 0: Shutdown decoders when video element is invisible
• Phase 1: Using a blank video decoder to replace video decoder instead of shutdown decoders directly. In this phase, the mechanism is applied to 1) Low resolution video (480P) and 2) Video film without audio track
• Phase 2: We're going to enhance the mechanism and make sure it can apply to all video files.

Success

• Effective cross discipline teams solving problems across platforms
• Validation of key assumptions through telmetrics
• video decode suspend in the hands of all our users

Engineering

Shutdown_Decoders [status: Green]

Current Goals:

• Ship disabling video decoders for ads
• Silent, looping videos where user doesn't really care the exact point

Next Milestone:

Milestones

Blank Previous Frame

• https://bugzil.la/1272919 - Dan
• Reviewed and needs rebasing before landing.

Optimizations

• For video elements off screen (using layout notification) - Kaku, http://bugzil.la/1282710
  • Not just for videos in background tabs but when the video element scrolls off the screen
  • mattwoodrow says it's possible to subscribe to visibility events via layout and to ask seth or tnikkel for details.
    • tnikkel suggests to start form here: http://searchfox.org/mozilla-central/source/layout/generic/nsIFrame.h#1177
    • nsVideoFrame has already implemented it: http://searchfox.org/mozilla-central/source/layout/generic/nsVideoFrame.cpp#664
  • Need to start video before it scrolls back into viewport
    • k17e: in async pan zoom the viewport is 3.5 times the height of the screen weighted in the direction of movement so we shouldn't need to do anything special here
  • k17e: Probably only want to do this for silent video where we can recover really quickly
• Resume video when mouse pointer starts hovering tab - https://bugzil.la/1274919 - Dan
  • In review. Needs response to mconley.
• Resume video when keyboard is used to change tags - Needs Bug
  • For example, if it's possible to detect the direction of cycling and videos are within, say, 5 tabs of the current tab, then start decoding again.
  • Need to hook into the tab navigation/change code and alert media elements, like bug 1274919
• Seek to nearest keyframe when video has no audio - http://bugzil.la/1282012 - Kaku
  • If there's no audio then no A/V sync to key, so just jump to the nearest keyframe.
  • This is a win for large background videos such as at:
    • http://www.gyg.com.au
    • https://www.paypal.com/nz/webapps/mpp/home
    • Many more examples - http://www.hongkiat.com/blog/fullsize-video-background-websites/
  • Tricky to detect no audio:
    • element is muted
    • video file has no audio track
    • video file has audio track of silence (alwu did work on this for tab audio indicator)
  • Don't suspend video decode if piping output through MSG.

Telemetry

• What telemetry should we collect about suspending video decode?
  • k17e: Amount of time hidden - measure of user value (Bucket results by resolution; i.e. are 720p videos hidden less often?)
    • http://bugzil.la/1285419 "Telemetry to support background video decoder suspend: Hidden play time" -> VIDEO_HIDDEN_PLAY_TIME_MS
      • Just something quick, based on existing VIDEO_PLAY_TIME_MS, to get some data soon, no bucketing on this one.
      • https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_HIDDEN_PLAY_TIME_MS
    • http://bugzil.la/1287987 "Percentage hidden/total play time, keyed by audio presence and height ranges" -> VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE
      • https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE
  • k17e: Recovery time - measure of user cost (separate for noisy vs silent videos) (Bucket results by resolution; i.e. do 720p videos take longer to recover?)
  • k17e: Key frame spacing - distribution allows better tuning
    • http://bugzil.la/1289668 "Telemetry to support background video decoder suspend: Inter-keyframe timings" -> measure=VIDEO_INTER_KEYFRAME_AVERAGE_MS
      • https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_INTER_KEYFRAME_AVERAGE_MS
      • https://telemetry.mozilla.org/new-pipeline/dist.html#max_channel_version=nightly%252F50&measure=VIDEO_INTER_KEYFRAME_MAX_MS

Mochitests

Currently no tests, need some to ensure that behaviour doesn't break http://bugzil.la/1284177 Add tests for suspending offscreen videos. Test no visible JS events because of suspending decode.

Team

Product owner:

Eng: Alastor, Gerald, JW, Kaku,

Program Management: Blake, Daniel, Josh

UX:

QA: whsu

Communications

IRC:

Email:

VidyoRoom: