MozillaWiki

Auto-tools/Projects/Platform Quality/External Media Tests

< Auto-tools‎ | Projects‎ | Platform Quality
Revision as of 21:35, 22 March 2016 by Sydpolk (talk | contribs) (→‎Notes: Fix on netflix instructions)

External Media Tests is a suite of Marionette Python tests for media playback in Mozilla Firefox. Its aim is to test MSE (Media Source Extensions) and EME (Encrypted Media Extensions) on specific websites such as YouTube and Netflix.

Source

How are these tests run

There are five ways to run External Media Tests:

  • From <mozilla-central>, type
./mach external-media-tests

Automation Information

External Media Tests currently runs on buildbot (limited tests), and pf-jenkins. There are plans to migrate some or all of the pf-jenkins to QA/Automation/Projects/Mozmill_Automation/Mozmill_CI Mozmill_CI, and to taskcluster.

Results Reported to Treeherder

These are Tier-2 jobs, meaning they are visible by default but not sheriff-managed.

buildbot

There are two short test jobs that run in buildbot

  • Repos: trunk
  • Platforms: Linux x64 (opt only), Windows 7, Windows 8 x64, OS X 10.10
  • Group Name: VideoPuppeteer
  • Group Symbol: VP
  • Jobs:
    • b-m - "media-tests"
      • short tests using standard video API on YouTube; equivalent to 'm' below
    • b-y - "media-youtube-tests"
      • short tests using YouTube JS API on YouTube; equivalent to 'y' below

pf-jenkins

All available test jobs also run in our Jenkins instance, pf-jenkins.

  • Repos
    • mozilla-central
    • mozilla-aurora
  • Platforms:
    • Mac OS X 10.10 opt
    • Windows 7 opt
    • Windows 8 opt
  • Group Name: VideoPuppeteer
  • Group Symbol: VP
  • Jobs:
    • m - short tests using standard video API on YouTube
    • y - short tests using YouTube JS API on YouTube
    • ym1-60 - hour-long tests using YouTube JS API on YouTube
    • e - short tests using standard video API to test EME
    • e-f - full tests using standard video API to test EME
    • *-e10s - job that is run with e10s enabled e.g. m-e10s, y-e10s
    • *-bw - tests that runs for various bandwidths

Debugging

You can RDC/VNC to individual machines in pf-jenkins. (User: mozilla). When there are problems, it often helps to trigger the job from pf-jenkins, and watch it run in your VNC window. You can also launch a mozilla-build shell and run scripts yourself. You should avoid running from the Jenkins workspace directories; you should be able to get the necessary assets to reproduce the tests on a directory on the local drives.

Command-lines for jobs can be found in the pf-jenkins GUI by selecting the project and either reading through the logs, or going into the configure pane, copying, and then pasting into your shell window.

Custom Profile

Netflix and network bandwidth limiting tests use a stored profile with custom logins and certificates installed.

  • Windows: c:\testing\eme_profile
  • Mac: /Users/mozilla/testing/eme_profile

How to disable an individual failing test on pf-jenkins.

Visit pf-jenkins (VPN required), open the summary page of the job you want to disable and click 'Disable Project'. When a job is reported to Treeherder, its Jenkins build url and build id are listed in the Job Details pane.

Command-lines for various tests

Notes

  • For all Netflix videos for anything thing built locally, you must copy plugin-container.exe and voucher.bin from a production Firefox installation into your <objdir>/dist/bin directory. You must also have a profile that has been logged in with a valid Netflix account.
  • Please see External Media Tests for information on setting up browsermob-proxy for network bandwidth limiting tests.

Default tests

./mach external-media-tests
  • Videos: Four shortish YouTube videos.
  • What it tests: Do the videos start, and can they be played for up to two minutes?

Default tests with e10s

./mach external-media-tests --e10s
  • Videos: Four shortish YouTube videos.
  • What it tests: Do the videos start, and can they be played for up to two minutes with e10s set?

Basic YouTube API

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini
  • Videos: Four shortish YouTube videos.
  • What it tests: Test various aspects of the API.

Basic YouTube with e10s

./mach external-media-tests dom/media/test/external/external_media_tests/playback/youtube/manifest.ini --e10s
  • Videos: Four shortish YouTube videos.
  • What it tests: Test various aspects of the API, with e10s enabled.

Run videos under various network configurations.

./mach external-media-tests --profile /Users/mozilla/eme_profile --browsermob-script /Users/mozilla/browsermob-proxy-2.1.0-beta-2/bin/browsermob-proxy --browsermob-port 999 dom/media/test/external/external_media_tests/playback/test_playback_limiting_bandwidth.py
  • Videos: Four shortish YouTube videos.
  • What it tests: Do the videos play for one minute
    • 1000 Kbps
    • 500 Kbps
    • 250 Kpbs
    • 100 Kpbs

Test Netflix playback.

./mach external-media-tests dom/media/test/external/external_media_tests/playback/eme.ini --profile /c/testing/eme_profile --urls=dom/media/test/external/external_media_tests/urls/netflix/default.ini
  • Videos: Three Netflix videos.
    • Unencrypted
    • Clearkey
    • DRM
  • What it tests: Playback starts; playback works up to two minutes.

Test Netflix with bandwidth limits in place.

./mach external-media-tests dom/media/test/external/external_media_tests/playback/netflix_limiting_bandwidth.ini --browsermob-script=/c/testing/browsermob-proxy-2.1.0-beta-2/bin/browsermob-proxy.bat --browsermob-port 999 --urls=dom/media/test/external/external_media_tests/urls/netflix/default.ini  --profile /c/testing/eme_profile
  • Videos: Three Netflix videos.
    • Unencrypted
    • Clearkey
    • DRM
  • What it tests: Do the videos play for one minute
    • 1000 Kbps
    • 500 Kbps
    • 250 Kpbs

Contacts

  • :SingingTree (primary contact)
  • :sydpolk

Where to file bugs

Test failures seen on Treeherder should be filed under Core::Audio/Video: Playback

Retrieved from "https://wiki.allizom.org/index.php?title=Auto-tools/Projects/Platform_Quality/External_Media_Tests&oldid=1123585"