Auto-tools/Projects/Platform Quality/External Media Tests

external-media-tests is a suite of Marionette Python tests for media playback in Mozilla Firefox. Its aim is to test MSE (Media Source Extensions) and EME (Encrypted Media Extensions) on specific websites such as YouTube and Netflix.

Source

• For Firefox 48 and later, the source code lives in hg. You can also get them from the common.tests.zip.
  • http://hg.mozilla.org/mozilla-central/file/default/dom/media/test/external.
  • From a common.tests.zip file, the tests are in external-media-tests.
• For Firefox 47 and before, the older version of the tests are at firefox-media-tests.

How are these tests run

There are five ways to run External Media Tests:

• From <mozilla-central>, type

./mach external-media-tests

• From <mozilla-central>/dom/media/test/external, follow the instructions at External Media Tests.
• The buildbot mozharness script is at firefox_media_tests_buildbot.py. The configuration files for buildbot are at buildbot_posix_config.py and buildbot_windows_posix_config.py. buildbot takes care of posting results to treeherder.
• The mozmill-ci-based mozharness script is at firefox_media_tests_jenkins.py. The configuration file is at jenkins_config.py. mozmill-ci will take care of posting results to treeherder.
• The pf-jenkins script is at run_media_tests.py. Unlike the other mozharness scripts, this script submits results to treeherder.

Automation Information

External Media Tests currently runs on buildbot (limited tests), and pf-jenkins. There are plans to migrate some or all of the pf-jenkins to QA/Automation/Projects/Mozmill_Automation/Mozmill_CI Mozmill_CI, and to taskcluster.

Results Reported to Treeherder

These are Tier-2 jobs, meaning they are visible by default but not sheriff-managed.

buildbot

There are two short test jobs that run in buildbot

• Repos: trunk
• Platforms: Linux x64 (opt only), Windows 7, Windows 8 x64, OS X 10.10
• Group Name: VideoPuppeteer
• Group Symbol: VP
• Jobs:
  • b-m - "media-tests"
    • short tests using standard video API on YouTube; equivalent to 'm' below
  • b-y - "media-youtube-tests"
    • short tests using YouTube JS API on YouTube; equivalent to 'y' below

pf-jenkins

All available test jobs also run in our Jenkins instance, pf-jenkins.

• Repos
  • mozilla-central
  • mozilla-aurora
• Platforms:
  • Mac OS X 10.10 opt
  • Windows 7 opt
  • Windows 8 opt
• Group Name: VideoPuppeteer
• Group Symbol: VP
• Jobs:
  • m - short tests using standard video API on YouTube
  • y - short tests using YouTube JS API on YouTube
  • ym1-60 - hour-long tests using YouTube JS API on YouTube
  • e - short tests using standard video API to test EME
  • e-f - full tests using standard video API to test EME
  • *-e10s - job that is run with e10s enabled e.g. m-e10s, y-e10s
  • *-bw - tests that runs for various bandwidths

Debugging

You can RDC/VNC to individual machines in pf-jenkins. (User: mozilla). When there are problems, it often helps to trigger the job from pf-jenkins, and watch it run in your VNC window. You can also launch a mozilla-build shell and run scripts yourself. You should avoid running from the Jenkins workspace directories; you should be able to get the necessary assets to reproduce the tests on a directory on the local drives.

Command-lines for jobs can be found in the pf-jenkins GUI by selecting the project and either reading through the logs, or going into the configure pane, copying, and then pasting into your shell window.

Custom Profile

Netflix and network bandwidth limiting tests use a stored profile with custom logins and certificates installed.

• Windows: c:\testing\eme_profile
• Mac: /Users/mozilla/testing/eme_profile

How to disable an individual failing test on pf-jenkins.

Visit pf-jenkins (VPN required), open the summary page of the job you want to disable and click 'Disable Project'. When a job is reported to Treeherder, its Jenkins build url and build id are listed in the Job Details pane.

Contacts

• :SingingTree (primary contact)
• :sydpolk

Where to file bugs

Test failures seen on Treeherder should be filed under Core::Audio/Video: Playback