Auto-tools/Projects/Platform Quality/External Media Tests: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎How are these tests run: More formatting)
Line 19: Line 19:
* From <mozilla-central>/dom/media/test/external, follow the instructions at [https://developer.mozilla.org/en-US/docs/Mozilla/QA/external-media-tests External Media Tests].
* From <mozilla-central>/dom/media/test/external, follow the instructions at [https://developer.mozilla.org/en-US/docs/Mozilla/QA/external-media-tests External Media Tests].
* The buildbot mozharness script is at [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/firefox_media_tests_buildbot.py firefox_media_tests_buildbot.py]. The configuration files for buildbot are at [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/configs/mediatests/buildbot_posix_config.py buildbot_posix_config.py] and [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/configs/mediatests/buildbot_windows_posix_config.py buildbot_windows_posix_config.py]. buildbot takes care of posting results to treeherder.
* The buildbot mozharness script is at [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/firefox_media_tests_buildbot.py firefox_media_tests_buildbot.py]. The configuration files for buildbot are at [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/configs/mediatests/buildbot_posix_config.py buildbot_posix_config.py] and [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/configs/mediatests/buildbot_windows_posix_config.py buildbot_windows_posix_config.py]. buildbot takes care of posting results to treeherder.
* The mozmill-ci-based mozharness script at [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/firefox_media_tests_jenkins.py firefox_media_tests_jenkins.py]. The configuration file is at [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/configs/mediatests/jenkins_config.py jenkins_config.py]. mozmill-ci will take care of posting results to treeherder.
* The mozmill-ci-based mozharness script is at [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/scripts/firefox_media_tests_jenkins.py firefox_media_tests_jenkins.py]. The configuration file is at [http://hg.mozilla.org/mozilla-central/file/default/testing/mozharness/configs/mediatests/jenkins_config.py jenkins_config.py]. mozmill-ci will take care of posting results to treeherder.
* The pf-jenkins script at [https://github.com/sydvicious/mozplatformqa-jenkins/blob/master/external-media-tests/run_media_tests.py run_media_tests.py]. Unlike the other mozharness scripts, this script submits results to treeherder.
* The pf-jenkins script is at [https://github.com/sydvicious/mozplatformqa-jenkins/blob/master/external-media-tests/run_media_tests.py run_media_tests.py]. Unlike the other mozharness scripts, this script submits results to treeherder.


== Automation Information ==
== Automation Information ==

Revision as of 21:07, 18 March 2016

external-media-tests is a suite of Marionette Python tests for media playback in Mozilla Firefox. Its aim is to test MSE (Media Source Extensions) and EME (Encrypted Media Extensions) on specific websites such as YouTube and Netflix.

Source

How are these tests run

There are five ways to run External Media Tests:

  • From <mozilla-central>, type
./mach external-media-tests

Automation Information

External Media Tests currently runs on buildbot (limited tests), and pf-jenkins. There are plans to migrate some or all of the pf-jenkins to QA/Automation/Projects/Mozmill_Automation/Mozmill_CI Mozmill_CI, and to taskcluster.

= Results Reported to Treeherder=

These are Tier-2 jobs, meaning they are visible by default but not sheriff-managed.

buildbot

There are two short test jobs that run in buildbot

  • Repos: trunk
  • Platforms: Linux x64 (opt only), Windows 7, Windows 8 x64, OS X 10.10
  • Group Name: VideoPuppeteer
  • Group Symbol: VP
  • Jobs:
    • b-m - "media-tests"
      • short tests using standard video API on YouTube; equivalent to 'm' below
    • b-y - "media-youtube-tests"
      • short tests using YouTube JS API on YouTube; equivalent to 'y' below

pf-jenkins

All available test jobs also run in our Jenkins instance, pf-jenkins.

  • Repos
    • mozilla-central
    • mozilla-aurora
  • Platforms:
    • Mac OS X 10.10 opt
    • Windows 7 opt
    • Windows 8 opt
  • Group Name: VideoPuppeteer
  • Group Symbol: VP
  • Jobs:
    • m - short tests using standard video API on YouTube
    • y - short tests using YouTube JS API on YouTube
    • ym1-60 - hour-long tests using YouTube JS API on YouTube
    • e - short tests using standard video API to test EME
    • e-f - full tests using standard video API to test EME
    • *-e10s - job that is run with e10s enabled e.g. m-e10s, y-e10s
    • *-bw - tests that runs for various bandwidths

How to disable an individual failing test on pf-jenkins.

Visit pf-jenkins (VPN required), open the summary page of the job you want to disable and click 'Disable Project'. When a job is reported to Treeherder, its Jenkins build url and build id are listed in the Job Details pane.

Contacts

  • :SingingTree (primary contact)
  • :sydpolk

Where to file bugs

Test failures seen on Treeherder should be filed under Core::Audio/Video: Playback

Retrieved from "https://wiki.allizom.org/index.php?title=Auto-tools/Projects/Platform_Quality/External_Media_Tests&oldid=1122943"