TestEngineering/Performance/Raptor/Youtube playback performance

YouTube playback performance test

The benchmark is measuring the playback performance of video streaming by assembling dropped frames during video playback. The test suite is a copy of the Playback Performance Tests from Google, and run in our own infrastructure.

Running locally

For information in how to setup your system to run the tests locally consult the general Raptor documentation first.

Once done the benchmark test can be run via the following command:

   $ ./mach raptor-test --test=raptor-youtube-playback 

Customizing the test execution

There are some variables you can tweak in the raptor-youtube-playback manifest file to customize the benchmark test.

Subset of tests

To only run a subset of tests the benchmark test page supports the tests argument, which allows to add any number of tests based on their index. Currently it needs to be a comma separated list, but will be soon extended for ranges too.

The following line will only cause the videos of index 6, 47, and 90 to run:

   test_url = http://...main.html?test_type=playbackperf-test&raptor=true&command=run&exclude=1,2&tests=6,47,90

This is helpful if specific video codecs, resolutions, or playback speeds are causing performance issues.

Profiling

TODO

Updating the test suite

In inregular intervals Google will update its test suite. As we also want to have those changes mirrored for our instance the perf-youtube-playback repository has to be used. Changing its content on the `raptor` branch will automatically be picked-up by yttest.prod.mozaws.net where the tests are hosted on.

Details in how to update the code can be found at https://github.com/mozilla/perf-youtube-playback/wiki.