Firefox OS/Performance/Datazilla: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
Line 14: Line 14:
=== Gaia Tests ===
=== Gaia Tests ===
* '''Startup Time''': This is gathered using the 'test-perf' make target with Gaia, and the data is submitted to Datazilla using [https://github.com/mozilla/b2gperf b2gperf].  The test-perf jobs can measure other arbitrary performance metrics that are designed by the test authors (like "rendering_WiFi_list_>_settings-panel-wifi-ready"); the job that submits data to datazilla can operate on arbitrary metric names without any changes to the datazilla infrastructure.  All the logic which runs these tests is part of Gaia; the output of 'make test-perf' includes a JSON structure which b2gperf submits to datazilla.
* '''Startup Time''': This is gathered using the 'test-perf' make target with Gaia, and the data is submitted to Datazilla using [https://github.com/mozilla/b2gperf b2gperf].  The test-perf jobs can measure other arbitrary performance metrics that are designed by the test authors (like "rendering_WiFi_list_>_settings-panel-wifi-ready"); the job that submits data to datazilla can operate on arbitrary metric names without any changes to the datazilla infrastructure.  All the logic which runs these tests is part of Gaia; the output of 'make test-perf' includes a JSON structure which b2gperf submits to datazilla.
Here is an example flow of an app using the startup time test today:
Process Startup -> Start Timer -> Make API calls -> Onload Event (Fire startup time event) -> API calls return, app useable.
There are a few problems with this test:
** The time between Process Startup and starting the timer is not constant. Ideally we would start the timer before we run startup. I was informed that this was difficult. TODO: File a bug if this does not exist.
** The startup time can be taken before the app is really ready. There are bugs open to add "startup path done" tests for the various apps.
** Test code: https://github.com/mozilla-b2g/gaia/blob/master/tests/performance/startup_events_test.js
** Test code: https://github.com/mozilla-b2g/gaia/blob/master/tests/performance/startup_events_test.js
* '''Startup Path Done''': Test which listens to a custom event of when the startup is done. Currently not implemented for every test. Triggered after the startup-path-done event.
* '''Startup Path Done''': Test which listens to a custom event of when the startup is done. Currently not implemented for every test. Triggered after the startup-path-done event.

Revision as of 00:38, 27 July 2013

Datazilla

This page details Datazilla and how we use it to triage performance problems in Firefox OS.

Collection

Data in Datazilla is collected from automated tests. Some of these tests reside within gaia in tests/performance, and there are also Python tests on github in the b2gperf repository. There is a Jenkins instance in Mountain View which runs the automated tests against physical devices.

Current Tests

  • Cold Load Time: b2gperf measures time taken from launching an app to the apploadtime event firing. This generates app startup times which are useful for identifying some kinds of performance regressions, but which are much shorter than camera-based startup times measured by Eideticker. This is the exact same test as startup time, but it runs more iterations.
  • Endurance: Gaia UI tests that perform the same few interactions repeatedly in order to identify memory leaks or performance issues. These are run infrequently due to the amount of time they take to run.
  • FPS: b2gperf performs a smooth scroll/swipe against the app and measures the frames per second using Gecko

Gaia Tests

  • Startup Time: This is gathered using the 'test-perf' make target with Gaia, and the data is submitted to Datazilla using b2gperf. The test-perf jobs can measure other arbitrary performance metrics that are designed by the test authors (like "rendering_WiFi_list_>_settings-panel-wifi-ready"); the job that submits data to datazilla can operate on arbitrary metric names without any changes to the datazilla infrastructure. All the logic which runs these tests is part of Gaia; the output of 'make test-perf' includes a JSON structure which b2gperf submits to datazilla.

Here is an example flow of an app using the startup time test today: Process Startup -> Start Timer -> Make API calls -> Onload Event (Fire startup time event) -> API calls return, app useable.

There are a few problems with this test:

    • The time between Process Startup and starting the timer is not constant. Ideally we would start the timer before we run startup. I was informed that this was difficult. TODO: File a bug if this does not exist.
    • The startup time can be taken before the app is really ready. There are bugs open to add "startup path done" tests for the various apps.

apploadtime: The event which we listen on for Startup Time and Cold Load Time tests. Gaia fires the apploadtime event, after listening to the gecko mozbrowserloadend event. Happens after all deferred scripts are loaded. This is the same as the load event. (There is no load event on a moz browser). We have to look at the code for each app to see exactly when this happens.

Triaging Regressions

Each point in datazilla has a gaia, gecko, and platform build revision id. You can use this information to triage and locate where the regression occurred.

Git Bisect

You can use git bisect to narrow down the range between two datapoints. This is useful for finding offending gaia commits. You can select two adjacent ranges and git bisect from one to another. Git bisect tutorial: http://webchick.net/node/99

Profiling

One way of doing this is to use the B2G profiler: https://github.com/mozilla-b2g/B2G/blob/master/profile.sh More information on profiling: https://developer.mozilla.org/en-US/docs/Performance/Profiling_with_the_Built-in_Profiler

Resources

Retrieved from "https://wiki.allizom.org/index.php?title=Firefox_OS/Performance/Datazilla&oldid=686279"