Firefox OS/Performance/Bisecting Regressions

Bisecting Regressions

This page will walk you through finding and bisecting performance regressions in Firefox OS.

Datazilla

Datazilla makes it easy to spot performance regressions in Firefox OS apps. The page that shows the performance test numbers is here: [1].

Performance regressions show up as spikes in the graphs like so (circled in red):

When you find a regression, you need to get the "regression range" for the regression. The regression range is the last known good revision to the first known bad revision for both Gecko and Gaia. To get the last known good revision IDs for both, click on the data point in the graph right before the spike. You will see something like this:

The Gaia and Gecko revisions used in the build and test that generated that data point are shown, circled in red. Write these down. You only need the first 7 characters of the IDs. So in this case you would write down d4655e7 for Gaia and a9f77f8 for Gecko.

To get the first know bad revision, click on the first data point after the spike. You will see something like this:

Again, you need to write down the revision IDs for both Gecko and Gaia, circled in red. So you would write down 9da1b9c for Gaia and b31f92b for Gecko.

The regression ranges are:

Gecko a9f77f8..b31f92b

Gaia d4655e7..9da1b9c

Is it in Gecko or Gaia?

The first step in bisecting is to determine if the regression is in Gecko or Gaia. To do this, you have to test all four combinations of Gecko and Gaia revisions from the regression range. You need to test:

• Gecko a9f77f8 + Gaia d4655e7
• Gecko a9f77f8 + Gaia 9da1b9c
• Gecko b31f92b + Gaia d4655e7
• Gecko b31f92b + Gaia 9da1b9c

To do that you start off by creating a build of the master branch for your device:

$ ./config.sh hamachi
$ ./build.sh -j
$ ./flash.sh

Next, you need to use git to check out the correct versions of Gecko and Gaia:

$ cd gecko
$ git checkout a9f77f8
$ cd ../gaia
$ git checkout d4655e7

Now that you have the correct versions of both checked out, time to do another build.

$ cd ..
$ ./build.sh -j
$ ./flash.sh

If the builds completed and the device rebooted correctly after the flash step, you know have a device running the same revisions of Gecko and Gaia as were used to generate the last known good numbers in datazilla.

Running b2gperf

This tutorial assumes that you have the b2gperf utility correctly set up on your host PC. If not, follow the instructions.

In our example, we are looking at the