Firefox OS/Performance/Bisecting Regressions

Bisecting Regressions

This page will walk you through finding and bisecting performance regressions in Firefox OS.

Datazilla

Datazilla makes it easy to spot performance regressions in Firefox OS apps. The page that shows the performance test numbers is here: [1].

Performance regressions show up as spikes in the graphs like so (circled in red):

When you find a regression, you need to get the "regression range" for the regression. The regression range is the last known good revision to the first known bad revision for both Gecko and Gaia. To get the last known good revision IDs for both, click on the data point in the graph right before the spike. You will see something like this:

The Gaia and Gecko revisions used in the build and test that generated that data point are shown, circled in red. Write these down. You only need the first 7 characters of the IDs. So in this case you would write down d4655e7 for Gaia and a9f77f8 for Gecko.

To get the first know bad revision, click on the first data point after the spike. You will see something like this:

Again, you need to write down the revision IDs for both Gecko and Gaia, circled in red. So you would write down 9da1b9c for Gaia and b31f92b for Gecko.

The regression ranges are:

Gecko a9f77f8..b31f92b

Gaia d4655e7..9da1b9c

Setting up for Bisecting

You typically start off by creating a build of the latest master branch code for your device. I'm using the hamachi device in this tutorial because it is the device that Datazilla runs its tests on. To build master, do the following:

$ ENABLE_MARIONETTE=1 ./config.sh hamachi
$ ENABLE_MARIONETTE=1 ./build.sh -j
$ ENABLE_MARIONETTE=1 ./flash.sh

Confirm that it builds, flashes, and the device reboots successfully.

Running b2gperf

This tutorial assumes that you have the b2gperf utility correctly set up on your host PC. If not, follow the instructions.

In our example, we are looking at the Contacts app. To run the tests that datazilla runs, you would execute b2gperf like so:

$ b2gperf --delay=10 --iterations=30 Contacts

These tests sometimes take a while to complete, so be patient. After it is done running, it will print out the results for you to compare to datazilla's numbers. I usually use the median value rather than the average value because it is influenced less by outliers in the gathered data.

Is it in Gecko or Gaia?

The first step in bisecting is to determine if the regression is in Gecko or Gaia. To do this, you have to test all four combinations of Gecko and Gaia revisions from the regression range. You need to test:

• Gecko a9f77f8 + Gaia d4655e7
• Gecko a9f77f8 + Gaia 9da1b9c
• Gecko b31f92b + Gaia 9da1b9c
• Gecko b31f92b + Gaia d4655e7

You use git to check out the last known good revisions of Gecko and Gaia:

$ cd gecko
$ git checkout a9f77f8
$ cd ../gaia
$ git checkout d4655e7
$ cd ..

Now that you have the correct revisions, time to do another build.

$ ENABLE_MARIONETTE=1 ./build.sh -j
$ ENABLE_MARIONETTE=1 ./flash.sh

Now, run b2gperf to see if the numbers match those on Datazilla:

$ adb forward tcp:2828 tcp:2828
$ b2gperf --delay=10 --iterations=30 Contacts
mozversion.mozversion.RemoteB2GVersion INFO | Unable to find system/sources.xml
2014-04-16 19:26:57,244 B2GPerfRunner WARNING | Missing required DataZilla field: build revision
2014-04-16 19:26:57,244 B2GPerfRunner WARNING | Missing required DataZilla field: device name
2014-04-16 19:26:57,244 B2GPerfRunner WARNING | Missing required DataZilla field: oauth key
2014-04-16 19:26:57,245 B2GPerfRunner WARNING | Missing required DataZilla field: project
2014-04-16 19:26:57,245 B2GPerfRunner WARNING | Missing required DataZilla field: gecko revision
2014-04-16 19:26:57,245 B2GPerfRunner WARNING | Missing required DataZilla field: oauth secret
2014-04-16 19:26:57,245 B2GPerfRunner WARNING | Missing required DataZilla field: machine name
2014-04-16 19:26:57,245 B2GPerfRunner WARNING | Missing required DataZilla field: branch
2014-04-16 19:26:57,245 B2GPerfRunner INFO    | Reports will not be submitted to DataZilla
2014-04-16 19:26:58,097 B2GPerfRunner INFO    | Running B2GPerfLaunchContactsTest
2014-04-16 19:26:58,397 B2GPopulate INFO | Populating 200 contacts
2014-04-16 19:28:28,570 B2GPerfRunner INFO    | Contacts [1/30]
2014-04-16 19:28:40,865 B2GPerfRunner INFO    | Contacts [2/30]
2014-04-16 19:28:52,123 B2GPerfRunner INFO    | Contacts [3/30]
2014-04-16 19:29:03,385 B2GPerfRunner INFO    | Contacts [4/30]
2014-04-16 19:29:14,700 B2GPerfRunner INFO    | Contacts [5/30]
2014-04-16 19:29:26,002 B2GPerfRunner INFO    | Contacts [6/30]
2014-04-16 19:29:37,328 B2GPerfRunner INFO    | Contacts [7/30]
2014-04-16 19:29:48,592 B2GPerfRunner INFO    | Contacts [8/30]
2014-04-16 19:29:59,823 B2GPerfRunner INFO    | Contacts [9/30]
2014-04-16 19:30:11,109 B2GPerfRunner INFO    | Contacts [10/30]
2014-04-16 19:30:22,466 B2GPerfRunner INFO    | Contacts [11/30]
2014-04-16 19:30:33,767 B2GPerfRunner INFO    | Contacts [12/30]
2014-04-16 19:30:45,062 B2GPerfRunner INFO    | Contacts [13/30]
2014-04-16 19:30:56,369 B2GPerfRunner INFO    | Contacts [14/30]
2014-04-16 19:31:07,664 B2GPerfRunner INFO    | Contacts [15/30]
2014-04-16 19:31:18,976 B2GPerfRunner INFO    | Contacts [16/30]
2014-04-16 19:31:30,280 B2GPerfRunner INFO    | Contacts [17/30]
2014-04-16 19:31:41,583 B2GPerfRunner INFO    | Contacts [18/30]
2014-04-16 19:31:52,900 B2GPerfRunner INFO    | Contacts [19/30]
2014-04-16 19:32:04,212 B2GPerfRunner INFO    | Contacts [20/30]
2014-04-16 19:32:15,524 B2GPerfRunner INFO    | Contacts [21/30]
2014-04-16 19:32:26,824 B2GPerfRunner INFO    | Contacts [22/30]
2014-04-16 19:32:38,122 B2GPerfRunner INFO    | Contacts [23/30]
2014-04-16 19:32:49,445 B2GPerfRunner INFO    | Contacts [24/30]
2014-04-16 19:33:00,803 B2GPerfRunner INFO    | Contacts [25/30]
2014-04-16 19:33:12,114 B2GPerfRunner INFO    | Contacts [26/30]
2014-04-16 19:33:23,431 B2GPerfRunner INFO    | Contacts [27/30]
2014-04-16 19:33:34,747 B2GPerfRunner INFO    | Contacts [28/30]
2014-04-16 19:33:46,057 B2GPerfRunner INFO    | Contacts [29/30]
2014-04-16 19:33:57,374 B2GPerfRunner INFO    | Contacts [30/30]
2014-04-16 19:33:57,376 B2GPerfRunner INFO    | Results for Contacts, cold_load_time: median:910, mean:935, std: 148, max:1732, min:877,
all:923,1732,890,898,915,911,913,896,877,917,886,908,904,909,910,911,904,902,909,916,919,910,909,918,942,910,916,902,914,908

So in this example the median value is 910, in Datazilla it is 885. They are within 5% of each other so I consider that a match. Now let's change the Gaia revision to the first known bad revision and re-run the tests to see what we get:

$ cd gaia
$ git checkout 9da1b9c
$ cd ..
$ ENABLE_MARIONETTE=1 ./build.sh -j
$ ENABLE_MARIONETTE=1 ./flash.sh
$ adb forward tcp:2828 tcp:2828
$ b2gperf --delay=10 --iterations=30 Contacts
.
.
.
2014-04-16 19:53:03,381 B2GPerfRunner INFO    | Results for Contacts, cold_load_time: median:1306, mean:1332, std: 95, max:1818, min:1282,
all:1326,1818,1312,1306,1303,1303,1305,1295,1312,1303,1294,1292,1306,1286,1320,1297,1282,1307,1310,1323,1304,1312,1325,1292,1379,1334,1311,1365,1440,1305

The median value has jumped up significantly. This certainly looks like the regression was caused by Gaia. Let's make sure by now updating Gecko to its first known bad revision and re-running b2gperf:

$ cd gecko
$ git checkout b31f92b
$ cd ..
$ ENABLE_MARIONETTE=1 ./build.sh -j
$ ENABLE_MARIONETTE=1 ./flash.sh
$ adb forward tcp:2828 tcp:2828
$ b2gperf --delay=10 --iterations=30 Contacts
.
.
.
2014-04-16 20:30:27,323 B2GPerfRunner INFO    | Results for Contacts, cold_load_time: median:1304, mean:1318, std: 84, max:1752, min:1208,
all:1323,1752,1305,1308,1296,1271,1302,1273,1271,1300,1282,1323,1299,1208,1295,1317,1313,1330,1326,1297,1318,1287,1308,1307,1343,1323,1304,1358,1300,1302

You can see that the median value is 1304, which is nearly identical to the median of the previous result. This tells you that the regression is definitely in Gaia and you don't need to test the fourth possible combination of Gecko and Gaia.

Bisecting Gecko/Gaia

To bisect the Gaia code, you will use the git bisect command to automate the selection of revisions to test. It does a binary search for the revision that caused the regression. Before you start your own bisect, I need to point out that the "git bisect start" command takes the revision IDs of your regression range in BAD then GOOD order like so:

$ git bisect start <first know bad> <last known good>

So to kick off the bisect of Gaia, I run:

$ git bisect start