Sheriffing/How To/Bisecting

Bisecting

Sheriffs are doing bisecting to find out what changeset cause a failure if we can’t determine this via code inspection.

Fictitious Scenario:

After a merge from integration to m-c the the xpcshell tests on linux-asan are busted and it’s not clear which changeset cause - inbound and autoland were fine.

So the Sheriff on duty starts bisecting:

Changes of the merge (complete Fictitious)

2c497462f25e Merge inbound to m-c a=merge 02851079c451Bug 1359458 - Increase assertion count range for test_bug437844.xul. r=jmaher B37b46c7f38f Bug 1358241 - [2.1] Add mutex locking around the library handles cache. r=jchen 751455b663d0 Bug 1358241 - [1.2] Make direct library reference counter atomic to avoid mutex locking issues. r=jchen 841fa5fb06a8 Bug 1355676 - Check for nulls when decoding icons. r=sebastian Ad9d525e6db7 Bug 1356243 - Enable Screenshots by default. r=Mossop 2e44294b9f5c Bug 1359273 - Split up DevTools' sort-arrows.svg to improve performance. r=jryans

Requirements for bisecting

-> Clone mozilla-central -> Add the try server settings (see https://wiki.mozilla.org/ReleaseEngineering/TryServer#Configuration ) -> Find the try syntax (trychooser might be a big help). In this scenario its try: -b o -p linux64-asan -u xpcshell -t none --rebuild 10 (The rebuild 10 means that you basically run the xpcshell tests 10 times and not once)

First step:

To make sure/give yourself confirmation that you set everything up correctly and that you can reproduce the problem on try, do a try push with m-c tip as topmost revision.

Make a dummy change (like touch clobber or so to have a change). Commit with something like -m “central tip try: -b o -p linux64-asan -u xpcshell -t none --rebuild 10” Push to try Check that you get the same failure etc

Second Step - now bisecting begins

Now it’s on you to start bisecting.

There are several approaches possible, one would be to split up the merge and check if the test failure already starts in the middle of the merge. So you decide to run a try run with 841fa5fb06a8 as topmost revision.

-> hg up -r 841fa5fb06a8 to update to the older topmost rev -> you can confirm this with doing: hg summary and it should show something like

   parent: 354839:841fa5fb06a8
    Bug 1355676 - Check for nulls when decoding icons. r=sebastian

-> hg commit -m "central rev 841fa5fb06a8 try: -b o -p linux64-asan -u xpcshell -t none --rebuild 10"

-> push to try

-> Check the results - let’s assume results for 841fa5fb06a8 are green, so you need to do the same steps again for the latter changes (02851079c451 and B37b46c7f38f / 751455b663d0 ) to check which of this 2 bugs cause the problem.

Note: in case of bisecting for an actual issue you can push up to 6 pushes to try at the same time to be able to have results asap.