QA/Automation/Projects/WebRTC/Crashtest Guide

Step by Step Guide to WebRTC Crashtests

Setup your system for WebRTC

There are two ways to get ready for WebRTC testing. The first one (Short-term setup) is most likely what you want to do for today but if you want to contribute code more often to the project you should have your own debug build of Firefox (Long-term setup). In any case, the first thing you should do in either case is to install mozdownload.

Download and install mozdownload

Go to http://pypi.python.org/pypi/mozdownload and install it.

For information on how to use mozdownload, please see the above website.

Short-term setup

Download one of the tinderbox builds in the alder branch

1. Download a platform appropriate build calling the following from command line: mozdownload -t tinderbox --branch=alder
2. Extract the alder-build into a suitable directory of your choice
3. Make a not of the url that mozdownload loads the build from and obtain the tests.zip archive from the same url
   • e.g. mozdownload output

   Downloading build: https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/alder-linux64/1355348662/firefox-20.0a1.en-US.linux-x86_64.tar.bz2

Therefore get the tests.zip archive from: https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/alder-linux64/1355348662/

Long-term setup

Build Firefox on your own (be warned that this will take longer)

1. Follow the instructions on https://developer.mozilla.org/En/Simple_Firefox_build to prepare your system and to build Firefox
2. As best create a debug, non-optimized build of Firefox. Here the content of a possible .mozconfig file

   . $topsrcdir/browser/config/mozconfig
   mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj
   mk_add_options MOZ_MAKE_FLAGS="-s -j8"   # the number behind -j depends on the available cores of your machine
   ac_add_options --enable-debug
   ac_add_options --enable-debugger-info-modules
   ac_add_options --enable-debug-symbols
   ac_add_options --disable-optimize
   export MOZ_WEBRTC_LEAKING_TESTS=1

1. Build Firefox off the mozilla-central branch aka Nightly

Other useful webpages are:

• https://developer.mozilla.org/en-US/docs/Simple_Firefox_build
• https://developer.mozilla.org/en-US/docs/Configuring_Build_Options
• https://developer.mozilla.org/en-US/docs/Developer_Guide/Mozilla_build_FAQ#Making_builds_faster
• https://developer.mozilla.org/en-US/docs/Developer_Guide/mach

Choose a bug (best one that is already verified) and reproduce the test case using an earlier version of firefox

The following is best explained with an example:

We take a bug, e.g Bug https://bugzilla.mozilla.org/show_bug.cgi?id=803535 critical, VERIFIED FIXED, WebRTC crash [@mozilla::MediaEngineDefault::EnumerateAudioDevices]

Take the submitted testcase and create a suitable file (usually HTML file) on your harddrive.

Now obtain a build from the day this bug was submitted using mozdownload. In this case it was: 2012-10-19 07:31 PDT (top right of the bug report)

Now call

   mozdownload --type=daily –date=2012-10-19

from the command line. This should download a file called 2012-10-19-03-05-51-mozilla-central-firefox-19.0a1.en-US.linux-x86_64.tar.bz2.

Unpack this version of firefox and start it from command line with

   home:~/[path]/firefox-2012-10-19$ ./firefox

Make sure, it is the nightly build. (Most common mistake to open it, while another non-nightly instance of firefox is already open).

Now access the previously created testcase file from this instance of firefox and check if the file crashes. In this case, the nightly becomes unresponsive but does not crash. If any of this does not happen, then download a build earlier in date and repeat until you can reproduce the error.

SIDE NOTE: If it crashes straight away, it is possible to check the stacktrace by starting the nightly build and going on the page about:crashes Also, please make sure that media.navigator.enabled and media.peerconnection.enabled in about:config is set to true before testing for the crash. If these are not enabled, nothing will happen.

Recreate the testcase in a firefox-debug build and compare to the callstack

Building the debug

We now need to test this scenario in a full debug build. In this case, firefox only froze, so we need to make sure it crashes in order to compare the given stacktrace (in the bug) and the one obtained. We will now revert our firefox nightly build to the version that it was before that specific bug report. We shall do this by backing out of this particular patch with a simple command line entry. In our case, go to this page and save the file on to your hard drive.

https://bug803535.bugzilla.mozilla.org/attachment.cgi?id=673403

In general, the file can be found by adjusting this webpage to reflect the current bug you are working on and the attachment number of the patch.

Now run the following command from the command line

   (home):~/path/to/dir$ patch -p1 -R < %filename%

as an example, I ran in this case patch -p1 -R < revert-patch-bug-803535.patch, however the file can be called whatever you want.

SIDE NOTE: In order to revert this change call either hg revert –all or patch -p1 < %filename%. It might also be wise to make sure at this point that the changes were truly implemented.

Now it is time, to (re-)build our firefox debug build. To do this, call the following from command line

   nice -19 make -f client.mk

This should only take a couple of minutes given that we are only changing a couple of files from the previous build. The time taken to build this, however, may take longer depending on the amount of changes that were made since then.

SIDE NOTE: If the building process takes a very long time, you may want to consult the following websites to potentially adjust the running time: https://developer.mozilla.org/en-US/docs/Developer_Guide/Mozilla_build_FAQ#Making_builds_faster https://developer.mozilla.org/en-US/docs/Developer_Guide/mach

Check the testcase in the debug

Here we have two possibilities: If the testcase crashes the browser, then we can retrieve the callstack from about:crashes in the browser. In this case, follow to the next subheading “Compare the callstack”. It needs to crash, a frozen screen like in our case is no good.

SIDE NOTE: Please make sure that media.navigator.enabled and media.peerconnection.enabled in about:config is set to true before testing for the crash. If these are not enabled, nothing will happen.

If this, however, fails the next option is to start firefox using gdb (for more information, please consult the following webpage: https://developer.mozilla.org/en-US/docs/Debugging_Mozilla_with_gdb ).

Therefore from the command line call:

   gdb %path_to_firefox%

and once you see the prompt, type:

   r -p %profilename%

SIDE NOTE: If you don't know your profile name, call ./firefox with the -ProfileManager flag, to see the available profiles, i.e. ./firefox -ProfileManager.

Once the browser becomes unresponsive or crashes, check the output in the gdb prompt and look for an exception. In our case, you would see something like this:

   Assertion failure: i < Length() (invalid array index), at ../../dist/include/nsTArray.h:538 
   Program received signal SIGSEGV, Segmentation fault. 
   [Switching to Thread 0x7fffc76fe700 (LWP 15827)] 
   0x00007ffff16d5420 in nsTArray<nsRefPtr<mozilla::MediaEngineAudioSource>, nsTArrayDefaultAllocator>::ElementAt (this=0x7fffd03ad850, i=0) 
   at ../../dist/include/nsTArray.h:538 
   538	    MOZ_ASSERT(i < Length(), "invalid array index"); 

At this point, the gdb prompt should reappear. Now type

   bt

and observe the output. This leads us then to the next section.

Compare the output of the crash to the callstack given in the bug

The output obtained in this via gdb, should look something like this (NOTE: If you arrived here via a crash, then consult about:crashes, find the report and scroll down. You should find the same output in pretty tabular form)

   #0  0x00007ffff16d5420 in nsTArray<nsRefPtr<mozilla::MediaEngineAudioSource>, nsTArrayDefaultAllocator>::ElementAt (this=0x7fffd6ea5b90, i=0) 
       at ../../dist/include/nsTArray.h:538 
   #1  0x00007ffff19ce182 in mozilla::MediaEngineDefault::EnumerateAudioDevices ( 
       this=0x7fffd6ea5b80, aASources=0x7fffd14fec80) 
       at /home/nebelhom/Mozilla-Dev/mozilla-central/content/media/webrtc/MediaEngineDefault.cpp:367 
   #2  0x00007ffff16d1155 in mozilla::GetUserMediaRunnable::SelectDevice ( 
       this=0x7fffc8efcb60) 
       at /home/nebelhom/Mozilla-Dev/mozilla-central/dom/media/MediaManager.cpp:505 
   #3  0x00007ffff16d0b36 in mozilla::GetUserMediaRunnable::Run ( 
       this=0x7fffc8efcb60) 
       at /home/nebelhom/Mozilla-Dev/mozilla-central/dom/media/MediaManager.cpp:408 
   #4  0x00007ffff25b7735 in nsThread::ProcessNextEvent (this=0x7fffc76a47e0, 
       mayWait=true, result=0x7fffd14fedff) 
       at /home/nebelhom/Mozilla-Dev/mozilla-central/xpcom/threads/nsThread.cpp:627 
   #5  0x00007ffff2544dec in NS_ProcessNextEvent_P (thread=0x7fffc76a47e0, 
       mayWait=true) 
       at /home/nebelhom/Mozilla-Dev/mozilla-central/obj/xpcom/build/nsThreadUtils---Type <return> to continue, or q <return> to quit--- 
       .cpp:221 
   #6  0x00007ffff25b6590 in nsThread::ThreadFunc (arg=0x7fffc76a47e0) 
       at /home/nebelhom/Mozilla-Dev/mozilla-central/xpcom/threads/nsThread.cpp:265 
   #7  0x00007ffff69e4f19 in _pt_root (arg=0x7fffc774f370) 
       at /home/nebelhom/Mozilla-Dev/mozilla-central/nsprpub/pr/src/pthreads/ptthread.c:156 
   #8  0x00007ffff73c4e9a in start_thread () 
       from /lib/x86_64-linux-gnu/libpthread.so.0 
   #9  0x00007ffff70f1cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6 
   #10 0x0000000000000000 in ?? () 

This output should be similar to the output given in the callstack of the bug (There may be some slight differences in the numbers). You have now successfully re-created the bug in question. Time to create a crashtest for this bug.

Creating a crashtest

Minimising the testcase

Now it is time to create the crashtest. The first thing to do, is to minimise the testcase.html to a minimal example in which the error still occurs. Of course, this is different for each testcase and has to be assessed on a case by case basis. In our case, it is possible to remove the following three lines from the <body></body> tags:

    var v2 = document.getElementById("pc1v"); 
    var v3 = document.getElementById("pc2v"); 
    var v4 = document.getElementById("pc3v");
    var v5 = document.getElementById("pc4a"); 
    var v6 = document.getElementById("pc5i");

    ...

    <video id="pc1v" width="100" height="100" controls></video>
    <video id="pc2v" width="100" height="100" controls></video>
    <video id="pc3v" width="100" height="100" controls></video>

SIDE NOTE: Make sure the “same” errors occur. Not just any error.

Preparing the crashtest

SIDE NOTE: For more information on this subject consult the following webpage: http://mxr.mozilla.org/mozilla-central/source/layout/tools/reftest/README.txt

Now that we have minimised our testcase, we can prepare it to be used in a crashtest. In order to do this, we must add the following to the <html> tag of the testcase file:

     <html class="reftest-wait">

Furthermore, we must add

    document.documentElement.removeAttribute("class");

to the javascript part of the testcase in order to let the crashtest framework know that the test is done now. Where to exactly put this can be gathered from the other crashtest files. If still in doubt, it is always a possibility to ask someone in IRC in the #automation channel. They are always happy to help.

To round it off, give the file a meaningful title, that explains what this test should actually test and transfer it to $HOME/mozilla-central/dom/media/tests/crashtests

To introduce the test to the framework, navigate to $OBJ_DIR/dom/media and run

   make

This should set everything up to test your crashtest.

Testing your crashtest

Now is the time to test your crashtest in a real case scenario. In order to do this, add the following line to crastest.list in the $HOME/mozilla-central/dom/media/tests/crashtests directory.

   pref(media.peerconnection.enabled,true) load %filename%

SIDE NOTE: If you just want to test your crashtest, you can comment out the other lines by preceding each line with a '#' (without inverted commas).

Now go to $HOME/mozilla-central/ and run the crasthest

   TEST_PATH=dom/media/tests/crashtests/ make -C obj/ crashtest

What we should now see, is a failed crashtest with the error message that you have seen earlier. To give you a guideline example again, I have shown the excerpt from the crashtest for this bug.

   ++DOMWINDOW == 9 (0x2ad2f35e7480) [serial = 9] [outer = 0x2ad2f1110c00] 
   WARNING: NS_ENSURE_TRUE(frame) failed: file /home/nebelhom/Mozilla-Dev/mozilla-central/layout/base/nsPresContext.cpp, line 1179 
   WARNING: NS_ENSURE_TRUE(frame) failed: file /home/nebelhom/Mozilla-Dev/mozilla-central/layout/base/nsPresContext.cpp, line 1179 
   Assertion failure: i < Length() (invalid array index), at ../../dist/include/nsTArray.h:538 
   TEST-UNEXPECTED-FAIL | file:///home/nebelhom/Mozilla-Dev/mozilla-central/dom/media/tests/crashtests/803535.html | Exited with code 11 during test run 
   INFO | automation.py | Application ran for: 0:00:07.677545 
   INFO | automation.py | Reading PID log: /tmp/tmpMHhXprpidlog 
   …
   REFTEST INFO | runreftest.py | Running tests: end. 
   crashtest failed: 
   TEST-UNEXPECTED-FAIL | file:///home/nebelhom/Mozilla-Dev/mozilla-central/dom/media/tests/crashtests/803535.html | Exited with code 11 during test run 
   TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | missing output line for total leaks! 
   make: *** [crashtest] Error 1 
   make: Leaving directory `/home/nebelhom/Mozilla-Dev/mozilla-central/obj' 

As you can see from the highlighted section, the error was raised again and we successfully created a crashtest! Well done!!!

Testing on the current version

Now all that is left to be done, is to test the crashtest on the current version of firefox and if everything is ok, create a patch file which can be submitted.

In order to revert this change call either hg revert –all (if you are using mercurial) or patch -p1 < %filename%. It might also be wise to make sure at this point that the changes were truly implemented. At this juncture just run another crashtest and see if the output is expected (in our case, the crashtest should pass)

   TEST_PATH=dom/media/tests/crashtests/ make -C obj/ crashtest

Create a patch

Now, all that is left is to create a patch and submit it to the bug report to be implemented.

How to find a good bug to write a crashtest for?

These are only guidelines and there is no easy rule to obey here.

• Find a bug that crashes straight away. If you have to do reloads or anything like this to re-create the crash, then it is probably not good for a start.
• If the bug has the keywords crash or in-testsuite, that is a good sign (in-testsuite may also mean mochitest, so be careful there).
• If you are unsure, you can always ask ;)