Confirmed users
91
edits
QA/Automation/Projects/WebRTC/Crashtest Guide: Difference between revisions
QA/Automation/Projects/WebRTC/Crashtest Guide (view source)
Revision as of 03:52, 29 June 2016
, 29 June 2016Replace mxr link with dxr link.
(Replace mxr link with dxr link.) |
|||
| (17 intermediate revisions by 2 users not shown) | |||
| Line 2: | Line 2: | ||
==Setup your system for WebRTC== | ==Setup your system for WebRTC== | ||
There are two ways to get ready for WebRTC testing. The first one ([https://wiki.mozilla.org/Auto-tools/Automation_Development/Projects/Crashtest_Guide#Short-term_setup Short-term setup]) is most likely what you want to do for a testday but if you want to contribute code more often to the project you should have your own debug build of Firefox ([https://wiki.mozilla.org/Auto-tools/Automation_Development/Projects/Crashtest_Guide#Long-term_setup Long-term setup]). In any case, the first thing you should do in either case is to install mozdownload. | |||
===Download and install mozdownload=== | |||
Go to [http://pypi.python.org/pypi/mozdownload PyPI] and install it or run 'pip install mozdownload'. If you don't want to install it globally you can use a [http://pypi.python.org/pypi/virtualenv virtualenv]. | |||
#Download a platform appropriate build from | For information on how to use mozdownload, please see the above website. | ||
===Short-term Setup=== | |||
Download one of the tinderbox builds of the [http://hg.mozilla.org/projects/alder/ alder branch]: | |||
#Download a platform appropriate build calling the following from command line: <tt>mozdownload -t tinderbox --branch=alder</tt> | |||
#Extract the alder-build into a suitable directory of your choice | |||
#Make a note of the URL that mozdownload uses to download the build from and obtain the tests.zip archive from the same location | |||
#*e.g. mozdownload output | |||
Downloading build: https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/alder-linux64/1355348662/firefox-20.0a1.en-US.linux-x86_64.tar.bz2 | |||
< | Therefor get the tests.zip archive from: <tt>https://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/alder-linux64/1355348662/</tt> | ||
===Long-term setup=== | |||
Build Firefox on your own (be warned that this will take longer) | Build Firefox on your own (be warned that this will take longer) | ||
<ol> | |||
<li>Follow the instructions on https://developer.mozilla.org/En/Simple_Firefox_build to prepare your system and to build Firefox</li> | |||
<li>As best create a debug, non-optimized build of Firefox. Here the content of a possible <tt>.mozconfig</tt> file<br> | |||
<pre> | |||
. $topsrcdir/browser/config/mozconfig | |||
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj | |||
mk_add_options MOZ_MAKE_FLAGS="-s -j8" # the number behind -j depends on the available cores of your machine | |||
ac_add_options --enable-debug | |||
ac_add_options --enable-debugger-info-modules | |||
ac_add_options --enable-debug-symbols | |||
ac_add_options --disable-optimize | |||
export MOZ_WEBRTC_LEAKING_TESTS=1 | |||
</pre> | |||
</li> | |||
<li>Build Firefox off the mozilla-central branch aka Nightly</li> | |||
</ol> | |||
Other useful webpages are: | Other useful webpages are: | ||
| Line 30: | Line 47: | ||
*https://developer.mozilla.org/en-US/docs/Developer_Guide/mach | *https://developer.mozilla.org/en-US/docs/Developer_Guide/mach | ||
== | ===Running existing Tests=== | ||
Another thing to note for later is how to run existing tests. Depending on how you got your Firefox build there are two ways in running the tests for WebRTC. For now we will only focus on crashtests because mochitests aren't ready yet. | |||
With packaged tests ([https://wiki.mozilla.org/Auto-tools/Automation_Development/Projects/Crashtest_Guide#Short-term_setup Short-term setup]): | |||
python reftest/runreftest.py reftest/tests/dom/media/tests/crashtests/crashtests.list --appname=%path_to_binary% | |||
With a self-build Firefox ([https://wiki.mozilla.org/Auto-tools/Automation_Development/Projects/Crashtest_Guide#Long-term_setup Long-term setup]): | |||
./mach crashtest dom/media/tests/crashtests | |||
The following is best explained with an example: | ==Creating a new Crash Test== | ||
The following process is best explained with an example. There is a brief section at the end on how to best choose a suitable bug candidate for a crashtest, so if you only want to use this as a guideline rather than work your way through it first time, feel free to look at the Section on [https://wiki.mozilla.org/Auto-tools/Automation_Development/Projects/Crashtest_Guide#How_to_find_a_good_bug_to_write_a_crashtest_for.3F 'How to find a suitable bug to write a crashtest for']. | |||
We take a | We take a [https://bugzilla.mozilla.org/show_bug.cgi?id=803535 803535] as an example (WebRTC crash [@mozilla::MediaEngineDefault::EnumerateAudioDevices]). | ||
Take the submitted testcase and | Take the submitted testcase and download it to your local harddrive. Then obtain a build from the day this bug was submitted using <tt>mozdownload</tt>. In this case it was: <tt>2012-10-19 07:31 PDT</tt> (top right of the bug report) | ||
Now call | Now call | ||
| Line 54: | Line 71: | ||
Unpack this version of firefox and start it from command line with | Unpack this version of firefox and start it from command line with | ||
home:~/[path]/firefox-2012-10-19$ ./firefox | home:~/[path]/firefox-2012-10-19$ ./firefox -no-remote -p | ||
This will ask you to choose a profile. I would suggest creating a new one with a fitting title such 'WebRTC'. Also you have to make sure that the preferences <tt>media.navigator.enabled</tt> and <tt>media.peerconnection.enabled</tt> in <tt>about:config</tt> are set to <tt>true</tt> before testing for the crash. If those are not enabled, nothing will happen. | |||
Now access the previously | Now access the previously downloaded testcase file from this instance of Firefox and check if the application crashes. In this specific case, the nightly becomes unresponsive but does not crash. If any of this does not happen, then download a build earlier in date and repeat until you can reproduce the error. | ||
Note: If it crashes straight away, it is possible to check the stacktrace by starting the nightly build and going on the page <tt>about:crashes</tt>. | |||
If it crashes straight away, it is possible to check the stacktrace by starting the nightly build and going on the page <tt>about:crashes</tt> | |||
==Recreate the testcase in a firefox-debug build and compare to the callstack== | ==Recreate the testcase in a firefox-debug build and compare to the callstack== | ||
===Building the debug=== | ===Building the debug=== | ||
We now need to test this scenario in a full debug build. In this case, firefox only froze, so we need to make sure it crashes in order to compare the given stacktrace (in the bug) and the one obtained. We will now revert our firefox nightly build to the version that it was before that specific bug report. We shall do this by backing out of this particular patch with a simple command line entry. In our case, go to this page and save the file on to your hard drive. | |||
We now need to test this scenario in a full debug build. In this case, firefox only froze, so we need to make sure it crashes in order to compare the given stacktrace (in the bug) and the one obtained. | |||
We will now revert our firefox nightly build to the version that it was before that specific bug report. We shall do this by backing out of this particular patch with a simple command line entry. In our case, go to this page and save the file on to your hard drive. | |||
https://bug803535.bugzilla.mozilla.org/attachment.cgi?id=673403 | https://bug803535.bugzilla.mozilla.org/attachment.cgi?id=673403 | ||
| Line 83: | Line 94: | ||
as an example, I ran in this case <tt>patch -p1 -R < revert-patch-bug-803535.patch</tt>, however the file can be called whatever you want. | as an example, I ran in this case <tt>patch -p1 -R < revert-patch-bug-803535.patch</tt>, however the file can be called whatever you want. | ||
Note: In order to revert this change call either <tt>hg revert –all</tt> or <tt>patch -p1 < %filename%</tt>. It might also be wise to make sure at this point that the changes were truly implemented. | |||
In order to revert this change call either <tt>hg revert –all</tt> or <tt>patch -p1 < %filename%</tt>. | |||
It might also be wise to make sure at this point that the changes were truly implemented. | |||
Now it is time, to (re-)build our firefox debug build. To do this, call the following from command line | Now it is time, to (re-)build our firefox debug build. To do this, call the following from command line | ||
nice -19 | nice -19 ./mach build | ||
This should only take a couple of minutes given that we are only changing a couple of files from the previous build. The time taken to build this, however, may take longer depending on the amount of changes that were made since then. | This should only take a couple of minutes given that we are only changing a couple of files from the previous build. The time taken to build this, however, may take longer depending on the amount of changes that were made since then. | ||
Note: If the building process takes a very long time, you may want to consult the following websites to potentially adjust the running time: | |||
If the building process takes a very long time, you may want to consult the following websites to potentially adjust the running time: | |||
*https://developer.mozilla.org/en-US/docs/Developer_Guide/Mozilla_build_FAQ#Making_builds_faster | *https://developer.mozilla.org/en-US/docs/Developer_Guide/Mozilla_build_FAQ#Making_builds_faster | ||
*https://developer.mozilla.org/en-US/docs/Developer_Guide/mach | *https://developer.mozilla.org/en-US/docs/Developer_Guide/mach | ||
===Check the testcase in the debug=== | ===Check the testcase in the debug build=== | ||
Here we have two possibilities: If the testcase crashes the browser, then we can retrieve the callstack from about:crashes in the browser. In this case, follow to the next subheading “Compare the callstack”. It needs to crash, a frozen screen like in our case is no good. | |||
Note: Make sure that the preferences <tt>media.navigator.enabled</tt> and <tt>media.peerconnection.enabled</tt> in <tt>about:config</tt> are set to <tt>true</tt> before testing for the crash. If those are not enabled, nothing will happen. | |||
If this, however, fails the next option is to start firefox using <tt>gdb</tt> (for more information, please consult the following webpage: https://developer.mozilla.org/en-US/docs/Debugging_Mozilla_with_gdb ). | If this, however, fails the next option is to start firefox using <tt>gdb</tt> (for more information, please consult the following webpage: https://developer.mozilla.org/en-US/docs/Debugging_Mozilla_with_gdb ). | ||
Therefore from the command line call: | Therefore from the command line call: | ||
gdb % | gdb %path_to_firefox_binary% | ||
and once you see the prompt, type: | and once you see the prompt, type: | ||
| Line 122: | Line 122: | ||
r -p %profilename% | r -p %profilename% | ||
Note: If you don't know your profile name, call <tt>./firefox</tt> with the <tt>-ProfileManager</tt> flag, to see the available profiles, i.e. <tt>./firefox -ProfileManager</tt>. | |||
If you don't know your profile name, call <tt>./firefox</tt> with the <tt>-ProfileManager</tt> flag, to see the available profiles, i.e. <tt>./firefox -ProfileManager</tt>. | |||
Once the browser becomes unresponsive or crashes, check the output in the <tt>gdb</tt> prompt and look for an exception. In our case, you would see something like this: | Once the browser becomes unresponsive or crashes, check the output in the <tt>gdb</tt> prompt and look for an exception. In our case, you would see something like this: | ||
| Line 180: | Line 176: | ||
===Minimising the testcase=== | ===Minimising the testcase=== | ||
Now it is time to create the crashtest. The first thing to do, is to minimize the <tt>testcase.html</tt> to a minimal example in which the error still occurs. Of course, this is different for each testcase and has to be assessed on a case by case basis. | |||
Now it is time to create the crashtest. The first thing to do, is to | In our case, it is possible to remove the following lines from the <tt><body></body></tt> tags: | ||
In our case, it is possible to remove the following | |||
<pre> | <pre> | ||
| Line 198: | Line 193: | ||
</pre> | </pre> | ||
Note: Make sure the “same” errors occur. Not just any error. | |||
Make sure the “same” errors occur. Not just any error. | |||
===Preparing the crashtest=== | ===Preparing the crashtest=== | ||
All the details about how to create a crashtest can be found at: https://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/README.txt | |||
Now that we have minimized our testcase, we can prepare it to be used as a crashtest. In order to do this, we must add the following to the <tt><html></tt> tag of the testcase file: | |||
Now that we have | |||
<pre> | <pre> | ||
| Line 224: | Line 210: | ||
</pre> | </pre> | ||
to the javascript part of the testcase in order to let the crashtest framework know that the test is done now. Where to exactly put this can be gathered from the other crashtest files. | to the javascript part of the testcase in order to let the crashtest framework know that the test is done now. Where to exactly put this can be gathered from the other crashtest files but will depend on the nature of the crashtest you are working on. If still in doubt, it is always a possibility to ask someone in IRC in the <tt>#automation</tt> channel. They are always happy to help. | ||
If still in doubt, it is always a possibility to ask someone in IRC in the <tt>#automation</tt> channel. They are always happy to help. | |||
To round it off, give the file a meaningful title, that explains what this test should actually test and transfer it to <tt>$HOME/mozilla-central/dom/media/tests/crashtests</tt> | To round it off, give the file a meaningful title, that explains what this test should actually test and transfer it to <tt>$HOME/mozilla-central/dom/media/tests/crashtests</tt> | ||
| Line 236: | Line 221: | ||
===Testing your crashtest=== | ===Testing your crashtest=== | ||
Now is the time to test your crashtest in a real case scenario. In order to do this, add the following line to <tt>crastest.list</tt> in the <tt>$HOME/mozilla-central/dom/media/tests/crashtests</tt> directory. | |||
load %filename% | |||
Note: If you just want to test your crashtest, you can comment out the other lines by preceding each line with a '#' (without inverted commas). | |||
Now go to <tt>$HOME/mozilla-central/</tt> and run the crasthest | Now go to <tt>$HOME/mozilla-central/</tt> and run the crasthest | ||
./mach crashtest dom/media/tests/crashtests | |||
If you use the Short-term build, you will have to call | |||
python reftest/runreftest.py reftest/tests/dom/media/tests/crashtests/crashtests.list --appname=%path_to_binary% | |||
What we should now see, is a failed crashtest with the error message that you have seen earlier. To give you a guideline example again, I have shown the excerpt from the crashtest for this bug. | What we should now see, is a failed crashtest with the error message that you have seen earlier. To give you a guideline example again, I have shown the excerpt from the crashtest for this bug. | ||
| Line 266: | Line 251: | ||
make: *** [crashtest] Error 1 | make: *** [crashtest] Error 1 | ||
make: Leaving directory `/home/nebelhom/Mozilla-Dev/mozilla-central/obj' | make: Leaving directory `/home/nebelhom/Mozilla-Dev/mozilla-central/obj' | ||
As you can see from the highlighted section, the error was raised again and we successfully created a crashtest! Well done!!! | As you can see from the highlighted section, the error was raised again and we successfully created a crashtest! Well done!!! | ||
===Testing on the current version=== | ===Testing on the current version=== | ||
Now all that is left to be done, is to test the crashtest on the current version of firefox and if everything is ok, create a patch file which can be submitted. | Now all that is left to be done, is to test the crashtest on the current version of firefox and if everything is ok, create a patch file which can be submitted. | ||
In order to revert this change call either <tt>hg revert –all</tt> (if you are using mercurial) or <tt>patch -p1 < %filename%</tt>. It might also be wise to make sure at this point that the changes were truly implemented. At this juncture just run another crashtest and see if the output is expected (in our case, the crashtest should pass) | In order to revert this change call either <tt>hg revert –all</tt> (if you are using mercurial) or <tt>patch -p1 < %filename%</tt>. It might also be wise to make sure at this point that the changes were truly implemented. At this juncture just run another crashtest and see if the output is expected (in our case, the crashtest should pass) | ||
./mach crashtest dom/media/tests/crashtests | |||
or | |||
python reftest/runreftest.py reftest/tests/dom/media/tests/crashtests/crashtests.list --appname=%path_to_binary% | |||
===Create a patch=== | ===Create a patch=== | ||
Now, all that is left is to create a patch and submit it to the bug report to be implemented. | Now, all that is left is to create a patch and submit it to the bug report to be implemented. | ||
==How to find a good bug to write a crashtest for?== | ==How to find a good bug to write a crashtest for?== | ||
These are only guidelines and there is no easy rule to obey here. | These are only guidelines and there is no easy rule to obey here. | ||
*Find a bug that crashes straight away. If you have to do reloads or anything like this to re-create the crash, then it is probably not good for a start. | * Find a bug that crashes straight away. If you have to do reloads or anything like this to re-create the crash, then it is probably not good for a start. | ||
*If the bug has the keywords <tt>crash</tt> or <tt>in-testsuite</tt>, that is a good sign (<tt>in-testsuite</tt> may also mean <tt>mochitest</tt>, so be careful there). | * If the bug has the keywords <tt>crash</tt> or <tt>in-testsuite</tt>, that is a good sign (<tt>in-testsuite</tt> may also mean <tt>mochitest</tt>, so be careful there). | ||
*If you are unsure, you can always ask ;) | * If you are unsure, you can always ask ;) | ||