|
|
| (16 intermediate revisions by 3 users not shown) |
| Line 1: |
Line 1: |
| __TOC__
| | Documentation moved in-tree: https://firefox-source-docs.mozilla.org/tools/code-coverage/index.html |
| | |
| == <FONT COLOR=GREEN>What is the progress so far!!</FONT> ==
| |
| | |
| We have completely instrumented Mozilla Firefox , both C/C++ portions and JavaScript portions, twice and generated code coverage results.
| |
| | |
| Results would be discussed in a separate section.
| |
| | |
| Using automated test suites, Mochitest, XPCShell, JSTest, Chrome, Browser-Chrome, RefTest , we got approximately 55% code coverage for C/C++ portion and 50% coverage on the JavaScript portion.
| |
| | |
| We have generated a lot of trend diagrams to study the interaction between code coverage data for a given file and the number of bug fixes went into the file; code coverage Vs the number of times the file is modified and file churn Vs bug fix rate.
| |
| | |
| Results would be discussed below.
| |
| | |
| The plan is to observe the temporal changes in the above trends as we fix more bugs as well as develop new test cases.
| |
| | |
| This is what is planned for the immediate future
| |
| | |
| * We are going to identify top 3 modules that needs further analysis
| |
| * We will identify at least 6 files in each of those modules that have active churn but not good coverage
| |
| * We will work with DEV and QA to come up with scenarios to improve coverage on those files
| |
| | |
| == <FONT COLOR=GREEN>Results and Findings </FONT> ==
| |
| Code Coverage report generated using '''lcov''' looks like this
| |
| | |
| For the full report to do your own study, please use the following URL
| |
| | |
| '''http://people.mozilla.org/~mnandigama/codecoverage_html/'''
| |
| | |
| For those who just want to have a peek, here are some screen shots.
| |
| | |
| http://people.mozilla.org/~mnandigama/images/Picture%202.png
| |
| | |
| If you click on any '''component''' for example '''js'''
| |
| | |
| http://people.mozilla.org/~mnandigama/images/Picture%204.png
| |
| | |
| If you want to drill down to a line level coverage
| |
| | |
| http://people.mozilla.org/~mnandigama/images/Picture%205.png
| |
| | |
| | |
| '''This is important ''': We have done a component/module by component/module code coverage Vs Bugs fix analysis.
| |
| | |
| * In a happy path case, all files should have 100% code coverage
| |
| * In a normal path case, all files that have high rate of bugs fixed in them, should have '''HIGH''' code coverage since there are the most active files in terms of change.
| |
| * Files that have no change since a long time can have either '''HIGH''' or '''LOW''' coverage in our first iteration since they are not undergoing any change so far.
| |
| | |
| For a sample module, '''JavaScript Engine''' the code coverage Vs Bug fixes looks like this.
| |
| | |
| http://people.mozilla.org/~mnandigama/images/covBugs.png
| |
| | |
| In a bubble chart format , the same can be visualized as follows
| |
| | |
| http://people.mozilla.org/~mnandigama/images/sample3.png
| |
| | |
| The '''bigger''' the bubble, the higher it should be and all bubbles should ideally bubble up.
| |
| | |
| == <FONT COLOR=GREEN>How to instrument the Mozilla Firefox Browser </FONT> ==
| |
| | |
| === Build config ===
| |
| This is what you do to instrument the Mozilla Firefox browser.
| |
| | |
| Create a .mozconfig-gold file in your home directory with the following statements
| |
| | |
| | |
| export CFLAGS="-fprofile-arcs -ftest-coverage"
| |
| export CXXFLAGS="-fprofile-arcs -ftest-coverage"
| |
| export LDFLAGS="-lgcov -static-libgcc"
| |
| export EXTRA_DSO_LDFLAGS="-lgcov -static-libgcc"
| |
| ac_add_options --enable-application=browser
| |
| mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../mozcentral-dbg
| |
| ac_add_options --enable-debug
| |
| ac_add_options --enable-tests
| |
| ac_add_options --enable-mochitest
| |
| ac_add_options --disable-optimize
| |
| mk_add_options MOZ_MAKE_FLAGS="-j3"
| |
| mk_add_options AUTOCONF=autoconf2.13
| |
| | |
| | |
| For more detailed build options and prerequisites, please visit the following URL
| |
| | |
| https://developer.mozilla.org/en/Configuring_Build_Options | |
| | |
| Checkout the source code from the mozilla-central code repository.
| |
| | |
| | |
| rm -rf /home/user/src
| |
| rm -rf /home/user/mozcentral-dbg
| |
| hg clone http://hg.mozilla.org/mozilla-central/ src
| |
| cd src
| |
| cp ../.mozconfig-gold .mozconfig
| |
| | |
| | |
| TIP: Save the above commands into a file named ‘startover’ and make it executable.
| |
| | |
| TIP:Change the ‘user’ part to your login account name.
| |
| | |
| | |
| | |
| | |
| | |
| The standard build command to build instrumented mozilla firefox is
| |
| | |
| | |
| make -f client.mk build
| |
| | |
| | |
| | |
| | |
| === Execute some Unit tests ===
| |
| Change Directory to ~/mozcentral-dbg/_tests/testing/mochitest and execute the following command.
| |
| | |
| | |
| python runtests.py to execute mochitest suite.
| |
| python runtests.py --chrome --browser-chrome to execute chrome tests.
| |
| python runtests.py --a11y to run accessibility tests
| |
| | |
| | |
| Please keep the focus on the Firefox browser , when running your tests, at all times or else your tests will fail.
| |
| | |
| | |
| In order to run jstests, please go to ~/src/js/tests and execute the following commands
| |
| | |
| | |
| ./jsDriver.pl -e smdebug -L lc2 -L lc3 -L spidermonkey-n-1.9.1.tests -s ~/mozcentral-dbg/dist/bin/js
| |
| | |
| ./jsDriver.pl -e smdebug -L lc2 -L lc3 -L spidermonkey-n-1.9.1.tests -s ~/mozcentral-dbg/dist/bin/js -o '-j'
| |
| | |
| Getopt-mixed is a required library by that script it needs to be added.
| |
| | |
| | |
| === Gather the data ===
| |
| After you are done with the test runs, you will end up with a bunch of *.gcda and *.gcno files in your build directory.
| |
| | |
| | |
| Now, come back into your home directory and execute the following command
| |
| | |
| | |
| lcov -c -i -d mozcentral-dbg -o app.info
| |
| | |
| | |
| The lcov tool will go through the entire build directory and generates an app.info file in your home directory
| |
| | |
| | |
| TIP: If you don’t know what is an ‘lcov’ tool, please feel free to ‘google’ the term.
| |
| | |
| | |
| Then, execute the following command
| |
| | |
| | |
| genhtml app.info
| |
| | |
| If by any chance the above command fails to complete, use the following option.
| |
| | |
| genhtml --no-source app.info
| |
| | |
| | |
| The resulting index.html is the landing page for all your code coverage data for the C/C++ code in the Firefox browser.
| |
| | |
| | |
| HINT: Look at the advanced tweaks section on how to modify, geninfo, genhtml scripts in order to generate full code coverage reports.
| |
| | |
| | |
| When we run the instrumented Firefox, there is now a "JSCoverage" menu in the "Tools" menu.
| |
| | |
| | |
| Tools -> JSCoverage -> View Coverage loads the jscoverage.html file
| |
| | |
| | |
| Tools -> JSCoverage -> Store Coverage saves the coverage data into a directory named "./jscoverage-report.
| |
| | |
| == <FONT COLOR=GREEN>Scripts and Works around required </FONT> ==
| |
| | |
| === Change to the mozilla code ===
| |
| As of now, there is a bug https://bugzilla.mozilla.org/show_bug.cgi?id=461270 which prevents the NSPR from getting linked in the code coverage mode.
| |
| | |
| | |
| The work around to address this problem is as follows:
| |
| | |
| | |
| Modify [ in the trunk you checked out ] src/nsprpub/config/rules.mk on line #352
| |
| Change it
| |
| from $(MKSHLIB) $(OBJS) $(RES) $(EXTRA_LIBS)
| |
| to $(MKSHLIB) $(OBJS) $(RES) $(LDFLAGS) $(EXTRA_LIBS)
| |
| | |
| | |
| Execute the usual build command 'make -f client.mk build'
| |
| | |
| | |
| Please make sure that you have all the build dependencies installed properly before attempting the build.
| |
| | |
| === LCOV ===
| |
| | |
| You'll need [http://ltp.sourceforge.net/coverage/lcov.php lcov].
| |
| | |
| on os X the bin/install.sh needs to be modified on the last line there is a install -D remove the -D switch as it's not supported on OS X.
| |
| | |
| The following needs to be done :
| |
| sudo vi /usr/bin/geninfo
| |
| Press the following key sequence:
| |
| Esc -> : -> %s/die/print/g -> Enter
| |
| | |
| sudo vi /usr/bin/genhtml
| |
| Press the following key sequence:
| |
| Esc -> : -> %s/die/print/g -> Enter
| |
| | |
| | |
| | |
| | |
| === other changes ===
| |
| | |
| A bunch of javascript files gets ‘jar’ed’ into the
| |
| | |
| Once the build is ready , all you have to do is to go into ~/mozcentral-dbg/dist/bin/chrome directory.
| |
| | |
| | |
| You have to unjar all the java script files and modify the jar manifest files to point to appropriate un-jarred directories instead of jar files.
| |
| | |
| | |
| Sounds confusing !! Don’t worry, please execute the following perl script expandJar.pl to do all that job for you.
| |
| | |
| | |
| Syntax : perl expandJar.pl ~/mozcentral-dbg/dist/bin/chrome
| |
| | |
| | |
| Now, we have to instrument the javascript files.
| |
| | |
| | |
| Code coverage for the Javascript part of the Firefox can be done in multiple ways. But using JSCoverage as the tool seemed to be the best way as JSCoverage is an open source tool.
| |
| | |
| | |
| I wish to fully acknowledge the great help Ed Kelly , who maintains the JSCoverage tool , provided in this instrumentation exercise. Ed was very generous in spending cycles to provide enhancements to the JSCoverage tool to meet Firefox instrumentation requirements.
| |
| | |
| | |
| So ... here is how to instrument the javascript portion of Firefox.
| |
| | |
| | |
| Do not download the binary of the JScoverage from
| |
| http://siliconforks.com/jscoverage/
| |
| | |
| You should check out the source code from its public Subversion repository:
| |
| | |
| svn co http://svn.siliconforks.com/jscoverage/trunk jscoverage
| |
| To compile, run bootstrap.sh (requires Automake and Autoconf):
| |
| | |
| ./bootstrap.sh
| |
| Then run make as usual.
| |
| Finally execute the make install command so that jscoverage is
| |
| installed in the /usr/local/bin
| |
| | |
| | |
| Why not use the binary from the site and why to build our own jscoverage !!
| |
| | |
| | |
| The following enhancements are present in our own build with latest revision.
| |
| | |
| 1. Instrumenting symbolic links in the case where the destination is a regular file.
| |
| | |
| For those who are curious, JSCoverage does not instrument javascript files that are symbolically referenced.
| |
| | |
| 2. It copies file permissions (so that executable files stay executable).
| |
| | |
| 3. It has the --mozilla option to copy the jscoverage.html file directly into Firefox's chrome.
| |
| | |
| The --mozilla option automatically sets the JavaScript version to 180,so it is not necessary to use both --mozilla and --js-version at the same time.
| |
| | |
| NOTE: --js-version=180 would allow instrumenting the symbolic links
| |
| | |
| | |
| So, starting from the top of the Firefox build (~/mozcentral-dbg )directory , here are the commands to instrument the whole Firefox directory and run the mochitests with the instrumented version:
| |
| | |
| | |
| cd /dist
| |
| mv bin bin-original
| |
| jscoverage \
| |
| --mozilla \
| |
| --no-instrument=defaults \
| |
| --no-instrument=greprefs \
| |
| bin-original bin
| |
| | |
| | |
| HINT: If you find a foobar.js file failing in the javascript instrumentation exercise, please add it to the no-instrument list using the following syntax --no-instrument=/path/to/foobar.js
| |
| | |
| | |
| It seems that it is necessary to use --no-instrument on the entire defaults and prefs directories because the preferences .js files should not contain arbitrary JavaScript.
| |
| | |
| See http://www.xulplanet.com/tutorials/xulqa/q_prefs.html or https://bugzilla.mozilla.org/show_bug.cgi?id=440777 for details.
| |
| | |
| == <FONT COLOR=GREEN>How to consolidate / merge results and generate graphs</FONT> ==
| |
| '''TO BE ADDED '''
| |