MozillaWiki

QA:CodeCoverage: Difference between revisions

Page Discussion

QA:CodeCoverage (view source)

Revision as of 01:22, 7 January 2009

1,243 bytes added ,  7 January 2009
→‎How to instrument the Mozilla Firefox Browser
No edit summary
Line 4: Line 4:
== <FONT COLOR=GREEN>Results and Findings </FONT> ==
== <FONT COLOR=GREEN>Results and Findings </FONT> ==
== <FONT COLOR=GREEN>How to instrument the Mozilla Firefox Browser </FONT> ==
== <FONT COLOR=GREEN>How to instrument the Mozilla Firefox Browser </FONT> ==
In order to perform code coverage exercise on the Firefox browser, we need to go through at least two iterations.


First : Instrument the C/C++ code in the Firefox browser using 'gcov' and run all tests against the browser.


Second: Instrument the javascript part of the Firefox browser and run all tests against the browser.
This is what you do to instrument the Mozilla Firefox browser.


Finally we will generate a combined report which would also have the path traversal capability so that we can look at the details of coverage at the source code line level.
Create a .mozconfig-gold file in your home directory with the following statements


*STEP 1


Create a .mozconfig file with the following statements
      export CFLAGS="-fprofile-arcs -ftest-coverage"
      export CXXFLAGS="-fprofile-arcs -ftest-coverage"
      export LDFLAGS="-lgcov -static-libgcc"
      export EXTRA_DSO_LDFLAGS="-lgcov -static-libgcc"
      ac_add_options --enable-application=browser
      mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../mozcentral-dbg
      ac_add_options --enable-debug
      ac_add_options --enable-tests
      ac_add_options --enable-mochitest
      ac_add_options --disable-optimize
      mk_add_options MOZ_MAKE_FLAGS="-j3"
      mk_add_options AUTOCONF=autoconf2.13


export CFLAGS="-fprofile-arcs -ftest-coverage"
export CXXFLAGS="-fprofile-arcs -ftest-coverage"
export EXTRA_DSO_LDFLAGS="-lgcov -static-libgcc"
export LDFLAGS="-lgcov -static-libgcc"
ac_add_options --enable-application=browser
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../mozcentral-dbg
ac_add_options --enable-tests
ac_add_options --enable-mochitest
ac_add_options --enable-debug
ac_add_options --disable-optimize
ac_add_options --enable-jsd
mk_add_options MOZ_MAKE_FLAGS="-j3"
mk_add_options AUTOCONF=autoconf2.13


For more detailed build options and prerequisites, please visit the following URL
https://developer.mozilla.org/en/Configuring_Build_Options
Checkout the source code from the mozilla-central code repository.
        rm -rf /home/user/src
        rm -rf /home/user/mozcentral-dbg
        hg clone http://hg.mozilla.org/mozilla-central/ src
        cd src
        cp ../.mozconfig-gold .mozconfig
TIP: Save the above commands into a file named ‘startover’ and make it executable.
TIP:Change the ‘user’ part to your login account name.
The standard build command to build instrumented mozilla firefox is
      make -f client.mk build
As of now, there is a bug https://bugzilla.mozilla.org/show_bug.cgi?id=461270 which prevents the NSPR from getting linked in the code coverage mode.


As of now, there is a bug https://bugzilla.mozilla.org/show_bug.cgi?id=461270 which prevents the NSPR from getting linked in the code coverage mode.


The work around to address this problem is as follows:
The work around to address this problem is as follows:
* Modify [ in the trunk you checked out ] src/nsprpub/config/rules.mk on line #352
** Change it from $(MKSHLIB) $(OBJS) $(RES) $(EXTRA_LIBS) to  $(MKSHLIB) $(OBJS) $(RES) $(EXTRA_LIBS) $(LDFLAGS)


Acknowledgement: Peter Ahe [ http://www.ahe.dk/peter/ ] for helping me crack the rules.mk  
 
      Modify [ in the trunk you checked out ] src/nsprpub/config/rules.mk on line #352
            Change it
            from $(MKSHLIB) $(OBJS) $(RES) $(EXTRA_LIBS)
            to $(MKSHLIB) $(OBJS) $(RES) $(LDFLAGS)  $(EXTRA_LIBS)
 


Execute the usual build command 'make -f client.mk build'
Execute the usual build command 'make -f client.mk build'


Please make sure that you have all the build dependencies installed properly before attempting the build.
Please make sure that you have all the build dependencies installed properly before attempting the build.


Once the build is ready , all you have to do is to go into  
 
A bunch of javascript files gets ‘jar’ed’ into the
 
Once the build is ready , all you have to do is to go into ~/mozcentral-dbg/dist/bin/chrome directory.
 
 
You have to unjar all the java script files and modify the jar manifest files to point to appropriate un-jarred directories instead of  jar  files.
 
 
Sounds confusing !! Don’t worry, please execute the following perl script expandJar.pl to do all that job for you.
 
 
    Syntax : perl expandJar.pl ~/mozcentral-dbg/dist/bin/chrome
 
 
Now, we have to instrument the javascript files.
 
 
Code coverage for the Javascript part of the Firefox can be done in multiple ways. But using JSCoverage as the tool seemed to be the best way as JSCoverage is an open source tool.
 
 
I wish to fully acknowledge the great help Ed Kelly , who maintains the JSCoverage tool , provided in this instrumentation exercise. Ed was very generous in spending cycles to provide enhancements to the JSCoverage tool to meet Firefox instrumentation requirements.
 
 
So ... here is how to instrument the javascript portion of Firefox.
 
 
Do not download the binary of the JScoverage from
http://siliconforks.com/jscoverage/
 
You should check out the source code from its public Subversion repository:
 
      svn co http://svn.siliconforks.com/jscoverage/trunk jscoverage
To compile, run bootstrap.sh (requires Automake and Autoconf):
 
      ./bootstrap.sh
      Then run make as usual.
      Finally execute the make install command so that jscoverage is
      installed in the /usr/local/bin
 
 
Why not use the binary from the site and why to build our own jscoverage !!
 
 
The following enhancements are present in our own build with latest revision.
 
1.  Instrumenting symbolic links in the case where the destination is a regular file.
 
For those who are curious, JSCoverage does not instrument javascript files that are symbolically referenced.
 
2. It copies  file permissions (so that executable files stay executable).
 
3. It has the  --mozilla option to copy the jscoverage.html file directly into Firefox's chrome.
 
The --mozilla option automatically sets the JavaScript version to 180,so it is not necessary to use both --mozilla and --js-version at the same time.
 
NOTE: --js-version=180 would allow instrumenting the symbolic links
 
 
So, starting from the top of the Firefox build (~/mozcentral-dbg )directory , here are the commands to instrument the whole Firefox directory and run the mochitests with the instrumented version:
 
 
          cd /dist
          mv bin bin-original
          jscoverage \
          --mozilla \
          --no-instrument=defaults \
          --no-instrument=greprefs \
          bin-original bin
 
 
HINT: If you find a foobar.js file failing in the javascript instrumentation exercise, please add it to the no-instrument list using the following syntax --no-instrument=/path/to/foobar.js
 
 
It seems that it is necessary to use --no-instrument on the entire defaults and prefs directories because the preferences .js files should not contain arbitrary JavaScript.
 
 
See http://www.xulplanet.com/tutorials/xulqa/q_prefs.html or https://bugzilla.mozilla.org/show_bug.cgi?id=440777 for details.
 


~/mozcentral-dbg/_tests/testing/mochitest and execute the following command.
~/mozcentral-dbg/_tests/testing/mochitest and execute the following command.
* python runtests.py


Please keep the focus on the Firefox browser at all times or else your tests will fail.  
 
    python runtests.py  to execute mochitest suite.
    python runtests.py --chrome --browser-chrome to execute chrome tests.
    python runtests.py --a11y to run accessibility tests
 
 
Please keep the focus on the Firefox browser , when running your tests, at all times or else your tests will fail.
 
 
In order to run jstests, please go to ~/src/js/tests  and execute the following commands
 
 
    ./jsDriver.pl -e smdebug -L lc2 -L lc3 -L spidermonkey-n-1.9.1.tests -s ~mozcentral-dbg/dist/bin/js
 
    ./jsDriver.pl -e smdebug -L lc2 -L lc3 -L spidermonkey-n-1.9.1.tests -s ~mozcentral-dbg/dist/bin/js -o '-j’
 
 
 


After you are done with the test runs, you will end up with a bunch of *.gcda and *.gcno files in your build directory.
After you are done with the test runs, you will end up with a bunch of *.gcda and *.gcno files in your build directory.


Now, come back into your home directory and execute the following command
Now, come back into your home directory and execute the following command


lcov -c -i -d mozcentral-dbg -o app.info


The lcov tool will go through the entire build directory and generates an app.info file in your home directory
      lcov -c -i -d mozcentral-dbg -o app.info


Then, execute the following command


genhtml app.info
The lcov tool will go through the entire build directory and generates an app.info file in your home directory
If by any chance the above command fails to complete, use the following option.
genhtml --no-source app.info




The resulting index.html is the landing page for all your code coverage data for the C/C++ code in the Firefox browser.
TIP: If you don’t know what is an ‘lcov’ tool, please feel free to ‘google’ the term.




* STEP 2
Then, execute the following command


Code coverage for the Javascript part of the Firefox can be done in multiple ways. But using JSCoverage as the tool seemed to be the best way as JSCoverage is an open source tool.


I wish to fully acknowledge the great help  Ed Kelly , who maintains the JSCoverage tool , provided in this instrumentation exercise. Ed was very generous in spending cycles to provide enhancements to the  JSCoverage tool to meet Firefox instrumentation requirements.
        genhtml app.info


So ... here is how to instrument the javascript portion of Firefox.
If by any chance the above command fails to complete, use the following option.


Do not download the binary of the JScoverage from http://siliconforks.com/jscoverage/
        genhtml --no-source app.info
You should check out the source code from its public Subversion repository:
svn co http://svn.siliconforks.com/jscoverage/trunk jscoverage
To compile, run bootstrap.sh (requires Automake and Autoconf):
./bootstrap.sh
Then run make as usual.
Finally execute the make install command so that jscoverage is
installed in the /usr/local/bin


Why  not use the binary from the site and why to build our own jscoverage ??


The resulting index.html is the landing page for all your code coverage data for the C/C++ code in the Firefox browser.
The following enhancements are present in our own build with
latest revision.
1.  Instrumenting symbolic links in the case where
the destination is a regular file. 
* For those who are curious, JSCoverage does not instrument
javascript files that are symbolically referenced.
2. It copies  file permissions (so that executable files stay executable).
3. It has the --mozilla option to copy the jscoverage.html file
directly into Firefox's chrome. 
The --mozilla option automatically sets the JavaScript version to 180,
so it is not necessary to use both --mozilla
and --js-version at the same time.
NOTE: --js-version=180 would allow instrumenting the symbolic links


So, starting from the top of the Firefox build directory , here are the commands to instrument the whole Firefox directory and run the mochitests with the instrumented version:


cd /dist
HINT: Look at the advanced tweaks section on how to modify, geninfo, genhtml scripts in order to generate full  code coverage reports.
mv bin bin-original
jscoverage \
--mozilla \
--no-instrument=defaults \
--no-instrument=greprefs \
bin-original bin
cd ../_tests/testing/mochitest
python runtests.py


It seems that it is necessary to use --no-instrument on the entire defaults and prefs directories because the preferences .js files should  not contain arbitrary JavaScript. 


See http://www.xulplanet.com/tutorials/xulqa/q_prefs.html or
When we run the instrumented Firefox, there is now a "JSCoverage" menu in the "Tools" menu.
https://bugzilla.mozilla.org/show_bug.cgi?id=440777 for details.


When we run the instrumented Firefox, there is now a "JSCoverage" menu  in the "Tools" menu.  Tools -> JSCoverage -> View Coverage loads the  jscoverage.html file, and Tools -> JSCoverage -> Store Coverage saves  the coverage data into a directory named "./jscoverage-report/".


Oh.. by the way ... you don't need to run the mochitests twice to get C/C++ and javascript code coverage. Simply build the Firefox as explained in STEP:1 and then execute the JSCoverage coomand on top of it. Then you can run your tests to get both C/C++ and JavaScript coverage at the same time..
Tools -> JSCoverage -> View Coverage loads the jscoverage.html file


Screen shots from the final results are given at this link


http://picasaweb.google.com/murali.nandigama/FirefoxCodeCoverage#
Tools -> JSCoverage -> Store Coverage saves the coverage data into a directory named "./jscoverage-report.


== <FONT COLOR=GREEN>Scripts and Works around required </FONT> ==
== <FONT COLOR=GREEN>Scripts and Works around required </FONT> ==
== <FONT COLOR=GREEN>How to consolidate / merge results and generate graphs</FONT> ==
== <FONT COLOR=GREEN>How to consolidate / merge results and generate graphs</FONT> ==
Mnandigama
183

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/122804"