QA:CodeCoverage

In order to perform code coverage exercise on the Firefox browser, we need to go through at least two iterations.

First : Instrument the C/C++ code in the Firefox browser using 'gcov' and run all tests against the browser.

Second: Instrument the javascript part of the Firefox browser and run all tests against the browser.

Finally we will generate a combined report which would also have the path traversal capability so that we can look at the details of coverage at the source code line level.

• STEP 1

      Create a .mozconfig file with the following statements

       export CFLAGS="-fprofile-arcs -ftest-coverage"
       export CXXFLAGS="-fprofile-arcs -ftest-coverage"
       export EXTRA_DSO_LDFLAGS="-lgcov -static-libgcc"
       export LDFLAGS="-lgcov -static-libgcc"
       ac_add_options --enable-application=browser
       mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../mozcentral-dbg
       ac_add_options --enable-tests
       ac_add_options --enable-mochitest
       ac_add_options --enable-debug
       ac_add_options --disable-optimize
       ac_add_options --enable-jsd
       mk_add_options MOZ_MAKE_FLAGS="-j3"
       mk_add_options AUTOCONF=autoconf2.13

As of now, there is a bug https://bugzilla.mozilla.org/show_bug.cgi?id=461270 which prevents the NSPR from getting linked in the code coverage mode.

The work around to address this problem is as follows:

• Modify [ in the trunk you checked out ] src/nsprpub/config/rules.mk on line #352
  • Change it from $(MKSHLIB) $(OBJS) $(RES) $(EXTRA_LIBS) to $(MKSHLIB) $(OBJS) $(RES) $(EXTRA_LIBS) $(LDFLAGS)

Execute the usual build command 'make -f client.mk build'

Please make sure that you have all the build dependencies installed properly before attempting the build.

Once the build is ready , all you have to do is to go into

~/mozcentral-dbg/_tests/testing/mochitest and execute the following command.

• python runtests.py

Please keep the focus on the Firefox browser at all times or else your tests will fail.

After you are done with the test runs, you will end up with a bunch of *.gcda and *.gcno files in your build directory.

Now, come back into your home directory and execute the following command

       lcov -c -i -d mozcentral-dbg -o app.info

The lcov tool will go through the entire build directory and generates an app.info file in your home directory

Then, execute the following command

        genhtml app.info
        If by any chance the above command fails to complete, use the following option.
        genhtml --no-source app.info

The resulting index.html is the landing page for all your code coverage data for the C/C++ code in the Firefox browser.

• STEP 2

Code coverage for the Javascript part of the Firefox can be done in multiple ways. But using JSCoverage as the tool seemed to be the best way as JSCoverage is an open source tool.

I wish to fully acknowledge the great help Ed Kelly , who maintains the JSCoverage tool , provided in this instrumentation exercise. Ed was very generous in spending cycles to provide enhancements to the JSCoverage tool to meet Firefox instrumentation requirements.

So ... here is how to instrument the javascript portion of Firefox.

    Do not download the binary of the JScoverage from http://siliconforks.com/jscoverage/ 
    You can also check out the source code from its public Subversion repository:
             svn co http://svn.siliconforks.com/jscoverage/trunk jscoverage
             To compile, run bootstrap.sh (requires Automake and Autoconf):
             ./bootstrap.sh
              Then run make as usual.
              Finally execute the make install command so that jscoverage is 
              installed in the /usr/local/bin

Why not use the binary from the site and why to build our own jscoverage ??

             The following enhancements are present in our own build with 
             latest revision.
             1.  Instrumenting symbolic links in the case where
                  the destination is a regular file.  
                       * For those who are curious, JSCoverage does not instrument 
                          javascript files that are symbolically referenced.
             2. It copies  file permissions (so that executable files stay executable).
             3. It has the  --mozilla option to copy the jscoverage.html file
                  directly into Firefox's chrome.  
                  The --mozilla option automatically sets the JavaScript version to 180, 
                  so it is not necessary to use both --mozilla 
                  and --js-version at the same time.
                  NOTE: --js-version=180 would allow instrumenting the symbolic links

So, starting from the top of the Firefox build directory , here are the commands to instrument the whole Firefox directory and run the mochitests with the instrumented version:

                 cd /dist
                 mv bin bin-original
                 jscoverage \
                --mozilla \
                --no-instrument=defaults \
                --no-instrument=greprefs \
                bin-original bin
                cd ../_tests/testing/mochitest
                python runtests.py

It seems that it is necessary to use --no-instrument on the entire defaults and prefs directories because the preferences .js files should not contain arbitrary JavaScript.

See http://www.xulplanet.com/tutorials/xulqa/q_prefs.html or https://bugzilla.mozilla.org/show_bug.cgi?id=440777 for details.

When we run the instrumented Firefox, there is now a "JSCoverage" menu in the "Tools" menu. Tools -> JSCoverage -> View Coverage loads the jscoverage.html file, and Tools -> JSCoverage -> Store Coverage saves the coverage data into a directory named "./jscoverage-report/".

Oh.. by the way ... you don't need to run the mochitests twice to get C/++ and javascript code coverage. Simply build the Firefox as explained in STEP:1 and then execute the JSCoverage coomand on top of it. Then you can run your tests to get both C/++ and JavaScript coverage at the same time..