Mobile/e10s/Testing

Fennec Mochitest and Reftest

Build Fennec with enable-tests.

• required patches:
  • bug 648104, 649776.

Running Tests

Make sure you have --enable-tests in your mozconfig as you build. You can then use normal Firefox desktop commands to run tests, like

TEST_PATH=modules/libpr0n/test/reftest/reftest.list make reftest

(do this in the OBJDIR). You will need to make a symlink called fennec-bin (pointing to fennec). (ln -s /builds/mozilla-central/obj-mobilebase-opt/dist/bin/fennec /builds/mozilla-central/obj-mobilebase-opt/dist/bin/fennec-bin)

Another way to build tests is (for example, for libpr0n)

make package-tests
cd dist/test-package-stage/reftest
python runreftest.py --app=../../bin/fennec tests/modules/libpr0n/test/reftest/pngsuite-chunkorder/reftest.list

This however requires make package-tests, and you will not have your tests symlinked (so changes to them will require another make package-tests each time).

If you run a debug build, you will hit lots of "assertion count X is more than expected Y assertions". For now you can ignore those (but we should fix them eventually!)

Also, you can more easily compare errors using the tool checked in at: file:///builds/mozilla-central/layout/tools/reftest/reftest-analyzer.xhtml

Reftests

abs-pos

successful

backgrounds

REFTEST TEST-UNEXPECTED-FAIL | file:///builds/mozilla-central/layout/reftests/backgrounds/background-size-zoom-no-repeat.html

bidi

Unexpected: 7

border-image

successful

border-radius

2 failures ====file:///builds/mozilla-central/layout/reftests/border-radius/scrollbar-clamping-1.html file:///builds/mozilla-central/layout/reftests/border-radius/scrollbar-clamping-2.html

box-ordinal

successful

box-properties

one failure: file:///builds/mozilla-central/layout/reftests/box-properties/CSS21-t100301.xhtml

box-shadow

Unexpected: 4

image

• background-image-zoom-1.html and image-zoom-1.html - looks like we zoom differently than Firefox? In Fennec the image begins so tiny it can't be seen. Doing some zooming in will show it properly in both tests.

libpr0n

Most of these tests fail due to fennec disabling color correction (due to performance concerns). And some other will fail if using a 16bbp display. For example, see bug 648148. Also note that reftests that require drawWindow support will fail in Fennec. (For that you will want the patch in bug 648104, as mentioned before.)

cjones was able to make these mostly pass back in Feb, but since have decayed.

layout/font*

All pass except for bug 649809.

layout/reftests/image*

Some failures due to color correction or scaling/smoothing:

• canvas-outside-document.html
• element-paint-native-widget.html
• element-paint-sharpness-01b.html
• element-paint-sharpness-02b.html

MochiTest

layout/base

Most of these failures are related to using the old netscape universal permissions api. Instead, we should be using SpecialPowers -- the new elevated permission api.

Many of the mochitests use some sort of elevated permission. For example, sending an keypress or composition event requires elevated permissions. I converted some of the EventUtils over to SpecialPowers in bug 649360.

To run, you can:

make package-tests
cd dist/test-package-stage/mochitest
python runreftest.py --app=../../bin/fennec --autorun --close-when-done --test-path=layout/base

layout/style is another test suite of interest. Currently this has over 90,000 tests and we fail <30 on fennec. Most of those are simple fixes for using SpecialPowers for preferences.

python runreftest.py --app=../../bin/fennec --autorun --close-when-done --test-path=layout/style