ReleaseEngineering/Mozharness/How to run tests as a developer

The main thing you need is to checkout mozharness and call it like on tbpl, however, you will need to use these options:

• --no-read-buildbot-configs (as you're running it without buildbot)
• --installer-url (The Firefox binaries) and --test-url (The tests zip)
  • You can find it in the log:

08:14:31     INFO - Found installer url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tar.bz2.
08:14:31     INFO - Found test url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tests.zip.

In the job in tbpl you can search for "scripts/scripts" and you should see how exactly the command was run in production and you can add the options mentioned above.

Clone Mozharness:

hg clone http://hg.mozilla.org/build/mozharness scripts

On Linux:

python scripts/scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --mochitest-suite plain1 --download-symbols ondemand \
--no-read-buildbot-config \
--installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1391797503/firefox-30.0a1.en-US.linux-i686.tar.bz2 \
--test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1391797503/firefox-30.0a1.en-US.linux-i686.tests.zip

On Mac:

python scripts/scripts/desktop_unittest.py --cfg unittests/mac_unittest.py --reftest-suite reftest --download-symbols ondemand \
--no-read-buildbot-config \
--installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1370616247/firefox-24.0a1.en-US.win32.zip \
--test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1370616247/firefox-24.0a1.en-US.win32.tests.zip

On Windows:

python -u scripts/scripts/desktop_unittest.py --cfg unittests/win_unittest.py --reftest-suite reftest --download-symbols ondemand \
--no-read-buildbot-config \
--installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1370616247/firefox-24.0a1.en-US.win32.zip \
--test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1370616247/firefox-24.0a1.en-US.win32.tests.zip

Emulator tests:

python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite reftest \
--this-chunk 5 --total-chunks 10 --blob-upload-branch elm --download-symbols ondemand \
--no-read-buildbot-config 
--installer-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/elm-generic/20140310074727/emulator.tar.gz 
--test-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/elm-generic/20140310074727/b2g-30.0a1.en-US.android-arm.tests.zip

B2G desktop tests (Using dev config):

python scripts/scripts/b2g_desktop_unittest.py --cfg b2g/desktop_automation_config.py \
 --cfg b2g/desktop_automation_config_dev.py --test-suite mochitest --this-chunk 1 \
 --total-chunks 1 --blob-upload-branch mozilla-inbound --download-symbols ondemand \
 --installer-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux64_gecko/1400225075/b2g-32.0a1.multi.linux-x86_64.tar.bz2 \
 --test-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux64_gecko/1400225075/b2g-32.0a1.multi.linux-x86_64.tests.zip

• If you use the above commands, you don't need a buildprops.json. Otherwise you need to create a buildprops.json (I think this goes in cwd); this is output in the log of the test you're trying to replicate.

Developer configs

In the long term, you should be able to simply add --cfg foo_dev.py to the line that is run in production and make it easier for a developer to run the tests.

See bug 1014914 for an example.

buildprops.json

In the previous section we mention to use --installer-url as well as --test-url, this is not necessary if you create a file called buildprops.json. You can find the information you need by loading a tbpl log and looking for "08:14:31 INFO - Using buildbot properties". Don't try to make sense of it. Just copy it without the datestamps (I know, it sucks).

Running mozharness on a loaner

Two variables that you want to set are:

• export DISPLAY=:0
• export MOZ_UPLOAD_DIR=/builds/slave/test/build/blobber_upload_dir

Running Mozharness with a local version of Talos and Firefox on Linux

These instructions can be more general, but this should be a useful starting point for any similar configuration. This allows you to use your own objdir instead of downloading an installer and a tests.zip.

python scripts/talos_script.py --cfg configs/talos/linux_config_local.py

configs/talos/linux_config_local.py

PYTHON = '/home/v/work/talos2/bin/python'                                          
VENV_PATH = '/home/v/work/talos2'
FIREFOX = "/home/v/work/gecko-dev/obj-linux-optimized/dist/bin/firefox"
     
config = {  
    "talos_json": "talos.json",
    "use_talos_json": True,
    "suite": "custom",
    "python_webserver": True,
    "binary_path": FIREFOX,
    "log_name": "talos",
    "buildbot_json_path": "buildprops.json", #optional
    "virtualenv_path": VENV_PATH,
    "default_actions": [
        "clobber",
        "run-tests",
    ],      
}

talos.json

{                                                                                  
    "global": {
        "talos_repo": "/home/v/work/talos2",
        "talos_revision": "tip"
    },
    "suites": {
        "custom": {
            "tests": ["ts"]
        }
    },
    "mobile-suites": {
    }
}

Use this (and enable it in the config) if you want to simulate a try syntax commit message.

buildprops.json

{                                                                                  
  "sourcestamp": {
    "changes": [ {
          "comments": "try: simulated try parameters go here",
          "files": [ {"name":""} ]
        }
      ]
  }
}