ReleaseEngineering/Mozharness/How to run tests as a developer

NOTE: You can now use your LDAP credentials to authenticate and download the files that are behind Http authentication (e.g. pvtbuilds).

Quick run-down

1. Find the command from the log. Copy/paste it.
2. Append --cfg developer_config.py
3. Append --installer-url/--test-url with the right values

To read more about mozharness you can reach these:

• https://developer.mozilla.org/en-US/docs/Mozharness_FAQ
• https://wiki.mozilla.org/ReleaseEngineering/Mozharness

Steps

Step 1 - Clone Mozharness

Clone Mozharness:

hg clone http://hg.mozilla.org/build/mozharness scripts

Step 2 - Find the command in the log

In the job in tbpl you can search for "Run as scripts/scripts" and you should see how exactly the command was run in production and you can add the options mentioned above. Prepend python before this command.

Step 3 - Append --cfg developer_config.py

This activates the LDAP authentication for private files. It also removes --read-buildbot-configs from the list of actions.

Step 4 - Append --installer-url and/or --test-url

Append --installer-url and/or --test-url. You can find the right values on the log:

08:14:31     INFO - Found installer url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tar.bz2.
08:14:31     INFO - Found test url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tests.zip.

NOTICE: Talos jobs do not require --test-url.

Example

We start with this production command:

/tools/buildbot/bin/python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 9 --blob-upload-branch mozilla-central --download-symbols ondemand

and we end up with this developer-friendly command:

python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 9 --blob-upload-branch mozilla-central --download-symbols ondemand \
--cfg developer_config.py \
--installer-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140910173159/emulator.tar.gz \
--test-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140910173159/b2g-35.0a1.en-US.android-arm.tests.zip

Running mozharness on your local machine

Follow all the steps mentioned above.

Running mozharness with your local build

TODO: I believe --installer-path instead of --installer-url can be used with --cfg dev_config.py, however, we have to verify this and document it properly.

Running mozharness on a loaner machine

Follow all the steps mentioned above plus add these variables:

• export DISPLAY=:0
• export MOZ_UPLOAD_DIR=/builds/slave/test/build/blobber_upload_dir
  • This will allow your job to upload files to blobber

Other examples

Running Mozharness with a local version of Talos and Firefox on Linux

These instructions can be more general, but this should be a useful starting point for any similar configuration. This allows you to use your own objdir instead of downloading an installer and a tests.zip.

python scripts/talos_script.py --cfg configs/talos/linux_config_local.py

Create the file configs/talos/linux_config_local.py

PYTHON = '/home/v/work/talos2/bin/python'                                          
VENV_PATH = '/home/v/work/talos2'
FIREFOX = "/home/v/work/gecko-dev/obj-linux-optimized/dist/bin/firefox"
     
config = {  
    "talos_json": "talos.json",
    "use_talos_json": True,
    "suite": "custom",
    "python_webserver": True,
    "binary_path": FIREFOX,
    "log_name": "talos",
    "buildbot_json_path": "buildprops.json", #optional
    "virtualenv_path": VENV_PATH,
    "default_actions": [
        "clobber",
        "run-tests",
    ],      
}

talos.json

{                                                                                  
    "global": {
        "talos_repo": "/home/v/work/talos2",
        "talos_revision": "tip"
    },
    "suites": {
        "custom": {
            "tests": ["ts"]
        }
    },
    "mobile-suites": {
    }
}

Use this (and enable it in the config) if you want to simulate a try syntax commit message.

buildprops.json

{                                                                                  
  "sourcestamp": {
    "changes": [ {
          "comments": "try: simulated try parameters go here",
          "files": [ {"name":""} ]
        }
      ]
  }
}

Deprecated

Step 3 - B) Create a buildprops.json

In the previous section we mention to use --installer-url as well as --test-url, this is not necessary if you create a file called buildprops.json since the URLs are already specified inside of it. You can find the information you need by loading a tbpl log and looking for "08:14:31 INFO - Using buildbot properties". Don't try to make sense of it. Just copy it without the datestamps (I know, it sucks).

Step 4 - B) --no-read-buildbot-config

Append --no-read-buildbot-config to the command. A developer config removes the action "read-buildbot-config" from the list of actions.
Some jobs could work with just this and not need a developer config, however, this is unlikely.

Step 4 - Use a developer config

You should be able to simply add --cfg foo_dev.py to the line that is run in production and make it easier for a developer to run the tests (e.g. --cfg android/androidarm.py --cfg android/androidarm_dev.py). All the files are under the configs directory.

If there's not a developer config available read the section "How to add a developer config".

How to create a developer config

Developer configs, if appended after a production config, will overwrite the values on the production config.

Developer configs have these things in common:

• They have the same name as the production one but instead end with "_dev.py"
• They overwrite the "exes" dict with an empty dict
  • This allows to use the binaries in your personal $PATH rather than infra-centric paths
• They overwrite the "default_actions" list
  • The main reason is to remove the action called read-buildbot-configs
  • WARNING: A production config could add new actions and should be added to the developer config upon review.
• They fix URLs to point to the right public reachable domains
  • WARNING: This should be fixed in the future as they can fall out of sync with production