ReleaseEngineering/Mozharness/How to run tests as a developer
NOTE: You can now use your LDAP credentials to authenticate and download the files that are behind Http authentication (e.g. pvtbuilds).
Quick run-down
- Find the command from the log. Copy/paste it.
- Append --cfg developer_config.py
- Append --installer-url/--test-url with the right values
To read more about mozharness you can reach these:
- https://developer.mozilla.org/en-US/docs/Mozharness_FAQ
- https://wiki.mozilla.org/ReleaseEngineering/Mozharness
Steps
Step 1 - Clone Mozharness
Clone Mozharness:
hg clone http://hg.mozilla.org/build/mozharness scripts
Step 2 - Find the command in the log
In the job in tbpl you can search for "Run as scripts/scripts" and you should see how exactly the command was run in production and you can add the options mentioned above. Prepend python before this command.
Step 3 - Append --cfg developer_config.py
This activates the LDAP authentication for private files. It also removes --read-buildbot-configs from the list of actions.
Step 4 - Append --installer-url and/or --test-url
Append --installer-url and/or --test-url. You can find the right values on the log:
08:14:31 INFO - Found installer url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tar.bz2. 08:14:31 INFO - Found test url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tests.zip.
NOTICE: Talos jobs do not require --test-url.
Example
We start with this production command:
/tools/buildbot/bin/python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 9 --blob-upload-branch mozilla-central --download-symbols ondemand
and we end up with this developer-friendly command:
python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 9 --blob-upload-branch mozilla-central --download-symbols ondemand \ --cfg developer_config.py \ --installer-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140910173159/emulator.tar.gz \ --test-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140910173159/b2g-35.0a1.en-US.android-arm.tests.zip
Running mozharness on your local machine
Follow all the steps mentioned above.
Running mozharness with your local build
TODO: I believe --installer-path instead of --installer-url can be used with --cfg dev_config.py, however, we have to verify this and document it properly.
Running mozharness on a loaner machine
Follow all the steps mentioned above plus add these variables:
- export DISPLAY=:0
- export MOZ_UPLOAD_DIR=/builds/slave/test/build/blobber_upload_dir
- This will allow your job to upload files to blobber
Other examples
Running Mozharness with a local version of Talos and Firefox on Linux
These instructions can be more general, but this should be a useful starting point for any similar configuration. This allows you to use your own objdir instead of downloading an installer and a tests.zip.
python scripts/talos_script.py --cfg configs/talos/linux_config_local.py
Create the file configs/talos/linux_config_local.py
PYTHON = '/home/v/work/talos2/bin/python'
VENV_PATH = '/home/v/work/talos2'
FIREFOX = "/home/v/work/gecko-dev/obj-linux-optimized/dist/bin/firefox"
config = {
"talos_json": "talos.json",
"use_talos_json": True,
"suite": "custom",
"python_webserver": True,
"binary_path": FIREFOX,
"log_name": "talos",
"buildbot_json_path": "buildprops.json", #optional
"virtualenv_path": VENV_PATH,
"default_actions": [
"clobber",
"run-tests",
],
}
talos.json
{
"global": {
"talos_repo": "/home/v/work/talos2",
"talos_revision": "tip"
},
"suites": {
"custom": {
"tests": ["ts"]
}
},
"mobile-suites": {
}
}
Use this (and enable it in the config) if you want to simulate a try syntax commit message.
buildprops.json
{
"sourcestamp": {
"changes": [ {
"comments": "try: simulated try parameters go here",
"files": [ {"name":""} ]
}
]
}
}
Deprecated
Step 3 - B) Create a buildprops.json
In the previous section we mention to use --installer-url as well as --test-url, this is not necessary if you create a file called buildprops.json since the URLs are already specified inside of it. You can find the information you need by loading a tbpl log and looking for "08:14:31 INFO - Using buildbot properties". Don't try to make sense of it. Just copy it without the datestamps (I know, it sucks).
Step 4 - B) --no-read-buildbot-config
Append --no-read-buildbot-config to the command.
A developer config removes the action "read-buildbot-config" from the list of actions.
Some jobs could work with just this and not need a developer config, however, this is unlikely.
Step 4 - Use a developer config
You should be able to simply add --cfg foo_dev.py to the line that is run in production and make it easier for a developer to run the tests (e.g. --cfg android/androidarm.py --cfg android/androidarm_dev.py). All the files are under the configs directory.
If there's not a developer config available read the section "How to add a developer config".
How to create a developer config
Developer configs, if appended after a production config, will overwrite the values on the production config.
Developer configs have these things in common:
- They have the same name as the production one but instead end with "_dev.py"
- They overwrite the "exes" dict with an empty dict
- This allows to use the binaries in your personal $PATH rather than infra-centric paths
- They overwrite the "default_actions" list
- The main reason is to remove the action called read-buildbot-configs
- WARNING: A production config could add new actions and should be added to the developer config upon review.
- They fix URLs to point to the right public reachable domains
- WARNING: This should be fixed in the future as they can fall out of sync with production