ReleaseEngineering/Mozharness/How to run tests as a developer

NOTE: If running a job does not easily run locally for you; don't hesitate to just stop (no need to try to fix it), ask for a loaner so you can make progress and file a bug in here for us to debug (please attach logs/log_info.log).

NOTE: You can now use your LDAP credentials to authenticate and download the files that are behind Http authentication (e.g. pvtbuilds).

Brief intro

Remember that there are jobs that download builds and run tests or performance tests, hence, requiring --installer-url and/or --test-url. Other jobs, simply create a build or run code analysis, hence, do not need --installer-url and/or --test-url.

To read more about mozharness you can reach these:

• https://developer.mozilla.org/en-US/docs/Mozharness_FAQ
• https://wiki.mozilla.org/ReleaseEngineering/Mozharness

Known issues

• Android emulator tests download a copy of the Android SDK from tooltool. Only Mozilla employees are authorized to download these files; non-employees will encounter a failure on download, but can easily work around the issue by skipping the download and using a local copy of the Android SDK.

Run mozharness ...

...on your local machine

Follow all the steps mentioned in "Common steps".

...on a loaner machine

• Don't use --cfg developer_config.py
• Use --no-read-buildbot-config instead
• If the job is...
  • A build job: nothing to append
  • A test job: append --installer-url and one of --test-url or --test-packages-url
  • A talos job: append --installer-url

Also add these variables:

• export DISPLAY=:0 # I think it applied for Linux hosts
• export MOZ_UPLOAD_DIR=/builds/slave/test/build/blobber_upload_dir
  • This will allow your job to upload files to blobber

...with binaries from your objdir

TODO: I believe --installer-path instead of --installer-url can be used with --cfg dev_config.py, however, we have to verify this and document it properly.

If you figure it out please add it here and we will review it.

Common steps

Step 1 - Check out mozharness

mozharness is now in the Gecko repository, under the testing/ directory. So if you do not already have a checkout,

• hg clone https://hg.mozilla.org/mozilla-central/

Step 2 - Create a tooltool token (fetches artifacts for you)

With the new tooltool, you need to create a token to run mozharness on your development machine. Note that this is not necessary for all mozharness scripts -- for example, the hazard builds do not require this as they only download public files via tooltool.

Steps:

• Go here
• Issue a new user token with tooltool.download.public and tooltool.download.internal selected
• The token will only be shown once
• Place it in ~/.mozilla/releng/relengapi.tok

Step 3 - Find the command in the log

In the job in treeherder you can search for "Run as scripts/scripts" and you should see how exactly the command was run in production and you can add the options mentioned above.

NOTE: Don't forget to prepend python before the command.

Step 4 - Append --cfg developer_config.py

Warning: If you have a loaned machine from Release Engineering, do not use this config.
This config is meant to be used only when you're running mozharness on your local machine.
(More Information)

This activates the LDAP authentication for private files. It also removes --read-buildbot-configs from the list of actions.

Unit/talos tests steps

If you're trying to reproduce a unit test suite or a talos suite follow this last step. Otherwise, go to the "Build and other jobs" section.

Step 5 - Append --installer-url and/or one of --test-url and --test-packages-url

Append --installer-url and one of --test-url or --test-packages-url. You can find the right values on the log:

08:14:31     INFO - Found installer url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tar.bz2.
08:14:31     INFO - Found test url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tests.zip.
…
17:51:04     INFO - Found a test packages url https://queue.taskcluster.net/v1/task/XHPBpieOSeO2e44zhkw64A/artifacts/public/build/test_packages.json.

NOTICE: Talos jobs do not require --test-url.

Build and other jobs

You don't have to use --installer-url, --test-url, or --test-packages-url.

Right now I have not been able to run too many of these jobs on my local machine since it requires setting up mock_mozilla and each of these jobs can have their own set of environments.

We have filed bug 1067354 to improve this situation.

Please ask in #releng for assistance as it might be possible to run the job locally with an extra brain. Otherwise, you will need to loan a machine.

Filing bugs

If you have tried mozharness and it is not doing what you need then file a bug and CC :armenzg.

Example

We start with this production command:

/tools/buildbot/bin/python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 9 --blob-upload-branch mozilla-central --download-symbols ondemand

and we end up with this developer-friendly command:

python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 9 --blob-upload-branch mozilla-central --download-symbols ondemand \
--cfg developer_config.py \
--installer-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140910173159/emulator.tar.gz \
--test-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140910173159/b2g-35.0a1.en-US.android-arm.tests.zip

Other examples

Running Mozharness with a local version of Talos and Firefox on Linux

These instructions can be more general, but this should be a useful starting point for any similar configuration. This allows you to use your own objdir instead of downloading an installer and a tests.zip.

python scripts/talos_script.py --cfg configs/talos/linux_config_local.py

Create the file configs/talos/linux_config_local.py

PYTHON = '/home/v/work/talos2/bin/python'                                          
VENV_PATH = '/home/v/work/talos2'
FIREFOX = "/home/v/work/gecko-dev/obj-linux-optimized/dist/bin/firefox"
     
config = {  
    "talos_json": "talos.json",
    "use_talos_json": True,
    "suite": "custom",
    "python_webserver": True,
    "binary_path": FIREFOX,
    "log_name": "talos",
    "buildbot_json_path": "buildprops.json", #optional
    "virtualenv_path": VENV_PATH,
    "default_actions": [
        "clobber",
        "run-tests",
    ],      
}

talos.json

{                                                                                  
    "global": {
        "talos_repo": "/home/v/work/talos2",
        "talos_revision": "tip"
    },
    "suites": {
        "custom": {
            "tests": ["ts"]
        }
    },
    "mobile-suites": {
    }
}

Use this (and enable it in the config) if you want to simulate a try syntax commit message.

buildprops.json

{                                                                                  
  "sourcestamp": {
    "changes": [ {
          "comments": "try: simulated try parameters go here",
          "files": [ {"name":""} ]
        }
      ]
  }
}

Deprecated

Step 3 - B) Create a buildprops.json

In the previous section we mention to use --installer-url as well as --test-url, this is not necessary if you create a file called buildprops.json since the URLs are already specified inside of it. You can find the information you need by loading a tbpl log and looking for "08:14:31 INFO - Using buildbot properties". Don't try to make sense of it. Just copy it without the datestamps (I know, it sucks). You will also need to add some dummy properties which you can find at https://searchfox.org/mozilla-central/source/taskcluster/docker/recipes/buildprops.json because the Python code that reads the file looks for these.

Step 4 - B) --no-read-buildbot-config

Append --no-read-buildbot-config to the command. A developer config removes the action "read-buildbot-config" from the list of actions.
Some jobs could work with just this and not need a developer config, however, this is unlikely.

Step 4 - Use a developer config

You should be able to simply add --cfg foo_dev.py to the line that is run in production and make it easier for a developer to run the tests (e.g. --cfg android/androidarm.py --cfg android/androidarm_dev.py). All the files are under the configs directory.

If there's not a developer config available read the section "How to add a developer config".

How to create a developer config

Developer configs, if appended after a production config, will overwrite the values on the production config.

Developer configs have these things in common:

• They have the same name as the production one but instead end with "_dev.py"
• They overwrite the "exes" dict with an empty dict
  • This allows to use the binaries in your personal $PATH rather than infra-centric paths
• They overwrite the "default_actions" list
  • The main reason is to remove the action called read-buildbot-configs
  • WARNING: A production config could add new actions and should be added to the developer config upon review.
• They fix URLs to point to the right public reachable domains
  • WARNING: This should be fixed in the future as they can fall out of sync with production