ReleaseEngineering/Mozharness/How to run tests as a developer

Quick run-down

1. Find the command from the log. Copy/paste it.
2. Append --installer-url/--test-url with the right values
3. Append --cfg same_file_path_but_ending_with_dev.py

Steps

Step 1 - Clone Mozharness

Clone Mozharness:

hg clone http://hg.mozilla.org/build/mozharness scripts

Step 2 - Find the command in the log

In the job in tbpl you can search for "Run as scripts/scripts" and you should see how exactly the command was run in production and you can add the options mentioned above. Prepend python before this command.

Step 3 - A) Append --installer-url and/or --test-url

Append --installer-url and/or --test-url. You can find the right values on the log:

08:14:31     INFO - Found installer url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tar.bz2.
08:14:31     INFO - Found test url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tests.zip.

NOTICE: Talos jobs do not require --test-url.

Step 3 - B) Create a buildprops.json

In the previous section we mention to use --installer-url as well as --test-url, this is not necessary if you create a file called buildprops.json since the URLs are already specified inside of it. You can find the information you need by loading a tbpl log and looking for "08:14:31 INFO - Using buildbot properties". Don't try to make sense of it. Just copy it without the datestamps (I know, it sucks).

Step 4 - A) Use a developer config

You should be able to simply add --cfg foo_dev.py to the line that is run in production and make it easier for a developer to run the tests (e.g. --cfg android/androidarm.py --cfg android/androidarm_dev.py). All the files are under the configs directory.

If there's not a developer config available read the section "How to add a developer config".

Step 4 - B) --no-read-buildbot-config

Append --no-read-buildbot-config to the command. A developer config removes the action "read-buildbot-config" from the list of actions.
Some jobs could work with just this and not need a developer config, however, this is unlikely.

How to add a developer config

Developer configs, if appended after a production config, will overwrite the values on the production config.

Developer configs have these things in common:

• They have the same name as the production one but instead end with "_dev.py"
• They overwrite the "exes" dict with an empty dict
  • This allows to use the binaries in your personal $PATH rather than infra-centric paths
• They overwrite the "default_actions" list
  • The main reason is to remove the action called read-buildbot-configs
  • WARNING: A production config could add new actions and should be added to the developer config upon review.
• They fix URLs to point to the right public reachable domains
  • WARNING: This should be fixed in the future as they can fall out of sync with production

Running mozharness on your local machine

Follow all the steps mentioned above.

Running mozharness on a loaner machine

Follow all the steps mentioned above plus add these variables:

• export DISPLAY=:0
• export MOZ_UPLOAD_DIR=/builds/slave/test/build/blobber_upload_dir
  • This will allow your job to upload files to blobber

Other examples

Running Mozharness with a local version of Talos and Firefox on Linux

These instructions can be more general, but this should be a useful starting point for any similar configuration. This allows you to use your own objdir instead of downloading an installer and a tests.zip.

python scripts/talos_script.py --cfg configs/talos/linux_config_local.py

Create the file configs/talos/linux_config_local.py

PYTHON = '/home/v/work/talos2/bin/python'                                          
VENV_PATH = '/home/v/work/talos2'
FIREFOX = "/home/v/work/gecko-dev/obj-linux-optimized/dist/bin/firefox"
     
config = {  
    "talos_json": "talos.json",
    "use_talos_json": True,
    "suite": "custom",
    "python_webserver": True,
    "binary_path": FIREFOX,
    "log_name": "talos",
    "buildbot_json_path": "buildprops.json", #optional
    "virtualenv_path": VENV_PATH,
    "default_actions": [
        "clobber",
        "run-tests",
    ],      
}

talos.json

{                                                                                  
    "global": {
        "talos_repo": "/home/v/work/talos2",
        "talos_revision": "tip"
    },
    "suites": {
        "custom": {
            "tests": ["ts"]
        }
    },
    "mobile-suites": {
    }
}

Use this (and enable it in the config) if you want to simulate a try syntax commit message.

buildprops.json

{                                                                                  
  "sourcestamp": {
    "changes": [ {
          "comments": "try: simulated try parameters go here",
          "files": [ {"name":""} ]
        }
      ]
  }
}