View source for ReleaseEngineering/Mozharness/How to run tests as a developer

'''NOTE: If running a job does not easily run locally for you; don't hesitate to just stop (no need to try to fix it), [https://wiki.mozilla.org/ReleaseEngineering/How_To/Request_a_slave ask for a loaner so you can make progress] and file a bug in [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Mozharness here] for us to debug (please attach logs/log_info.log).'''

'''NOTE: You can now use your LDAP credentials to authenticate and download the files that are behind Http authentication (e.g. pvtbuilds).'''

= Brief intro =
Remember that there are jobs that download builds and run tests or performance tests, hence, requiring --installer-url and/or --test-url.
Other jobs, simply create a build or run code analysis, hence, do not need --installer-url and/or --test-url.

To read more about mozharness you can reach these:
* https://developer.mozilla.org/en-US/docs/Mozharness_FAQ
* https://wiki.mozilla.org/ReleaseEngineering/Mozharness

= Known issues =
* Android emulator tests download a copy of the Android SDK from tooltool. Only Mozilla employees are authorized to download these files; non-employees will encounter a failure on download, but can easily work around the issue by skipping the download and using a local copy of the Android SDK.

= Run mozharness ... =
== ...on your local machine ==
Follow all the steps mentioned in "Common steps".

== ...on a loaner machine ==
* <font color='red'>Don't use --cfg developer_config.py</font>
* Use --no-read-buildbot-config instead
* If the job is...
** A build job: nothing to append
** A test job: append --installer-url and one of --test-url or --test-packages-url
** A talos job: append --installer-url

Also add these variables:
* export '''DISPLAY=:0''' # I think it applied for Linux hosts
* export '''MOZ_UPLOAD_DIR=/builds/slave/test/build/blobber_upload_dir'''
** This will allow your job to upload files to blobber

== ...with binaries from your objdir ==
TODO: I believe --installer-path instead of --installer-url can be used with --cfg dev_config.py, however, we have to verify this and document it properly.

If you figure it out please add it here and we will review it.

== Common steps ==
=== Step 1 - Check out mozharness ===
mozharness is now in the Gecko repository, under the testing/ directory. So if you do not already have a checkout,

* hg clone https://hg.mozilla.org/mozilla-central/

=== Step 2 - Create a tooltool token (fetches artifacts for you) ===
With the new tooltool, you need to create a token to run mozharness on your development machine. Note that this is not necessary for all mozharness scripts -- for example, the hazard builds do not require this as they only download public files via tooltool.

Steps:
* Go [https://api.pub.build.mozilla.org/tokenauth here]
* Issue a new user token with tooltool.download.public and tooltool.download.internal selected
* The token will only be shown once
* Place it in ~/.mozilla/releng/relengapi.tok

=== Step 2 - Find the command in the log ===
In the job in treeherder you can search for '''"Run as scripts/scripts"''' and you should see how exactly the command was run in production and you can add the options mentioned above.

NOTE: Don't forget to prepend ''python'' before the command.

=== Step 4 - Append --cfg developer_config.py ===

{{warning|If you have a loaned machine from Release Engineering, do not use this config.<br/>This config is meant to be used only when you're running mozharness on your local machine.<br/>(<small>[[How_to_run_tests_as_a_developer#...on_a_loaner_machine|More Information]]</small>)}}

This activates the LDAP authentication for private files.
It also removes --read-buildbot-configs from the list of actions.

== Unit/talos tests steps ==
If you're trying to reproduce a unit test suite or a talos suite follow this last step. Otherwise, go to the "Build and other jobs" section.

=== Step 5 - Append --installer-url and/or one of --test-url and --test-packages-url ===
Append ''--installer-url'' and one of ''--test-url'' or ''--test-packages-url''. You can find the right values on the log:
<pre>
08:14:31     INFO - Found installer url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tar.bz2.
08:14:31     INFO - Found test url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tests.zip.
…
17:51:04     INFO - Found a test packages url https://queue.taskcluster.net/v1/task/XHPBpieOSeO2e44zhkw64A/artifacts/public/build/test_packages.json.
</pre>

NOTICE: Talos jobs do not require --test-url.

== Build and other jobs ==
You don't have to use --installer-url, --test-url, or --test-packages-url.

Right now I have not been able to run too many of these jobs on my local machine since it requires setting up mock_mozilla and each of these jobs can have their own set of environments.

We have filed {{bug|1067354}} to improve this situation.

Please ask in #releng for assistance as it might be possible to run the job locally with an extra brain. Otherwise, you will need to loan a machine.

= Filing bugs =
If you have tried mozharness and it is not doing what you need then [https://bugzilla.mozilla.org/enter_bug.cgi?product=Release%20Engineering&component=Mozharness file a bug] and CC :armenzg.

= Example =
We start with this production command:
 /tools/buildbot/bin/python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 9 --blob-upload-branch mozilla-central --download-symbols ondemand

and we end up with this developer-friendly command:
 python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite mochitest --this-chunk 1 --total-chunks 9 --blob-upload-branch mozilla-central --download-symbols ondemand \
 --cfg developer_config.py \
 --installer-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140910173159/emulator.tar.gz \
 --test-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-central-emulator/20140910173159/b2g-35.0a1.en-US.android-arm.tests.zip

= Other examples =
== Running Mozharness with a local version of Talos and Firefox on Linux ==

These instructions can be more general, but this should be a useful starting point for any similar configuration.
This allows you to use '''your own objdir''' instead of downloading an installer and a tests.zip.

<pre>
python scripts/talos_script.py --cfg configs/talos/linux_config_local.py
</pre>

Create the file configs/talos/linux_config_local.py

<pre>
PYTHON = '/home/v/work/talos2/bin/python'                                          
VENV_PATH = '/home/v/work/talos2'
FIREFOX = "/home/v/work/gecko-dev/obj-linux-optimized/dist/bin/firefox"
     
config = {  
    "talos_json": "talos.json",
    "use_talos_json": True,
    "suite": "custom",
    "python_webserver": True,
    "binary_path": FIREFOX,
    "log_name": "talos",
    "buildbot_json_path": "buildprops.json", #optional
    "virtualenv_path": VENV_PATH,
    "default_actions": [
        "clobber",
        "run-tests",
    ],      
}
</pre>

talos.json

<pre>
{                                                                                  
    "global": {
        "talos_repo": "/home/v/work/talos2",
        "talos_revision": "tip"
    },
    "suites": {
        "custom": {
            "tests": ["ts"]
        }
    },
    "mobile-suites": {
    }
}
</pre>

Use this (and enable it in the config) if you want to simulate a try syntax commit message.

buildprops.json
<pre>
{                                                                                  
  "sourcestamp": {
    "changes": [ {
          "comments": "try: simulated try parameters go here",
          "files": [ {"name":""} ]
        }
      ]
  }
}
</pre>

= Deprecated =
== Step 3 - B) Create a buildprops.json ==
In the previous section we mention to use --installer-url as well as --test-url, this is not necessary if you create a file called buildprops.json since the URLs are already specified inside of it.
You can find the information you need by loading a tbpl log and looking for ''"08:14:31     INFO - Using buildbot properties"''.
Don't try to make sense of it. Just copy it without the datestamps (I know, it sucks). You will also need to add some dummy properties which you can find at https://searchfox.org/mozilla-central/source/taskcluster/docker/recipes/buildprops.json because the Python code that reads the file looks for these.

== Step 4 - B) --no-read-buildbot-config ==
Append --no-read-buildbot-config to the command.
A developer config removes the action "read-buildbot-config" from the list of actions.<br />
Some jobs could work with just this and not need a developer config, however, this is unlikely.

== Step 4 - Use a developer config ==
You should be able to simply add --cfg foo_dev.py to the line that is run in production and make it easier for a developer to run the tests (e.g. --cfg android/androidarm.py --cfg android/androidarm'''_dev.py'''). All the files are under the ''configs'' directory.

If there's not a developer config available read the section "How to add a developer config".

== How to create a developer config ==
Developer configs, if appended after a production config, will overwrite the values on the production config.<br />

Developer configs have these things in common:
* They have the same name as the production one but instead end with '''"_dev.py"'''
* They overwrite the '''"exes"''' dict with an empty dict
** This allows to use the binaries in your personal $PATH rather than infra-centric paths
* They overwrite the '''"default_actions"''' list
** The main reason is to remove the action called read-buildbot-configs
** WARNING: A production config could add new actions and should be added to the developer config upon review.
* They fix URLs to point to the right public reachable domains
** WARNING: This should be fixed in the future as they can fall out of sync with production