ReleaseEngineering/Mozharness/How to run tests as a developer: Difference between revisions

no edit summary
mNo edit summary
No edit summary
Line 1: Line 1:
The main thing you need is to checkout mozharness and call it like on tbpl, however, you will need to use these options:
== Quick run-down ==
* --no-read-buildbot-configs (as you're running it without buildbot)
# Find the command from the log. Copy/paste it.
* --installer-url (The Firefox binaries) and --test-url (The tests zip)
# Append --installer-url/--test-url with the right values
** You can find it in the log:
# Append --cfg same_file_path_but_ending_with_dev.py
 
== Steps ==
=== Step 1 - Clone Mozharness ===
Clone Mozharness:
hg clone http://hg.mozilla.org/build/mozharness scripts
 
=== Step 2 - Find the command in the log ===
In the job in tbpl you can search for '''"Run as scripts/scripts"''' and you should see how exactly the command was run in production and you can add the options mentioned above.
Prepend ''python'' before this command.
 
=== Step 3 - A) Append --installer-url and/or --test-url ===
Append ''--installer-url'' and/or ''--test-url''. You can find the right values on the log:
<pre>
<pre>
08:14:31    INFO - Found installer url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tar.bz2.
08:14:31    INFO - Found installer url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux32_gecko/1400225075/en-US/b2g-32.0a1.en-US.linux-i686.tar.bz2.
Line 8: Line 20:
</pre>
</pre>


In the job in tbpl you can search for "scripts/scripts" and you should see how exactly the command was run in production and you can add the options mentioned above.
NOTICE: Talos jobs do not require --test-url.


Clone Mozharness:
=== Step 3 - B) Create a buildprops.json ===
hg clone http://hg.mozilla.org/build/mozharness scripts
In the previous section we mention to use --installer-url as well as --test-url, this is not necessary if you create a file called buildprops.json since the URLs are already specified inside of it.
On Linux:
You can find the information you need by loading a tbpl log and looking for ''"08:14:31    INFO - Using buildbot properties"''.
python scripts/scripts/desktop_unittest.py --cfg unittests/linux_unittest.py --mochitest-suite plain1 --download-symbols ondemand \
Don't try to make sense of it. Just copy it without the datestamps (I know, it sucks).
--no-read-buildbot-config \
--installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1391797503/firefox-30.0a1.en-US.linux-i686.tar.bz2 \
--test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1391797503/firefox-30.0a1.en-US.linux-i686.tests.zip


On Mac:
=== Step 4 - A) Use a developer config ===
python scripts/scripts/desktop_unittest.py --cfg unittests/mac_unittest.py --reftest-suite reftest --download-symbols ondemand \
You should be able to simply add --cfg foo_dev.py to the line that is run in production and make it easier for a developer to run the tests (e.g. --cfg android/androidarm.py --cfg android/androidarm'''_dev.py'''). All the files are under the ''configs'' directory.
--no-read-buildbot-config \
--installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1370616247/firefox-24.0a1.en-US.win32.zip \
--test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1370616247/firefox-24.0a1.en-US.win32.tests.zip


On Windows:
If there's not a developer config available read the section "How to add a developer config".
python -u scripts/scripts/desktop_unittest.py --cfg unittests/win_unittest.py --reftest-suite reftest --download-symbols ondemand \
--no-read-buildbot-config \
--installer-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1370616247/firefox-24.0a1.en-US.mac64.dmg \
--test-url http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/1370616247/firefox-24.0a1.en-US.mac64.tests.zip


Emulator tests:
=== Step 4 - B) --no-read-buildbot-config ===
python scripts/scripts/b2g_emulator_unittest.py --cfg b2g/emulator_automation_config.py --test-suite reftest \
Append --no-read-buildbot-config to the command.
--this-chunk 5 --total-chunks 10 --blob-upload-branch elm --download-symbols ondemand \
A developer config removes the action "read-buildbot-config" from the list of actions.<br />
--no-read-buildbot-config  
Some jobs could work with just this and not need a developer config, however, this is unlikely.
--installer-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/elm-generic/20140310074727/emulator.tar.gz
--test-url http://pvtbuilds.pvt.build.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/elm-generic/20140310074727/b2g-30.0a1.en-US.android-arm.tests.zip


B2G desktop tests (Using dev config):
== How to add a developer config ==
python scripts/scripts/b2g_desktop_unittest.py --cfg b2g/desktop_automation_config.py \
Developer configs, if appended after a production config, will overwrite the values on the production config.<br />
  --cfg b2g/desktop_automation_config_dev.py --test-suite mochitest --this-chunk 1 \
  --total-chunks 1 --blob-upload-branch mozilla-inbound --download-symbols ondemand \
  --installer-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux64_gecko/1400225075/b2g-32.0a1.multi.linux-x86_64.tar.bz2 \
  --test-url http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/mozilla-inbound-linux64_gecko/1400225075/b2g-32.0a1.multi.linux-x86_64.tests.zip


* If you use the above commands, you don't need a buildprops.json.  Otherwise you need to create a buildprops.json (I think this goes in cwd); this is output in the log of the test you're trying to replicate.
Developer configs have these things in common:
* They have the same name as the production one but instead end with '''"_dev.py"'''
* They overwrite the '''"exes"''' dict with an empty dict
** This allows to use the binaries in your personal $PATH rather than infra-centric paths
* They overwrite the '''"default_actions"''' list
** The main reason is to remove the action called read-buildbot-configs
** WARNING: A production config could add new actions and should be added to the developer config upon review.
* They fix URLs to point to the right public reachable domains
** WARNING: This should be fixed in the future as they can fall out of sync with production


== Developer configs ==
== Running mozharness on your local machine ==
In the long term, you should be able to simply add --cfg foo_dev.py to the line that is run in production and make it easier for a developer to run the tests.
Follow all the steps mentioned above.
 
See {{bug|1014914}} for an example.
 
== buildprops.json ==
In the previous section we mention to use --installer-url as well as --test-url, this is not necessary if you create a file called buildprops.json.
You can find the information you need by loading a tbpl log and looking for "08:14:31    INFO - Using buildbot properties".
Don't try to make sense of it. Just copy it without the datestamps (I know, it sucks).


== Running mozharness on a loaner ==
== Running mozharness on a loaner machine ==
Two variables that you want to set are:
Follow all the steps mentioned above plus add these variables:
* export DISPLAY=:0
* export '''DISPLAY=:0'''
* export MOZ_UPLOAD_DIR=/builds/slave/test/build/blobber_upload_dir
* export '''MOZ_UPLOAD_DIR=/builds/slave/test/build/blobber_upload_dir'''
** This will allow your job to upload files to blobber


== Running Mozharness with a local version of Talos and Firefox on Linux ==
== Other examples ==
=== Running Mozharness with a local version of Talos and Firefox on Linux ===


These instructions can be more general, but this should be a useful starting point for any similar configuration.
These instructions can be more general, but this should be a useful starting point for any similar configuration.
This allows you to use your own objdir instead of downloading an installer and a tests.zip.
This allows you to use '''your own objdir''' instead of downloading an installer and a tests.zip.


<pre>
<pre>
Line 70: Line 69:
</pre>
</pre>


configs/talos/linux_config_local.py
Create the file configs/talos/linux_config_local.py


<pre>
<pre>
Confirmed users
3,990

edits