MozillaWiki

ReleaseEngineering/TryChooser

< ReleaseEngineering
Revision as of 13:04, 14 June 2017 by Standard8 (talk | contribs) (→‎What if I want to run a sub-set of tests for a job?: Add warning that tag builds are broken)

TryChooser lets you select specific jobs you want to run on tryserver. You use TryChooser by putting a try: block at the end of the topmost commit message when you push to try.

Syntax Builder

To simplify using the TryChooser, you can use the TryChooser Syntax Builder website or the mercurial extension (which also functions as a command-line application, but is not maintained by RelEng).

Defaults

If you don't specify an option or if you give an invalid value for an option, TryChooser will use its default for that option: 

try: --build do
     --platform all
     --unittests all
     --talos none

Examples

try: -b do -p all -u all -t none
Run all platforms, all tests, no talos. You probably want to narrow down the test scope though if your change does not affect all platforms and all unit tests.:
try: -b do -u crashtest
Debug and optimized builds, all available platforms, only crashtests, no talos:
try: -b d -p linux -u none
Linux debug build, no unittests, no talos:
try: -b do -u all -t chrome,nochrome
Debug and optimized builds for all platforms, all unittests, select talos:
try: -b do -u all -t chrome,nochrome mozharness: --spsProfile
Same as above but run talos in profiling mode so that it produces zip files with profiles that can be opened in Cleopatra. See Talos Profiling.
try: -b o -u jsreftest,crashtest,mochitest-1 -t tp4,scroll
Opt builds on all available platforms, select unittests, select talos
try: -b o -u all -n --post-to-bugzilla bug 456234
Opt builds on all available platforms, all unittests, no emails, results posted to bug 456234
try: -b d -p macosx64 -u all[10.7]
Debug builds for 64-bit OSX, run all tests only on OSX 10.7 machines
try: -b d -p macosx64 -u all[-10.7]
Debug builds for 64-bit OSX, run all tests but not on all OSX test platforms except OSX 10.7
try: -b do -p all -u all[x64]
"T push" -- good replacement for -p all -u all. Builds on all platforms, runs all tests on linux64.
try: -b do -p all -u all[10.8]
Same as above, but run the tests on OSX 10.8 instead

You may precede the try: section with other text: 

 bug 12345 attempt 2, try: -b d -p linux64 -t jsreftest

Workflow

Without MQ

If you're not using Mercurial Queues, you can use TryChooser by including a "try:" string in the message of the commit you push to try: 

$ (edit your code)
$ hg commit -m "Bug XXXXXX - Change foo to bar.  try: -b d -p all"
$ hg push -f ssh://hg.mozilla.org/try

With MQ

Note: If you are running Mercurial 2.1 or higher, see this note about phases. (bug 725362)

If you're using Mercurial Queues, you can use TryChooser by adding an empty patch with a "try:" string to the top of your queue: 

hg qgoto your-patch
hg qnew -m "try: <Insert options here>" try
hg push -f -rtip ssh://hg.mozilla.org/try

After pushing, you can remove the empty patch: 

hg qpop
hg qremove try

FAQ

What if I want PGO for my build

Until bug 691673 is resolved, you'll have to do a small hack if you want PGO to work on your try push. Add 

mk_add_options MOZ_PGO=1

to build/mozconfig.common.override.

What if I don't put "try:" in my commit message?

If you don't give a "try:" string in your commit message, you'll get an error message on attempts to push to try as per bug 649402 where try syntax use was made mandatory.

What if make a syntax error in my commit message?

If you pass an invalid parameter, TryChooser will ignore it and substitute the default in its place. For instance, if you specify try: -b f -u crashtest, you'll get both debug and optimized builds, since "f" is not a recognized argument. You'll only get crashtest run on those builds, however.

Where's the full list of unit tests and talos tests I can request?

Try should have the same test/talos suites available to it as mozilla-central does so you can refer to Treeherder for complete listings.

To match up the Talos suite name to the tests it runs you can go here. (eg: 'chrome' runs ts:tdhtml:twinopen:tsspider)

If you need help building the syntax line you can also go to the TryChooser Syntax Builder page.

What do I do if I need to add a job after I push?

Use try extender on Treeherder. To be announced in November.

What do I do if I need to remove a job after I push?

Check out Self Serve!. With Self-Serve you can cancel pending builds, re-build/run completed builds/test/talos runs.

What do I do if I used the wrong syntax and I'm missing a job?

With the arbitrary jobs API you can trigger jobs that would have not been triggered according to your Try syntax. You can even specify a different build or tests.zip.

Where are my builds/test/talos?

If you think Treeherder is missing results or it feels like it's been a while since you kicked the build off and you're getting antsy for results. Here are some things you can look into before pinging the person on buildduty:

  • Did the build compile successfully? If it didn't, you won't get unittests or talos results.
    • Look for your directory in the builds FTP server and see what has been uploaded. (You can also find here the email-changeset string that you'll need if you end up pinging releng.)
  • Check Running and Pending to view your build's status, since these are where Treeherder gets the job data from, so are closer to the source of truth.

What if I want to run a sub-set of tests for a job?

Note: This is currently broken, see bug 1336606

Tests can be tagged via the *.ini files, e.g. 

[default]
tags = search

or 

[default]

[browser_426329.js]
tags = search

You can then run just the required tag by appending `--tag search` onto your try command, e.g. 

./mach try -b d -p linux64 -u mochitest-e10s-bc -t none --artifact --tag search

Parsing details

The parser for TryChooser is here.

The basic algorithm is:

  • The commit message is split on 'try:'.
  • Whatever follows after 'try:' is passed to try_parser as a string.
  • The string is processed into a list of args handed to parse_known_args (thus keeping anything that is not a recognized argument from causing errors).

The email preferences are parsed out in the try_mailer script, and the --post-to-bugzilla flag is currently a work in progress but will land in the RelEng tools repo as part of the Bugzilla Autolanding project.

Buildduty Issues

How to Deploy TryChooser changes

ReleaseEngineering/How_To/Update_TryChooser

Jobs not scheduled at all?

Recreate the comment of their change with https://mozilla-releng.net/trychooser/ and compare it to make sure is correct.

Then do a sendchange and tail the scheduler master: 

 buildbot sendchange --master buildbot-master10:9301 --revision 923103d5a656 \
   --branch try --username mpalmgren@mozilla.com \
   --comments "try: -b d -p linux -u all" doit

Using the TryChooser to submit build/test requests

Buildduty can also use the same TryChooser syntax as developers use to (re)submit build and testing requests. Here is an example: 

 buildbot sendchange --master buildbot-master10:9301 --revision 923103d5a656 \
   --branch try --username mpalmgren@mozilla.com \
   --comments "try: -b d -p linux -u all" doit

Unable to schedule builds due to missing 'requests' package

As with pip 8.0, the '--download-cache' option has been removed (it was already deprecated since 6.0). The fix for this issue was removing the argument in mozharness config (see https://hg.mozilla.org/try/rev/1c396281c01b). In order to build with a base revision prior to '1c396281c01b', you will need to rebase your patch to use a newer version.

Source code

The trychooser syntax builder lives in the buildtools repository, under the subdirectory "trychooser":

Retrieved from "https://wiki.allizom.org/index.php?title=ReleaseEngineering/TryChooser&oldid=1173484"