ReleaseEngineering/TryServer

Try Server

The try server is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.

To use try server, you need a Mozilla hg account (level 1 is sufficient).

How to push to try

Running a subset of builds/test/talos available to Try

You must use Build:TryChooser to choose which builds, tests, and talos you would like run on your push to try. Make sure you place the try chooser text in your topmost commit. The TryChooser web page can help you build a commit message for custom requests, so can the mercurial extension

% hg qref --message "try: -b o -p all -u all -t none"
% hg push -f try

Pushing to try

To submit your changes to the try server (assuming they're modifications to mozilla-central or a similar branch, e.g. tracemonkey), you have a few options:

• hg push -f ssh://hg.mozilla.org/try/, or
• hg push -f ssh://<username@host>@hg.mozilla.org/try/ ?, or
• hg push -f ssh://hg.mozilla.org/try/ -e 'ssh -l <username@host>

Creating an alias

To save yourself some typing, you can add an alias to your hgrc:

[paths]
try = ssh://hg.mozilla.org/try

and then push with

hg push -f try

~/.ssh/config

ssh host settings to make life easier

Host hg.mozilla.org
  User <commit_user_email_address>
  IdentityFile ~/.ssh/alternate_public_identity_key_file

hg phases

hg phases were introduced in Mercurial 2.1. Their purpose is to keep you from accidentally modifying changesets you've shared with others, or from accidentally sharing your patch queue with others. In particular, this keeps you from qpop'ing patches you've pushed to try!

When you run hg qpop after pushing to try, you'll get

abort: popping would remove an immutable revision

You can force hg to let you qpop your patch queue by running

hg phase -f --draft qbase:tip

Disable hg phases with a post-push hook

Alternatively, you can avoid this rigamarole by disabling hg phases with an outgoing hook. Add the following to your personal .hgrc file (~/.hgrc):

 [hooks]
 post-push = hg phase --force --draft "mq()"

Now you should be able to perform hg push -f <whatever> without it affecting the phase.

These extra steps should no longer be necessary once the server is upgraded to Mercurial 2.1 and configured as "non-publishing" (bug 725362).

If you use an alias (e.g. "hg try" instead of "hg push -f ssh://hg.mozilla.org/try")

If you use an alias (e.g. you use hg try to push to try), this won't work -- you need to actually specify the post-command that you want it to perform after.

For example, if you alias your push command to hg try, as

  [alias]
  try = push -f ssh://hg.mozilla.org/try

Then you need to use the following instead of the above post-push hook:

[hooks]
post-try = hg phase --force --draft "mq()"

Viewing the results

You can see the results of your tryserver build in a number of ways:

• You'll get an email on a successful push with a link to tbpl for your revision as well as emails on any non-successful build/test/talos results (this setting can be adjusted using Build:TryChooser args for email notification)
• You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax (see: Build:TryChooser for examples)
• Look for your changeset on Try TBPL. You can add &pusher=YOUR.EMAIL to only see your pushes.
• Compare Talos perf numbers using Pike's talos-node or mconnor's compare-talos.
• Download your completed builds from firefox/tryserver-builds on ftp.m.o.

If you're using Mercurial queues, the push -f command pushes any patches that are currently applied, and the Try server will build the result. (This is an awesome feature, not a bug!)

You don’t need to clone or pull from the try repo, and you probably don’t want to. You’d get every half-baked changeset anybody ever tested.

See Jorendorff's blog for more details.

Using a custom mozconfig

The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.

If you want to apply the same mozconfig changes to multiple platforms, you can edit build/mozconfig.common.override instead. This file is included at the end of each of the in-tree mozconfig files.

Android mozconfigs are in mobile/android/config/mozconfigs.

Note:

• TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.
  • Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.

Getting debug symbols

By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1 to the in-tree mozconfigs. You can do this for all platforms by importing this patch into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.

Debugging with debug symbols

• Windows
  • Unzip the crashreporter-symbols-full.zip package somewhere, let's call it c:\try_symbols
  • Set your symbol path to cache*c:\symcache;SRV*http://msdl.microsoft.com/download/symbols;SRV*c:\try_symbols where c:\symcache is a writable directory where the debugger can store symbols.
• Linux
  • TODO
• Mac
  • TODO

Server Status

• Try server load can be seen at http://build.mozilla.org/builds/pending/try.html
• Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending
• In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running

Other Notes

• Finished builds will live on http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/{your_email_changeset} for 4 days
  • They are moved to http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/old/{your_email_changeset} for an additional 14 days before removal
• Try repo is reset occasionally, and you may not be able to use TBPL to see your test results after it has been reset.
• If you have any problems please file a bug
• Windows builds have symbols uploaded to http://build.mozilla.org/tryserver-symbols. Windbg and the Visual Studio debugger may use them to help debug crashing try server builds. Instructions for setting this up can be found here: http://developer.mozilla.org/en/docs/Using_the_Mozilla_symbol_server. Make sure you use the aforementioned URL instead of http://symbols.mozilla.org/firefox.

• Suggestions for the future can be made here or file a blocking bug against bug try_enhancements

HOWTO: Preserve a commit message between try pushes

Simply create a new patch in your queue for the try attempt commit message. Future pushes can simply push/pop the try patch onto the applied queue when needed.

% hg qref --message "bug xxx - a spell/patch of improvements"
% hg qnew patch.try
% hg qpush patch.try
% hg qref --message "try: -b o -e -p all -u all -t none"
% hg push -f try
% hg qpop patch.try

Other Mozilla Try Servers

• Thunderbird Try Server for the comm-central repository

Problem Diagnosis

Test your account & configuration

• ssh hg.mozilla.org, response: "No Interactive shells allowed here!"
• ssh hg.mozilla.org clone invalid_sandbox, response: menu display and interactive prompting.

Buildduty issues

How do I trigger additional talos/test runs for a given try build?

See ReleaseEngineering/How_To/Trigger_Talos_Jobs

How do I cancel existing jobs?

TBD

TryChooser

See the TryChooser wiki page.

Bug Commenter

This is on cruncher and is run in a crontab in lsblakk's account:

source /home/lsblakk/autoland/bin/activate && cd /home/lsblakk/autoland/tools/scripts/autoland \
&& time python schedulerDBpoller.py -b try -f -c schedulerdb_config.ini -u None -p None -v

You can see quickly if things are working by looking at:

/home/lsblakk/autoland/tools/scripts/autoland/postedbugs.log  # this shows what's been posted lately
/home/lsblakk/autoland/tools/scripts/autoland/try_cache  # this shows what the script thinks is 'pending' completion

See Also

• http://developer.mozilla.org/en/Creating_Mercurial_User_Repositories
• http://trychooser.pub.build.mozilla.org/
• https://wiki.mozilla.org/Build:TryChooser
• https://wiki.mozilla.org/ReleaseEngineering:Autoland
• Manage Submissions [try]