ReleaseEngineering/TryServer

Try Server

The try server is an easy way to test a patch without actually checking the patch into the core repository. Your code will go through the same tests as a mozilla-central push, and you'll be able to download builds if you wish.

To use try server, you need a Mozilla hg account (level 1 is sufficient).

How to push to try

Running a subset of builds/test/talos available to Try

You must use Build:TryChooser to choose which builds, tests, and talos you would like run on your push to try. Make sure you place the try chooser text in your topmost commit. The TryChooser web page can help you build a commit message for custom requests, so can the mercurial extension

% hg qref --message "try: <your-computed-syntax-here>"
% hg push -f try

Where the computed syntax will be something like -b d -p linux64 -u all -t none. Please note that running all tests on all platforms is very expensive (at least 300 compute-hours), and probably overkill.

Pushing to try

To submit your changes to the try server (assuming they're modifications to mozilla-central or a similar branch, e.g. tracemonkey), you have a few options:

• hg push -f ssh://hg.mozilla.org/try/, or
• hg push -f ssh://<username@host>@hg.mozilla.org/try/ ?, or
• hg push -f ssh://hg.mozilla.org/try/ -e 'ssh -l <username@host>

Creating an alias

To save yourself some typing, you can add an alias to your hgrc:

[paths]
try = ssh://hg.mozilla.org/try

and then push with

hg push -f try

~/.ssh/config

ssh host settings to make life easier

Host hg.mozilla.org
  User <commit_user_email_address>
  IdentityFile ~/.ssh/private_key_file

Viewing the results

You can see the results of your tryserver build in a number of ways:

• You'll get an email on a successful push with a link to tbpl for your revision as well as emails on any non-successful build/test/talos results (this setting can be adjusted using Build:TryChooser args for email notification)
• You can have the results of your try run posted to bug(s) automatically at the completion of the run using the --post-to-bugzilla flag in your try syntax (see: Build:TryChooser for examples)
• Look for your changeset on Try TBPL. You can add &pusher=YOUR.EMAIL to only see your pushes.
• Compare Talos perf numbers using Pike's talos-node or mconnor's compare-talos.
• Download your completed builds from firefox/tryserver-builds on ftp.m.o.

If you're using Mercurial queues, the push -f command pushes any patches that are currently applied, and the Try server will build the result. (This is an awesome feature, not a bug!)

You don’t need to clone or pull from the try repo, and you probably don’t want to. You’d get every half-baked changeset anybody ever tested.

See Jorendorff's blog for more details.

Using a custom mozconfig

The mozconfigs for recent mozilla-central clones are located in the browser/config/mozconfigs directory. Edit those as you please.

If you want to apply the same mozconfig changes to multiple platforms, you can edit build/mozconfig.common.override instead. This file is included at the end of each of the in-tree mozconfig files.

Android mozconfigs are in mobile/android/config/mozconfigs.

Note:

• TryServer purpose is to tell what will happen on Tinderbox, not to check every possible build option/configuration.
  • Any non-standard feature is implicitly unsupported. You may try them, but don't complain if they break.

Using a custom Gaia

For Gaia-related tests on B2G desktop builds, you can use an arbitrary branch on any github Gaia fork in your try job. To do this, you need to modify $src/b2g/config/gaia.json to point to your repo. Normally, gaia.json looks something like this:

{
    "revision": "6fbf376b98130fca7198d7d2e5e588cb37dd07c4", 
    "repo_path": "/integration/gaia-central"
}

You should edit this file to include some additional fields:

{
    "revision": "6fbf376b98130fca7198d7d2e5e588cb37dd07c4", 
    "repo_path": "/integration/gaia-central",
    "git": {
       "remote": "",
       "branch": "",
       "git_revision": ""
    }
}

You must specify remote, and can specify either branch or revision. If you specify both, revision will be used and branch ignored. For example:

{
    "revision": "6fbf376b98130fca7198d7d2e5e588cb37dd07c4", 
    "repo_path": "/integration/gaia-central",
    "git": {
       "remote": "https://github.com/jonallengriffin/gaia.git",
       "branch": "gaia_unit_crash",
       "git_revision": ""
    }
}

When the 'git' dict in gaia.json is populated in this way, the hg attributes for revision and repo_path will be ignored, and the Gaia will be cloned from the given remote and used to run the tests.

Getting debug symbols

By default native debug symbols are not uploaded for Try server builds because of their size. If you want to debug your builds locally you must add MOZ_CRASHREPORTER_UPLOAD_FULL_SYMBOLS=1 to the in-tree mozconfigs. You can do this for all platforms by importing this patch into your mq and pushing it along with your changes to Try. This will cause a ...crashreporter-symbols-full.zip package to be uploaded to the builds directory for each platform built.

Debugging with debug symbols

• Windows
  • Unzip the crashreporter-symbols-full.zip package somewhere, let's call it c:\try_symbols
  • Set your symbol path to cache*c:\symcache;SRV*http://msdl.microsoft.com/download/symbols;SRV*c:\try_symbols where c:\symcache is a writable directory where the debugger can store symbols.
    • If using WinDBG, you can type .sympath ... with the above path in the command window.
    • If using Visual Studio, you can set the _NT_SYMBOL_PATH environment variable to the above.
• Linux
  • TODO
• Mac
  • TODO

Server Status

• Try server load can be seen at https://secure.pub.build.mozilla.org/builddata/reports/pending/try.html
• Pending builds by revision are at https://secure.pub.build.mozilla.org/buildapi/pending
• In-progress builds by revision are are https://secure.pub.build.mozilla.org/buildapi/running

Other Notes

• Finished builds will live in http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/<your_ldap_email>-<revision> for 14 days before deletion
• Try repo may be reset every 6 weeks to avoid it slowing down with thousands of heads if needed. You can use links like https://tbpl.mozilla.org/?tree=Try&rev=<revision> to reach your results even after a reset (subject to TBPL data retention).
• TBPL data for try repos is purged after 30 days. After that, the trick above will no longer work.
• If you have any problems please file a bug

• Suggestions for the future can be made here or file a blocking bug against bug try_enhancements

HOWTO: Preserve a commit message between try pushes

Simply create a new patch in your queue for the try attempt commit message. Future pushes can simply push/pop the try patch onto the applied queue when needed.

% hg qref --message "bug xxx - a spell/patch of improvements"
% hg qnew patch.try
% hg qref --message "try: -b o -e -p all -u all -t none"
% hg push -f try
% hg qpop patch.try

If you know that your patches commit message is correct you can simply use:

% hg qref
% hg qnew patch.try
% hg qref --message "try: -b o -e -p all -u all -t none"
% hg push -f try
% hg qpop patch.try

Other Mozilla Try Servers

• Thunderbird Try Server for the comm-central repository

• You can push b2g18 to the regular Firefox tryserver, but you will get a lot of oranges. There is currently no work-around. It's possible to eke out some useful information from such a try push -- ask a sheriff, who should be familiar with the expected failures, or otherwise push your qtip rev to try and compare the results to the patched version. Good luck.

Problem Diagnosis

Can not access try server

Test your account & configuration

• ssh hg.mozilla.org, response: "No Interactive shells allowed here!"
• ssh hg.mozilla.org clone invalid_sandbox, response: menu display and interactive prompting.

Pushes to try take a very long time

Note: if a fellow developer cancelled their try push, they have saddled you with the cost of rebuilding the cache. (See caching details.)

If you're experiencing excessive wait times (> 45min) pushing to try, please file a bug asking IT to reset the try repository using this template (please include specifics of your experience). They will coordinate with sheriffs and release engineering as needed.

Waiting for Lock

If you get a message similar to:

   remote: waiting for lock on repository /repo/hg/mozilla/try/ held by 'hgssh1.dmz.scl3.mozilla.com:23974'
   remote: abort: repository /repo/hg/mozilla/try/: timed out waiting for lock held by hgssh1.dmz.scl3.mozilla.com:30549

It means several developers are trying to push to try at the same time. In the case above, nothing appears to be wrong, as the PID changes between the messages.

Waiting for Lock multiple times with the same pid

Similar to the above case, but with the same pid when you retry over and over again.

Please retry your push. If you see messages indicating the same process has been pushing for more than 15 minutes, treat as above.

Buildduty issues

How do I trigger additional talos/test runs for a given try build?

If your trychooser syntax included the tests you'd like more of, then select the job you want on TBPL and use the + button.

For test suites you didn't request originally you can ask Release Engineering to do a sendchange', look for someone who is <person>|buildduty in #releng on IRC. Alternatively you can push again with a different try syntax.

How do I cancel existing jobs?

For individual jobs, select the relevant one on TBPL and use the red X icon. To cancel all jobs, hover on the left side the push list on TBPL and click on the dim red X icon.

TryChooser

See the TryChooser wiki page.

See Also

• https://developer.mozilla.org/en/Creating_Mercurial_User_Repositories
• http://trychooser.pub.build.mozilla.org/
• https://wiki.mozilla.org/Sheriffing/How:To:Recommended_Try_Practices
• https://wiki.mozilla.org/Build:TryChooser
• https://wiki.mozilla.org/ReleaseEngineering:Autoland
• Manage Submissions [try]
• Scoreboard