User:Lukasblakk/TryServerSyntax

Syntax for Custom Try Requests

The goal of this syntax is to allow user to select particular build/unittest/talos combinations in their hg commit message, a config.info file in the $topsrcdir or from a web interface.

Syntax

try: --build [d,o,do,all] --p [(desktop_platforms,...),none] --m [(mobile_platforms,...),none] --u [(test_suites,...),all] --t [(talos_suites,...),none]

This covers:

• Build Types both(do) || opt(o) || debug(d) || m-c all override (all)
• Platforms none || --p {linux,linux64,osx,osx64,win32,win64}
• Mobile none || --m {mobile-maemo5-qt, android, etc}
• Unittest Suites all || --u {crashtest,reftest,mochitest{1,2,3,4,5},xpcshell,etc}
• Talos Suites all || --t {tp4,tscroll,tspider,etc}

Order

Specify build types (opt/debug/both) first then either all platforms or specify what you want then all unittest suites, no unittest suites, or specify a suite (then run for each platform built) then all talos suites, no talos suites, or specify suite (then run for each platform built)

This allows us to cut off at any point and just to a default 'None' for any info not submitted. If the user submits just a build type, they get a "Does is compile?" try run.

Usage Examples

In an hg commit message

 # to run what mozilla-central's automation does ie: all platforms, all tests/talos
 hg commit -m "bug 432432 testing something try: --build all" 

 # both opt/debug builds, all available {desktop,mobile} platforms, no unittests/talos
 hg commit -m "bug 432432 testing something try: --build do" 

 # linux debug build, all available mobile platforms, all unittests, no talos
 hg commit -m "bug 432432 testing something try: --build d --p linux --u all"

 # mobile-only builds
 hg commit -m "bug 432432 testing something try: --build o --p none"

 # all builds, all available desktop platforms, no mobile, all unittests & talos
 hg commit -m "bug 432432 testing something try: --build do --m none --u all --t all"

 # opt builds on all available {desktop,mobile} platforms, no unittests, talos tp4 & tscroll 
 hg commit -m "bug 432432 testing something try: --build o --t tp4,tscroll"

In a try.info file

try.info file pushed to $topsrc_dir would look like:

 try: --build do --m none --u all --t all

 OR

 try: --build all

 OR

 try: --build d --m none --u reftest

Scheduler/Builder Parsing

• splits on 'try:'
• then splits on ' ' to create a list to pass to argparse which runs parse_known_args (thus keeping anything that is not recognized from causing errors)
• we also have allowed for manual sendchanges to send in comments with "try: ..." in order to make it possible to (re)run a particular test/talos/platform and not all things specified in a commit message

Default Set

If you only specify try: or no commit comment is present (or info file) then the default set would be a "Does it compile?" set comprised of:

All try desktop platforms' debug builds, all mobile builds, no tests, no talos

Optional arguments

If the following exist they will be handled as follows:

build (default is debug)

Options are o (opt) || d (debug) || do (gets you both) || all (override for m-c style run)
We use this to build the desktop and test builder names

desktop_platforms (if not present default is all)

Now we can create a builderName for each combo of opt/debug platform builders specified

mobile_platforms (if not present default is all)

Now we can create a builderName for each mobile platform builder specified

unittests (if not present default is none)

--u variables are used to add test builderNames (if ['all'] then all available testBuilders will be added for that platform ie: no mac a11y if that's not a test suite on mac) 

talos (if not present default is none)

--t variables are used to add talos builderNames (if ['all'] then all available talosBuilders will be added for that platform) 

How do I know what test/talos suites I can list?

Note: we are using a flat file for right now, LINK TO VALID_BUILDERS but bug 589847 has been filed and in the next iteration we will have a way to generate the lists of valid builder info dynamically from config.py which will keep it in sync with try's configs