Sheriffing/How To/Recommended Try Practices: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (ChrisCooper moved page Sheriffing/How:To:Recommended Try Practices to Sheriffing/How To/Recommended Try Practices: More intuitive naming scheme)
(linked to Firefox docs for further information about Try server some cleanup)
 
(2 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Sheriffing How To|Recommended Try practices}}
{{Sheriffing How To|Recommended Try practices}}
The [https://wiki.mozilla.org/ReleaseEngineering/TryServer TryServer] is a critical piece of our infrastructure for ensuring that patches are suitable for landing prior to being pushed. However, proper usage is important to avoid pushing under- or over-tested patches. This page is intended to provide some recommended practices for finding the right balance. The [http://trychooser.pub.build.mozilla.org/ TryChooser Syntax Builder] offers control over which builds and tests are run.
The [https://wiki.mozilla.org/ReleaseEngineering/TryServer TryServer] is a critical piece of our infrastructure for ensuring that patches are suitable for landing prior to being pushed. However, proper [https://firefox-source-docs.mozilla.org/tools/try/index.html usage] is important to avoid pushing under- or over-tested patches. This page is intended to provide some recommended practices for finding the right balance.


Note that the guidelines below are not a substitute for using your best judgement. The guidelines below are intended to be general recommendations but not absolute musts.
Note that the guidelines below are not a substitute for using your best judgment. The guidelines below are intended to be general recommendations but not absolute musts.


<ol>
<ol>
   <li>In many cases, a "try: -b do -p all -u all -t none" is overkill and <b>should be avoided</b>. A full Try run consumes <b><u>over 500hrs of machine time</u></b> to generate a full set of builds and run the full suite of tests across them. This can have a real impact on wait times across resource-constrained platforms, including production runs.</li>
   <li>In many cases, running all builds and tests is overkill and <b>should be avoided</b>. A full Try run consumes <b><u>over 500hrs of machine time</u></b> to generate a full set of builds and run the full suite of tests across them. This can have a real impact on wait times across resource-constrained platforms, including production runs.</li>
   <li>If in doubt, run both debug and opt builds.</li>
   <li>If in doubt, run both debug and opt builds.</li>
   <li>If the patch only affects one platform, only that platform should have builds and tests.
   <li>If the patch only affects one platform, only that platform should have builds and tests.</li>
    <ul><li>For example, a B2G-only Try run should look something like "try: -b do -p emulator,emulator-jb,emulator-kk,linux32_gecko,linux64_gecko,macosx64_gecko,win32_gecko -u all -t none"</li></ul>
   <li>Be aware of which test suites are likely to be affected by your patch and avoid running others that won't.
  </li>
     <ul><li>For example, a devtools-only change is unlikely to break any tests outside of mochitest-dt and xpcshell.</li></ul></li>
   <li>Be aware of which test suites are likely to be affected by your patch and avoid running others that won't.</li>
   <li>If the code is cross-platform, a <i>T-style</i> run (build on all platforms, test on one) will find most problems while making best use of resources. (Use copy several filter changes with <code>./mach try chooser</code> and copy and paste if necessary).
     <ul><li>For example, a devtools-only change is unlikely to break any tests outside of mochitest-dt and xpcshell.</li></ul>
   <li>If the code is cross-platform, a <i>T-style</i> run (build on all platforms, test on one) will find most problems while making best use of resources.
     <ul>
     <ul>
       <li>T runs can be created by using the "Restrict tests to platform(s)" checkbox on TryChooser.</li>
       <li>Linux64 is the best choice for the test platform as it is the least resource-constrained.</li>
      <li>Linux32 is the best choice for the test platform as it is the least resource-constrained.</li>
      <ul><li>"try: -b do -p all -u all[Ubuntu,-x64] -t none" will run builds on all platforms and tests only on Linux32.</li></ul>
      <li>Multiple platforms can be specified as-needed (i.e. Linux + Android or any other combo as needed).</li>
     </ul>
     </ul>
   </li>
   </li>
   <li>If you no longer need builds/tests from a run, cancel the remaining jobs.</li>
   <li>If you no longer need builds/tests from a run, cancel the remaining jobs.
     <ul>
     <ul>
       <li>Use the stop button on Treeherder's job details panel.</li>
       <li>Use the stop button on Treeherder's job details panel.</li>
Line 35: Line 30:
</ol>
</ol>


[https://secure.pub.build.mozilla.org/builddata/reports/reportor/daily/highscores/highscores.html High scores!]
[https://catlee.github.io/highscores/highscores.html High scores!]

Latest revision as of 15:30, 28 February 2020

The TryServer is a critical piece of our infrastructure for ensuring that patches are suitable for landing prior to being pushed. However, proper usage is important to avoid pushing under- or over-tested patches. This page is intended to provide some recommended practices for finding the right balance.

Note that the guidelines below are not a substitute for using your best judgment. The guidelines below are intended to be general recommendations but not absolute musts.

  1. In many cases, running all builds and tests is overkill and should be avoided. A full Try run consumes over 500hrs of machine time to generate a full set of builds and run the full suite of tests across them. This can have a real impact on wait times across resource-constrained platforms, including production runs.
  2. If in doubt, run both debug and opt builds.
  3. If the patch only affects one platform, only that platform should have builds and tests.
  4. Be aware of which test suites are likely to be affected by your patch and avoid running others that won't.
    • For example, a devtools-only change is unlikely to break any tests outside of mochitest-dt and xpcshell.
  5. If the code is cross-platform, a T-style run (build on all platforms, test on one) will find most problems while making best use of resources. (Use copy several filter changes with ./mach try chooser and copy and paste if necessary).
    • Linux64 is the best choice for the test platform as it is the least resource-constrained.
  6. If you no longer need builds/tests from a run, cancel the remaining jobs.
    • Use the stop button on Treeherder's job details panel.
    • Unneeded jobs are a frequent source of frustration-inducing backlog.
  7. Any existing jobs on a push can be retriggered without a new push.
    • Use the retrigger button on Treeherder's job details panel.
    • You can trigger as many additional runs as needed (i.e. trying to track down an intermittent failure).
    • Retriggering a build will cause all tests to be run again after the build completes.

High scores!

Retrieved from "https://wiki.allizom.org/index.php?title=Sheriffing/How_To/Recommended_Try_Practices&oldid=1224359"