View source for ReleaseEngineering:TestingTechniques

= Code review guidelines =

* See the [[Code_Review]] documentation on how to be an effective code reviewer, and to how to prepare patches effectively as an author.
* Also see Eric Rose's presentation on constructive code review https://air.mozilla.org/erik-rose-constructive-code-review/
* For authors:
** If possible, please include a try run with your patches for review
** If you are working on a high priority bug, please include your work in progress patches on the bug at the end of your workday so that people in other timezones can pick work on it during their day if required.

= Taskcluster =

==Testing changes to the decision task==

* From https://firefox-source-docs.mozilla.org/python/taskgraph.html?highlight=taskgraph#module-taskgraph

"The decision task is the first task created when a new graph begins. It is responsible for creating the rest of the task graph.

The decision task for pushes is defined in-tree, in .taskcluster.yml. That task description invokes mach taskcluster decision with some metadata about the push. That mach command determines the optimized task graph, then calls the TaskCluster API to create the tasks.

Note that this mach command is not designed to be invoked directly by humans. Instead, use the mach commands described below, supplying parameters.yml from a recent decision task. These commands allow testing everything the decision task does except the command-line processing and the queue.createTask calls."

[https://docs.taskcluster.net/ Taskcluster ] creates a large graph of tasks, some which depend on each other.  Each task has different properties. Depending on the code tree that the task runs on, different tasks may run.  This decision task will vary depending on the code tree it is running on.
For every push to a code tree, a Gecko Decision Task runs which generates several files associated with it, include parameters.yml which includes metadata for that push.  

[[Image:Screen Shot 2017-05-29 at 11.02.17 AM.png]]

==Comparing the list of tasks after a patch is applied==
* You can download the parameters.yml file associated with a push on a particular tree.  This file can be used locally with the mach taskgraph command.
* Documentation on using mach taskgraph http://gecko.readthedocs.io/en/latest/taskcluster/taskcluster/taskgraph.html

* For example, if you have a local clone of a repo, in this case mozilla-beta, you can output the list of tasks to a file. Then apply your patch, output the list of tasks to a file, and then diff to ensure your patch resulted in the expected changes. In this case, the patch is to enable talos jobs for linux64 devedition builds.

 Kims-MacBook-Pro:mozilla-beta kmoir$ pwd
 /Users/kmoir/hg/mozilla-beta
 Kims-MacBook-Pro:mozilla-beta kmoir$ ./mach taskgraph target -p ~/Downloads/parameters-m-b.yml &> old
 apply patch
 Kims-MacBook-Pro:mozilla-beta kmoir$ hg diff
 diff --git a/taskcluster/ci/test/test-platforms.yml b/taskcluster/ci/test/test-platforms.yml
 --- a/taskcluster/ci/test/test-platforms.yml
 +++ b/taskcluster/ci/test/test-platforms.yml
 @@ -71,16 +71,17 @@ linux64-pgo/opt:
 
  linux64-devedition/opt:
      build-platform: linux64-devedition-nightly/opt
      test-sets:
          - common-tests
          - web-platform-tests
          - opt-only-tests
          - desktop-screenshot-capture
 +        - talos
          - awsy
 
  linux64-asan/opt:
      build-platform: linux64-asan/opt
      test-sets:
          - common-tests
  
  # Stylo builds only run a subset of tests for the moment. So give them
 diff --git a/taskcluster/taskgraph/transforms/job/mozharness_test.py b/taskcluster/taskgraph/transforms/job/mozharness_test.py
 --- a/taskcluster/taskgraph/transforms/job/mozharness_test.py
 +++ b/taskcluster/taskgraph/transforms/job/mozharness_test.py
 @@ -22,16 +22,17 @@ ARTIFACTS = [
      ("public/test", "artifacts/"),
      ("public/test_info/", "build/blobber_upload_dir/"),
  ]
 
  BUILDER_NAME_PREFIX = {
      'linux64-pgo': 'Ubuntu VM 12.04 x64',
      'linux64': 'Ubuntu VM 12.04 x64',
      'linux64-nightly': 'Ubuntu VM 12.04 x64', 
 +    'linux64-devedition-nightly': 'Ubuntu VM 12.04 x64',
      'linux64-asan': 'Ubuntu ASAN VM 12.04 x64',
      'linux64-ccov': 'Ubuntu Code Coverage VM 12.04 x64',
      'linux64-jsdcov': 'Ubuntu Code Coverage VM 12.04 x64',
      'linux64-stylo': 'Ubuntu VM 12.04 x64',
      'macosx64': 'Rev7 MacOSX Yosemite 10.10.5',
      'android-4.3-arm7-api-15': 'Android 4.3 armv7 API 15+',
      'android-4.2-x86': 'Android 4.2 x86 Emulator',
      'android-4.3-arm7-api-15-gradle': 'Android 4.3 armv7 API 15+',
 Kims-MacBook-Pro:mozilla-beta kmoir$ ./mach taskgraph target -p ~/Downloads/parameters-m-b.yml &> new
 Kims-MacBook-Pro:mozilla-beta kmoir$ diff old new
 ....
 > test-linux64-devedition/opt-talos-chrome
 > test-linux64-devedition/opt-talos-chrome-e10s
 > test-linux64-devedition/opt-talos-dromaeojs
 > test-linux64-devedition/opt-talos-dromaeojs-e10s
 > test-linux64-devedition/opt-talos-g1
 > test-linux64-devedition/opt-talos-g1-e10s
 > test-linux64-devedition/opt-talos-g2
 > test-linux64-devedition/opt-talos-g2-e10s
 > test-linux64-devedition/opt-talos-g3
 > test-linux64-devedition/opt-talos-g3-e10s
 > test-linux64-devedition/opt-talos-g4
 > test-linux64-devedition/opt-talos-g4-e10s
 > test-linux64-devedition/opt-talos-other
 > test-linux64-devedition/opt-talos-other-e10s
 > test-linux64-devedition/opt-talos-svgr
 > test-linux64-devedition/opt-talos-svgr-e10s
 > test-linux64-devedition/opt-talos-tp5o
 > test-linux64-devedition/opt-talos-tp5o-e10s

* The diff indicates that new talos jobs are added. You should include a diff like this when asking for review of your taskcluster patches on the bug or pull request.
Note: You can also specify the branch of the tree instead of downloading the paramaters.yml - see  Ability to use latest parameters.yml file from given tree (https://bugzilla.mozilla.org/show_bug.cgi?id=1378212)

==Generating the decision task locally in JSON==
* You can also output the JSON of the taskgraph that will be generated.  This is a useful troubleshooting tool when you are changing the dependencies of the tasks within a graph, or the payload

 ./mach taskgraph target  --json -p ~/Downloads/parameters-m-b.yml > task.json
 more task.json
 {
   "android-api-15-gradle-dependencies": {
     "attributes": {
       "build_platform": "android-api-15-gradle-dependencies",
       "build_type": "opt",
       "kind": "android-stuff",
       "run_on_projects": [
         "all"
       ]
     },
     "dependencies": {},
     "kind_implementation": "taskgraph.task.transform:TransformTask",
     "label": "android-api-15-gradle-dependencies",
     "task": {
       "created": {
         "relative-datestamp": "0 seconds"
       },
       "deadline": {
         "relative-datestamp": "1 day"
       },
       "expires": {
         "relative-datestamp": "1 year"
       },
 .....

==Changing the target==

You can also modify the properties of your parameters.yml file to reflect a different target then the default one to look at the jobs generated for this target task.  Target tasks are defined in tree here taskcluster/taskgraph/target_tasks.py.  For instance, here I change the default target to nightly_linux from the default push to see the tasks generated for that target.
 Kims-MacBook-Pro:mozilla-central kmoir$ cat ~/Downloads/parameters-m-c.yml 
 base_repository: https://hg.mozilla.org/mozilla-central
 build_date: 1494018407
 filters:
 - check_servo
 - target_tasks_method
 head_ref: ff83fde8be946eabcf27ea97d4676f601c122194
 head_repository: https://hg.mozilla.org/mozilla-central/
 head_rev: ff83fde8be946eabcf27ea97d4676f601c122194
 include_nightly: false
 level: '3'
 message: ' '
 moz_build_date: '20170505210647'
 optimize_target_tasks: true
 owner: kwierso@gmail.com
 project: mozilla-central
 pushdate: 1494018407
 pushlog_id: '31771'
 #target_tasks_method: default
 target_tasks_method: nightly_linux
 triggered_by: push
 
 Kims-MacBook-Pro:mozilla-central kmoir$ ./mach taskgraph target -p ~/Downloads/parameters-m-c.yml &> default

== See also==
[https://bugzilla.mozilla.org/show_bug.cgi?id=1359328#c2 Bug 1359328]