ReleaseEngineering

About Us

We ship Firefox to the world!

Release Engineering does this by:

• Developing world-class, scalable, open, secure, and maintainable infrastructure for application builds and tests, on a diverse set of desktop and mobile platforms
• Building fast and reliable pipelines to transform source code into shipped products

Contacting Release Engineering

Our team is located on 3 continents and in 5 time zones. This allows us to provide better service, especially if you contact us using one of the following methods:

• To request a new service, please file a bug
• To ask a general question, ping in #firefox-ci on Matrix, or email release@mozilla.com
• To request status on an existing request, set the "needinfo" flag in the bug.

Our team works normal business hours for our respective time zones. If you need to escalate an issue that is both urgent and important, use the following methods:

1. Use #firefox-ci on Matrix or email release@mmozilla.com, starting your message with 'URGENT:'. If no response in 15 minutes, then
2. Be sure the bug severity is set to "blocker"
3. Call one or more of the following people (nagios page preferred if you have access):
   • Chris AtLee Eastern Time
   • Jordan Lund Pacific Time

If you get voice mail, please leave your callback number.

Contributing

If you are interested in getting involved with Release Engineering, please see our contributing page! We'd love to hear from you.

Team

Mozilla's Release Engineering:

• twitter: @MozRelEng
• blogs: Planet Release Engineering
• calendar: Google Calendar
• mailing list: release-engineering@lists.mozilla.org

Name	Profile	Twitter	Blog
Ben Hearsum	bhearsum	@bhearsum	Ben's blog
Aki Sasaki	aki	@escapewindow	escapewindow
Mihai Tabara	mtabara	@mihaitabara	Mihai Tabara's blog

Interns

• Interns

Releng Offerings

As mentioned above, Releng offer pipelines to transform source code into shipped products. The following sections break down how that happens. Each section can be understood as both isolated and combined parts.

Initiating a Release

To start a release, Release Management submit a form in Ship It with what product they would like to release, what revision to use, and a number of other inputs.

Configuring & Scheduling

Ship It then creates Taskcluster tasks that are needed to take a builds and release them to users. These tasks make up the release graph that are derived from templates within taskgraph and are submitted via action tasks.

Signing

The signing of build artifacts and binaries. Some features of this:

• signing is done through a client/server model
• integrated with Taskcluster thanks to signingscript in conjunction with scriptworker, a securely managed Taskcluster worker type. signingscript tasks are part of the task graph that is generated from Ship It
• keys are generated with modern standards
• keys are secured by our infra and rotated regularly
• coming soon: self served signing through exposed API

Uploading

As important release artifacts are created, they are then uploaded to archive.m.o where they can served from and live permanently for historical or longstanding support reasons.

Like signing, uploading is integrated with Taskcluster thanks to beetmoverscript in conjunction with scriptworker, a securely managed Taskcluster worker type. signingscript tasks are part of the task graph that is generated from Ship It

Install & Update Tests

Like product tests, releases likewise need tests. This ensures that initial downloads install correctly on all supported platforms and updates work as expected, all before actually releasing. Right now this is done through our Update and Final Verify tests. Which, like all other sections are part of the release graph.

TODO: link to documentation and source of these

Shipping & Updates

Finally, we release the product via both initial downloads and as new updates.

For Desktop Firefox:

• Initial installs are served via updating the download page, mozilla.com (Bedrock) and updating Bouncer, Mozilla's mirroring and redirection server.
• Updates are handled by our own application, Balrog. Balrog is our server side component of the update system
  • this is integrated with Taskcluster thanks to balrogscript in conjunction with scriptworker, a securely managed Taskcluster worker type. balrogscript tasks are part of the task graph that is generated from Ship It

For Android Products, including Firefox:

• releases are submitted to the Play Store
  • this is integrated with Taskcluster thanks to pushapkscript in conjunction with scriptworker, a securely managed Taskcluster worker type. pushapkscript tasks are part of the task graph that is generated from Ship It

Development

Best Practices

• Python Standards - Our Python coding standards
• Development Best Practices - read this when you're starting something new
• Deployment - How we deploy our tools, and how do we update them.
• Wiki Guidelines - how to wiki garden for fun and profit

Bugzilla

• Flags
• Whiteboard
• Triage

Operations

Release Engineering has historically been involved with the Build and Release pipeline of a product. As part of that, we have some responsibility to help ensure that the build (continuous integration) and release (deployment) pipeline infrastructure is in good running health. We have two functional groups that monitor and respond to any operational issues that come up.

Buildduty

Buildduty provides support for Firefox's build and test infrastructure

The role of Buildduty has grown beyond Release Engineering's scope. There is a dedicated team of 6 people, providing 24/7 support of Mozilla's build and test infrastructure.

See the Buildduty home page and manifesto for more information.

CI Duty

CI Duty provides support for Firefox's release infrastructure and is currently handled by Releng

See the CIDuty for more information.

Releng Applications

• Applications - Various applications and services that RelEng provides
• TODO update application list with: Services, Scriptworker, ChainOfTrust

Mozilla Applications

• Taskcluster - our task execution tool for providing continuous integration and release pipelines
• Treeherder - aggregates all the build, test, and release tasks as a reporting dashboard
• PuppetAgain - Release Operations tool and puppet configs for configurating build and release machines. Both for workers and hosting applications
• TODO - add Release Operations replacement for Slaveapi: Controller

External Third Party Dependencies

• AWS
• Heroku
• Papertrail
• Nagios
• Sentry

Deprecated

below pages should be deleted or be given a "warning" outdated box where appropriate.

VCS Sync

• HowTo
• History

Releases

• Release Checklist and Build Notes
• Releng Post Mortem Meetings
• What is releaseduty?
• Release Promotion
• Update Generation
  • Patcher tags for release updates

Platform Support

• Reference Platforms
• Slave Allocator
• Official Platform Support Checklist

Tryserver

• Tryserver As Branch
• TryServer
• TryChooser

Testing

• Intermittent Errors
• Requesting a loaner
• Testing a new Signing Server
• Testing your changes

Subpages

• Presentations
• How Tos
• Releng Docs outdated
• Special:PrefixIndex/ReleaseEngineering/ - all subpages of ReleaseEngineering in the wiki
• ReleaseEngineering/Obsolete Pages
• day 1 checklist
• day -1 checklist

Run the tasks to staging pools (OSX/Linux)

• Clone mozilla-central to local machine: hg clone ssh://hg.mozilla.org/mozilla-central
• Add the following changes to taskcluster/taskgraph/transforms/task.py file:

diff --git a/taskcluster/taskgraph/transforms/task.py b/taskcluster/taskgraph/transforms/task.py
--- a/taskcluster/taskgraph/transforms/task.py
+++ b/taskcluster/taskgraph/transforms/task.py
@@ -1717,16 +1717,48 @@ def add_index_routes(config, tasks):
         index_type = index.get('type', 'generic')
         task = index_builders[index_type](config, task)

         del task['index']
         yield task


 @transforms.add
+def set_worker_type(config, tasks):
+    for task in tasks:
+        level = str(config.params['level'])
+
+        provisioner_id, worker_type = get_worker_type(
+            config.graph_config,
+            task['worker-type'],
+            level,
+        )
+        task['worker-type'] = '/'.join([provisioner_id, worker_type])
+        yield task
+
+
+@transforms.add
+def use_beta_workertypes(config, tasks):
+    """
+    This transform uses a dictionary of worker type names to worker type names
+    in order to map one set of worker types to another set. This is typically
+    useful when you want to run a try push against a staging set of worker
+    types, that you want to test before rolling them out to production.
+    """
+    find_replace_dict = {
+        "releng-hardware/gecko-t-osx-1014":      "releng-hardware/gecko-t-osx-1014-staging",
+        "releng-hardware/gecko-t-linux-talos":   "releng-hardware/gecko-t-linux-talos-b"
+    }
+    for task in tasks:
+        if task['worker-type'] in find_replace_dict:
+            task['worker-type'] = find_replace_dict[task['worker-type']]
+        yield task
+
+
+@transforms.add
 def build_task(config, tasks):
     for task in tasks:
         level = str(config.params['level'])

         provisioner_id, worker_type = get_worker_type(
             config.graph_config,
             task['worker-type'],
             level,

• Commit the changes using the following commit message: hg commit -m "<commit_message>; try: -b do -p macosx64 -u all -t none". The <commit_message> will be a message that will describe your action, like: Run all tests on OSX staging workers.
  • To run different tests or on a different platform, use https://mozilla-releng.net/trychooser/
  • Do _not_ run unnecessay tests or you may end up on the hiscore board.

• Push the changes into try repository: hg push -f -r . ssh://hg.mozilla.org/try

• You will receive an email like this:

Thank you for your try submission. It's the best!

Results will be displayed on Treeherder as they come in:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6657878a04024803ce0aae0601627810b2d8d95e

It looks like this submission has talos jobs. You can compare the performance of your push against a baseline revision here:
https://treeherder.mozilla.org/perf.html#/comparechooser?newProject=try&newRevision=6657878a04024803ce0aae0601627810b2d8d95e

Once completed, all build and log artifacts will be available through the Treeherder URL above.

All Taskcluster based artifacts can also be found in the Taskcluster Artifact Index:
e.g.  https://tools.taskcluster.net/index/artifacts/gecko.v2.try.revision.6657878a04024803ce0aae0601627810b2d8d95e

Summary: Test linux generic-worker; try: -b do -p linux64 -u none -t chromez-e10s

Notes

• A useful link for try syntax: https://mozilla-releng.net/trychooser/
• To see the hosts from the staging pool:

* OSX hosts: https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-osx-1010-beta
* Linux hosts: https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-linux-talos-b