Sheriffing/How To/Merges

Goal

Autoland and Inbound repositories are opened for developers who work on bug fixes, improvements and other. Usually, twice a day, clean code is transferred form Autoland and Inbound to Central, and back.
No sync is performed, only differences are transferred (merge).

Merges are done between 8:00 PM – 10 PM UTC and 8 AM – 10 AM. At 11AM and 11PM UTC time, Nightly builds start (they take between 2 to 4 hours to build) and it’s desired to transfer as much clean code as possible. Ideally, merges should be done before 10 AM/PM UTC time to be able to fix bad code before Nightly builds start.

Each sheriff should perform a merge at the beginning of their work day. Effectively, this results in two sets of merges every day to/from autoland and mozilla-inbound to mozilla-central.

Note: in winter Nightlies, always start at 12PM/AM Romanian time (+1 or 2 minutes after that), so having merges done before that helps with getting the changes into these builds. How long it takes to merge depends, mostly on merge conflicts.

Choosing a changeset to merge

For merges from the integration trees to mozilla-central, you should choose a changeset with a green PGO run to avoid bustages on nightly builds that also run PGO. You need to also make sure that are *NOT* any backouts *after* that changeset.

A more in depth view of the conditions a merge candidate must fulfill:

1. Candidate revision must be 100% completed or at least 98-99%, with maximum 15 jobs running.
2. No jobs classified as fixed by commit
3. Few or no unclassified jobs
4. Windows 7 pgo platform must have all jobs finished (32 minimum – M-e10s, see screenshot as an example)
   
5. No backouts after merge candidate
6. If a merge candidate has more than 15 jobs running but they are autophone jobs, the candidate is good to use

Order of operations

Integration branches should be merged to mozilla-central first, then mozilla-central should be merged back to the integration branches. These merge instructions were copied from Sheriffing From Unified Repos document.

Merge pattern:

1. autoland > central
2. inbound > central
3. central > autoland
4. central > inbound

Merging from autoland and mozilla-inbound to mozilla-central

Steps on how to do the autoland to mozilla-central AND mozilla-inbound to mozilla-central merge:

1. Have your revision ids ready
2. In the console switch to Mozilla-unified directory: cd mozilla-unified
3. Download the Central repo: hg pull central
4. Switch and update to it: hg update central
5. Autoland to central merge command: hg merge -r <autoland_revision_id>
6. Add a commit message: hg commit -m "Merge autoland to mozilla-central. a=merge"
7. Transfer the files to Central repo: hg push -r . central
8. Inbound to Central merge command: hg merge -r <inbound_revision_id>
9. Add a commit message: hg commit -m "Merge inbound to mozilla-central. a=merge"
10. Transfer the files to Central repo: hg push -r . central

After the merges are in Central, bugs need to be marked with Bugherder: go to Central and in the top right side of each of your pushes to Central, click on Action Menu (arrow button next to the pin button) and choose Mark With Bugherder.

Navigate through all the pages, check for any security bugs that can’t be updated by Bugherder and mention them to someone who can, check that Resolve and Fixed checkboxes are ticked and the correct version is set. At the last step click Submit and use your API key to finish the process. Leave the tab opened until the loadbar is not displayed anymore.

Monitor the pushes and check that the Gecko Decision Task opt job is running. If it fails, retrigger it, if it fails again the tree should be closed and the issue escalated.

Merging back from mozilla-central to autoland and mozilla-inbound

Steps on how to do the mozilla-central to autoland AND mozilla-central to inbound merge:

1. Close inbound and autoland from Treestatus
2. In the console switch to Mozilla-unified directory: cd mozilla-unified
3. Download the repos: hg pull fxtrees
4. Update autoland: hg update autoland
5. Merge from central to autoland: hg merge central
6. Add a commit message: hg commit -m "Merge mozilla-central to autoland. a=merge CLOSED TREE"
7. Transfer the files to Autoland: hg push -r . autoland
8. Update inbound: hg update inbound
9. Merge from central to inbound: hg merge central
10. Add a commit messagel: hg commit -m "Merge mozilla-central to mozilla-inbound. a=merge CLOSED TREE"
11. Transfer the files to inbound: hg push -r . inbound

These commands *might* result in race conditions on the tree when it is busy. Under those circumstances, this commands avoid delays between steps:

cd mozilla-unified
hg pull && hg update inbound && hg merge central && hg commit -m "Merge mozilla-central to mozilla-inbound" && hg push -r . inbound

Note: For correcting a commit message, use hg commit --amend

When there is nothing to merge

You run hg update to get a working copy information, but there are no changes on mozilla-central and so nothing to merge. The workflow for this case is:

1. Run hg update
   • You can run hg out to list all the changesets to merged to mozilla-central.
2. Because there's been no merge, you won't have a commit containing a approval message, e.g. "merge a to b a=me." This means you will run afoul of the hg hook telling you that the m-c repo is set to approval-only. You need to temporarily set the mozilla-central tree to "open" via Treestatus to be able to push.
3. Push your changes from Step 1.
4. Set the m-c tree back to "approval-required" in Treestatus. This is important because if we miss this, we risk unexpected pushes to mozilla-central since some people might think the open tree is intentional.
5. Use Bugherder to mark and failures as usual for your push. If all bugs failed to get marked, entered Bugzilla API might be wrong. Close the tab and try with a new one. If only some failed, open Bugherder for that push again and append &resume=1 to the url.
6. Now when you merge from mozilla-inbound you get the note that you need to merge.

Issues

Push rejected: DOM peer review needed

(fix still needs to be verified in production)

If the server rejects the push with the merge and complains that a changeset needs review by a DOM peer, you can disable this check. The issue should only affect merges with changesets which initially landed on autoland. Temporarily add this to the hgrc file in the .hg folder of the repository:

mozilla.check.webidl_check = false

How to run Bugherder if the tool is not working

1. Go to https://bugherder.mozilla.org/
2. Save the whole page (menu File > Save Page As > complete page)
3. Save it as bugherder.htm
4. In the folder "bugherder_files", you will find the file: bz-0.js. Open that with an editor
5. Go to line 265 and change the url to https://bugzilla.mozilla.org/latest/configuration
6. Save the file
7. Open bugherder.htm in the browser
8. Type in the revision number from central (first autoland revision and then mozilla-inbound revision), hit submit
9. Check to see if everything is ok and submit the changes