Sheriffing/How To/Unified Repos: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Clean up rebase instructions)
(add internal merge tool setup and xcode install as requirements for macOS)
 
(46 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== Unified Repos ==
{{Sheriffing How To|Unified repos}}
I(KWierso)'ll clean this up as I go. Rough IRC logs for now.
= Unified Repos =
 
=== Setting up the repo ===
* http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/unifiedrepo.html
* http://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/unifiedrepo.html
* https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html
* https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/firefoxtree.html


Here's how I set it up:
== Setting up the repo ==
* Install Mercurial 3.2 or higher
* Set up your ssh key yourself
* Install [https://www.mercurial-scm.org/downloads Mercurial 4.2] or higher
* Make a copy of your .hgrc file for safekeeping and delete the original
* Make a copy of your .hgrc file for safekeeping and delete the original
* Use default hg tool to resolve code conflicts:
** Open .hgrc file in home folder.
** Search for line <code>[ui]</code>.
** Search for line starting with <code>merge =</code> between the line above and the next one starting with <code>[</code>.
** Add or update it to match <code>merge = internal:merge3</code>
* hg clone https://hg.mozilla.org/mozilla-central
* hg clone https://hg.mozilla.org/mozilla-central
* hg clone https://hg.mozilla.org/hgcustom/version-control-tools
* cd mozilla-central
* cd mozilla-central
* ./mach mercurial-setup
* ./mach mercurial-setup
** Configure mercurial as desired (do not enable the mq extension)
* On macOS
* Edit your global .hgrc file to add the following to the [extensions] section:
** Install xcode which is be required for the execution of some commands: <code>xcode-select --install</code><br>This will open an install dialog which might be minimized. Check the list of open applications and confirm the installation.
** firefoxtree = /path/to/version-control-tools/hgext/firefoxtree
 
* The following will pull all (or at least most) of the branches sheriffs need to have on hand:
Configure mercurial as desired. At a minimum, you need the following:
** hg pull inbound && hg pull b2ginbound && hg pull fx-team && hg pull aurora && hg pull beta && hg pull esr31 && hg pull b2g34 && hg pull b2g32 && hg pull b2g30
* Set your username and email.
** If you need other branches, you can find their names in the [https://hg.mozilla.org/hgcustom/version-control-tools/file/default/pylib/mozautomation/mozautomation/repository.py firefoxtree extension source].
* Enable the firefoxtree extension.
* Enable the histedit extension.
 
The following will pull all (or at least most) of the branches sheriffs need to have on hand:
* hg pull autoland && hg pull beta && hg pull release && hg pull esr52
If you need other branches, you can find their names in the [https://hg.mozilla.org/hgcustom/version-control-tools/file/default/pylib/mozautomation/mozautomation/repository.py firefoxtree extension source].
 
You should now have everything set up for proper use! <br />
 
If you want, you can rename the mozilla-central folder to something like "unified", since it is more than just mozilla-central at this point.


You should now have everything set up for proper use!
The `hg fxheads` command should list references to each of the relevant branch names, and the current revision and commit message for each.
The `hg fxheads` command should list references to each of the relevant branch names, and the current revision and commit message for each.


=== Merges ===
To pull in newly landed changes, you can use one of the following commands:
This assumes you have your unified repo all set up so that `hg fxheads` lists each of central, b2ginbound, fx-team, and inbound.
* hg pull fxtrees # this will pull from all branches listed with the fxheads command
* hg pull treename # this will pull only from the tree with the name "treename"
 
== Merges ==
This assumes you have your unified repo all set up so that `hg fxheads` lists each of central and autoland.


To merge a specific non-tip <revision> from mozilla-inbound to mozilla-central:
To merge a specific non-tip <revision> from autoland to mozilla-central:
# hg pull central
# hg pull central
# hg pull inbound
# hg up central
# hg up central
# hg merge -r <revision>
# hg merge -r <revision>
# hg commit -m "Merge mozilla-inbound to mozilla-central a=merge"
# hg commit -m "Merge autoland to mozilla-central a=merge"
# hg push -r . central
# hg push -r . central


To merge mozilla-central into mozilla-inbound:
To merge mozilla-central into autoland:
# hg pull central
# hg pull central
# hg pull inbound
# hg pull autoland
# hg up inbound
# hg up autoand
# hg merge central
# hg merge central
# hg commit -m "Merge mozilla-central to mozilla-inbound a=merge"
# hg commit -m "Merge mozilla-central to autoland. CLOSED TREE"
# hg push -r . inbound
# hg push -r . autoland
 
* If the `hg merge central` command results in "abort: nothing to merge", you should instead use `hg update -r <mozilla-central's tip revision>` to do a non-merging update, then you can do `hg push -r . autoland` to push it as usual.
 
== Uplifts ==
If branch-specific patches are posted, follow import the patches onto the relevant branch.
 
If you're uplifting directly from an mozilla-central checkin to mozilla-beta:
 
Pull and update to prepare for the uplifts:
* hg pull central
* hg pull aurora
* hg up aurora
Graft the mc commit and bring up the editor to edit the commit message, adding "a=foo" as needed:
* hg graft --edit -r <revision>
Repeat the previous step as needed for all uplifts as needed.
Verify the outgoing changes and push:
* hg out -r . aurora
* hg push -r . aurora
 
You can graft a range of commits at once if that's easier:
* hg graft -r <toprevision>::<bottomrevision>
 
Read the documentation for [https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/unifiedrepo.html#uplifting-backporting-commits graft] for further help.
 
== Backouts ==
Better tools are coming:
* https://bugzilla.mozilla.org/show_bug.cgi?id=1117632
* https://bugzilla.mozilla.org/show_bug.cgi?id=1121211
<br />
 
The standard backout command makes backing out revisions more difficult than it should, as it won't pre-fill the backout commit message with a bug number:
* Copy backout <revision> from treeherder, note the bug <number>, and remember the <reason>
* hg backout -r <revision>
** This should open up your editor with a completely empty commit message. Type in the backout message like "Backout revision <revision> (bug <number>) for <reason>" (possibly with "CLOSED TREE" in there to get around a closure hook).
* hg push -r . <tree>
 
[https://hg.mozilla.org/hgcustom/version-control-tools/file/4e7ab9638cf0/hgext/qbackout qbackout] can still be used, though you will want to use `hg oops` in place of `hg qbackout` (eg if you use `hg share` to get multiple working directories, qbackout and other uses of mq do not play nice with shared repos):
* Copy the backout <revision> from treeherder
* hg oops -e -r <revision>
** This should open up your editor with a prepopulated commit message like "Backed out <revision> (bug <number>)". Add in the <reason> and possibly "CLOSED TREE" to get around a closure hook.
* hg out -r . <tree>
** to double-check what you will be pushing
* hg push -r . <tree>
 
qbackout can also back out a range of commits in a single backout commit:
* hg oops -e -s -r <toprevision>:<bottomrevision>
** This will open your editor with a prepopulated commit message like "Backed out <toprevision>,<anymiddlerevisions>,<bottomrevision> (bug <number>,any other bug <numbers>)". Add in the <reason> and possibly "CLOSED TREE" to get around a closure hook.
* hg push -r . <tree>
(Omit oops's -s flag to back out each individual revision in the range as a separate commit.)
 
=== Revert a backout ===
* `hg graft`, see https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmozilla/common.html#revive-a-commit-that-was-backed-out
 
== Rebasing after losing a push race ==
This assumes you have commits you're attempting to push to autoland when you lose a push race with someone. Change `autoland` to the correct branch name as needed.
 
* hg pull autoland
* hg rebase -d autoland
* hg push -r . autoland
 
== Recovering from mistakes ==
The histedit subcommand of hg lets you remove, edit and reorder changesets (in addition to other actions):
* hg histedit
If a merge commit needs to be removed:
* hg strip --no-backup -r .
<big>WARNING:</big> The hg strip command below removes any commits that haven't been pushed to the remote repositories, regardless of what branch/label/tag those commits are on. (So if you have local changes as patches for other issues - even based on other trees - they will be removed.) Only do this if you don't care about any of those commits!
* hg strip --no-backup 'not public()'
 
 
You may also need to use some combination of the following to get things back to a known-good state:
* hg purge # http://mercurial.selenic.com/wiki/PurgeExtension
* hg up -C .
 
== See what you're about to push ==
`hg out` without any flags added will show you a LOT of unrelated commits. To see just what you'll be pushing to a given branch, use `hg out -r . <branch>`
* hg out -r . autoland
** This will only display the things you're about to push to the autoland branch


* If the `hg merge central` command results in "abort: nothing to merge", you should instead use `hg update -r <mozilla-central's tip revision>` to do a non-merging update, then you can do `hg push -r . inbound` to push it as usual.
== View incoming changes ==
Since your unified repo will likely have all changesets from the various branches pulled locally, `hg in <somebranch> -r <somerev>` will likely not help you see what would be pulled into your current branch from <somebranch> up to that <somerev>.
Instead, you can use the following to print out the equivalent. Say you want to see what would be merged onto mozilla-central from autoland's revision ca142ec8ba0f:


=== Checkin-neededs ===
Make sure m-c is fully up to date:
=== Uplifts ===
* hg pull central
* hg up central
Get all of autoland's changesets:
* hg pull autoland
Do a preview of the merge without actually performing the merge:
* hg merge -r ca142ec8ba0f -P
This will print out everything on autoland (up to and including revision ca142ec8ba0f) that is not already on mozilla-central.


=== Rebasing after losing a push race ===
== See the log for a particular branch ==
This assumes you have commits you're attempting to push to mozilla-inbound when you lose a push race with someone. Change `inbound` to the correct branch name as needed.
* hg log -fr [branch]


* hg pull inbound
= Extras =
* hg rebase -d inbound
== hg aliases ==
* hg push -r . inbound
=== mergetocentral/mergetointegration ===
I set up the following aliases in my global .hgrc file to make merging things around trunk a little faster:
[alias]
# merge tree $1, revision $2 to central, optionally appending $3 (for "CLOSED TREE", etc)
mergetocentral = !$HG pull central ; $HG up central ; $HG pull $1 ; $HG merge $2 ; $HG commit -m "Merge $1 to central, a=merge $3" ; $HG push -r . central
# merge central tip to tree $1, optionally appending $2 (for "CLOSED TREE", etc)
mergetointegration = !$HG pull central ; $HG pull $1 ; $HG up $1 ; $HG merge central ; $HG commit -m "Merge m-c to $1, a=merge $2" ; $HG push -r . $1


=== Recovering from mistakes ===
With these set up, I can merge autoland's revision 04a3d9130aa0 over to mozilla-central with the following command:
* [20:55] KWierso gps: actually, one last question :)
hg mergetocentral autoland 04a3d9130aa0
* [20:56] KWierso at the moment with all of the separate repositories, cleaning up stuff that's not right is some combination of | hg pull -u | and | hg strip | to get rid of non-pushed local changes and bring in newly-pushed things
and then I can merge mozilla-central's tip back to autoland with the following command:
* [20:57] KWierso how would I do that in a unified repo?
hg mergetointegration autoland
* [21:05] gps KWierso: generally speaking, if you have to "clean up stuff that's not right," that's probably a sign you are doing something wonky
This cuts down on a lot of repetitive typing, saving a bit of time.
* [21:06] gps i rarely find myself running `hg strip`
* [21:06] gps although that's partially because i use evolve
* [21:07] gps running `hg strip` in a shared repo is somewhat dangerous. if another working copy is using changesets that were stripped, that could corrupt the working copy
* [21:07] gps if you have old screw-ups, it's best to leave them be
* [21:07] gps if you have evolve, you can `hg prune <rev>` to mark them hidden
* [21:07] gps they kinda just vanish, without the bad consequences
* [21:10] gps as long as you don't have a shared repo referencing commits that you strip, you should be OK
* [21:10] gps e.g. if you `hg pull` and then want to `hg strip` what you just pulled, that's fine
* [21:10] gps but if you go back in history and start removing random things, watch out!
* [21:11] KWierso gps: I probably should have put the whole command, not just "hg strip"
* [21:11] KWierso | hg strip --no-backup 'roots(outgoing())' |
* [21:12] gps well, if you have a unified repo, outgoing() will report changesets from all repos, and that will almost certainly strip a lot of commits!
* [21:12] gps depending on what you've pulled, of course
* [21:13] gps if you have aurora and central in the same repo and out outgoing against central, all aurora commits will be in that set
* [21:13] KWierso ick
* [21:14] gps |not public()| may be a better selector
* [21:14] gps but that assumes you don't have any local commits that haven't landed yet
* [21:15] KWierso most of the time for sheriffing-related stuff, it's not our own commits, but stuff we're checking in for others or uplifting
* [21:16] KWierso so it's stuff others already put up in patches on bugzilla or landed on m-c and need uplifted to aurora/beta/etc
* [21:16] KWierso so losing local changes isn't devastating
* [21:16] gps in that case, you can probably repopulate it easily enough, so |not public()| might be better
* [21:17] KWierso so | hg strip --no-backup 'roots(not public())' | ?
* [21:17] KWierso or hg strip --no-backup 'not public()' ?
* [21:20] gps the latter.
* [21:20] gps you know --no-backup is somewhat dangerous, righ?
* [21:22] KWierso yep
* [21:22] KWierso if the hg strip/hg pull combo doesn't get me back to a known-good state, I'm more than willing to delete and reclone that repository
* [21:23] KWierso less willing now if I'm using a unified repo...
* [21:24] gps generally speaking, you should *never* have to reclone
* [21:25] gps if you have extra local commits, just deal with them
* [21:25] gps not doing so ignores the realities of how distributed version control works

Latest revision as of 15:49, 12 July 2022

Unified Repos

Setting up the repo

  • Set up your ssh key yourself
  • Install Mercurial 4.2 or higher
  • Make a copy of your .hgrc file for safekeeping and delete the original
  • Use default hg tool to resolve code conflicts:
    • Open .hgrc file in home folder.
    • Search for line [ui].
    • Search for line starting with merge = between the line above and the next one starting with [.
    • Add or update it to match merge = internal:merge3
  • hg clone https://hg.mozilla.org/mozilla-central
  • cd mozilla-central
  • ./mach mercurial-setup
  • On macOS
    • Install xcode which is be required for the execution of some commands: xcode-select --install
      This will open an install dialog which might be minimized. Check the list of open applications and confirm the installation.

Configure mercurial as desired. At a minimum, you need the following:

  • Set your username and email.
  • Enable the firefoxtree extension.
  • Enable the histedit extension.

The following will pull all (or at least most) of the branches sheriffs need to have on hand:

  • hg pull autoland && hg pull beta && hg pull release && hg pull esr52

If you need other branches, you can find their names in the firefoxtree extension source.

You should now have everything set up for proper use!

If you want, you can rename the mozilla-central folder to something like "unified", since it is more than just mozilla-central at this point.

The `hg fxheads` command should list references to each of the relevant branch names, and the current revision and commit message for each.

To pull in newly landed changes, you can use one of the following commands:

  • hg pull fxtrees # this will pull from all branches listed with the fxheads command
  • hg pull treename # this will pull only from the tree with the name "treename"

Merges

This assumes you have your unified repo all set up so that `hg fxheads` lists each of central and autoland.

To merge a specific non-tip <revision> from autoland to mozilla-central:

  1. hg pull central
  2. hg up central
  3. hg merge -r <revision>
  4. hg commit -m "Merge autoland to mozilla-central a=merge"
  5. hg push -r . central

To merge mozilla-central into autoland:

  1. hg pull central
  2. hg pull autoland
  3. hg up autoand
  4. hg merge central
  5. hg commit -m "Merge mozilla-central to autoland. CLOSED TREE"
  6. hg push -r . autoland
  • If the `hg merge central` command results in "abort: nothing to merge", you should instead use `hg update -r <mozilla-central's tip revision>` to do a non-merging update, then you can do `hg push -r . autoland` to push it as usual.

Uplifts

If branch-specific patches are posted, follow import the patches onto the relevant branch.

If you're uplifting directly from an mozilla-central checkin to mozilla-beta:

Pull and update to prepare for the uplifts:

  • hg pull central
  • hg pull aurora
  • hg up aurora

Graft the mc commit and bring up the editor to edit the commit message, adding "a=foo" as needed:

  • hg graft --edit -r <revision>

Repeat the previous step as needed for all uplifts as needed. Verify the outgoing changes and push:

  • hg out -r . aurora
  • hg push -r . aurora

You can graft a range of commits at once if that's easier:

  • hg graft -r <toprevision>::<bottomrevision>

Read the documentation for graft for further help.

Backouts

Better tools are coming:


The standard backout command makes backing out revisions more difficult than it should, as it won't pre-fill the backout commit message with a bug number:

  • Copy backout <revision> from treeherder, note the bug <number>, and remember the <reason>
  • hg backout -r <revision>
    • This should open up your editor with a completely empty commit message. Type in the backout message like "Backout revision <revision> (bug <number>) for <reason>" (possibly with "CLOSED TREE" in there to get around a closure hook).
  • hg push -r . <tree>

qbackout can still be used, though you will want to use `hg oops` in place of `hg qbackout` (eg if you use `hg share` to get multiple working directories, qbackout and other uses of mq do not play nice with shared repos):

  • Copy the backout <revision> from treeherder
  • hg oops -e -r <revision>
    • This should open up your editor with a prepopulated commit message like "Backed out <revision> (bug <number>)". Add in the <reason> and possibly "CLOSED TREE" to get around a closure hook.
  • hg out -r . <tree>
    • to double-check what you will be pushing
  • hg push -r . <tree>

qbackout can also back out a range of commits in a single backout commit:

  • hg oops -e -s -r <toprevision>:<bottomrevision>
    • This will open your editor with a prepopulated commit message like "Backed out <toprevision>,<anymiddlerevisions>,<bottomrevision> (bug <number>,any other bug <numbers>)". Add in the <reason> and possibly "CLOSED TREE" to get around a closure hook.
  • hg push -r . <tree>

(Omit oops's -s flag to back out each individual revision in the range as a separate commit.)

Revert a backout

Rebasing after losing a push race

This assumes you have commits you're attempting to push to autoland when you lose a push race with someone. Change `autoland` to the correct branch name as needed.

  • hg pull autoland
  • hg rebase -d autoland
  • hg push -r . autoland

Recovering from mistakes

The histedit subcommand of hg lets you remove, edit and reorder changesets (in addition to other actions):

  • hg histedit

If a merge commit needs to be removed:

  • hg strip --no-backup -r .

WARNING: The hg strip command below removes any commits that haven't been pushed to the remote repositories, regardless of what branch/label/tag those commits are on. (So if you have local changes as patches for other issues - even based on other trees - they will be removed.) Only do this if you don't care about any of those commits!

  • hg strip --no-backup 'not public()'


You may also need to use some combination of the following to get things back to a known-good state:

See what you're about to push

`hg out` without any flags added will show you a LOT of unrelated commits. To see just what you'll be pushing to a given branch, use `hg out -r . <branch>`

  • hg out -r . autoland
    • This will only display the things you're about to push to the autoland branch

View incoming changes

Since your unified repo will likely have all changesets from the various branches pulled locally, `hg in <somebranch> -r <somerev>` will likely not help you see what would be pulled into your current branch from <somebranch> up to that <somerev>. Instead, you can use the following to print out the equivalent. Say you want to see what would be merged onto mozilla-central from autoland's revision ca142ec8ba0f:

Make sure m-c is fully up to date:

  • hg pull central
  • hg up central

Get all of autoland's changesets:

  • hg pull autoland

Do a preview of the merge without actually performing the merge:

  • hg merge -r ca142ec8ba0f -P

This will print out everything on autoland (up to and including revision ca142ec8ba0f) that is not already on mozilla-central.

See the log for a particular branch

  • hg log -fr [branch]

Extras

hg aliases

mergetocentral/mergetointegration

I set up the following aliases in my global .hgrc file to make merging things around trunk a little faster:

[alias]
# merge tree $1, revision $2 to central, optionally appending $3 (for "CLOSED TREE", etc)
mergetocentral = !$HG pull central ; $HG up central ; $HG pull $1 ; $HG merge $2 ; $HG commit -m "Merge $1 to central, a=merge $3" ; $HG push -r . central
# merge central tip to tree $1, optionally appending $2 (for "CLOSED TREE", etc)
mergetointegration = !$HG pull central ; $HG pull $1 ; $HG up $1 ; $HG merge central ; $HG commit -m "Merge m-c to $1, a=merge $2" ; $HG push -r . $1

With these set up, I can merge autoland's revision 04a3d9130aa0 over to mozilla-central with the following command:

hg mergetocentral autoland 04a3d9130aa0

and then I can merge mozilla-central's tip back to autoland with the following command:

hg mergetointegration autoland

This cuts down on a lot of repetitive typing, saving a bit of time.