ReleaseEngineering/How To/VCSSync
This is documentation on the mozharness-based vcs sync processes.
See Hal's docs for the legacy process.
Quick notes
Machines
2013.10.14: We're currently using the following machines:
| Machine | Process | Location |
| vcssync1.srv.releng.usw2.mozilla.com | beagle (gecko-dev) | /opt/vcs2vcs |
| vcssync2.srv.releng.usw2.mozilla.com | project-branches (gecko-projects) | /opt/vcs2vcs |
| github-sync2.dmz.scl3.mozilla.com | staging (initial conversion, gecko-git, l10n) | /home/asasaki/{initial2,gecko,l10n} |
| github-sync4.dmz.scl3.mozilla.com | staging (project branches) | /home/asasaki/projects |
Plus a few other github-sync machines for Hal's legacy processes.
Branch
2013.10.14: We're currently based off Aki's vcs_multi_repo branch off github. Once all of bug 847727 lands, we'll go to the mozharness production branch.
Specific instructions
Emails
Maintenance
How to add a repo to gecko-dev or gecko-git
How to add a project to project-branch conversion
If the project is under hg.m.o/projects, add it here.
If the project isn't under hg.m.o/projects, add it here.
How to add a locale to l10n conversion
L10n conversions are based on this config file.
These point at gecko all-locales and gaia languages_dev.json files. To add a locale, update the appropriate all-locales or languages_dev.json file.
If the b2g versions are changing against gecko trains (as they will every 6 weeks), you need to update this section.
How to adjust email notifications
The notify_config's are in each config file, like this.
This is a list-of-dictionaries. Each dictionary has a "to" email address.
If failure_only is set to True, no emails with a successful outcome will be sent to this address.
If skip_empty_messages is set to True, no emails without a message body will be sent (these should only be successful noop runs with no warnings).
Allowing both of these to be False will result in a lot of email, approximately one per minute. Just setting skip_empty_messages to True will send email per successful push, any sort of warning, or failures.
How to force the process to pull/bookmark/convert/push a repo, even if nothing's changed
There is a --no-check-incoming commandline option to the vcs_sync.py script.
This can also be set in the config file, as check_incoming set to True.
This can also be set in the config file per-repository, as check_incoming inside each of the specific conversion_repos where we want to skip the incoming check.
How to start a mozilla-central based repo conversion from scratch
- First, you need a copy of initial3.tar.bz2. This is mozilla-central-with-cvs-history, already converted to git for you, up to Spring, 2013. Without this file, you will need to run the initial conversion with this config file (or beagle) and let it run for about a week.
- The sha512 sum is
0a3243fe5a6c8ffa4e47131e0eb0243e1f5676ea3cacd535d11b424f8f601511130a2e941670950b63c0b00726dafa9bd30bf7c3d040752fce824158021ef014 - This file exists on vcssync{1,2} at /opt/vcs2vcs/initial3.tar.bz2 , on github-sync{2,4} at /home/asasaki/initial3.tar.bz2 , and on Aki's laptop.
- The sha512 sum is
- Second, you need a config file for the conversion type. This is probably covered with the beagle, gecko-git, and project-branches config files here.
Troubleshooting
How to deal with non-ffwd
- Determine what happened. This can generally be found with
hg heads
. If there are multiple heads for a branch, you'll need to follow the below instructions. - Determine if you want to discard one of the heads, or merge them. (Do you want the changes from both heads, or do you want one head's changes to stick and discard the other one?)
- If you want to merge the two heads, you can do so with
hg merge,hg commit, andhg push - If we want to discard one,
- If you want to merge the two heads, you can do so with
# we're on good revision X # we have old [bad] revision Y, which is a different head hg debugsetparents X Y; hg commit -m "merging X and Y via debugsetparents" # Create a new changeset that has parents X and Y hg glog | more # see that the history has merged hg diff -r X # see that the code is the same as revision X, no diffs hg push
If you hit a relbranch issue you may need to follow these instructions instead.
How to deal with project branch reset
To some degree, the integration-gecko-projects repo should be self-healing. If the incoming changesets to a repo are incompatible (so an hg pull doesn't work), the script should blow away the source repo and reclone. The integration-gecko-projects target has force_push set to True, so it should be pushing with git push -f.
However, project branch resets may become an issue; time will tell.
We can probably:
- pause the project branches job (vcs2vcs@vcssync2 cron),
- delete the branch on github https://github.com/mozilla/integration-gecko-projects/branches
- delete the branch on disk (vcs2vcs@vcssync2:/opt/vcs2vcs/build/stage_source/BRANCH)
- restart the job
and it'll clone a new copy of the branch, convert, and push.
If there are still issues, we can debug further to try to get it to work. A final nuke-the-site-from-orbit solution is documented here, but may take the repo down for a day or so.
How to deal with completely resetting gecko-projects
- Make the decision that this is the solution to the problem.
- Make sure people are aware this is going to happen.
- Stop the conversion of gecko-projects. This is currently on vcs2vcs@vcssync2, and is run via cron.
- Save a copy of https://raw.github.com/mozilla/integration-gecko-projects/master/README.md
- Delete the repo! This option is available in the "Danger Zone" area of https://github.com/mozilla/integration-gecko-projects/settings
- Recreate the repo. There's a "Create a new repo" icon in the top right. This will need the same name, should be public, and you want to initialize the repo with a README. (gecko-projects is the only repo where we want the README: essentially, we want an empty master branch because github doesn't allow you to delete the first populated branch in a repo.)
- Edit the README with the contents you downloaded in step 4.
- Start the conversion back up. You may only need to delete the build/target and build/stage_source/BAD_REPO directories. Or you may need to blow away the whole thing from orbit to be sure, in which case you should follow the docs for how to start a mozilla-central based repo conversion from scratch.
- Note, just cloning the hg repos for the projects repo can take many many hours, so this will take the gecko-projects repo down for a long time.
How to deal with GECKO90_2011121217_RELBRANCH
We were getting email with the following failure:
11:14:01 ERROR - remote: error: denying non-fast-forward refs/heads/GECKO90_2011121217_RELBRANCH (you should pull first) 11:14:01 ERROR - ! [remote rejected] GECKO90_2011121217_RELBRANCH -> GECKO90_2011121217_RELBRANCH (non-fast-forward) 11:14:01 ERROR - error: failed to push some refs to '/opt/vcs2vcs/build/target/beagle/.git' 11:14:02 ERROR - Return code: 256 11:14:02 ERROR - mozilla-beta: Can't push /opt/vcs2vcs/build/conversion/beagle to /opt/vcs2vcs/build/target/beagle/.git! 11:14:02 ERROR - This was a test push that failed; not proceeding any further with mozilla-beta!
To debug:
# vcs2vcs@vcssync1 cd /opt/vcs2vcs/build/target/beagle git show GECKO90_2011121217_RELBRANCH # this gave 1003ec79451969335008880ad82e305d93b89642 cd /opt/vcs2vcs/build/conversion/beagle git show GECKO90_2011121217_RELBRANCH # this gave fac7279c040d643fb4c35105fa85b9335ba2c2f9 git merge-base fac7279c040d643fb4c35105fa85b9335ba2c2f9 1003ec79451969335008880ad82e305d93b89642
The merge-base returned fac7279c040d643fb4c35105fa85b9335ba2c2f9. That means that fac727 is a parent to 1003ec. No sha divergence, only stale history.
I noticed that http://hg.mozilla.org/releases/mozilla-beta/rev/GECKO90_2011121217_RELBRANCH pointed at a9221b332d8a (dec 15) and http://hg.mozilla.org/releases/mozilla-release/rev/GECKO90_2011121217_RELBRANCH pointed at 4e309e63c279 (jan 3); Calendar must have done a release off the relbranch off mozilla-release. Since the branch name is shared across mozilla-beta and mozilla-release, converting the branch from mozilla-beta effectively does a non-fastforward push and is rejected.
To fix, I exported the two revisions from mozilla-release to mozilla-beta:
cd mozilla-release hg export -r 79350 > ../cal2 hg export -r 79349 > ../cal1 # Doublecheck to make sure those look good cd ../mozilla-beta hg pull -u hg up -r GECKO90_2011121217_RELBRANCH hg import ../cal1 hg ident # sha matched m-r hg import ../cal2 hg ident # sha matched m-r hg out hg push