Bugzilla:Migrating to git

It has been proposed to move the Bugzilla source from bzr.mozilla.org to git.mozilla.org with a read-only mirror on GitHub.

Rationale

There's no single killer reason to move to git; however, there are a handful of benefits:

• git is pretty much the de facto standard for VCSs these days. This means many people are familiar with it and like its flexibility.

• On the other side, Bazaar is almost dead. There are maybe 2-3 commits to trunk every month. The time to fix bugs in Bazaar also seems to be quite long, generally.

• GitHub is extremely popular, and for good reason, since it's a very powerful system. Exposure via GitHub could mean increased contributions (whether we can keep up with them or not is, of course, another question).

• Mozilla would very much like to retire Tinderbox, since Bugzilla is the only system still using it. Moving to GitHub means we could use travis-ci, which is really nice.

• Mozilla would also very much like to retire the Bazaar server, which is used only for Bugzilla.

Limitations

After the migration, the git repo will be the repo of record, that is, while the bzr repo will be maintained, it will be read-only. Similarly, there will be a one-way mirror to GitHub. Users will be able to fork the project, but they will have to push to git.mozilla.org, which in turn will push the changes to GitHub.

Doing a one-time export using bzr fast-export and git fast-import is fairly straightforward. It just requires a bit of translation to preserve commit metadata, which git doesn't support. This is pretty much just the bug ID.

Keeping the two repos in sync, however, is a bit trickier. The bzr-git plugin looked like a good solution but in fact can't automatically pull recent changes from git and apply them to bzr. Launchpad can mirror projects from GitHub, but it is still a manual process and not at all suited for the number of branches Bugzilla uses.

Luckily there is already a script that is currently used to do the bzr-cvs one-way mirror. We can create a new script based on this to copy changes from git to bzr.

It was originally proposed to keep the bzr repo going for only another 9 months, but it was also suggested that we keep it running until all current versions (up to 4.4) are EOLed, since the suggested upgrade path is to pull changes directly from the repo, rather than applying a tarball or diff. However, we do publish tarballs and diffs, so this is about convenience in upgrades, since it is always possible to upgrade via tarball or diff.

Plan

There are a bunch of things we'll need to do to complete this. Some can be done in parallel.

The tracking bug for the migrations is bug 929685.

Pre-migration:

1. (DONE) fast-export translation script to preserve bug ID metadata.
2. (DONE) git-to-bzr.pl script to keep bzr in sync with git.
3. (DONE) Write migration instructions similar to https://wiki.mozilla.org/Bugzilla:Moving_From_CVS_To_Bazaar
4. (DONE, though see bug 967697) Set up repos (bugzilla and bmo) on git.mozilla.org.
5. (DONE, approx 1h15m) Determine which branches should be migrated over (presumably all, but there might be obsolete branches) and estimate how long this will take.
6. (DONE) Do a trial migration.
7. (DONE, 11 March 2014) Announce date for migration.

Migration:

1. Disallow commits to bzr aside from those originating from the git-to-bzr script.
2. Create git repos, restricting write access to just the account used for the migration.
3. Perform fast-import/-export migration on all (supported) branches of bugzilla and bmo.
4. Run git-to-bzr script once on all mirrored branches (see below) to initialize the .gitrev tracking file.
5. Start running git-to-bzr periodically (every 5 minutes should be good).
6. Commit .gitignore file based on .bzrignore to git (at least to supported/active repos/branches), and confirm that it has been mirrored.
7. Enable write access on all git repos for all authorized accounts.

Soon after:

1. Set up repos on GitHub and configure git.m.o repos to mirror there.
2. Set up a travis-ci instance with a test migration to GitHub. Note: someone out there has apparently done this for the unit tests (https://travis-ci.org/dnozay/bugzilla); we still need support for the Selenium tests in the qa branches.

Eventually:

1. Turn off tinderbox when we are sure that the test results from travis-ci and tinderbox match.
2. Turn off bzr (bug 967642).

Migration scripts are in a GitHub repo.

We will mirror the following bzr branches from git. There are three reasons to mirror: to support existing deployment scripts, to support existing bzr-based installations, and to have tinderbox continue to test commits until we switch to travis-ci.

For existing installations:

• bugzilla/4.0 (switch off after 5.0 is released, at which point we can also stop mirroring to cvs)
• bugzilla/4.2
• bugzilla/4.4

For tinderbox CI testing (switch off after travis-ci is set up for both packaged tests and qa tests) and deployment scripts:

• bugzilla/trunk
• bugzilla/qa/4.0
• bugzilla/qa/4.2
• bugzilla/qa/4.4
• bmo/4.2 (also needed until deployment scripts are updated, bug 973298)
• bmo/4.2-dev (also needed until deployment scripts are updated, bug 973298)
• bmo/qa/4.2

Mapping

Almost all the Bazaar branches on bzr.mozilla.org will be migrated to one or more branches in various repositories on git.mozilla.org.

All branches of the Bugzilla source will be migrated to individual branches in one git repo, /bugzilla/bugzilla.git. Similarly, the QA branches will be migrated to bugzilla/qa.git. Branches in extensions and misc will be migrated to repos in /bugzilla/extensions, keeping related branches together.

The BMO source will be migrated to /webtools/bmo/bugzilla.git, and the BMO qa repo as /webtools/bmo/qa.git.

/bzr-plugins and /securemail will not be migrated, since the former is unneeded in git, and the latter was a mistake.

Bazaar trunk branches are migrated to git as master, following convention. Note that not all Bazaar branches of the same project contain a trunk branch (e.g. bmo); git references may need to be updated to prevent warnings during cloning.

Full mapping follows:

/bugzilla/2.*, 3.*, 4.*, trunk, sightings

/bugzilla/bugzilla.git (as individual branches, with trunk -> master)

/bugzilla/qa/*

/bugzilla/qa.git

/bugzilla/extensions/*

/bugzilla/extensions/*.git

Full list of git extension repos:

• /bugzilla/extensions/Browse.git
• /bugzilla/extensions/browserid.git
• /bugzilla/extensions/DescribeUser.git
• /bugzilla/extensions/Developers.git
• /bugzilla/extensions/ExtraValues.git
• /bugzilla/extensions/GNOME.git
• /bugzilla/extensions/PatchReport.git
• /bugzilla/extensions/ProductInterests.git
• /bugzilla/extensions/profanivore.git
• /bugzilla/extensions/requestwhiner.git
• /bugzilla/extensions/sitemap.git
• /bugzilla/extensions/StockAnswers.git
• /bugzilla/extensions/sync.git
• /bugzilla/extensions/typesniffer.git
• /bugzilla/extensions/vcs.git
• /bugzilla/extensions/WeeklyBugSummary.git
• /bugzilla/extensions/cannedcomments.git
• /bugzilla/extensions/rest.git
• /bugzilla/extensions/trackingflags.git
• /bugzilla/extensions/InlineHistory.git
• /bugzilla/extensions/securemail.git
• /bugzilla/extensions/splinter.git
• /bugzilla/extensions/testopia.git

/bugzilla/misc/*

/bugzilla/misc/*.git

Full list of git misc repos:

• /bugzilla/misc/active-installs.git
• /bugzilla/misc/bugzilla-bugbot.git
• /bugzilla/misc/build.git
• /bugzilla/misc/landfill.git
• /bugzilla/misc/tinderbox-bugbot.git
• /bugzilla/misc/tinderbox-client.git

/bmo/3.*, 4.*

/webtools/bmo/bugzilla.git

/bmo/qa/*

/webtools/bmo/qa.git

Testing

Initial Migration

When the git repos are created, we can do a full migration to look for errors. The migration script (migrate.sh) calculates the md5sum for all files in the head of each branch in both bzr and the resulting git repository and compares them along with file metadata (as reported by 'ls -l'). After the trial migration we can examine the log for differences. Limitations: this only checks heads of branches and does not verify any commit metadata.

The first trial migration was completed on 2014/02/12. It was successful aside from a few files that were removed in bzr commits but present in the final git repository. It appears that at least some of these are a result of a directory being renamed and files in that directory being removed in the same commit; bzr records this (given old name "A" and new name "B") as "R A B" followed by "D A", meaning that the delete is recorded on the original name, not the new name. This results in the rename being successful but the delete being ignored. This is not critical, since they can be removed after the fact, but it might be worth adjusting the fastexport translation script to account for this.

Continuous Mirroring

Initial testing involves creating a new bzr branch from a current one (e.g. from bugzilla/trunk), migrating that over to git, then making various changes to git and running the git-to-bzr.pl script (either regularly on cron, or manually for more fine-grained testing). The resulting commits should have identical commit metadata and changes to file metadata and contents. We can manually inspect for the first two and use the same process as the initial migration for the latter.

Changes that should be tested:

• Commits from various authors.
• Creating a new file in an existing directory.
• Creating a new file in a new directory (git has no concept of empty directories).
• Modifying a file's contents.
• Modifying a file's metadata.
• Removing a file.
• Removing the only file in a directory (mirror script should automatically delete the empty directory to match git's behaviour).
• Moving/renaming a file.

We can leave the script running on the test branch for continued testing.