Mozilla.com/Workflow-improvements

This is an article about critiquing mozilla.com's workflow and improving it.

Overview

Work on mozilla.com is done by a team of 3-5 people, with additional localization changes from contributors. There is an even ratio between high frequency small changes and larger less frequent additions.

Everything is managed through SVN, with three different branches: trunk, stage, and production. All committers commit everything to trunk (many commits per day). QA tests bugs when they are finished off the trunk branch, and when verified, the committer selects the changes with SVN and merges it onto the stage branch, and pushes the new revision from stage to production. QA tests it on production, and if fixed, the bug is verified fixed.

There is not a clear process for rolling back, besides using SVN to roll files back to a specified revision.

A closer look

The current process goes something like this:

Bugfix cycle:

       +------------------------------------------------------------+
       |                                                            |
       v                                                            |        
* Developer -> fixes bug -> commits to trunk -> marks qawanted      |        
* QA -> tests on trunk-based dev site                               |        
     -> fix good? -> mark as qa-trunk-resolved                      |        
     -> fix bad? -> remove qawanted, reopen bug  -------------------+        
                                                                             

Rollout cycle:

* Developer or QA -> mark bug as push-needed                                 
* Developer -> pushes to staging branch -> pushes to production branch -> remove push-needed
* QA -> test on live site                                                    
     -> fix good? -> mark as verified fixed                                  
     -> fix bad? -> remove push-needed -> reopen bug -> start bugfix cycle

The bugfix cycle looks pretty standard. The rollout cycle is a little repetitive, but that's mainly because the staging branch isn't currently being used. This cycle gets laborious when there are many bugs. Also note how there isn't a step to rollback a bad bug on the live site, but that's as simple as reverting the files in SVN on the production branch.

Improvements

• staging needs to be authoritative. Developers should be responsible for merging in changes to staging, and QA only uses staging to test. Rolling out to production is then just one command.

• Reduce the number of ad-hoc rollouts, and use rolling release cycles. This reduces the man-hours spent rolling out code, makes it trivial to robustly push out changes (simply mirror production onto staging).

With these changes, we have a typical SVN workflow.

Other concerns

• All developers committing to trunk. I know this is typical for SVN, but it really does make it more confusing than it should be. Developers should be encouraged to commit frequently, but with a global trunk, changes might get clobbered, pulled down, moved around, etc. by accident. It muddles the central repository with possibly very experimental code.
  • There are a few bad situations with this. Here's one:

Bob and Tim have updated their repository to the latest revision, r100.

Bob is working on feature A in index.html       Tim is working on feature B in index.html
              |                                                 |
              v                                                 v
      commits partial fix                                 commits bug fix
            r101                                               r102
                                                                |
                                                                v
              +---------------------------------------pushes out r102 to production
              |
              v
   r101 goes live accidentally

• • Possible easy fix: Each developer works on his own SVN branch. This probably wouldn't work though because SVN doesn't track revisions across branches and it would make tracking code extremely confusing.

• SVN's handling of branches. Revision numbers aren't kept in sync at all; each branch has completely separate revisions (merge tracking is coming it seems, but it's a big hack with bad edge cases right now).
  • Pushing code between branches frequently merges several revisions into one with completely separate numbers.
  • The user must manually track the movement of code by including the origin branch and revision in the new commit (every single time).
  • In general, the user has to perform a lot of careful tracking of revision numbers for bookkeeping. This is prone to error.
  • Hard fix: Switch to git

• SVN revisions. Another gripe I have with revisions is that you have to know what revision you are working with to do anything. Because SVN doesn't have lightweight branches, a lot of work is done on one one branch, and any merging/reverting/etc. should be carefully done with the revision you desire. This is another reason why careful tracking of revision numbers with bug fixes and branch merges is absolutely necessary.
  • Hard fix: Switch to git

Our current solution, with the 2 improvements listed at the top, would work ok. But it would require a lot of manual labor, and that's prone to error. Also, I'm lazy.

Dramatic Proposal: Git

Git solves a lot of this, and makes it easier to sleep at night (ok, maybe not). We will reap the benefits for years if we switch to git now. It will be easy to move code around and rationalize about what's going on, and there a lot of cool tools to show you branching/merging statuses, history, etc.

We will also be more in sync with the other teams at Mozilla.

This is a rough proposal of what our system could look like with git:

Bugfix cycle:

       +---------------------------------------------------------------+
       |                                                               |
       v                                                               |        
* Developer -> create branch for bug -> fixes bug -> publishes branch  |
* QA -> tests on site running from bugfix branch                       |        
     -> fix good? -> mark as qa-trunk-resolved                         |        
     -> fix bad? -> remove qawanted, reopen bug  ----------------------+        

Rollout cycle:

* Developer or QA -> mark bug as push-needed                                 
* Developer -> updates bugfix branches -> merges specific bugfix to master -> publish master
* QA -> test on live site                                                    
     -> fix good? -> mark as verified fixed                                  
     -> fix bad? -> remove push-needed -> reopen bug -> reset master -> start bugfix cycle

The key here is that each of those actions to create, merge, publish, etc. a branch are all literally one git command in the form of "git merge/push/fetch/etc. origin <branch>".

Benefits

• Git is decentralized, which mean you make commits locally, no dependance on a centralized server. You just publish commits when you are ready to the central server.

• Branching is so painless that we can create a separate branch for every single bugfix. This isolates the work and makes the context of each bug very clear.

• No more having to specify revision numbers, commit hashes, magic pony locations, etc. It's all done for you.

• Since it's all done for you, git keeps a very good history of what went on. This lets us generate cool graphs and statistics about the codebase to see what's going on and helps us to improve our process more and more.

Releases & one-offs

A big problem with mozilla.com is that there is a high number of small changes which always need to go out ASAP, as well as larger changes that trickle in over time. We want to start doing rolling releases ever 2 weeks, with the occasional one-off.

The above process describes what would happen with a one-off, so a one-off changes are easily supported. For a release cycle, the developer would merge his/her bugfix branch into master but not push it out. When time comes to publish a release, all it takes is publishing the master branch (an easy git command).

Localizers

One problem is that localization contributors submit patches through SVN, and it wouldn't be good to tell them to use git. However, git-svn lets us pull in commits from SVN where localizations are coming through.

Other notes

The above process is just an idea, there are many ways we could use git. We could publish commits on the master branch in the centralized server, but then we need a separate process to support one-offs because you can't publish only part of a branch.

git-svn is a great tool which comes with git. It's a bridge between git and SVN that works very well; it actually converts git commits to SVN revisions and back. I've been using it to work with the mozilla.com codebase. We can use it to convert the whole SVN repository into git.

The biggest obstacle will be the learning curve for people who don't know git and making the transition easy, if we did this. I think we'd gain a lot of benefits though.