Bugzilla:Release Process

Bugzilla Releases

Security Releases

All security bugs to be triaged one business day by a Assistant Project Leader or Project Leader. Along with the Mozilla Security Team, it should be determined if the issue is valid, and the severity (the sec-low, sec-moderate and sec-high keywords is normally used for this).

A high security bug would include (but is not limited to) data leakage (including CSRF/XSS attacks) or when the exploit is already publicly available (as was the case with bug 1036213). This would also include non-security bugs that could cause data loss or corruption. In these case, a new point release would be done as soon as possible.

A medium security bug is a bug which could cause user harm indirectly or if some heighten level of permissions are required. These include bugs such as bug 1054702 and bug 873932. They should be fixed as soon as possible, but will be included with the next scheduled release.

Low security bugs are issues which are hard to reproduce and only occur under some circumstance. Examples are bug 761043 and bug 301686. These should be fixed as developer time permits, and will be included in the next scheduled release if completed.

Version Numbers

Major version increases (e.g. 5.0) reflect large, visible, and/or breaking changes. These would include features being removed, interfaces being changed, dependencies being updated, significant UI changes, major new features, etc.

Minor version increases (e.g. 5.2) reflect noncritical bugfixes, small new features, and other nonbreaking changes. These would include additions and/or backward-compatible changes to interfaces, support for new technologies or platforms, and general improvements and fixes. Note that official releases will always have an even-numbered minor version (e.g. 5.0, 5.2, 5.4); odd numbers are reserved for development releases. This applies only to the minor version number.

Patch version increases (e.g. 5.2.1) usually reflect critical fixes, such as to security or other severe bugs. While we generally wait until there is a critical fix to release a patch version, these versions may also contain other minor fixes committed in the meantime. We may also occasionally release a patch version without critical fixes if there is a sufficiently large backlog of minor fixes, or for development snapshots (of an odd-numbered minor version).

When to Release

In May 2014 we decided to try to release approximately every 6 months with whatever is ready at the time. This gets new features out the door more quickly. Branching should be done when master is relatively stable, but sometimes branching with known bugs is acceptable if it allows other large, potentially unstable features to land (for the next release). If there are bugs related to new features in the release branch, the feature should be backed out of the release branch (if it's very recent), or fixes should be committed to both the release branch and master. Bugs blocking release should be denoted with a blocking<version> flag, e.g. blocking5.0.

The version number will be determined at time of branching depending on the changes included, as per the section above. Note that this means we may have back-to-back major releases.

What to Release

As stated above, generally master will be branched approximately six months after the previous release, but there may be additional bugs to be fixed. In August 2014, at the public meeting and follow-up newsgroup discussion we decided that the Target Milestone should be used to denote things that should be fixed before release, i.e., blockers, or bugs being actively worked on. Release candidates should only be created after there are zero open bugs with the Target Milestone set to that release. Furthermore, only the bug's assignee or a Bugzilla reviewer/approver should set or unset the Target Milestone.

The existing policy of having approvers set or update the Target Milestone, if necessary, on bugs that are committed before an upcoming release continues unchanged.

Release Process Details

File Bugs

File bugs for the following things...

• "Release Bugzilla X.YY, Z.AA" and mark it as severity "blocker." It should block bug 286269, which has the alias bz-release. (Sample: bug 1042086)
• Security Advisory: It should block the first bug, and and Security bugs targeted at this release should be dependencies. (Sample: bug 1042093)
• Release Notes: Point releases and rc1 versions need release notes. Each version being released needs a separate Release Notes bug filed. These bugs should block the first bug. For final releases of major versions, you need a "clean up release notes" bug. (Sample: bug 1042088)
• New Features Page (Release Candidates and Final Releases Only): Make this bug depend on the appropriate Release Notes bug. (Sample: bug 286278)

Motivate People

Basically, just make it really easy to find and access all the bugs that need to be fixed for the release. I usually do this a few days after I file the above bugs.

This means:

• Do a search for blockers for the release, and make the query link into a tinyurl. When creating a tinyurl, make sure that no "order by" information is part of the URL so that you don't break people's LASTORDER cookies.
• Put that tinyurl at the very end of the topic line of the #mozwebtools IRC channel on irc.mozilla.org, with a brief explanation like "4.0.11 Blockers: tinyurl".
• Send out an email to developers@bugzilla.org explaining that we're releasing, and include the tinyurl. Having just one link in the email gets people to focus a bit more.
• If you are desperate: Send out an email to support-bugzilla with a tinyurl pointing to all bugs targeted at your release, sorted by Importance. Explain that you'd like some help, and link them also to the Contributor's Guide
• Do not include any sensitive security information in the email announcements yet.

Pre-Release

Basically, you just have to get all the bugs to have patches with review+ on them. A few of them require some special handling:

Security Advisory

Format

The actual "issues" in a security advisory normally look something like this:

  Class:       Cross-site scripting
  Versions:    2.15 through 2.18rc3 and 2.19.1 (from cvs)
  Description: It is possible to blah blah blah
               And here are some details on how it can be made less bad...
  Reference:   https://bugzilla.mozilla.org/show_bug.cgi?id=272620
  CVE Name:    CVE-2014-1061

The different values for Class can be found in older Security Advisories. The "CVE Name" is, obviously, not always needed.

Determining Affected Versions

It also can be a bit tricky to determine the earliest version affected by a security bug. What I do is use Loggerhead's "Files" interface to find the place where the patched code was first introduced. (That is, the code the Security bug is modifying.) Then, by looking at the Changes log in Loggerhead, you can match that up with the dates on the Release Information page.

Getting it Reviewed

Security Advisories get sent to security@bugzilla.org for review. Allow at least two or three days for the members of the list to respond.

If you don't get a response after three or four days, that probably means the advisory is fine. But still check with people on IRC and get them to look at it.

Release Notes

The actual process of writing Release Notes is described in the Release Noting Guide. The important thing to remember about Release Notes is that they should be sent to reviewers@bugzilla.org for comment. Give the list about two days to respond.

They should also be attached normally to the bug, and you should ask some Documentation reviewer to look over them.

Web Site Updates

Check Out From CVS (needs to be put into Git)

In order to update the website, the first thing you have to do is check out its code from CVS. The best place to do this is on landfill, because it already has the environment set up to be able to fully build the website. Make sure you have checkin access on cvs.mozilla.org.

The CVS info is:

  CVSROOT=\:ext\:user%domain.com@cvs.mozilla.org\:/www
  cvs -z3 checkout bugzilla-org

Or, if you don't have checkin access, but you want to work on the web site updates anyhow, the CVSROOT for anonymous access is:

  CVSROOT=\:pserver:anonymous@cvs-mirror.mozilla.org\:/www

Automatic Website Updater

There is now a script that does most of the web site updates for a release, called bin/do-release.pl in the website checkout. You use it like this:

bin/do-release.pl --security=3.2.6 3.7.1 3.6.1 3.4.7 3.2.7

That updates your local checkout as though you were going to do a release of 3.7.1, 3.6.1, 3.4.7, and 3.2.7, with a security advisory where the lowest unfixed version being fixed was 3.2.6.

This script will update the files in your checkout appropriately for the release, and will also print a lot of messages like "** Remember to update some/file.html". Those "remember" messages are files that are not updated by the script, and will have to be updated by you. The updates that need to be done to those files are described below in the "Manual Website Updating" section below.

Now, search the src/ directory for "FIXME": grep -R src/ FIXME

The results show places where you probably need to update the files. Some of the FIXMEs are dates, and you don't know the date of the release yet, so you can leave those as FIXME. But the other FIXMEs need to be fixed now.

Manual Website Updating

Before we had the automatic updater script, we used to do all the website updates by hand. This section preserves the full instructions for how to update the website for a release, in case you need to understand exactly what the automatic script is doing, or in case you need to understand how to update one of the files that the script doesn't update.

• Create a directory for the Security Advisory. The directory is always numbered by the lowest unfixed version that is being fixed. For example, if you release versions 2.16.9, 2.18.1, and 2.19.3, and they all have a single Security Advisory, the security advisory goes into the 2.16.8/ directory.
• For each new release, create a "Release Info" page:
  • Create a directory in the releases/ directory for the new release.
  • Copy over the index.html.tmpl file from the previous release on that branch.
  • Edit any links to point to the right place for the new version.
  • Change any text that refers to the previous version to instead refer to this version. This includes at least the title and the text at the top.
  • Open Firefox and go to the branch installation on landfill. Open up the release notes and save them as "Web Page, complete"--this will make all relative links into absolute links so that you can copy the page to the website. Just remove the Bugzilla page header and page footer, and you can pase the code basically directly as the Release Notes page. Do make sure that the docs_urlbase parameter on the branch installation points to the correct docs on bugzilla.org first, though.
  • Link to the appropriate new Security Advisory.
• Update lib/releases-list.txt, which controls the main Release Information page.
• Patch the /status/changes.html.tmpl to reflect the new releases. If the patch has a "FIXME" date in it, remember to fix it!
• Post the News announcement in /news/.
• Update the main index.html.tmpl to point to the new News item, and remove the last News item from being listed on the main page.
• Now actually put the Security Advisory into the directory. You could have done this earlier, but it usually takes a while to get the Security Advisory complete and reviewed, so I put it last, here
• Update security/index.html.tmpl to include a link to the new Security Advisory.
• Update docs/index.html.tmpl to display the latest version numbers for the links to the docs.

And now you should be done with the web site update! Don't check in your changes yet -- that comes later.

Check-In Security Bugs and Release Notes

Locate approved patches on Security bugs, and check them in now to git. If the person who wrote the patches is around, have them do it. Otherwise use them as the author value for the checkin. Also check in the release notes changes as well.

Once you hit this point, you are committed to releasing, because you've essentially exposed Security bugs publically without a release that fixes them.

Freeze the Tree

This is only here so that you remember to do it. Basically, just make sure that from this point until after you post the tarballs, there aren't any checkins to any branch, from anybody but you.

Update Version Numbers

For each branch that you are releasing:

• Update Bugzilla/Constants.pm's BUGZILLA_VERSION variable.
• Update docs/en/xml/Bugzilla-Guide.xml. bz-ver, bz-date, bz-nextver, and current-year need to be updated. You may also have to update landfillbase if you just branched.
• For 4.3 and newer, instead of docs/en/xml/Bugzilla-Guide.xml you need to update docss/bugzilla.ent.tmpl.

Tag the Releases in Bzr, and Git

TODO: This should all be a script at some point

To tag a release in Bzr for the 4.0 branch:

Note: bzr.bugzilla.org now resides on the community infrastructure and requires shell access to make the tags manually. Find dkl or justdave to get access. The BZR branches are located in /var/www/html/bzr.bugzilla.org/bugzilla.

  bzr tag bugzilla-4.0.14
  bar tag --force bugzilla-stable

To tag a release in Git for the 4.0 branch:

  git checkout 4.0
  git tag bugzilla-4.0.14 <commitid>
  git tag release-4.0.14 <commitid>
  git push origin bugzilla-4.0.14
  git push origin release-4.0.14
  git push --delete origin bugzilla-4.0.14 (to remove a tag if needed)

Note: Once 5.0 is released, we will only be using the release-x.x.x tag form for 5.0 and future releases.

Creating stable release branch in git initially:

  git checkout 4.0
  git branch release-4.0-stable <commitid>
  git push origin release-4.0-stable

Updating stable release branch after bumping version number step:

  git checkout release-4.0-stable
  git merge origin/release-4.0.14
  git push

Generate Files to Release

Create the Tarballs

For building tarballs for 4.2.x and newer:

  ./build.pl 4.2.10
  ./build.pl 4.4.5
  ./build.pl 4.5.5

Make the Diff Files

In addition to tarballs, you also need to generate the .diff files. For this, you'll need an unpacked tarball of every Bugzilla release from each series you're releasing against.

There's a script called makediffs.pl in the misc/build bzr repository that will generate diff files for every .tar.gz file in your current directory.

Development versions don't need .diff files, and neither do Release Candidates.

Post the Files

Using sftp or scp, upload the tarballs and .diff.gz files to stage.mozilla.org, to the /pub/mozilla.org/webtools directory.

Remember to fix the bugzilla-LATEST and bugzilla-STABLE symlinks, if necessary.

Also verify that the uploaded files are world readable so that downloads will work properly.

Check-In the Website Updates

Now, you commit the changes you made to the website, above. Double-check everything to make sure that all the dates and times are correct anywhere that you had to put a date/time.

For the commit message, use the following format:

  Bug 1072487 - (bz-release-446) Release Bugzilla 4.5.6, 4.4.6, 4.2.11 and 4.0.15
  r=LpSolit

Send Announcements

Security Advisory

Double-check the Security Advisory before you send it. Make sure that any links you have in the Advisory actually work.

Usually, we GPG sign the Security Advisory with a key that has its public half available from either public keyservers or the bugzilla.org website.

Summary (List the affected versions, not the fixed ones):

 Security advisory for Bugzilla 5.0, 4.4.9, and 4.2.14

The Security Advisory is sent to:

• BugTraq
• announce@bugzilla.org
• support-bugzilla@lists.mozilla.org

Send each as a separate email. This is because you will get a lot of bounce messages, and you don't want them to be spread around to the other lists. (Some things sending bounce messages are really stupid and will attempt to do this.)

Release Announcement

This is the Release Announcement that you created on the bug that you filed.

Make sure that you put [ANN] at the beginning of the Subject line. List the fixed versions.

 [ANN] Release of Bugzilla 5.0.1, 4.4.10, and 4.2.15

Send it to:

• announce@bugzilla.org
• support-bugzilla@lists.mozilla.org

Once again, send each as a separate email.

Update updates.bugzilla.org

Update bugzilla-update.xml on landfill so that the notification system will detect new releases.

  ssh dlawrence@updates.bugzilla.org
  sudo bash
  vi /var/www/html/bugzilla-update.xml
  Change the versions and dates

Post-Release

Stuff you do after you've posted the release and sent the announcements:

• Unlock the Security Bugs. Just remove them from the bugzilla-security group
• Update the VERSION number in Bugzilla/Constants.pm on each branch, to have a "+" after it.
• Update topic on the #bugzilla IRC channel on irc.mozilla.org to indicate latest released versions (and note any new EOL versions).