Bugzilla:Release Process

Bugzilla Releases

Security Releases

All security bugs should be triaged as soon as possible by the Bugzilla team. Along with the Mozilla Security Team, it should be determined if the issue is valid, and the severity (the sec-low, sec-moderate and sec-high keywords is normally used for this).

A high security bug would include (but is not limited to) data leakage (including CSRF/XSS attacks) or when the exploit is already publicly available (as was the case with bug 1036213). This would also include non-security bugs that could cause data loss or corruption. In these case, a new point release would be done as soon as possible.

A medium security bug is a bug which could cause user harm indirectly or if some heighten level of permissions are required. These include bugs such as bug 1054702 and bug 873932. They should be fixed as soon as possible, but will be included with the next scheduled release.

Low security bugs are issues which are hard to reproduce and only occur under some circumstance. Examples are bug 761043 and bug 301686. These should be fixed as developer time permits, and will be included in the next scheduled release if completed.

Version Numbers

Major version increases (e.g. 5.0) reflect large, visible, and/or breaking changes. These would include major features being removed or added, interfaces being changed, significant UI changes, etc.

Minor version increases (e.g. 5.2) reflect smaller new features, and other nonbreaking changes. These would include additions and/or backward-compatible changes to interfaces, support for new technologies or platforms, and general improvements and fixes. Note that official releases will always have an even-numbered minor version (e.g. 5.0, 5.2, 5.4); odd numbers are reserved for development releases. This applies only to the minor version number.

Patch version increases (e.g. 5.2.1) usually reflect critical fixes, such as to security or other severe bugs. While we generally wait until there is a critical fix to release a patch version, these versions may also contain other minor fixes committed in the meantime. We may also occasionally release a patch version without critical fixes if there is a sufficiently large backlog of minor fixes, or for development snapshots (of an odd-numbered minor version).

When to Release

We release approximately once per year with whatever is ready at the time. Branching should be done when master is relatively stable, but sometimes branching with known bugs is acceptable if it allows other large, potentially unstable features to land (for the next release). If there are bugs related to new features in the release branch, the feature should be backed out of the release branch (if it's very recent), or fixes should be committed to both the release branch and master. Bugs blocking release should be denoted with a blocking<version> flag, e.g. blocking5.0.

The version number will be determined at time of branching depending on the changes included, as per the section above.

Release Process Details

File Bugs

File bugs for the following things...

• "Release Bugzilla X.YY, Z.AA" and mark it as severity "blocker." It should block bug 286269, which has the alias bz-release. (Sample: bug 1042086)
• Security Advisory: It should block the first bug, and and Security bugs targeted at this release should be dependencies. (Sample: bug 1042093)
• Release Notes: Point releases and rc1 versions need release notes. Each version being released needs a separate Release Notes bug filed. These bugs should block the first bug. For final releases of major versions, you need a "clean up release notes" bug. (Sample: bug 1042088)
• New Features Page (Release Candidates and Final Releases Only): Make this bug depend on the appropriate Release Notes bug. (Sample: bug 286278)

Pre-Release

Basically, you just have to get all the bugs to have patches with review+ on them. A few of them require some special handling:

Security Advisory

Format

The actual "issues" in a security advisory normally look something like this:

  Class:       Cross-site scripting
  Versions:    2.15 through 2.18rc3 and 2.19.1
  Description: It is possible to blah blah blah
               And here are some details on how it can be made less bad...
  Reference:   https://bugzilla.mozilla.org/show_bug.cgi?id=272620
  CVE Name:    CVE-2014-1061

The different values for Class can be found in older Security Advisories. The "CVE Name" is, obviously, not always needed.

Determining Affected Versions

It also can be a bit tricky to determine the earliest version affected by a security bug. git blame can help you find the place where the patched code was first introduced (that is, the code the Security bug is modifying) and which bug introduced it.

Getting it Reviewed

Security Advisories must be reviewed by a member of the Bugzilla team. If you don't get a response after two or three days, check with people on IRC and get them to look at it.

Release Notes

The actual process of writing Release Notes is described in the Release Note Guide. They should also be attached normally to the bug, and you should ask some reviewer to look over them.

Web Site Updates

Check Out From Git

In order to update the website, the first thing you have to do is check out its code from Git. Make sure you have checkin access on git.mozilla.org:

ssh://gitolite3@git.mozilla.org/www/bugzilla.org.git.

Or, if you don't have checkin access, but you want to work on the web site updates anyhow:

https://git.mozilla.org/www/bugzilla.org.git

Automatic Website Updater

There is now a script that does most of the web site updates for a release, called bin/do-release.pl in the website checkout. You use it like this:

bin/do-release.pl --security=4.2.15 4.4.10 5.0.1

That updates your local checkout as though you were going to do a release of 4.2.15, 4.4.10 and 5.0.1, with a security advisory where the lowest unfixed version being fixed was 4.2.14.

This script will update the files in your checkout appropriately for the release, and will also print a lot of messages like "** Remember to update some/file.html". Those "remember" messages are files that are not updated by the script, and will have to be updated by you. The updates that need to be done to those files are described below in the "Manual Website Updating" section below.

Now, search the src/ directory for "FIXME": grep -R src/ FIXME

The results show places where you probably need to update the files. Some of the FIXMEs are dates, and you don't know the date of the release yet, so you can leave those as FIXME. But the other FIXMEs need to be fixed now.

Manual Website Updating

Before we had the automatic updater script, we used to do all the website updates by hand. This section preserves the full instructions for how to update the website for a release, in case you need to understand exactly what the automatic script is doing, or in case you need to understand how to update one of the files that the script doesn't update.

• Create a directory for the Security Advisory. The directory is always numbered by the lowest unfixed version that is being fixed. For example, if you release versions 4.2.15, 4.4.10 and 5.0.1, and they all have a single Security Advisory, the security advisory goes into the 4.2.14/ directory.
• For each new release, create a "Release Info" page:
  • Create a directory in the releases/ directory for the new release.
  • Copy over the index.html.tmpl file from the previous release on that branch.
  • Edit any links to point to the right place for the new version.
  • Change any text that refers to the previous version to instead refer to this version. This includes at least the title and the text at the top.
  • Open Firefox and go to the branch installation on landfill. Open up the release notes and save them as "Web Page, complete"--this will make all relative links into absolute links so that you can copy the page to the website. Just remove the Bugzilla page header and page footer, and you can pase the code basically directly as the Release Notes page. Do make sure that the docs_urlbase parameter on the branch installation points to the correct docs on bugzilla.org first, though.
  • Link to the appropriate new Security Advisory.
• Update lib/releases-list.txt, which controls the main Release Information page.
• Patch the /status/changes.html.tmpl to reflect the new releases. If the patch has a "FIXME" date in it, remember to fix it!
• Post the News announcement in /news/.
• Update the main index.html.tmpl to point to the new News item, and remove the last News item from being listed on the main page.
• Now actually put the Security Advisory into the directory. You could have done this earlier, but it usually takes a while to get the Security Advisory complete and reviewed, so I put it last, here
• Update security/index.html.tmpl to include a link to the new Security Advisory.
• Update docs/index.html.tmpl to display the latest version numbers for the links to the docs.

And now you should be done with the web site update! Don't check in your changes yet -- that comes later.

Check-In Security Bugs and Release Notes

Locate approved patches on Security bugs, and check them in now to git. If the person who wrote the patches is around, have them do it. Otherwise use them as the author value for the checkin. Also check in the release notes changes as well.

Once you hit this point, you are committed to releasing, because you've essentially exposed Security bugs publically without a release that fixes them.

Freeze the Tree

This is only here so that you remember to do it. Basically, just make sure that from this point until after you post the tarballs, there aren't any checkins to any branch, from anybody but you.

Update Version Numbers

For each branch that you are releasing:

• Update Bugzilla/Constants.pm's BUGZILLA_VERSION variable.
• For Bugzilla 4.4, you must also update docs/bugzilla.ent.tmpl. bz-ver, bz-date, and current-year need to be updated. You may also have to update landfillbase if you just branched.

Tag the Releases in Bzr, and Git

TODO: This should all be a script at some point

To tag a release in Bzr for the 4.4 branch:

Note: bzr.bugzilla.org now resides on the community infrastructure and requires shell access to make the tags manually. Find dkl or justdave to get access. The BZR branches are located in /var/www/html/bzr.bugzilla.org/bugzilla.

  bzr tag bugzilla-4.4.11
  bar tag --force bugzilla-stable

To tag a release in Git for the 4.4 branch:

  git checkout 4.4
  git tag bugzilla-4.4.11 <commitid>
  git tag release-4.4.11 <commitid>
  git push origin bugzilla-4.4.11
  git push origin release-4.4.11
  git push --delete origin bugzilla-4.4.11 (to remove a tag if needed)

Note: Since 5.0, we only use the release-x.x.x tag form for 5.0 and future releases.

Creating stable release branch in git initially:

  git checkout 4.4
  git branch release-4.4-stable <commitid>
  git push origin release-4.4-stable

Updating stable release branch after bumping version number step:

  git checkout release-4.4-stable
  git merge origin/release-4.4.11
  git push

Generate Files to Release

Create the Tarballs

For building tarballs for 4.2.x and newer:

  ./build.pl 4.2.15
  ./build.pl 4.4.10
  ./build.pl 5.0.1

Make the Diff Files

In addition to tarballs, you also need to generate the .diff files. For this, you'll need an unpacked tarball of every Bugzilla release from each series you're releasing against.

There's a script called makediffs.pl in the misc/build bzr repository that will generate diff files for every .tar.gz file in your current directory.

Development versions don't need .diff files, and neither do Release Candidates.

Post the Files

Mozilla recently migrated specific files on ftp.mozilla.org to be served from Amazon S3 buckets. In order to upload the tar files and diffs to S3, you will need to have access keys assigned to your account. dkl and justdave currently have access keys so either of them can upload the files.

You will need to download and install the aws-cli client from github.

http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-bundle-other-os

Create an .aws directory in your user's home directly.

Generate an .aws/credentials file containing your access keys assigned to you.

 [default]
 aws_access_key_id=<ACCESS_KEY>
 aws_secret_access_key=<SECRET_ACCESS_KEY>

Generate an .aws/config file that specifies the region desired.

 [default]
 region=us-east-1
 output=json

To copy a file to S3 available for download. Need to do for each file:

 $ aws s3 cp <local_file> s3://net-mozaws-prod-delivery-contrib/pub/webtools/

The file can then be accessed by the general public at the same URL as before (over https only):

 https://ftp.mozilla.org/pub/mozilla.org/webtools/<file>

To remove a file from S3:

 $ aws s3 rm s3://net-mozaws-prod-delivery-contrib/pub/webtools/<remote_file>

To list the current files in S3:

 $ aws s3 ls s3://net-mozaws-prod-delivery-contrib/pub/webtools

To copy a file on S3 to another name. This needs to be done to mimic the old symlinks for bugzilla-STABLE.tar.gz and bugzilla-LATEST.tar.gz:

 $ aws s3 cp s3://net-mozaws-prod-delivery-contrib/pub/webtools/<src_file> s3://net-mozaws-prod-delivery-contrib/pub/webtools/<dest_file>

Check-In the Website Updates

Now, you commit the changes you made to the website, above. Double-check everything to make sure that all the dates and times are correct anywhere that you had to put a date/time.

For the commit message, use the following format:

  Bug 1072487 - (bz-release-446) Release Bugzilla 4.5.6, 4.4.6, 4.2.11 and 4.0.15
  r=LpSolit

Send Announcements

Security Advisory

Double-check the Security Advisory before you send it. Make sure that any links you have in the Advisory actually work.

Usually, we GPG sign the Security Advisory with a key that has its public half available from either public keyservers or the bugzilla.org website.

Summary (List the affected versions, not the fixed ones):

 Security advisory for Bugzilla 5.0, 4.4.9, and 4.2.14

The Security Advisory is sent to:

• BugTraq
• announce@bugzilla.org
• support-bugzilla@lists.mozilla.org

Send each as a separate email. This is because you will get a lot of bounce messages, and you don't want them to be spread around to the other lists. (Some things sending bounce messages are really stupid and will attempt to do this.)

Release Announcement

This is the Release Announcement that you created on the bug that you filed.

Make sure that you put [ANN] at the beginning of the Subject line. List the fixed versions.

 [ANN] Release of Bugzilla 5.0.1, 4.4.10, and 4.2.15

Send it to:

• announce@bugzilla.org
• support-bugzilla@lists.mozilla.org

Once again, send each as a separate email.

Update updates.bugzilla.org

Update bugzilla-update.xml on updates.bugzilla.org so that the notification system will detect new releases.

  ssh dlawrence@updates.bugzilla.org
  sudo bash
  vi /var/www/html/bugzilla-update.xml
  Change the versions and dates

Post-Release

Stuff you do after you've posted the release and sent the announcements:

• Unlock the Security Bugs. Just remove them from the bugzilla-security group
• Update the VERSION number in Bugzilla/Constants.pm on each branch, to have a "+" after it.
• Update topic on the #bugzilla IRC channel on irc.mozilla.org to indicate latest released versions (and note any new EOL versions).