IT/ChangeControl

Document Status	Last Reviewed
DRAFT	Aug 13 2012

Introduction

As Mozilla's Infrastructure has grown, so has the complexity of the systems. It is apparent that we need to be more proactive about regular system updates and maintenance so we can continue supporting the Mozilla Mission.

To address these needs, Mozilla has established a Change Advisory Board whose mandate is to manage the Change Control Policy and manage Scheduled Maintenance Approvals.

Change Advisory Board

The Change Advisory Board (CAB) is the entity that defines, and owns, Mozilla's Schedule Maintenance & Change Control Policies.

The CAB is comprised of the relevant stakeholders of each critical IT/Infrastructure service. These stakeholders are the people who can best make decisions about changes because of their understanding of the business goals, as well as technical and operational risks.

Core CAB Members

• Director IT/Operations (:mrz)
• Release Engineering, Director (joduinn)
• Services Ops Director (mmayo)
• Security Assurance, Director (mcoates)
• Mozilla SRE/Oncall, Manager (shyam)
• Systems, Manager (cshields)
• Netops, Manager (rpina)

CAB Meeting Schedule

The CAB will meet every week to review proposed Change/Maintenance Requests. The meeting will be chaired by Director of IT/Operations and the SRE Manager.

Meeting details:

• Tuesday 8:30a Pacific (1630 UTC)
• Video: mrz's vidyo room (x219)
• Audio: +1 800 707 2533, pin 369, conf 9219#

Change Management Process

Mozilla IT/Operations & Service Operations has the responsibility to design, manage, and maintain an infrastructure that provides the highest levels of availability, reliability, performance, and service in support of the Mozilla Mission.

An effective Change Management Process is an integral part of a comprehensive strategy to meet this responsibility.

It is the responsibility of the Change Advisory Board to implement a change control process that will provide management of all actions and modifications that may affect the stability of any production network, system, or service.

Submitting a Change Request

• File a bug in Server Operations: Change Requests
• Set the bug flag cab-review to ?
• Include the following:
  • data, time, duration of maintenance
  • system(s) affected
  • end-user impact
  • maintenance plan and timeline (link to a wiki or etherpad is fine)
  • rollback plan / rollback point (at which point will you determine to roll back)
  • notification mechanisms
  • who will be point, who else will be involved

CAB Approval Process

Informational Only

Information Only Approvals can be granted for limited-scope changes for things where you have blanket approval.

Example: Some changes may be pre-approved as part of a larger project. These CAB requests are merely information that "we are doing this upgrade". It is not necesssary to wait for an approval; a veto may be issued if there are extenuating circumstances.

Minor Changes

Minor Changes needed limited approval because they are limited in scope and shouldn't hold up for a long time waiting for the next meeting. These Changes require a single CAB member signoff where there's little to no risk and/or impact limited to known scope.

Approved Minor Changes are to be reported at the next CAB Meeting by the CAB Member who approved the Minor Change.

Major Changes

Major Changes require the board in quorum. The Change Requester (or proxy) will be invited to the CAB Meeting to defend the request/change. Undefended requests will be deferred.

The CAB will mark the Request:

• Denied
• Approved
• Pending (requires contingencies for the Requester to fullfill (shifting start time, improved rollout/rollback plans)).

Emergency Changes

Emergency/breakfix changes will be handled on an as-needed basis with the primary purpose focused on restoring service. These changes require no pre-authorization.

Mozilla SRE will inform the CAB via email with a short description of the change and the reason for it.

Mozilla SRE will attend the next CAB Meeting to walk through Root Cause Analysis.