Continuous Integration

Program Description

The purpose of the Firefox OS Quality initiative is to drive the engineering team to Continuous Integration through testing platform improvements, test improvements, test automation and code coverage, where every commit on Gecko and Gaia triggers a full test run, with failures and performance issues reported on Treeherder for immediate backout.

Requirements:

The following Firefox OS 2.6 work must be completed to achieve Continuous Integration:

 * Test improvements
   * Convert integration tests from GIP to GIJ
   * Fix intermittently failing tests
   * Fix tests that work on mulet but not on device
   * Create a Test Matrix to define which platform the tests are going to run on (device, mulet, emulator)
   * Make sure every single commit in gecko/gaia triggers automated tests
   * Automate MozTrap scenarios using GIJ when feasible
   * Actively maintain working Mulet tests with reporting on Treeherder 
   * Ensure code coverage 
 * Marionette Improvements
   * Remove python from Marionette harness
   * Run Marionette.js tests on devices in the Bitbar testing farm with reporting on Treeherder
   * Improve Marionette stability and stability of sandbox environment
 * Emulator Testing Improvements
   * Fix kitkat emulator testing issues
   * Put emulators in place for testing with reporting on Treeherder
   * Port emulator to next android x86 version
 * Performance Testing Improvements
   * Run performance tests on devices with reporting on Raptor and Treeherder (Bitbar testing farm) 
   * Integrate the device service into Raptor
   * Achieve ability for sheriffs to back out commits if post-commit data is bad
   * Ability to block commits with pre-commit testing

Program Status

Automated Test Coverage

Milestone	Due Date	Status	Notes
Convert integration tests from GIP to GIJ	12/31/15	AT RISK	All but 14 have been converted. Engineers out on PTO.
Intermittently failing tests fixed	01/31/15	CHALLENGED	Engineers are disabling the tests that fail intermittently and will fix them before re-enabling. Need more engineering help with this.
Tests that work on mulet work on device	01/31/15	CHALLENGED	It's being reported that some of the flagged tests actually work on device. Need to reassess all.
Test Matrix created that defines platform(s) the tests are going to run on	12/31/15	AT RISK	Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Won't get this done until January.
Automation of tests according to Test Matrix	05/31/16	ON TARGET	Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Won't get this done until January.
Every single commit in gecko/gaia triggers automated tests	05/31/16	ON TARGET	There's still work that needs to be identified and scheduled.
MozTrap scenarios automated using GIJ when feasible	05/31/16	ON TARGET
Working Mulet tests actively maintained with reporting on Treeherder	05/31/16	ON TARGET
Code Coverage Tool developed	12/31/15	AT RISK	Originally targeted for end of Q4, but assistance at Mozlando didn't happen.

Marionette Improvements

Milestone	Due Date	Status	Notes
Removal of python from the Marionette harness		AT RISK	New marionette-client promises driver developed. Now working on tests for driver.
Device testing on TaskCluster/Treeherder with Bitbar		ON TARGET
Marionette stability		ON TARGET	Bugs have been entered and work is slotted for Q1.
Sandbox environment stability		ON TARGET	Work is slotted for Q1.

Emulator Testing Improvements

Milestone	Due Date	Status	Notes
Milestone 1		ON TARGET
Milestone 2		CHALLENGED
Milestone 3		AT RISK
Milestone 4		ON TARGET
Milestone 5		CHALLENGED
Milestone 6		AT RISK
Milestone 7		ON TARGET
Milestone 8		CHALLENGED
Milestone 9		AT RISK

Performance Testing Improvements

Milestone	Due Date	Status	Notes
Integrate the fxos device service into Raptor		DONE
Enable automated testing of devices at Bitbar		NOT STARTED
		AT RISK

Status Key

Color	Status	Key
	On Target	The project or deliverable is expected to meet its due date.
	Challenged	The project or deliverable is facing an issue that might cause it to miss its due date, but a “get well” plan has been developed to get it back on track.
	At Risk or Late	The project or deliverable is blocked or facing an issue that might cause it to miss its due date, and there’s no “get well” plan to get it back on track, or it is already late.
	Done	The project or deliverable has been completed.
	On Hold	The project or deliverable has been placed on hold.

---

Program Timeline

---

MVP Scope

Querying by 2.6+ features

(please add correct bug tracking number)

No results.

0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);

---

Dependency Tracking

---

Detailed Program Plan

Action Item	Engineering Owner	QA Owner	UX Owner	Bugzilla ID	Planned Done	Actual Done

---

Program Stakeholders

Role	Name	IRC
EPM
EM
PM
TL
UX
QA

• EPM = Engineering Program Manager
• EM = Engineering Manager
• PM = Product Manager
• TL = Tech Lead
• UX = User Experience
• QA = Quality Assurance

---

---