MozillaWiki

Continuous Integration: Difference between revisions

Page Discussion

Continuous Integration (view source)

Revision as of 18:10, 18 February 2017

777 bytes added ,  18 February 2017
→‎Dependency Tracking: added TBD
(→‎Marionette Improvements: Made Marionette Improvements clickable)
 
(15 intermediate revisions by the same user not shown)
Line 1: Line 1:
<div style="float: right; margin: 0px 0px 10px 25px;">[[File:Fxossmall.png|link=]]</div>
<div style="float: right; margin: 0px 0px 10px 25px;">[[File:Fxossmall.png|link=]]</div>
'''This program ended before work was completed.The content within this wiki remains for archival purposes.'''


=='''Program Description'''==
=='''Program Description'''==
The purpose of the Firefox OS Quality initiative is to drive the engineering team to Continuous Integration through testing platform improvements, test improvements, test automation and code coverage, where every commit on Gecko and Gaia triggers a full test run, with failures and performance issues reported on Treeherder for immediate backout.   
The purpose of the Firefox OS Quality initiative was to drive the engineering team to Continuous Integration through testing platform improvements, test improvements, test automation and code coverage, where every commit on Gecko and Gaia triggers a full test run, with failures and performance issues reported on Treeherder for immediate backout. Work on these projects ended with the decision to sunset Firefox OS.   


===Requirements:===
===Requirements:===


The following Firefox OS 2.6 work must be completed to achieve Continuous Integration:
The following Firefox OS 2.6 work must be completed to achieve Continuous Integration:
   * Test improvements
   * [[Automated Test Coverage]]
     * Convert integration tests from GIP to GIJ
     * Convert integration tests from GIP to GIJ
     * Fix intermittently failing tests
     * Fix intermittently failing tests
Line 16: Line 18:
     * Actively maintain working Mulet tests with reporting on Treeherder  
     * Actively maintain working Mulet tests with reporting on Treeherder  
     * Ensure code coverage  
     * Ensure code coverage  
   * Marionette Improvements
   * [[Marionette Improvements]]
     * Remove python from Marionette harness
     * Remove python from Marionette harness
     * Run Marionette.js tests on devices in the Bitbar testing farm with reporting on Treeherder
     * Run Marionette.js tests on devices in the Bitbar testing farm with reporting on Treeherder
     * Improve Marionette stability and stability of sandbox environment
     * Improve Marionette stability and stability of sandbox environment
   * Emulator Testing Improvements
   * [[Kitkat Emulator Improvements]]
     * Fix kitkat emulator testing issues
     * Fix kitkat emulator testing issues
     * Put emulators in place for testing with reporting on Treeherder
     * Put emulators in place for testing with reporting on Treeherder
     * Port emulator to next android x86 version
     * Port emulator to next android x86 version
   * Performance Testing Improvements
   * [[Raptor Activation on Treeholder]]
     * Run performance tests on devices with reporting on Raptor and Treeherder (Bitbar testing farm)  
     * Run performance tests on devices with reporting on Raptor and Treeherder (Bitbar testing farm)  
     * Integrate the device service into Raptor
     * Integrate the device service into Raptor
Line 39: Line 41:
|-
|-
| Convert integration tests from GIP to GIJ
| Convert integration tests from GIP to GIJ
| 12/31/15
| 03/25/16
! style="background:#FF2800;" | AT RISK
! style="background:#FFFF00;" | CHALLENGED
| All but 14 have been converted. Engineers out on PTO.
| Due date revised to end of Sprint 10. All but 10 have been converted. This deliverable is challenged because intermittents are a competing priority.
|-
|-
| Intermittently failing tests fixed
| Intermittently failing tests fixed
| 01/31/15
| 02/26/16
! style="background:#FFFF00;" | CHALLENGED
! style="background:#00EC00;" | ON TARGET
| Engineers are disabling the tests that fail intermittently and will fix them before re-enabling. Need more engineering help with this.
| Due date revised to end of Sprint 8. Fixed parts of Marionette that were causing issues. Disabled all the tests were failing intermittently. Module owners need to ensured they're fixed before re-enabling them. Need support to prioritize this work with the engineers.
|-
|-
| Tests that work on mulet work on device
| Tests that work on mulet work on device
| 01/31/15
| 02/26/16
! style="background:#FFFF00;" | CHALLENGED
! style="background:#FFFF00;" | CHALLENGED
| It's being reported that some of the flagged tests actually work on device.  Need to reassess all.
| It's being reported that some of the flagged tests actually work on device.  Need to reassess all. This deliverable isn't being given much attention.
|-
|-
| Test Matrix created that defines platform(s) the tests are going to run on
| Test Matrix created that defines platform(s) the tests are going to run on
| 12/31/15
| 02/26/16
! style="background:#FF2800;" | AT RISK
! style="background:#00EC00;" | ON TARGET
| Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Won't get this done until January.
| Due date pushed out to one that is more realistic. Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment. Meeting with eng. mgmt. scheduled for Jan. 27th to review test matrix.
|-
|-
| Automation of tests according to Test Matrix
| Automation of tests according to Test Matrix
| 05/31/16
| 05/31/16
! style="background:#00EC00;" | ON TARGET
! style="background:#FFFF00;" | CHALLENGED
| Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment.  Won't get this done until January.
| Draft of matrix completed but there are still some tests that need to be evaluated for appropriate testing environment.  Need to finalize matrix before automation can begin.
|-
|-
| Every single commit in gecko/gaia triggers automated tests
| Every single commit in gecko/gaia triggers automated tests
Line 79: Line 81:
|-
|-
| Code Coverage Tool developed
| Code Coverage Tool developed
| 12/31/15
| 05/31/15
! style="background:#FF2800;" | AT RISK
! style="background:#00EC00;" | ON TARGET
| Originally targeted for end of Q4, but assistance at Mozlando didn't happen.
| Originally targeted for end of Q4, but assistance at Mozlando didn't happen. Re-prioritized and given new due date.
|}
|}


Line 92: Line 94:
|-
|-
| Removal of python from the Marionette harness
| Removal of python from the Marionette harness
|  
| 02/26/16
! style="background:#FF2800;" | AT RISK
! style="background:#00EC00;" | ON TARGET
| New marionette-client promises driver developed. Now working on tests for driver.
| New marionette-client promises driver developed. Now working on test for new api. Due date revised (pushed out).
|-
|-
| Device testing on TaskCluster/Treeherder with Bitbar
| Device testing on TaskCluster/Treeherder with Bitbar
|
|05/31/16
! style="background:#00EC00;" | ON TARGET
! style="background:#C1C6CA;" | ON HOLD
|
|Uncertaintly about the future of phone testing at Bitbar has resulted in questions regarding how to proceed.
|-
|-
| Marionette stability
| Marionette stability
|
|03/31/16
! style="background:#00EC00;" | ON TARGET
! style="background:#00EC00;" | ON TARGET
| Bugs have been entered and work is slotted for Q1.
| Bugs have been entered and work is slotted for Q1. Fixes have been made to tackle problem associated with "intermittents".
|-
|-
| Sandbox environment stability
| Sandbox environment stability
|
|03/31/16
! style="background:#00EC00;" | ON TARGET
! style="background:#00EC00;" | ON TARGET
| Work is slotted for Q1.
| Work is slotted for Q1.
|}
|}


=== Emulator Testing Improvements ===
=== [[Kitkat Emulator Improvements]] ===
{| class="wikitable"
{| class="wikitable"
! style="text-align: center;" | Milestone
! style="text-align: center;" | Milestone
Line 165: Line 167:
|}
|}


=== Performance Testing Improvements ===
=== [[Raptor Activation on Treeherder]] ===
{| class="wikitable"
{| class="wikitable"
! style="text-align: center;" | Milestone
! style="text-align: center;" | Milestone
Line 184: Line 186:
|  
|  
|
|
! style="background:#FF2800;" | AT RISK
|  
|  
|  
|}
|}
Line 214: Line 216:
|}
|}


----
=='''Program Timeline'''==
[[File:Release Timeline.png]]
----


=='''MVP Scope'''==
=='''MVP Scope'''==
Line 237: Line 233:
----
----


=='''Dependency Tracking'''==
=='''Dependency Tracking (TBD)'''==
[[File:Dependency Gantt Chart.png]]
[[File:Dependency Gantt Chart.png]]


Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1110954...1163253"