QA/Automation/Projects/Mozmill Automation/Mozmill CI

Overview

The Mozmill Automation team currently performs update testing triggered by a daily cron job. This approach has some weaknesses:

• If no build lands, we still attempt to perform update testing.
• If a build lands off schedule, we attempt to update test it at the wrong time.
• There's no way to use this style of trigger to pre-qualify a build (or even promptly post-qualify it) so results have limited impact.
• Because we update build-over-build, using the same installation, anomalies in testing (or bugs in a given build) can cause the process to get "off-track" and poison subsequent update tests.

Instead, we'd like to move towards performing update testing as soon as the build lands. We'd also like to independently download and verify each day's build, rather than relying on using the same install day over day.

The ultimate goal is to build a prototype within QA walls, then hand off the process to Release Engineering for ongoing maintenance.

Leads:	Geo Mealer Henrik Skupin
Team Assistance:	n/a

Proposed Design

Process

1. Pulse Trigger received on build landing, contains current build id
2. Trigger polling script on VM catches trigger
3. Trigger polling script adds build to queue (directory full of action snippets or a single flat file) for processing.
   • Initial Proof of Concept will not include a queue.
4. Queue polling script pulls the job off the queue, does two things:
   1. Downloads current build and saves it off for next trigger.
   2. Build saved off from last trigger will be updated and compared to expected *build id.

• Initial reporting to brasstacks, possibly email on failures
• Initial version for trunk, en_US only. Implementing OS X version first, other platforms to come very shortly thereafter.

Resources already present

• Script to install a build
• Script to update a build and do some basic heuristics for testing exists.
• Agreement from RelEng to send Pulse trigger, interim trigger already in place based on FTP scraping
• Sample code for Pulse interaction that can be adapted to our purpose.

Things needed now

• Update test script needs to be able to take a particular build id for verification
• Final details on Trigger information to be decided.
• Polling script

Things needed eventually

• Queuing mechanism (this will primarily be needed for L10N testing, where all locales land simultaneously and need to be queued on platform)
• RelEng wants to do this in their infrastructure eventually if at all possible.
  • If not possible, we'll need a way to report directly back to RelEng
  • Not there yet, though. For now want to run it on our side to make sure of robustness before putting in critical path.