User:Zeller

Ship It

Bug 826753 - release automation should update ship it at certain points

Notes

• /status - Lists all available releases, that have status data, in JSON format and can be queried with parameters (ie /status?var1=1&var2=10). Analogous to /releases
• /statuses.html – Pretty GUI view of the info shown in /status. Analogous to /releases.html
• /status/<release-name>
  • GET: Lists release status info in JSON. Analogous to /releases/<release-name>
  • POST: covered in Bug 1032985
• /status/<release-name>.html – Pretty GUI view of the info shown in /status/ GET, possibly including a visual timeline of the 6 steps: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease. (status steps taken from Bug 826753 comment 0)

Questions

• Why does /release 404?
• Should /statuses be available through a 3rd tab (ie Reviewed/Completed/Statuses), or through a new "view statuses" link?
• Does a ViewStatuses() need to be added to kickoff.js? In addition to the ViewReleases() function?

To Do

• Fix line 331 in model.py to sort the product properly
• When RelMan hits 'go', add proper row to database
• Make system diagram for interactions in shipit additions
• Make /status GET queriable (other than var1 and var2). Other release tables use the column variables ready and complete.
• Run a JOIN via the name column between release_status and *_releases, and print that JSON info upon /status/<release-name> GET request.
• Take progress bars from bootstrap rather than change the entire css

Bug 1032985 - Add REST API entry point to shipit that allows shipit-agent to enter release data into shipit database

https://bugzilla.mozilla.org/show_bug.cgi?id=1032985#c2

Add REST API entry point for updating update.db at /status/<release-name> when POST

Notes

• Use name field in release_status as an unofficial foreign key to the 3 *_releases tables (Can't use JOIN for 3 primary keys (all name) in the *_releases tables)

Questions

• How exactly should I divide the release pulse messages into the 6 statuses: Tagging completed, All builds/repacks completed, Updates on betatest, Ready for releasetest, Ready for release and Postrelease.

To Do

• Have StatusAPI update the database tables *-release, and check that name is in release_data
• Check to see if name already exists in tables *-releases, and check that name is in release_data before adding to release_status... use as switch between creating a new row and updating rows.
• DATETIME stamps need to not have T or +01:00
• Sort data into the 6 statuses

Bug 1032978 - Add a standalone process that listens to pulse for release related buildbot messages

Long-running standalone script (shipit-agent.py) that listens to pulse and uses / status/<release-name> with POST to update update.db for <release-name> status

Notes

• pulse.m.o gives buildbot events
  • Look for build.release-*.finished (Found in ['_meta']['routing_key']?)
  • Use FF candidate buildlogs to help determine the names of the builder I am interested in (ie the first part before "bmNN" is the name of the builder)
• If collecting, release_data, then the table will gain ~0.5mb/release, which I estimate makes it ~117Mb/year.
• If version = "None", then routing_key doesn't have build.release-*
• Some pulse fields are unnecessary, such as 'exchange', and 'serializer'
• Nothing says *.finished
• Ignore the following messages:
  • Any routing_key that doesn't match build.release-* (including unittest.release-*, talos.release-*, etc... they are tests we don't usually look at.
  • Any version or build_number that is None... there was a bug for that, they are test related. These also seem to only show up on release messages that don't match build.release-*, like unittest.release-*, talos.release-*, etc.
  • Any xulrunner jobs, they are implicitly run by Firefox
• [Examples https://bugzilla.mozilla.org/show_bug.cgi?id=1032985#c2] given by rail on how to sort pulse messages into the 6 statuses

Questions

• What do do with release messages that don't fit the expected:
  • Have a product other than "firefox" when supposedly it's a firefox build (ie "mobile" or "xulrunner"
• Did I scrap all of Firefox-31.0b5-build1 or am I missing something?
• Should I look at routing_key, or key, or?
• Do we want to keep all buildbot release pulse data, to be able to use it later to deduce new things, or forgoe all that entirely? (feature creep)
• Should protocol be configurable via puppet or with command line option?
• Is it possible to use the topic option with pulse to listen to only build.release-* messages?
• Should unique_label have a set name?
• Should I be using a different consumer like BuildConsumer? NormalizedBuildConsumer will miss some builds, pulsetranslator has a hardcoded list of recognized platforms, and will drop others. http://hg.mozilla.org/automation/mozillapulse/file/default/mozillapulse/consumers.py
  • Dump out the full message stream for a while until you see one of the ones you care about.
• Should we send the concat'd data instead of payload, and have the common_keys listed in the REST API and hold all the logic for the None's and such there?
  • No because the API should only be hit with messages we care about, and so should be filtered before ever propagating a POST request to the REST API

To Do

• Give rail an example of the product "mobile" showing up when I expect "firefox"
• Run with supervisord if supposed to be run in the foreground
• Read credentials from a file deploted by puppet, rather than relying on .netrc for username and password
• Make protocol configurable via puppet or with command line option
• Use selfserve-agent.py as the template for this
• Mark it as durable

Bug 1032975 - Add new table(s) to shipit database

New table(s) in update.db

Questions

• Should we have 2 tables (release_status and release_data), or just 1 (release_status)?
• Is release_status schema adequete?

To Do

• Give rail the estimate for how much data will be added to the release_data table
• Show sheeri the schema for help in optimizing columns, and to get other feedback once ready

Questions

• Can only store events with results=0 because otherwise (name, event_name) has duplicates when results!=0