Partnering:Projects:BYOB:PilotBuildQueueDetails: Difference between revisions
Jump to navigation
Jump to search
LesOrchard (talk | contribs) (Created page with '== Overview == This is a summary of the moving parts involved in the PHP/MySQL based build queue in use for the pilot launch of BYOB. __TOC__ == Repack states == The workflow …') |
LesOrchard (talk | contribs) |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 7: | Line 7: | ||
== Repack states == | == Repack states == | ||
The workflow status of a repack is state machine, where each state describes what actions are currently available and to which states it may transition next. | The workflow status of a repack is state machine, where each state describes | ||
what actions are currently available and to which states it may transition | |||
next. A history log of these state transitions is recorded in the repack | |||
data, along with comments. | |||
{| border="1" cellpadding="4" width="100%" | {| border="1" cellpadding="4" width="100%" | ||
| Line 18: | Line 21: | ||
| requested || user has requested a build and review || cancelled; started; failed; pending | | requested || user has requested a build and review || cancelled; started; failed; pending | ||
|- | |- | ||
| cancelled || user has cancelled a repack build || edited; requested; deleted | | cancelled || user has cancelled a repack build and review request || edited; requested; deleted | ||
|- | |- | ||
| started || build has started || failed; pending | | started || build has started || failed; pending | ||
| Line 26: | Line 29: | ||
| pending || build completed, review pending || cancelled; released; rejected | | pending || build completed, review pending || cancelled; released; rejected | ||
|- | |- | ||
| rejected || build rejected | | rejected || build rejected for public release || edited; requested; deleted | ||
|- | |- | ||
| released || build | | released || build approved for public release || reverted | ||
|- | |- | ||
| reverted || public release of build taken down || edited; requested; deleted | | reverted || public release of build taken down || edited; requested; deleted | ||
| Line 41: | Line 44: | ||
* Repacks get copied on edit as an unreleased version when in these states: | * Repacks get copied on edit as an unreleased version when in these states: | ||
** released | ** released | ||
== Build queue tasks in BYOB == | == Build queue tasks in BYOB == | ||
In reaction to certain state changes, build tasks get enqueued into a PHP/MySQL | In reaction to certain state changes, build tasks get enqueued into a PHP/MySQL | ||
based offline work queue that gets exhausted every 5 minutes or so. The | based offline work queue that gets exhausted at least every 5 minutes or so. The | ||
current build queue tasks are: | current build queue tasks are: | ||
| Line 66: | Line 54: | ||
* Enqueue triggered by repack transitioning to '''requested''' state. | * Enqueue triggered by repack transitioning to '''requested''' state. | ||
* Fire up the repack build script | * Process: | ||
* Move the built repack to private review hosting | ** Fire up the repack build script | ||
* If successful, transition the repack to '''pending''' state. | ** Move the built repack to private review hosting | ||
* If failed, transition the repack to '''failed''' state. | * Result: | ||
** If successful, transition the repack to '''pending''' state. | |||
** If failed, transition the repack to '''failed''' state. | |||
=== Release build === | === Release build === | ||
* Enqueue triggered by repack transitioning to '''released''' state. | * Enqueue triggered by repack transitioning to '''released''' state. | ||
* Move a built repack from private review to public release hosting | * Process: | ||
** Move a built repack from private review to public release hosting | |||
* Result: | |||
** In any case, repack remains in '''released''' state. | |||
** (No failure case currently implemented.) | |||
=== Delete build === | === Delete build === | ||
* Enqueue triggered by repack transitioning to '''deleted''' state. | * Enqueue triggered by repack transitioning to '''deleted''' state. | ||
* Delete a built repack from both private and public hosting | * Process: | ||
** Delete a built repack from both private and public hosting | |||
* Result: | |||
** In any case, the repack data has been deleted from the BYOB database. | |||
** (No failure case currently implemented.) | |||
== Build queue implementation details == | |||
The build queue consists of: | |||
* [https://svn.mozilla.org/projects/byob/trunk/modules/messagequeue/config/schema-mysql.sql a MySQL table listing tasks to be performed]; | |||
* [https://svn.mozilla.org/projects/byob/trunk/bin/exhaust-queue.sh-dist a shell script] that runs in a crontab or [https://svn.mozilla.org/projects/byob/trunk/bin/org.mozilla.byob.cron.plist-dist as an OS X launchd scheduled task]; | |||
** every 5 minutes or so, it sets a lock to prevent concurrent runs and exhausts the queue | |||
* [https://svn.mozilla.org/projects/byob/trunk/modules/messagequeue/controllers/messagequeue.php a PHP controller that processes the queue], using [https://svn.mozilla.org/projects/byob/trunk/modules/messagequeue/models/messagequeue.php a model library] to perform the work. | |||
The web app uses this build queue by: | |||
# [https://svn.mozilla.org/projects/byob/trunk/application/hooks/builds.php Registering DeferredEvent callback handlers with unique IDs] - these do the actual work | |||
# [http://viewvc.svn.mozilla.org/vc/projects/byob/trunk/application/models/repack.php?revision=66815&view=markup#l882 Firing Events] that correspond with [http://viewvc.svn.mozilla.org/vc/projects/byob/trunk/application/hooks/builds.php?revision=66819&view=markup#l14 registered DeferredEvent handlers] | |||
# Firing Events results in enqueued tasks, which the queue processor handles via calls to registered DeferredEvent callbacks. | |||
Latest revision as of 17:22, 7 May 2010
Overview
This is a summary of the moving parts involved in the PHP/MySQL based build queue in use for the pilot launch of BYOB.
Repack states
The workflow status of a repack is state machine, where each state describes what actions are currently available and to which states it may transition next. A history log of these state transitions is recorded in the repack data, along with comments.
| state | description | next states |
|---|---|---|
| new | freshly created repack | edited; requested; deleted |
| edited | repack with changes | requested; deleted |
| requested | user has requested a build and review | cancelled; started; failed; pending |
| cancelled | user has cancelled a repack build and review request | edited; requested; deleted |
| started | build has started | failed; pending |
| failed | build failed for some reason | edited; requested; deleted |
| pending | build completed, review pending | cancelled; released; rejected |
| rejected | build rejected for public release | edited; requested; deleted |
| released | build approved for public release | reverted |
| reverted | public release of build taken down | edited; requested; deleted |
| deleted | repack to be deleted from system | -- |
Notes
- Repacks are locked as read-only (ie. editing via wizard is disabled) when in these states:
- requested; started; pending; approved
- Repacks get copied on edit as an unreleased version when in these states:
- released
Build queue tasks in BYOB
In reaction to certain state changes, build tasks get enqueued into a PHP/MySQL based offline work queue that gets exhausted at least every 5 minutes or so. The current build queue tasks are:
Perform build
- Enqueue triggered by repack transitioning to requested state.
- Process:
- Fire up the repack build script
- Move the built repack to private review hosting
- Result:
- If successful, transition the repack to pending state.
- If failed, transition the repack to failed state.
Release build
- Enqueue triggered by repack transitioning to released state.
- Process:
- Move a built repack from private review to public release hosting
- Result:
- In any case, repack remains in released state.
- (No failure case currently implemented.)
Delete build
- Enqueue triggered by repack transitioning to deleted state.
- Process:
- Delete a built repack from both private and public hosting
- Result:
- In any case, the repack data has been deleted from the BYOB database.
- (No failure case currently implemented.)
Build queue implementation details
The build queue consists of:
- a MySQL table listing tasks to be performed;
- a shell script that runs in a crontab or as an OS X launchd scheduled task;
- every 5 minutes or so, it sets a lock to prevent concurrent runs and exhausts the queue
- a PHP controller that processes the queue, using a model library to perform the work.
The web app uses this build queue by:
- Registering DeferredEvent callback handlers with unique IDs - these do the actual work
- Firing Events that correspond with registered DeferredEvent handlers
- Firing Events results in enqueued tasks, which the queue processor handles via calls to registered DeferredEvent callbacks.