Software Update:Nightly update infrastructure: Difference between revisions
m (bug 337910 added mapping for 2.0a3) |
(fix up example, tweaks) |
||
| Line 1: | Line 1: | ||
=Overview= | =Overview= | ||
Updates are available for nightly builds of Firefox & Thunderbird | Updates are available for nightly builds of Firefox & Thunderbird. Keeping this system running requires the coordination of tinderbox configuration, partial update generation, and the AUS server that provides update information the client applications. This document describes how that all works, to the extent that publicly available information allows. Hopefully the build crew and Mike Morgan can fill out the details to make this page more complete. | ||
Scope: Currently, updates are setup for the en-US locale on the branches for 1.5.x.y and 2.0, and on the trunk. The three Tier-1 platforms of Windows, Linux and Macintosh (ppc) are supported for the two main toolkit apps, Firefox & Thunderbird. | |||
=Tinderbox= | =Tinderbox= | ||
| Line 31: | Line 31: | ||
| Variable || Example value || Use | | Variable || Example value || Use | ||
|- | |- | ||
| $update_package || 1 || Whether to create a complete update | | $update_package || 1 || Whether to create a complete update (other conditions apply too) | ||
|- | |- | ||
| $update_pushinfo || 1 || Whether to push the update snippet to AUS | | $update_pushinfo || 1 || Whether to push the update snippet to AUS | ||
| Line 50: | Line 50: | ||
|} | |} | ||
The examples reflect current real world values (**CHECK**) rather than | The examples reflect current real world values (**CHECK**) rather than those in tinder-defaults.pl. Aside: [https://bugzilla.mozilla.org/show_bug.cgi?id=334011 Bug 334011] aims to avoid the need to manually bump $update_appv and $update_extv each time the app versions increases. | ||
The snippet is scp'ed to | The snippet is scp'ed to | ||
| Line 56: | Line 56: | ||
$update_version/$update_platform/$buildid/$locale/complete.txt | $update_version/$update_platform/$buildid/$locale/complete.txt | ||
A complete-snippet for a Firefox build from the Mozilla1.8 branch looks like: | |||
complete | complete | ||
http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-05-10-02-mozilla1.8/firefox-2.0a2.en-US.win32.mar | http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-05-10-02-mozilla1.8/firefox-2.0a2.en-US.win32.mar | ||
| Line 68: | Line 68: | ||
And here is the relevant part of the tinderbox log (from almost at the very end): | And here is the relevant part of the tinderbox log (from almost at the very end): | ||
Gathering complete update info... | Gathering complete update info... | ||
Got build ID | Got build ID 2006051002. | ||
Not pushing first-gen update info... | Not pushing first-gen update info... | ||
Pushing third-gen update info... | Pushing third-gen update info... | ||
ssh -i /home/cltbld/.ssh/aus cltbld@aus-staging.mozilla.org mkdir -p /opt/aus2/build/0/Firefox/2.0/WINNT_x86-msvc/ | ssh -i /home/cltbld/.ssh/aus cltbld@aus-staging.mozilla.org mkdir -p /opt/aus2/build/0/Firefox/2.0/WINNT_x86-msvc/2006051002/en-US | ||
scp -i /home/cltbld/.ssh/aus | scp -i /home/cltbld/.ssh/aus /cygdrive/c/builds/tinderbox/Fx-Mozilla1.8/WINNT_5.2_Depend/mozilla/dist/update/update.snippet.1 cltbld@aus-staging.mozilla.org:/opt/aus2/build/0/Firefox/2.0/WINNT_x86-msvc/2006051002/en-US/complete.txt | ||
Completed pushing update info... | Completed pushing update info... | ||
| Line 86: | Line 86: | ||
Not much information is available, apart from | Not much information is available, apart from | ||
* the work is done on Prometheus, using some sort of cron job | * the work is done on Prometheus, using some sort of cron job | ||
* it pushes partial updates to the FTP server, and presumably a partial snippet to AUS | * it pushes partial updates to the FTP server (to the from path of any from-->to update), and presumably a partial snippet to AUS | ||
* the code for partial generation lives in [http://lxr.mozilla.org/mozilla/source/tools/update-packaging/ mozilla/tools/update-packaging] but also uses mbsdiff and mar. | * the code for partial generation lives in [http://lxr.mozilla.org/mozilla/source/tools/update-packaging/ mozilla/tools/update-packaging] but also uses mbsdiff and mar. | ||
* Chase Phillips wrote the code; Paul Reed (preed) is a current build engineer who knows how to drive it | * Chase Phillips wrote the code; Paul Reed (preed) is a current build engineer who knows how to drive it | ||
| Line 92: | Line 92: | ||
=AUS= | =AUS= | ||
This server returns update information to the client, based on the snippets it has from the build systems. It was written in PHP by Mike Morgan (morgamic) | This server returns update information to the client, based on the snippets it has from the build systems. It was written in PHP by Mike Morgan (morgamic). | ||
==Application update request== | ==Application update request== | ||
The application makes a request to the url specified by the preference app.update.url ( | The application makes a request to the url specified by the global preference app.update.url (unless app.update.url.override is defined), which is: | ||
https://aus2.mozilla.org/update/1/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/update.xml | https://aus2.mozilla.org/update/1/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/update.xml | ||
This is preprocessed to make the required substitutions. Continuing the example above of a Firefox Mozilla1.8 branch nightly, this would be: | This is preprocessed to make the required substitutions. Continuing the example above of a Firefox Mozilla1.8 branch nightly, this would be: | ||
| Line 118: | Line 118: | ||
==AUS itself== | ==AUS itself== | ||
To provide this content, AUS maps the client version (%VERSION%) from the request to $update_version of snippet store. This was the mapping on May | To provide this content, AUS maps the client version (%VERSION%) from the request to $update_version of snippet store. This was the mapping on May 15 2006: | ||
'1.0+', '1.4', '1.4.1', '1.5' => '1.5', | '1.0+', '1.4', '1.4.1', '1.5' => '1.5', | ||
'1.5.0.1' => '1.5.0.1', | '1.5.0.1' => '1.5.0.1', | ||
| Line 128: | Line 128: | ||
(This is a more compact form of the actual definition, each mapping should be on it's own line) | (This is a more compact form of the actual definition, each mapping should be on it's own line) | ||
So there is a many to one relationship between versions from a | So there is a many to one relationship between application versions from a code branch to an AUS disk location. Eg: The Mozilla1.8 branch is being used for developement of v2.0 and will have versions 2.0a1, a2, a3, b1,b2, RC1, ... and finally 2.0. The situation with the Mozilla1.8.0 branch needs fixing up (partly [https://bugzilla.mozilla.org/show_bug.cgi?id=328497 Bug 328497]), an $update_version of "1.5.x.y" would be flexible. | ||
Revision as of 16:03, 15 May 2006
Overview
Updates are available for nightly builds of Firefox & Thunderbird. Keeping this system running requires the coordination of tinderbox configuration, partial update generation, and the AUS server that provides update information the client applications. This document describes how that all works, to the extent that publicly available information allows. Hopefully the build crew and Mike Morgan can fill out the details to make this page more complete.
Scope: Currently, updates are setup for the en-US locale on the branches for 1.5.x.y and 2.0, and on the trunk. The three Tier-1 platforms of Windows, Linux and Macintosh (ppc) are supported for the two main toolkit apps, Firefox & Thunderbird.
Tinderbox
The tinderbox produces the nightly build & mar file for a complete update, and publishes them to the FTP. It also creates a complete-update snippet, which describes the update, and copies that to AUS.
mozconfig
The update channel is set to "default" (no updates) unless this line is in the mozconfig:
ac_add_options --enable-update-channel=nightly
For 1.5 the beta releases shipped with a "beta" channel, while the release candidates and final shipped with "release".
To pull the code for complete update generation you need:
mk_add_options MOZ_CO_MODULE="mozilla/tools/update-packaging"
This line is also used
ac_add_options --enable-update-packaging
but it's not obvious why, because mbsdiff (the binary diff executable) is not required for a complete update, only mozilla/tools/update-packaging/{make_full_update.sh,common.sh}. (**CHECK**: Depends on trunk/branch ?)
The mar executable (mozilla/modules/libmar) is built by default.
Snippet configuration
A handful of variables in tinder-config.pl control the update and snippet generation on the tinderbox. The code is mozilla/tools/post-mozilla-rel.pl::update_create_package and runs after the successful completion of the build and any tests.
| Variable | Example value | Use |
| $update_package | 1 | Whether to create a complete update (other conditions apply too) |
| $update_pushinfo | 1 | Whether to push the update snippet to AUS |
| $update_product | "Firefox" | Product identifier for AUS |
| $update_version | "2.0" | "Branch" identifier for AUS |
| $update_platform | "WINNT_x86-msvc" | Platform ABI for AUS |
| $update_hash | "SHA1" | Hash function for download verfication |
| $update_filehost | "mozilla.osuosl.org" | The server that provides the mar files to the client |
| $update_appv | "2.0a2" | Version that appears in the update notification dialog |
| $update_extv | "2.0a2" | Version used to check extension compatibility (this seems to not work properly all the time) |
The examples reflect current real world values (**CHECK**) rather than those in tinder-defaults.pl. Aside: Bug 334011 aims to avoid the need to manually bump $update_appv and $update_extv each time the app versions increases.
The snippet is scp'ed to
aus-staging.mozilla.org:/opt/aus2/incoming/0/$update_product/... $update_version/$update_platform/$buildid/$locale/complete.txt
A complete-snippet for a Firefox build from the Mozilla1.8 branch looks like:
complete http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-05-10-02-mozilla1.8/firefox-2.0a2.en-US.win32.mar SHA1 202f8c4a5c6af9bd00569ed087e245dec87d67fb 6925417 2006051002 2.0a2 2.0a2
And here is the relevant part of the tinderbox log (from almost at the very end):
Gathering complete update info... Got build ID 2006051002. Not pushing first-gen update info... Pushing third-gen update info... ssh -i /home/cltbld/.ssh/aus cltbld@aus-staging.mozilla.org mkdir -p /opt/aus2/build/0/Firefox/2.0/WINNT_x86-msvc/2006051002/en-US scp -i /home/cltbld/.ssh/aus /cygdrive/c/builds/tinderbox/Fx-Mozilla1.8/WINNT_5.2_Depend/mozilla/dist/update/update.snippet.1 cltbld@aus-staging.mozilla.org:/opt/aus2/build/0/Firefox/2.0/WINNT_x86-msvc/2006051002/en-US/complete.txt Completed pushing update info... Update build completed.
The section "Pushing third-gen update info..." is missing if $update_pushinfo is not set.
Partial Update Generation
Not much information is available, apart from
- the work is done on Prometheus, using some sort of cron job
- it pushes partial updates to the FTP server (to the from path of any from-->to update), and presumably a partial snippet to AUS
- the code for partial generation lives in mozilla/tools/update-packaging but also uses mbsdiff and mar.
- Chase Phillips wrote the code; Paul Reed (preed) is a current build engineer who knows how to drive it
AUS
This server returns update information to the client, based on the snippets it has from the build systems. It was written in PHP by Mike Morgan (morgamic).
Application update request
The application makes a request to the url specified by the global preference app.update.url (unless app.update.url.override is defined), which is:
https://aus2.mozilla.org/update/1/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/update.xml
This is preprocessed to make the required substitutions. Continuing the example above of a Firefox Mozilla1.8 branch nightly, this would be:
https://aus2.mozilla.org/update/1/Firefox/2.0a2/2006050905/WINNT_x86-msvc/en-US/nightly/update.xml
which returns
<updates> <update type="minor" version="2.0a2" extensionVersion="2.0a2" buildID="2006051002"> <patch type="complete" URL="http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-05-10-02-mozilla1.8/firefox-2.0a2.en-US.win32.mar" hashFunction="SHA1" hashValue="202f8c4a5c6af9bd00569ed087e245dec87d67fb" size="6925417"/> <patch type="partial" URL="http://mozilla.osuosl.org/pub/mozilla.org/firefox/nightly/2006-05-09-05-mozilla1.8/firefox-2.0a2.en-US.win32.partial.2006050905-2006051002.mar" hashFunction="SHA1" hashValue="6e277c58c232f021b034d595ce72fc240f011dab" size="525151"/> </update> </updates>
Other %BUILD_TARGET%/$update_platform combinations are
- Linux: Linux_x86-gcc3
- Mac Universal: Darwin_Universal-gcc3
- Mac PPC: Darwin_ppc-gcc3
AUS itself
To provide this content, AUS maps the client version (%VERSION%) from the request to $update_version of snippet store. This was the mapping on May 15 2006:
'1.0+', '1.4', '1.4.1', '1.5' => '1.5', '1.5.0.1' => '1.5.0.1', '1.5.0.2' => '1.5.0.2', '1.5.0.3' => '1.5.0.3', '1.5.0.4' => '1.5.0.4', '2.0', '2.0a1', '2.0a2', '2.0a3', '2.0b1', '2.0b2' => '2.0', '1.6a1', '3.0a1' => 'trunk'
(This is a more compact form of the actual definition, each mapping should be on it's own line)
So there is a many to one relationship between application versions from a code branch to an AUS disk location. Eg: The Mozilla1.8 branch is being used for developement of v2.0 and will have versions 2.0a1, a2, a3, b1,b2, RC1, ... and finally 2.0. The situation with the Mozilla1.8.0 branch needs fixing up (partly Bug 328497), an $update_version of "1.5.x.y" would be flexible.