ReleaseEngineering/Applications/Tooltool

Tooltool basics

Tooltool is a client side program written in Python that uses a file manifest in concert with HTTP servers to retrieve sets of files and perform integrity checks on them based on hashcode verification. The manifests are JSON files which list details of individual files (see below). Each file is represented in the JSON by a dictionary with the keys “filename”, “digest”, “size” and “algorithm”.

A set of files to be fetched by tooltool is specified via a tooltool manifest, usually a file with tt extension.

The following is an example of a valid tooltool manifest:

 [
 {
 "size": 139308,
 "digest": "b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4",
 "algorithm": "sha512",
 "filename": "file2.pdf"
 },
 {
 "size": 3017536,
 "digest": "630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d",
 "algorithm": "sha512",
 "filename": "file1.pdf"
 },
 {
 "size": 3420686,
 "digest": "931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4",
 "algorithm": "sha512",
 "filename": "file3.pdf"
 }
 ]

The simplest usecase for tooltool is to provide a manifest and the url of a tooltool server, and run a "fetch" command to download the files mentioned in the manifest:

 python  tooltool.py fetch --url http://tooltool.pub.build.mozilla.org/temp-sm-stuff -m my-manifest.tt

If a manifest name is not provided, tooltool will default to manifest.tt.

Tooltool will build an url for each of the mentioned files concatenating the server url and the file digest, and try to download.

In this example, the urls will be:

 http://tooltool.pub.build.mozilla.org/temp-sm-stuff/58941214a8334331e52114aab851fc3d8d5da5dd14983f933da8735c24b0ddcac134e8f13692553199c4d9a14a4b3188b62878a30b9d696edda1204666b60837
 http://tooltool.pub.build.mozilla.org/temp-sm-stuff/b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4
 http://tooltool.pub.build.mozilla.org/temp-sm-stuff/630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d
 http://tooltool.pub.build.mozilla.org/temp-sm-stuff/931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4

After downloading the files, their digest is verified according to the algorithm specified in the manifest, and they are finally renamed according to the filename specified in the manifest.

The tooltool servers are simply apache server folders with a flat structure (all files are stored at root level with no subfolders).

There are global options and command arguments. All terminal interactions after the option parser finishes is done through the Python logging API. The default is to print logging.INFO and higher messages. Currently, the following global options exist:

   -q/--quiet tells Tooltool to print only logging.ERROR and higher messages
   -v/--verbose specifies to print logging.INFO and higher
   -m/--manifest <file> instructs Tooltool to reference a manifest file located at the specified path
   -d/--algorithm <algorithm> instructs Tooltool to use the specified algorithm
   -o/--overwrite tells Tooltool to overwrite a local file if the filename matches the manifest but the hash value is different to the manifest
   --url specifies the base url to be used for remote operations

Where's tooltool code?

Tooltool currently lives in https://github.com/mozilla/build-tooltool.

Listing and Validating

The two most basic commands list a manifest and validate the local files against the manifest. The list command lists out all of the files in the manifest as well as whether they are present and/or valid. The return code from listing is zero unless there was an error in listing the files. Absent or invalid files will still result in an exit code of zero if there was no error in the listing process. The validate command is used to check if all the files in the manifest are present and valid. The exit code for validating is zero if all files in the manifest are present and their hash matches the manifest. It is non-zero if any file is missing locally or the file does not have the same hash as the manifest.

How to upload to tooltool

Tooltool uploads have been traditionally managed by simply scp'ing files to the folder server by the apache instance working as tooltool download server. A new upload mechanism has been implemented to allow better tracking of what is uploaded to tooltool (see "Bug 772190 - tooltool upload mechanism").

The new upload procedure allow users to upload artifacts via rsync to a dedicated upload server, contextually providing some metadata about the upload. The artifacts uploaded to the upload servers will then be distributed to the relevant tooltool download servers (and their mirrors, if any) by a separate sync script which is cron'd on the upload server itself.

Preliminary notes:

• The server used for uploads (AKA tooltool upload server) is tooltool-uploads.pub.build.mozilla.org (no VPN required)
• Currently, artifacts uploaded to the tooltool upload server can be made visible to two tooltool download servers:

• Public tooltool download server: http://tooltool.pub.build.mozilla.org/temp-sm-stuff/sha512/

• all uploads to /tooltool/uploads/user/pub on the upload server (more details below) will be visible here

• Private tooltool download server http://tooltool.pvt.build.mozilla.org/tooltool/sha512 AKA http://runtime-binaries.pvt.build.mozilla.org/tooltool/sha512 AKA https://secure.pub.build.mozilla.org/tooltool/pvt/build/sha512/

• all uploads to /tooltool/uploads/user/pvt on the upload server (more details below) will be visible here

Pre-requisites to run an upload

• You need a unix account on the tooltool upload server, and an upload folder for each visibility level currently two visibility levels are supported (pub and pvt), so what is needed is folders /tooltool/uploads/$user/pub and /tooltool/uploads/$user/pvt
• You need ssh properly configured to access the tooltool upload server
• You need rsync
• You need tooltool! https://github.com/mozilla/build-tooltool/blob/master/tooltool.py

How to request a tooltool upload account

Request addition to "vpn_tooltooleditors" group, via "Infrastruture & Operations::Infrastructure:LDAP". Once that's done, it's an hour or two for the change to propagate from LDAP to Puppet to the upload servers. They should request an ACK on the bug from someone in release engineering for that.

Let's update a bunch of files!

• Create a local folder (name it as you like) and put in it all the files you want to upload
• Run the tooltool distribute command (see example below)

That's it!

The distribute command will take care of

• determining the sha512 checksum of all the files
• creating a tooltool manifest file for those files, which will be included in a newly created folder with the same name as the upload folder plus the .TOOLTOOL_PACKAGE suffix
• transferring the files to the tooltool upload server using rsync

Example:

 # Folder /Users/sbruno/upload contains file1.tar.gz file2.tar.gz and file3.tar.gz
 tooltool.py distribute \
 --folder /Users/$user/upload \
 --message "Bug 123456 - artifacts needed for this and that" \
 --user $user \
 --host tooltool-uploads.pub.build.mozilla.org \
 --path "/tooltool/uploads/$user/pub"

The parameters are:

 --folder: the folder containing the files you want to upload
 --message: any comment you may want to write about this upload
 --user: the user to access the tooltool upload server
 --host: the upload server host or IP
 --path: the target folder on the upload server

Use /tooltool/uploads/<your_username>/pub to upload to the public download server Use /tooltool/uploads/<your_username>/pvt to upload to the private download server

What now?

You should receive in a few minutes an email confirming that your upload has been correctly processed on the upload server, and the files should appear in the relevant download server with their correct digest (and in all mirrors, if any, setup for the same visibility level).

The default email address used for these notifications is user@mozilla.com, where user is the user you use to access the upload server.

All metadata about the upload is stored in the upload server.

See ReleaseEngineering:Buildduty:Other Duties#How to upload to Tooltool

Need to download files from it?

If you want to download a file from tooltool and you're not part of ReleaseEngineering then add your name to bug 1019011.

That should grant you access to https://secure.pub.build.mozilla.org/tooltool/pvt/build