View source for ReleaseEngineering/Applications/Tooltool

= Tooltool basics =

Some of the jobs in the releng infrastructure make use of generic binary artifacts, which are stored in dedicated artifacts repositories.

Tooltool is the client program, written in Python, which is used to retrieve these artifacts from the servers, and to perform integrity checks on them based on hashcode verification. Hence, the servers storing these artifacts are named "tooltool servers".

A set of files to be fetched by tooltool is specified via a tooltool manifest, usually a file in JSON format with tt extension.

The following is an example of a valid tooltool manifest:

  [
  {
  "size": 139308,
  "digest": "b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4",
  "algorithm": "sha512",
  "filename": "file2.pdf"
  },
  {
  "size": 3017536,
  "digest": "630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d",
  "algorithm": "sha512",
  "filename": "file1.pdf"
  },
  {
  "size": 3420686,
  "digest": "931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4",
  "algorithm": "sha512",
  "filename": "file3.pdf"
  }
  ]

The simplest usecase for tooltool is to provide a manifest and the url of a tooltool server, and run a "fetch" command to download the files mentioned in the manifest:

  python  tooltool.py fetch --url http://tooltool.pub.build.mozilla.org/temp-sm-stuff -m my-manifest.tt

If a manifest name is not provided, tooltool will default to manifest.tt.

Tooltool will build an url for each of the mentioned files concatenating the server url, the hashing algorithm and the file digest, and try to download.

In this example, the urls will be:

  http://tooltool.pub.build.mozilla.org/temp-sm-stuff/sha512/58941214a8334331e52114aab851fc3d8d5da5dd14983f933da8735c24b0ddcac134e8f13692553199c4d9a14a4b3188b62878a30b9d696edda1204666b60837
  http://tooltool.pub.build.mozilla.org/temp-sm-stuff/sha512/b2a463249bb3a9e7f2a3604697b000d2393db4f37b623fc099beb8456fbfdb332567013a3131ad138d8633cb19c50a8b77df3990d67500af896cada8b6f698b4
  http://tooltool.pub.build.mozilla.org/temp-sm-stuff/sha512/630d01a329c70aedb66ae7118d12ff7dc6fe06223d1c27b793e1bacc0ca84dd469ec1a6050184f8d9c35a0636546b0e2e5be08d9b51285e53eb1c9f959fef59d
  http://tooltool.pub.build.mozilla.org/temp-sm-stuff/931eb84f798dc9add1a10c7bbd4cc85fe08efda26cac473411638d1f856865524a517209d4c7184d838ee542c8ebc9909dc64ef60f8653a681270ce23524e8e4

After downloading the files, their digest is verified according to the algorithm specified in the manifest, and they are finally renamed according to the filename specified in the manifest.

The tooltool download servers are simply apache server folders with a flat structure (all files are stored at root level with no subfolders).

There are global options and command arguments.  All terminal interactions after the option parser finishes is done through the Python logging API.  The default is to print logging.INFO and higher messages.  Currently, the following global options exist:

    -q/--quiet tells Tooltool to print only logging.ERROR and higher messages
    -v/--verbose specifies to print logging.INFO and higher
    -m/--manifest <file> instructs Tooltool to reference a manifest file located at the specified path
    -d/--algorithm <algorithm> instructs Tooltool to use the specified algorithm
    -o/--overwrite tells Tooltool to overwrite a local file if the filename matches the manifest but the hash value is different to the manifest
    --url specifies the base url to be used for remote operations

= Where's tooltool code? =

Tooltool's main dev repo is [https://github.com/mozilla/build-tooltool https://github.com/mozilla/build-tooltool].

The version of tooltool which is actually deployed in our infrastructure is in https://hg.mozilla.org/build/puppet/file/default/modules/packages/templates/tooltool.py. The tooltool.py file in the puppet repo should be in sync with the main dev repo (they should be identical, apart from the she-bang line).

= Listing and Validating =

The two most basic commands list a manifest and validate the local files against the manifest.  The list command lists out all of the files in the manifest as well as whether they are present and/or valid.  The return code from listing is zero unless there was an error in listing the files.  Absent or invalid files will still result in an exit code of zero if there was no error in the listing process.  The validate command is used to check if all the files in the manifest are present and valid.  The exit code for validating is zero if all files in the manifest are present and their hash matches the manifest.  It is non-zero if any file is missing locally or the file does not have the same hash as the manifest.

= Other tooltool features =

== The tooltool cache ==

This has been implemented in bug [https://bugzilla.mozilla.org/show_bug.cgi?id=858635 https://bugzilla.mozilla.org/show_bug.cgi?id=858635]

When connecting to a tooltool server to fetch a file, it is now possible to use a local cache specifying the local cache folder with the -c option, e.g.:

  python  tooltool.py fetch --url http://tooltool.pub.build.mozilla.org/temp-sm-stuff -c ~/cache

In the previous example, folder ~/cache will be inspected for artifacts before connecting to the specified tooltool server. If the specified cache folder does not exist, tooltool will create it (if the user running tooltool has permissions to do so!).
Purging the cache with the purge command

A mechanism to purge the tooltool cache has been implemented as a separate command. Let's see some examples of purge commands:

  python  tooltool.py purge -c ~/cache
  python  tooltool.py purge -c ~/cache -s 34
  python  tooltool.py purge -c ~/cache --size 34

In the first example, we are cleaning up the tooltool cache completely, i.e.: all the content will be wiped out

In the second and third example the extra parameter -s (or, in extended form, --size) is provided, which specifies the number of Gigs we want to be free at the end of the purging process. Tooltool will delete files in the cache folder, starting with the oldest ones, until the specified amount of gigs is free on disk. When there is more free space that the specified gigs when the command is invokes, no files will be deleted at all.

=== Enabling automated tooltool cache purge within purge_build.py ===

Since the purge_builds.py script (within the build-tools repo) is already responsible of clean-up operations, the option of cleaning the tooltool cache has been added to that script as well.

In order for the automated cleanup to take place, two environment variables need to be setup when purge_builds.py is invoked, and namely:

  TOOLTOOL_HOME: the folder containing tooltool.py
  TOOLTOOL_CACHE: the local folder being used as tooltool cache (which will be cleaned up)

= How Tooltool is invoked in build/test jobs =

Two wrapper scripts exist which are usually being used to invoke tooltool in build/test jobs:
* tooltool_wrapper.sh: http://hg.mozilla.org/build/tools/file/327616201e98/scripts/tooltool/tooltool_wrapper.sh
* TooltoolMixin class for mozharness calls: https://hg.mozilla.org/build/mozharness/file/default/mozharness/mozilla/tooltool.py (despite being an Homonym of tooltool itself, this is a python module defining TooltoolMixin class to be used in mozharness calls).

= How To Upload To Tooltool  =

{{note|Make sure you're using the latest tooltool.py from https://github.com/mozilla/build-tooltool}}

First, plan out your upload.  Are all of the files you need to upload public?  If not, tooltool also offers storage of "internal" files, which are not made publicly available but can be downloaded by those with proper permissions.  Internal files might be under a non-redistribution license (e.g., Google SDKs, Microsoft DLLs), but '''must not include secrets''' such as passwords or private keys.

Use the `tooltool.py` client to build a manifest containing the files you would like to upload, annotating each file with its visibility level:

    python tooltool.py add --visibility public gcc.tar.gz

Next, you will need credentials for the upload.  Tooltool is hosted on RelengAPI, so the credentials take the form of a RelengAPI token.  To get such a token, to to https://api.pub.build.mozilla.org, log in, click on "Tokens", and issue a user token with the permissions ''tooltool.upload.internal'' and/or ''tooltool.upload.public'', depending on the nature of your upload.  If you don't have internal permission, then the option won't be available to you.  When you click "Issue", you'll get a long string which you should copy/paste into a (protected) file on your system, and pass to tooltool.py with ''--authentication-file''.

Once all of that is in place, you're ready to perform the upload.  Just like for a Git or Mercurial commit, you'll need to provide a message to indicate why you're making this upload.  Including a bug reference is always helpful.

    python tooltool.py upload --authentication-file=~/.tooltool-token --message "Bug 1234567: add new frobnicator binaries"

The tool will begin uploading and, when it's finished, may wait a minute or two longer until the files are verified by RelengAPI.  If your upload fails for some reason, just restart it.  Any files already completely uploaded will not need to be uploaded again.

You can view the available files and uploads at https://api.pub.build.mozilla.org/tooltool.

= Tooltool uploads: the old school =

'''''This is the old-school, deprecated, manual way to upload artifacts to tooltool. Please use the official procedure described above whenever possible, in order to keep information about what is uploaded, by whom, and why (the manual procedure doesn't allow to store any metadata about uploads).'''''

If you don't want to upload from your own laptop (because, eg, you have a slow uplink) you can do this from cruncher.

Access cruncher with your credentials:
 ME="your_short_ldap_username" # or `whoami`
 ssh -A $ME@cruncher.build.mozilla.org
Download the file and then:
 scp filename.tar.xz $ME@relengwebadm.private.scl3.mozilla.com:
Login to relengwebadmn:
 ssh $ME@relengwebadm.private.scl3.mozilla.com
And deploy the file to tooltool:
<pre>
FILE=~/emulator.zip # or whatever you're uploading
export SHA512=`openssl sha512 $FILE | cut -d' ' -f2`
sudo mv -i $FILE /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
sudo chmod 644  /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
ls -l  /mnt/netapp/relengweb/tooltool/pvt/build/sha512/${SHA512}
</pre>

* Add the filename, filesize, and sha512 digest to the bug you are working on.  These can be added to the tooltool manifests later.