View source for Packaging Android host utilities

== Packaging Android host utilities ==

Host utilities are executable files that run on a *host* machine. These utilities provide services to an Android *target* device, such as running a web server or a certificate authority. Host utilities must be periodically updated with a good build from mozilla-central or autoland due to addition of new tests targeted at Android and/or changing of component behavior.

This page will provide instruction on how to package new versions of the host utilities.

=== Generate new archive ===

On Treeherder, identify a target build, preferably from mozilla-central or autoland. Check the test output to ensure a reasonably good build (one without too many failures).

example: for the version 66.0 update, [https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=c9fa642e9e3aecaa8852f5c647904eecddaa4450&selectedJob=216676591 this build] was selected. For more details, please see [https://bugzilla.mozilla.org/show_bug.cgi?id=1514075 bug 1514075].

==== Linux ====

For Linux, the steps are fairly straightforward. Follow these steps to locate and download the necessary files.

# From the selected treeherder build, identify '''Linux x64 opt'''.
# Click on the green B icon, which will bring up a pane covering bottom third of the window.
# On the bottom left pane is a header named '''Task'''. Click on the hash.
# A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".
# Download target.common.tests.tar.gz.
# Download target.tar.bz2.
# Follow the contents of the script below:

<pre>
tar xvf target.tar.bz2
tar xvf target.common.tests.tar.gz -C 'temp_common'
rm firefox/firefox*
rm -r firefox/browser
mv 'temp_common'/bin/* firefox
mv firefox host-utils-66.0a1.en-US.linux-x86_64
tar cvf host-utils-66.0a1.en-US.linux-x86_64.tar host-utils-66.0a1.en-US.linux-x86_64
gzip host-utils-66.0a1.en-US.linux-x86_64.tar
</pre>

Repeat for Linux 32bit, substituting x86 for x86_64 where necessary.

==== macOS ====

Preparing for macOS is not quite so simple. Due to requirements of macOS, it's best to do this on a Mac OS X machine so that you can codesign the host utility binaries. If binaries are unsigned, users will be prompted at every invocation to "allow network connections". For more information on signing macOS binaries, see [http://apple.stackexchange.com/a/150711 here].

# Navigate to [https://ftp.mozilla.org/pub/firefox/nightly/ Mozilla FTP].
# Drill down to the appropriate year, month. Select a date that is not l10n.
# Ensure files in the directory are prefixed with the version of Firefox that is desired.
# Download firefox-66.0a1.en-US.mac.common.tests.tar.gz.
# Download firefox-66.0a1.en-US.mac.dmg.
# Follow the contents of the script below:

<pre>
tar xvf firefox-66.0a1.en-US.mac.common.tests.tar.gz 'bin/*'
open firefox-66.0a1.en-US.mac.dmg
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/MacOS/* bin
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/Resources/* bin
find bin -type f -perm +111 -print | grep -v \\. | xargs sudo codesign --force --deep --sign -
mv bin host-utils-66.0a1.en-US.mac
tar cvf host-utils-66.0a1.en-US.mac.tar host-utils-66.0a1.en-US.mac/*
gzip host-utils-66.0a1.en-US.mac.tar
</pre>

==== Windows (experimental) ====
Similar to Linux. Follow these steps to locate and download the necessary files

# From the selected treeherder build, identify '''Windows 2012 debug''' ('''Windows 2012 opt''' isn't being built any longer).
# Click on the green B icon, which will bring up a pane covering bottom third of the window.
# On the bottom left pane is a header named '''Task'''. Click on the hash.
# A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".
# Download target.common.tests.tar.gz.
# Download target.zip
# Follow the contents of the script below:

<pre>
unzip target.zip
mkdir temp_common
tar xvf target.common.tests.tar.gz -C 'temp_common'
rm firefox/firefox*
rm -r firefox/browser
mv 'temp_common'/bin/* firefox
mv firefox host-utils-66.0a1.en-US.win32
tar cvf host-utils-66.0a1.en-US.win32.tar host-utils-66.0a1.en-US.win32
gzip host-utils-66.0a1.en-US.win32.tar
</pre>

==== Uploading to ToolTool ====

# Compare contents of current archive to the new archive. For instructions on existing archive, see the section below.
# Ensure uploading user has a valid token at [https://mozilla-releng.net/tokens/ Mozilla Releng].
# Add file to temporary manifest
<pre>
python tooltool.py add --unpack --visibility public [file]
</pre>
# Upload the archive:
<pre>
python tooltool.py upload --authentication-file=[token_location] --message [commit_message]
</pre>
# Update the manifest in testing/config/tooltool-manifests/macosx64/hostutils.manifest.

If updating host utilities for Linux, repeat using 32bit/x86 archives.

Do the same for Windows. testing/config/tooltool-manifests/win32/hostutils.manifest

=== Verification ===

ToolTool will return an updated manifest once the archives are uploaded and processed. This new manifest must first be tested, follow the steps below:

==== Linux ====

Changes to host utilities for Linux can be tested on the try server.

# Update to current tip by hg pull and hg update -r tip -C
# Copy and paste the updated manifest to the hostutils.manifest file for Linux 64 and Linux 32.
# Commit change and run moz-phab.
# Initiate a try run of all Android tests:

<pre>
# all (use this by default)
./mach try fuzzy --no-artifact -q='test-android-'

# only hardware
# ./mach try fuzzy --no-artifact -q=test-android-hw-
# only emulator
# ./mach try fuzzy --no-artifact -q=test-android-em-
</pre>

==== macOS ====

Changes to host utilities for macOS cannot be tested on try server, therefore the following steps are taken:

# Update to current tip by hg pull and hg update -r tip -C
# Copy and paste the updated manifest to the hostutils.manifest file for Mac.
# Commit change and run moz-phab.
# Ensure the currently selected MOZCONFIG designates an Android build target. 
# Sanitize and rebuild with the following:
<pre>
./mach clobber
./mach build
</pre>
# Once build is complete, package the apk using ./mach package
# Install on device with ./mach install
# Run a short test with ./mach mochitest testing/mochitest/tests/Harness_sanity
# Accept offer to update host utilities.

If tests complete without issues, the generated host utilities are deemed to be good.

== Download existing archive ==

It is possible to download existing host utilities.

# Locate the existing manifest file using [https://searchfox.org/mozilla-central/search?q=hostutils.manifest&path=SearchFox].
# Using ToolTool, download the host utilities:
<pre>
python tooltool.py fetch -m old_hostutils_manifest.tt
</pre>