Packaging Android host utilities: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Created page with "== Packaging Android host utilities == The host utilities are executable files that run on a *host* machine. The utilities provide services to an Android *target* device, in...")
 
(updating build to use (best guess) for windows)
 
(25 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== Packaging Android host utilities ==
== Packaging Android host utilities ==


The host utilities are executable files that run on a *host* machine. The utilities provide services to an Android *target* device, including a web server and certificate authority. For historical reason, the host utilities include an essentially complete version of Firefox.  Here's how to package new versions of the host utilities.
Host utilities are executable files that run on a *host* machine. These utilities provide services to an Android *target* device, such as running a web server or a certificate authority. Host utilities must be periodically updated with a good build from mozilla-central or autoland due to addition of new tests targeted at Android and/or changing of component behavior.


First, identify the target build.  Generally, prefer Aurora channel builds to Nightly channel builds.  That usually means fetching target archives from http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/
This page will provide instruction on how to package new versions of the host utilities.


=== Linux ===
=== Generate new archive ===


Preparing for Linux is dead simple. If you have shell access to <tt>people.mozilla.org</tt>, use it -- the connection to <tt>ftp.mozilla.org</tt> is lightning fast.
On Treeherder, identify a target build, preferably from mozilla-central or autoland. Check the test output to ensure a reasonably good build (one without too many failures).
 
example: for the version 66.0 update, [https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=c9fa642e9e3aecaa8852f5c647904eecddaa4450&selectedJob=216676591 this build] was selected. For more details, please see [https://bugzilla.mozilla.org/show_bug.cgi?id=1514075 bug 1514075].
 
==== Linux ====
 
For Linux, the steps are fairly straightforward. Follow these steps to locate and download the necessary files.
 
# From the selected treeherder build, identify '''Linux x64 opt'''.
# Click on the green B icon, which will bring up a pane covering bottom third of the window.
# On the bottom left pane is a header named '''Task'''. Click on the hash.
# A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".
# Download target.common.tests.tar.gz.
# Download target.tar.bz2.
# Follow the contents of the script below:


<pre>
<pre>
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-37.0a2.en-US.linux-x86_64.tar.bz2
tar xvf target.tar.bz2
tar jxf firefox-37.0a2.en-US.linux-x86_64.tar.bz2
tar xvf target.common.tests.tar.gz -C 'temp_common'
rm -rf firefox/firefox* firefox/components
rm firefox/firefox*
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-37.0a2.en-US.linux-x86_64.tests.zip
rm -r firefox/browser
unzip firefox-37.0a2.en-US.linux-x86_64.tests.zip "bin/*"
mv 'temp_common'/bin/* firefox
mv bin/* firefox
mv firefox host-utils-66.0a1.en-US.linux-x86_64
mv firefox host-utils-37.0a2.en-US.linux-x86_64
tar cvf host-utils-66.0a1.en-US.linux-x86_64.tar host-utils-66.0a1.en-US.linux-x86_64
gzip host-utils-66.0a1.en-US.linux-x86_64.tar
</pre>
</pre>


=== Mac OS X ===
Repeat for Linux 32bit, substituting x86 for x86_64 where necessary.
 
==== macOS ====
 
Preparing for macOS is not quite so simple. Due to requirements of macOS, it's best to do this on a Mac OS X machine so that you can codesign the host utility binaries. If binaries are unsigned, users will be prompted at every invocation to "allow network connections". For more information on signing macOS binaries, see [http://apple.stackexchange.com/a/150711 here].


Preparing for Mac OS X is not quite so simple. Due to changes in the way that Mac OS X codesigns binaries, it's best to do this on a Mac OS X machine so that you can codesign the host utility binaries. If you don't codesign them, you will be prompted every invocation to "allow network connections". See http://apple.stackexchange.com/a/150711 for a discussion of the issue and the codesigning invocation used below.  You will be prompted for your password (by <tt>sudo</tt>), but it's possible super user permissions are not needed.
# Navigate to [https://ftp.mozilla.org/pub/firefox/nightly/ Mozilla FTP].
# Drill down to the appropriate year, month. Select a date that is not l10n.
# Ensure files in the directory are prefixed with the version of Firefox that is desired.
# Download firefox-66.0a1.en-US.mac.common.tests.tar.gz.
# Download firefox-66.0a1.en-US.mac.dmg.
# Follow the contents of the script below:


<pre>
<pre>
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-37.0a2.en-US.mac.tests.zip
tar xvf firefox-66.0a1.en-US.mac.common.tests.tar.gz 'bin/*'
unzip firefox-37.0a2.en-US.linux-x86_64.tests.zip "bin/*"
open firefox-66.0a1.en-US.mac.dmg
wget http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-aurora/firefox-37.0a2.en-US.mac.dmg
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/MacOS/* bin
open firefox-37.0a2.en-US.mac.dmg
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/Resources/* bin
cp -R /Volumes/FirefoxDeveloperEdition/FirefoxDeveloperEdition.app/Contents/MacOS/* bin
cp -R /Volumes/FirefoxDeveloperEdition/FirefoxDeveloperEdition.app/Contents/Resources/* bin
find bin -type f -perm +111 -print | grep -v \\. | xargs sudo codesign --force --deep --sign -
find bin -type f -perm +111 -print | grep -v \\. | xargs sudo codesign --force --deep --sign -
mv bin host-utils-37.0a2.en-US.mac
mv bin host-utils-66.0a1.en-US.mac
tar cvf host-utils-66.0a1.en-US.mac.tar host-utils-66.0a1.en-US.mac/*
gzip host-utils-66.0a1.en-US.mac.tar
</pre>
 
==== Windows (experimental) ====
Similar to Linux. Follow these steps to locate and download the necessary files
 
# From the selected treeherder build, identify '''Windows 2012 debug''' ('''Windows 2012 opt''' isn't being built any longer).
# Click on the green B icon, which will bring up a pane covering bottom third of the window.
# On the bottom left pane is a header named '''Task'''. Click on the hash.
# A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".
# Download target.common.tests.tar.gz.
# Download target.zip
# Follow the contents of the script below:
 
<pre>
unzip target.zip
mkdir temp_common
tar xvf target.common.tests.tar.gz -C 'temp_common'
rm firefox/firefox*
rm -r firefox/browser
mv 'temp_common'/bin/* firefox
mv firefox host-utils-66.0a1.en-US.win32
tar cvf host-utils-66.0a1.en-US.win32.tar host-utils-66.0a1.en-US.win32
gzip host-utils-66.0a1.en-US.win32.tar
</pre>
 
==== Uploading to ToolTool ====
 
# Compare contents of current archive to the new archive. For instructions on existing archive, see the section below.
# Ensure uploading user has a valid token at [https://mozilla-releng.net/tokens/ Mozilla Releng].
# Add file to temporary manifest
<pre>
python tooltool.py add --unpack --visibility public [file]
</pre>
# Upload the archive:
<pre>
python tooltool.py upload --authentication-file=[token_location] --message [commit_message]
</pre>
# Update the manifest in testing/config/tooltool-manifests/macosx64/hostutils.manifest.
 
If updating host utilities for Linux, repeat using 32bit/x86 archives.
 
Do the same for Windows. testing/config/tooltool-manifests/win32/hostutils.manifest
 
=== Verification ===
 
ToolTool will return an updated manifest once the archives are uploaded and processed. This new manifest must first be tested, follow the steps below:
 
==== Linux ====
 
Changes to host utilities for Linux can be tested on the try server.
 
# Update to current tip by hg pull and hg update -r tip -C
# Copy and paste the updated manifest to the hostutils.manifest file for Linux 64 and Linux 32.
# Commit change and run moz-phab.
# Initiate a try run of all Android tests:
 
<pre>
# all (use this by default)
./mach try fuzzy --no-artifact -q='test-android-'
 
# only hardware
# ./mach try fuzzy --no-artifact -q=test-android-hw-
# only emulator
# ./mach try fuzzy --no-artifact -q=test-android-em-
</pre>
 
==== macOS ====
 
Changes to host utilities for macOS cannot be tested on try server, therefore the following steps are taken:
 
# Update to current tip by hg pull and hg update -r tip -C
# Copy and paste the updated manifest to the hostutils.manifest file for Mac.
# Commit change and run moz-phab.
# Ensure the currently selected MOZCONFIG designates an Android build target.
# Sanitize and rebuild with the following:
<pre>
./mach clobber
./mach build
</pre>
# Once build is complete, package the apk using ./mach package
# Install on device with ./mach install
# Run a short test with ./mach mochitest testing/mochitest/tests/Harness_sanity
# Accept offer to update host utilities.
 
If tests complete without issues, the generated host utilities are deemed to be good.
 
== Download existing archive ==
 
It is possible to download existing host utilities.
 
# Locate the existing manifest file using [https://searchfox.org/mozilla-central/search?q=hostutils.manifest&path=SearchFox].
# Using ToolTool, download the host utilities:
<pre>
python tooltool.py fetch -m old_hostutils_manifest.tt
</pre>
</pre>

Latest revision as of 22:37, 2 November 2023

Packaging Android host utilities

Host utilities are executable files that run on a *host* machine. These utilities provide services to an Android *target* device, such as running a web server or a certificate authority. Host utilities must be periodically updated with a good build from mozilla-central or autoland due to addition of new tests targeted at Android and/or changing of component behavior.

This page will provide instruction on how to package new versions of the host utilities.

Generate new archive

On Treeherder, identify a target build, preferably from mozilla-central or autoland. Check the test output to ensure a reasonably good build (one without too many failures).

example: for the version 66.0 update, this build was selected. For more details, please see bug 1514075.

Linux

For Linux, the steps are fairly straightforward. Follow these steps to locate and download the necessary files.

  1. From the selected treeherder build, identify Linux x64 opt.
  2. Click on the green B icon, which will bring up a pane covering bottom third of the window.
  3. On the bottom left pane is a header named Task. Click on the hash.
  4. A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".
  5. Download target.common.tests.tar.gz.
  6. Download target.tar.bz2.
  7. Follow the contents of the script below:
tar xvf target.tar.bz2
tar xvf target.common.tests.tar.gz -C 'temp_common'
rm firefox/firefox*
rm -r firefox/browser
mv 'temp_common'/bin/* firefox
mv firefox host-utils-66.0a1.en-US.linux-x86_64
tar cvf host-utils-66.0a1.en-US.linux-x86_64.tar host-utils-66.0a1.en-US.linux-x86_64
gzip host-utils-66.0a1.en-US.linux-x86_64.tar

Repeat for Linux 32bit, substituting x86 for x86_64 where necessary.

macOS

Preparing for macOS is not quite so simple. Due to requirements of macOS, it's best to do this on a Mac OS X machine so that you can codesign the host utility binaries. If binaries are unsigned, users will be prompted at every invocation to "allow network connections". For more information on signing macOS binaries, see here.

  1. Navigate to Mozilla FTP.
  2. Drill down to the appropriate year, month. Select a date that is not l10n.
  3. Ensure files in the directory are prefixed with the version of Firefox that is desired.
  4. Download firefox-66.0a1.en-US.mac.common.tests.tar.gz.
  5. Download firefox-66.0a1.en-US.mac.dmg.
  6. Follow the contents of the script below:
tar xvf firefox-66.0a1.en-US.mac.common.tests.tar.gz 'bin/*'
open firefox-66.0a1.en-US.mac.dmg
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/MacOS/* bin
cp -R /Volumes/Firefox\ Nightly/Firefox\ Nightly.app/Contents/Resources/* bin
find bin -type f -perm +111 -print | grep -v \\. | xargs sudo codesign --force --deep --sign -
mv bin host-utils-66.0a1.en-US.mac
tar cvf host-utils-66.0a1.en-US.mac.tar host-utils-66.0a1.en-US.mac/*
gzip host-utils-66.0a1.en-US.mac.tar

Windows (experimental)

Similar to Linux. Follow these steps to locate and download the necessary files

  1. From the selected treeherder build, identify Windows 2012 debug (Windows 2012 opt isn't being built any longer).
  2. Click on the green B icon, which will bring up a pane covering bottom third of the window.
  3. On the bottom left pane is a header named Task. Click on the hash.
  4. A new tab will open and load Taskcluster. Click on tab named "Run Artifacts".
  5. Download target.common.tests.tar.gz.
  6. Download target.zip
  7. Follow the contents of the script below:
unzip target.zip
mkdir temp_common
tar xvf target.common.tests.tar.gz -C 'temp_common'
rm firefox/firefox*
rm -r firefox/browser
mv 'temp_common'/bin/* firefox
mv firefox host-utils-66.0a1.en-US.win32
tar cvf host-utils-66.0a1.en-US.win32.tar host-utils-66.0a1.en-US.win32
gzip host-utils-66.0a1.en-US.win32.tar

Uploading to ToolTool

  1. Compare contents of current archive to the new archive. For instructions on existing archive, see the section below.
  2. Ensure uploading user has a valid token at Mozilla Releng.
  3. Add file to temporary manifest
python tooltool.py add --unpack --visibility public [file]
  1. Upload the archive:
python tooltool.py upload --authentication-file=[token_location] --message [commit_message]
  1. Update the manifest in testing/config/tooltool-manifests/macosx64/hostutils.manifest.

If updating host utilities for Linux, repeat using 32bit/x86 archives.

Do the same for Windows. testing/config/tooltool-manifests/win32/hostutils.manifest

Verification

ToolTool will return an updated manifest once the archives are uploaded and processed. This new manifest must first be tested, follow the steps below:

Linux

Changes to host utilities for Linux can be tested on the try server.

  1. Update to current tip by hg pull and hg update -r tip -C
  2. Copy and paste the updated manifest to the hostutils.manifest file for Linux 64 and Linux 32.
  3. Commit change and run moz-phab.
  4. Initiate a try run of all Android tests:
# all (use this by default)
./mach try fuzzy --no-artifact -q='test-android-'

# only hardware
# ./mach try fuzzy --no-artifact -q=test-android-hw-
# only emulator
# ./mach try fuzzy --no-artifact -q=test-android-em-

macOS

Changes to host utilities for macOS cannot be tested on try server, therefore the following steps are taken:

  1. Update to current tip by hg pull and hg update -r tip -C
  2. Copy and paste the updated manifest to the hostutils.manifest file for Mac.
  3. Commit change and run moz-phab.
  4. Ensure the currently selected MOZCONFIG designates an Android build target.
  5. Sanitize and rebuild with the following:
./mach clobber
./mach build
  1. Once build is complete, package the apk using ./mach package
  2. Install on device with ./mach install
  3. Run a short test with ./mach mochitest testing/mochitest/tests/Harness_sanity
  4. Accept offer to update host utilities.

If tests complete without issues, the generated host utilities are deemed to be good.

Download existing archive

It is possible to download existing host utilities.

  1. Locate the existing manifest file using [1].
  2. Using ToolTool, download the host utilities:
python tooltool.py fetch -m old_hostutils_manifest.tt