ReleaseEngineering/Platforms/Android

Introduction

Mozilla releases a version of Firefox for Android. Mozilla's Release Engineering team support this initiative with continuous integration systems. This page explains what is involved on all the pieces related to these systems.

Currently, we build on EC2 instances (bld-linux64-ec2-XXX machines) and in-house machines (bld-linux64-ix-XXX machines). We create builds for three different architectures: Android API9, Android API10 and Android 4.2 x86. We run unit tests and performance test differently for each build. You can see all jobs running in [1].

Builds & tests

Armv7 API 9

For this architecture we build both optimized and debug builds. NOTE: We don't yet run tests on the debug builds.

More info needed as to how we build this.

Android 2.3 API 9

Jobs: unit tests tests
Setup configuration: Emulator (single-core)
Hosts: EC2 Ubuntu 64-bit instances (tst-linux64-spot-XXX or tst-linux64-ec2-XXX).
Scripts: android_emulator_unittest.py
Configs: androidarm.py
Artifacts: SUT agent, watcher, Android emulator, Android SDK, AVD definitions

Armv7 API 10

For this architecture we build both optimized and debug builds. NOTE: We don't yet run tests on the debug builds.

More info needed as to how we build this.

Android 4.0 API 10

Jobs: unit tests' and performance tests
Setup configuration: Foopies-board setup and fixed allocation (no mozpool)
Hosts: PandaES development boards.
Scripts: android_panda.py & android_panda_talos.py
Configs: android_panda_releng.py and android_panda_talos_releng.py
Artifacts: SUT agent, watcher

Android x86

More info needed as to how we build this.

Android 4.2

Jobs: unit tests
Setup configuration: Emulator parralellization
Hosts: In-house Ubuntu 64-bit hosts (talos-linux64-ix-XXX).
Scripts: android_emulator_unittest.py
Configs: androidarm.py Artifacts: SUT agent, watcher, Android emulator, Android SDK, AVD definitions

Older builds/tests

Armv6

More info needed as to how we build this.

Android 2.2

Jobs: unit tests tests
Setup configuration: Foopies-board setup and fixed allocation (no mozpool).
Hosts: Tegra boards (tegra-XXX)
Scripts: factory.py
Configs: android_panda_releng.py and android_panda_talos_releng.py
Artifacts: SUT agent, watcher

Armv7

Android 2.2

Jobs: unit tests tests
Setup configuration: Foopies-board setup and fixed allocation (no mozpool).
Hosts: Tegra boards (tegra-XXX)
Scripts: factory.py
Configs: android_panda_releng.py and android_panda_talos_releng.py
Artifacts: SUT agent, watcher

Running locally

TBD

Testing setup configurations

All of our Mobile testing is driven by Mozharness code to the exception of Tegras, which run code from factory.py

Foopy-Board setup

TODO: document

Mozpool allocation

Mozharness uses MozpoolClient to talk to the Mozpool server stack which IT wrote to manage & allocate the Panda boards.

Emulator parallelization

The Android x86 jobs run inside Android emulators. Android 2.3 only triggers one emulator since it runs on single-core machines. Mozharness starts up Android emulators for each different test suite. We can run up to four different test jobs on the same host cutting the load on the pool.

Each emulator starts up with a specific AVD definition that already has the SUT agent and Watcher installed. These AVD definitions are stored on a tar ball, which we store on ToolTool. We download the AVD definitions to the host if it is not already cached. We untar it on *every* job to have a clean state for the emulators to start from. TODO: explain what AVD definitions are. We deploy the Android SDK 18 (TODO add link) with puppet (TODO add link) before the machine starts a job.

Artifacts

Android emulator

We have emulators for Android 2.3 (un-patched) and Android x86.

The emulator that ships in stock SDKs downloaded from AOSP, at http://developer.android.com/sdk/ , contains the newest (and most-debugged) implementation of OpenGL host rendering and ARM emulation (from ARMv5TE through ARMv7A). We therefore use stock emulators.

For Android x86, we patched the AOSP "tools_r22" emulator. The original problem was documented in bug 917562, and we distributed new emulator binaries in bug 933918.

AVD definition

AVDs ("Android Virtual Devices") are directories full of disk images and config files stored in a user's AVD directory, typically ~/.android/avd/avdname.avd

Each running emulator process during testing will have a separate AVD associated with it. When the emulator is invoked, you must pass the AVD name on the command line, such as emulator -avd test-1. Typically we make 4 AVDs called test-1, test-2, test-3, and test-4; then execute 4 emulator processes on the test host, each running a quarter of the testsuite.

Android SDKs do not ship with AVDs, but do ship with sufficient pieces to instantiate them easily. Unfortunately the AVDs that the SDK creates are based on disk images that contain either an ARMv5TE userspace or an x86 userspace. The SDK does not ship with disk images for an ARMv7a userspace, which is what we need to run ARMv6 and ARMv7a Firefox binaries. So we construct our own from AOSP source.

NOTE: AVDs do not fully self-identify their (sub-)architecture. In particular, when running an ARM AVD, one must guess at the correct CPU model to pass to the emulator command, as a startup argument to qemu. For ARMv7, typically cortex-a8 or cortex-a9 works. So the emulator should be invoked as emulator -avd test-1 -qemu -cpu cortex-a8

NOTE: OpenGL is emulated by serializing the GL command stream out of the emulator and applying it to host-side GL libraries, either software rendering or hardware. So it tends to run quite quickly, compared to the (emulated) CPU instructions. Unfortunately this host-GL-emulation technology was only developed in the Android 4.x SDK series timeframe. It was backported to the gingerbread branch, but it only exists on the symbolic branch name 'gingerbread', a development branch, not any of the specific releases in the android-2.3.x series. If you check out a 2.3.x series branch, you will not be able to run firefox. For this reason, the script below (for AVD-building) is somewhat forceful about trying to make you use the 'gingerbread' branch, unless you are absolutely certain you want a 2.3.x branch.

How to generate the AVD definitions

This is the process that graydon followed (the next section shows how to create the same VM that he used):

• Start a new VM (AWS is easiest) running Ubuntu 12.04LTS x86_64
• Log in
• Ensure you have 30+GB of free space to work in
• Check out https://hg.mozilla.org/build/mozharness
• run scripts/android_emulator_build.py and wait several hours
• AVDs should (eventually) be built and left in android_emulator_build/AVDs-*.tar.gz

If anything goes wrong in the process (it's quite involved), the easiest solution is probably just to read the source of the script; it contains our best current understanding of how to build AVDs from AOSP sources. It also has quite a lot of options and can restart at particular steps, if part of the build fails.

Create EC2 instance based on graydon's initial AMI

This will help anyone trying to build the emulator again.

• Go to Amazon's AWS console
• Under EC2's dashboard click on "Images->AMIs"
• Search for "2014-02-05-android-AVD-builder"
• Select the AMI and launch an instance
• Choose m3.medium
• Click "Next: Configure Instance Details" instead of "Review and Launch"
• Change "Network" to "vpc-b42100df"
  • choose this subnet: subnet-fb97bc8f
• Go to section 6 and choose security group: "tests"
• Click on "Review and Launch" to go to section 7
• Click "Launch"
  • As key pairs select "Choose an existing key pair" and "aws-releng"
• Find the instance and name it so you can find it later
• Once running, load the aws-releng key on cruncher: setup environment
• You can then ssh into the machine using the private ip e.g:

ssh ubuntu@10.134.54.50
sudo su -
passwd ubuntu # so you can change the password and loaned it

• Also make sure to add the developers pub key to .ssh/authorized_keys
• File an IT bug to get the developer access to the slave, and mark it as blocking the developer request bug.

Deployment of artifacts

• TODO: Link to how to deploy a new version of the Android SDK
• How to upload a new AVD definition to ToolTool

Hosts involved

TODO: This section should be a page of its own.

Mozpool

TODO: add info

Tegra mobile boards

NOTE: Tegra boards are armv7, however, we can run armv6 builds on it.

These are mobile development boards which are named following this format: tegra-XXX. These boards currently have installed Android 2.2. These boards are located at various locations at the Mountain View office and will be moved to the new Mountain View office. The will be supported until the end of The Tegra boards are driven by Linux hosts called Foopies (TODO link to docs) which run multiple buildbot slave instances (TODO link to docs) named after the tegra boards. More info needed. Add specs.

Panda mobile boards

These are mobile development boards which are named following this format: panda-XXXX. These boards currently have installed Android 4.0. The Panda boards are driven by Linux hosts called Foopies (TODO link to docs) which run multiple buildbot slave instances (TODO link to docs) named after the panda boards. More info needed. Add specs.

Foopies

More info needed. Add specs.

Ubuntu 64-bit in-house test hosts

More info needed. Add specs. These hosts live on scl3, they are made by iX systems, we started buying them on 2013 and they have external graphic cards installed.

Ubuntu 64-bit EC2 test hosts

More info needed.