CI Automation/windows10 aarch64: Difference between revisions
(→Example 3: added example) |
(Emphasized warning) |
||
| Line 115: | Line 115: | ||
= Run tests on Try = | = Run tests on Try = | ||
This is probably what you came to the document for. How to run tests against the windows10-aarch64 hardware currently available. Note, the number of hardware is limited so please exercise caution when scheduling tests. | This is probably what you came to the document for. How to run tests against the windows10-aarch64 hardware currently available. | ||
'''Note, the number of hardware is limited so please exercise caution when scheduling tests - a careless try can easily backlog other developers looking to test their changes.''' | |||
== Overview == | == Overview == | ||
Revision as of 00:56, 11 April 2019
Overview
Since mid-January 2019 the CI-A team has been working to enable existing test harnesses, continuous integration tests and other tools to run on Windows 10 ARM64.
General Information
Hardware
- Make: Lenovo
- Model: C630 YOGA
- Processor: Qualcomm Snapdragon 850 3.0GHz
- Cores: 8
- Memory: 8GB
- Disk: 128GB SSD
Hosting
Currently an array of 9 machines are hosted at Bitbar in the United States.
Setup
Tests that are run against windows10-aarch64 execute using Taskcluster Generic-Worker. These are installed as a service on the Windows 10 ARM64 manually or via OpenCloudConfig.
A brief walkthrough of the steps to have Taskcluster Generic-Worker running on Windows 10 ARM64 will be provided.
Using only generic-worker
Follow this step to install Taskcluster Generic-Worker on the hardware, and have it launch as a service. After following these steps, the hardware should be ready to accept any tasks started on Taskcluster.
Instruction originally from 1522997.
Prerequisites
- disable Windows S mode
- disable User Account Control
- disable Windows Firewall
- download NSSM to C:\nssm-2.24\
- create "Remote Desktop Users" group:
net localgroup "Remote Desktop Users" /add
- log in to Taskcluster
- request scope `assume:project:taskcluster:generic-worker-tester`
Steps
- download the current 386 release of `generic-worker-windows-386.exe` from taskcluster generic-worker.
- download the latest 386 version of livelog.exe and taskcluster-proxy.exe.
- create new directory C:\generic-worker.
- move the three executable files under C:\generic-worker.
- rename generic-worker-windows-386.exe to generic-worker.exe.
- generate two signing keys:
generic-worker new-openpgp-keypair --file <unique_file_name> generic-worker new-ed25519-keypair --file <unique_file_name>
- create generic-worker.config and include the following:
"accessToken": "<access token tied to taskcluster>", "clientId": "<client ID tied to taskcluster>", "ed25519SigningKeyLocation": "<file location you wrote ed25519 private key in step 6>", "livelogSecret": "<any text>", "openpgpSigningKeyLocation": "<file location you wrote gpg private key kn step 6>", "provisionerId": "test-provisioner", "publicIP": "<ideally an IP address of one of your network interfaces>", "rootURL": "https://taskcluster.net", "workerGroup": "test-worker-group", "workerId": "test-worker-id", "workerType": "<a unique string that only you will use for your test worker(s)>"
- launch cmd.exe with Administrator rights.
- cd c:\generic-worker
- generic-worker.exe install service --config generic-worker.config --nssm c:\nssm-2.24\win32\nssm.exe
- reboot once installed.
- launch cmd.exe with Administrator rights.
- sc query "Generic Worker"
Using OpenCloudConfig
This is the method that is used in production.
Steps originally taken from 1520432.
- Invoke-Expression (New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/aarch64/userdata/rundsc.ps1')
Currently Running
Currently supported list of tests include:
- awsy
- mochitest (all flavors, including e10s)
- web-platform-tests (all flavors)
- reftests (including crashtest, jsreftest)
- xpcshell
Supported, requires non-artifact build:
- jittest
- gtests
- cppunittest
There is remaining work needed to get these test suites running:
- talos
- raptor
- marionette
For an up-to-date list of tests, please refer to this file.
Run tests Locally
Theoretically, you can run tests locally with mach from a local build environment. However, since our aarch64 builds are usually cross-compiled in an x86 environment, you probably don't have a local build environment!
The recommended alternative is to use mozharness to download, install, and test a build from try or continuous integration. A handy script is provided as an attachment to bug 1520867 that greatly simplifies running tests from mozharness; let's call that script 'moztest'.
Run moztest from a MozillaBuild shell. You need only a few parameters:
- The task-id of the Windows-aarch64 build that you want to test: Click on the aarch64 build in treeherder, and copy the "Task" shown in the treeherder detail pane; it might look like "Q-CE8DFvSAWmc08vw6bd6A".
- The name of the test suite you want to run: one of (cppunit, gtest, xpcshell, mochitest, mochitest-chrome, mochitest-clipboard, mochitest-dt, mochitest-gpu, mochitest-media, crashtest, jsreftest, reftest, jittest, web-platform, web-platform-reftest, web-platform-wdspec, raptor-speedometer, raptor-tp6, talos-g5, talos-chromez)
- Optionally, the test "chunk" number to run and the number of test chunks to split the suite into.
For example:
- moztest Q-CE8DFvSAWmc08vw6bd6A cppunit
- moztest Q-CE8DFvSAWmc08vw6bd6A xpcshell 1 3
Run tests on Try
This is probably what you came to the document for. How to run tests against the windows10-aarch64 hardware currently available.
Note, the number of hardware is limited so please exercise caution when scheduling tests - a careless try can easily backlog other developers looking to test their changes.
Overview
Follow these steps to be able to enable windows10-aarch64 tests for the try server. These steps are required as of 2019-02-25; it will become obsolete when windows10-aarch64 tests are released to the general public.
Prerequisites
- try access (commit access level 1)
- up-to-date mozilla-central codebase
Steps
- open the file at taskcluster/ci/test/test-platforms.yml
- search for 'windows10-aarch64/opt'
- uncomment all or some of the items under 'test-sets'
- make changes to the local codebase that needs testing
- ./mach try fuzzy
- select tests that need to be run (e.g. 'windows10-aarch64 xpcshell')
- enter
Tests will appear in Treeherder under the heading windows10-aarch64 opt.
Greening tests
Since Windows on ARM64 is a new platform/architecture combination, failures unique to this combination is to be expected. It will be necessary to fix, correct or update the tests in order to obtain a green run.
Example 1
As part of 1525743, the timeout for mochitest-browser-chrome was extended to 4x the default value if the platform combination of Windows and ARM64 is detected.
See change: https://phabricator.services.mozilla.com/D19882
This change greened the test that was previously failing due to a timeout.
Example 2
Some tests provide a manifest file in the form of <test_category>.ini, such as mochitest.ini.
For bug 1525665 it was determined to disable a certain a11y test while windows10-aarch64 a11y support was being investigated.
See change: https://phabricator.services.mozilla.com/D22363
This change meant the failing test is now disabled for windows10-aarch64, and the test would have been green had it not been for another failure elsewhere.
Example 3
Another example of manipulating the manifest of a category of tests, this time with web-platform-tests.
For Bug 1533912 the manifest was modified to disable the test if it was running on aarch64 hardware.
See change: https://phabricator.services.mozilla.com/D23003
Note that web-platform-tests use a slightly different format in order.
Bugs
These are the top-level tracking bugs; the recommended view is tree (login required).
32 Total; 5 Open (15.63%); 27 Resolved (84.38%); 0 Verified (0%);