CI Automation/windows10 aarch64: Difference between revisions
(Some minor updates) |
(Merged 'Running tests Locally' with new section on setup) |
||
| Line 33: | Line 33: | ||
# place the script in the mozilla-build directory | # place the script in the mozilla-build directory | ||
# copy the task ID from treeherder for the Build | # copy the task ID from treeherder for the windows10-aarch64 Build (artifact is supported) | ||
# invoke start-shell.bat, which will launch a bash-like command line | # invoke start-shell.bat, which will launch a bash-like command line | ||
# from mozilla-build directory, run the test runner script as follows: | # from mozilla-build directory, run the test runner script as follows: | ||
bash script.sh task_id test_type <chunk_to_run> <total_chunks> | bash script.sh task_id test_type <chunk_to_run> <total_chunks> | ||
Example: | |||
bash script.sh Q-CE8DFvSAWmc08vw6bd6A xpcshell 1 8 | |||
=== Mozilla-central === | === Mozilla-central === | ||
| Line 132: | Line 135: | ||
For an up-to-date list of tests, please refer to [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#222 this file]. | For an up-to-date list of tests, please refer to [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#222 this file]. | ||
= Run tests on Try = | = Run tests on Try = | ||
Revision as of 20:44, 12 April 2019
Overview
Since mid-January 2019 the CI-A team has been working to enable existing test harnesses, continuous integration tests and other tools to run on Windows 10 ARM64.
General Information
Hardware
- Make: Lenovo
- Model: C630 YOGA
- Processor: Qualcomm Snapdragon 850 3.0GHz
- Cores: 8
- Memory: 8GB
- Disk: 128GB SSD
Hosting
Currently an array of 28 machines are hosted at Bitbar in the United States.
Setup
Test environment
There are two methods by which developers are able to run tests with a windows10-aarch64 hardware locally.
Prequisites
- download Mozilla-Build 2.2.0
- install Mozilla-Build 2.2.0
- download the test runner script from bug 1520867
Mozilla-Build
This method installs an older version of Mozilla-Build on the system to execute tests.
- place the script in the mozilla-build directory
- copy the task ID from treeherder for the windows10-aarch64 Build (artifact is supported)
- invoke start-shell.bat, which will launch a bash-like command line
- from mozilla-build directory, run the test runner script as follows:
bash script.sh task_id test_type <chunk_to_run> <total_chunks>
Example:
bash script.sh Q-CE8DFvSAWmc08vw6bd6A xpcshell 1 8
Mozilla-central
This method is taken from this guide.
- invoke start-shell.bat, which will launch a bash-like command line
- clone the repository with hg clone https://hg.mozilla.org/mozilla-central/
- run ./mach bootstrap and pick artifact build
- download python3 as embeddable zip, then extract to mozilla-build/ directory
- comment out this line
- download 32bit NodeJS zip and extract to .mozbuild/node
- inside mozilla-build, remove watchman directory
- retry ./mach bootstrap
After the artifact build succeeds, it is possible to run most suites of tests as normal:
./mach mochitest
CI environment
Tests that are run in Taskcluster environment against windows10-aarch64 execute using Taskcluster Generic-Worker. These are installed as a service on via OpenCloudConfig.
OpenCloudConfig
This is the method used in production.
Steps originally taken from 1520432.
$gitBranchOrRef = 'master'
Invoke-Expression (New-Object Net.WebClient).DownloadString(('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/{0}/userdata/rundsc.ps1?{1}' -f $gitBranchOrRef, [Guid]::NewGuid()))
Manually install Taskcluster Generic-Worker [Not recommended]
Follow these step to install Taskcluster Generic-Worker on the hardware, and have it launch as a service.
Instruction originally from 1522997.
Prerequisites
- disable Windows S mode
- disable User Account Control
- disable Windows Firewall
- download NSSM to C:\nssm-2.24\
- create "Remote Desktop Users" group:
net localgroup "Remote Desktop Users" /add
- log in to Taskcluster
- request scope `assume:project:taskcluster:generic-worker-tester`
Steps
- download the current 386 release of `generic-worker-windows-386.exe` from taskcluster generic-worker.
- download the latest 386 version of livelog.exe and taskcluster-proxy.exe.
- create new directory C:\generic-worker.
- move the three executable files under C:\generic-worker.
- rename generic-worker-windows-386.exe to generic-worker.exe.
- generate two signing keys:
generic-worker new-openpgp-keypair --file <unique_file_name> generic-worker new-ed25519-keypair --file <unique_file_name>
- create generic-worker.config and include the following:
"accessToken": "<access token tied to taskcluster>", "clientId": "<client ID tied to taskcluster>", "ed25519SigningKeyLocation": "<file location you wrote ed25519 private key in step 6>", "livelogSecret": "<any text>", "provisionerId": "test-provisioner", "publicIP": "<ideally an IP address of one of your network interfaces>", "rootURL": "https://taskcluster.net", "workerGroup": "test-worker-group", "workerId": "test-worker-id", "workerType": "<a unique string that only you will use for your test worker(s)>"
- launch cmd.exe with Administrator rights.
- cd c:\generic-worker
- generic-worker.exe install service --config generic-worker.config --nssm c:\nssm-2.24\win32\nssm.exe
- reboot once installed.
- launch cmd.exe with Administrator rights.
- sc query "Generic Worker"
Currently Running
Currently supported list of tests include:
- awsy
- mochitest (all flavors)
- reftests
- xpcshell
- raptor
- marionette
Supported, requires non-artifact build:
- cppunittest
There is remaining work needed to get these test suites running:
- talos
- web-platform-tests (all flavors)
For an up-to-date list of tests, please refer to this file.
Run tests on Try
This is probably what you came to the document for. How to run tests against the windows10-aarch64 hardware currently available.
Note, the number of hardware is limited so please exercise caution when scheduling tests - a careless try can easily backlog other developers looking to test their changes.
Overview
Follow these steps to be able to enable windows10-aarch64 tests for the try server. These steps are required as of 2019-02-25; it will become obsolete when windows10-aarch64 tests are released to the general public.
Prerequisites
- try access (commit access level 1)
- up-to-date mozilla-central codebase
Steps
- open the file at taskcluster/ci/test/test-platforms.yml
- search for 'windows10-aarch64/opt'
- uncomment all or some of the items under 'test-sets'
- make changes to the local codebase that needs testing
- ./mach try fuzzy
- select tests that need to be run (e.g. 'windows10-aarch64 xpcshell')
- enter
Tests will appear in Treeherder under the heading windows10-aarch64 opt.
Greening tests
Since Windows on ARM64 is a new platform/architecture combination, failures unique to this combination is to be expected. It will be necessary to fix, correct or update the tests in order to obtain a green run.
Example 1
As part of 1525743, the timeout for mochitest-browser-chrome was extended to 4x the default value if the platform combination of Windows and ARM64 is detected.
See change: https://phabricator.services.mozilla.com/D19882
This change greened the test that was previously failing due to a timeout.
Example 2
Some tests provide a manifest file in the form of <test_category>.ini, such as mochitest.ini.
For bug 1525665 it was determined to disable a certain a11y test while windows10-aarch64 a11y support was being investigated.
See change: https://phabricator.services.mozilla.com/D22363
This change meant the failing test is now disabled for windows10-aarch64, and the test would have been green had it not been for another failure elsewhere.
Example 3
Another example of manipulating the manifest of a category of tests, this time with web-platform-tests.
For Bug 1533912 the manifest was modified to disable the test if it was running on aarch64 hardware.
See change: https://phabricator.services.mozilla.com/D23003
Note that web-platform-tests use a slightly different format in order.
Bugs
These are the top-level tracking bugs; the recommended view is tree (login required).
32 Total; 5 Open (15.63%); 27 Resolved (84.38%); 0 Verified (0%);