MozillaWiki

CI Automation/windows10 aarch64: Difference between revisions

Page Discussion

CI Automation/windows10 aarch64 (view source)

Revision as of 21:35, 3 April 2020

834 bytes removed ,  3 April 2020
→‎Currently running on CI: updated
(Shuffled)
 
(15 intermediate revisions by 2 users not shown)
Line 1: Line 1:
=Overview=
=Overview=
Since mid-January 2019 the CI-A team has been working to enable existing test harnesses, continuous integration tests and other tools to run on Windows 10 ARM64.
Since mid-January 2019 the [https://wiki.mozilla.org/CI_Automation CI-A team] has been working to enable existing test harnesses, continuous integration tests and other tools to run on Windows 10 ARM64, aka aarch64.


= General Information =
= General Information =
Line 15: Line 15:
== Hosting ==
== Hosting ==


Currently an array of 28 machines are hosted at [https://bitbar.com/ Bitbar] in the United States.
Currently an array of ~30 machines are hosted at [https://bitbar.com/ Bitbar] in the United States.


= Setup =
= Setup - local environment =
== Test environment ==
Developers wishing to run tests locally have two methods.


There are two methods by which developers are able to run tests with a windows10-aarch64 hardware locally.
== Prequisites ==


=== Prequisites ===
# download and install [https://ftp.mozilla.org/pub/mozilla/libraries/win32/MozillaBuildSetup-2.2.0.exe Mozilla-Build 2.2.0]


# download [https://ftp.mozilla.org/pub/mozilla/libraries/win32/MozillaBuildSetup-2.2.0.exe Mozilla-Build 2.2.0]
=== Using mozilla-build ===
# install Mozilla-Build 2.2.0
# download the test runner script from [https://bug1520867.bmoattachments.org/attachment.cgi?id=9046793 bug 1520867]


=== Mozilla-Build ===
This method uses a script to download test archives in order to run tests locally.


This method installs an older version of Mozilla-Build on the system to execute tests.
# download <code>script for running mozharness on Yoga</code> from [https://bugzilla.mozilla.org/show_bug.cgi?id=1520867 bug 1520867]
# place the test runner script in the <code>C:\mozilla-build</code> directory
# from treeherder, identify a changeset that contains a successful <code>build-win64-aarch64/opt</code>
# copy the task ID of the build
# invoke start-shell.bat, which will launch a bash-like commandline
# from mozilla-build directory, run the test runner script as follows:
<code>bash script.sh task_id test_type <chunk_to_run> <total_chunks></code>


# place the script in the mozilla-build directory
Example:
# copy the task ID from treeherder for the Build
<code>bash script.sh Q-CE8DFvSAWmc08vw6bd6A xpcshell 1 8</code>
# invoke start-shell.bat, which will launch a bash-like command line
# from mozilla-build directory, run the test runner script as follows:
bash script.sh task_id test_type <chunk_to_run> <total_chunks>


=== Mozilla-central ===
=== Using mozilla-central ===


This method is taken from [https://www.gijsk.com/blog/2019/02/getting-firefox-artifact-builds-working-on-an-arm64-aarch64-windows-device/ this guide].
This method is taken from [https://www.gijsk.com/blog/2019/02/getting-firefox-artifact-builds-working-on-an-arm64-aarch64-windows-device/ this guide] and uses mozilla-central with a build artifact.


# invoke start-shell.bat, which will launch a bash-like command line
# invoke start-shell.bat, which will launch a bash-like commandline
# clone the repository with hg clone https://hg.mozilla.org/mozilla-central/
# clone the repository using <code>hg clone https://hg.mozilla.org/mozilla-central/</code>
# run ./mach bootstrap and pick artifact build
# run <code>./mach bootstrap</code> and pick artifact build
# download python3 as embeddable zip, then extract to mozilla-build/ directory
# download python3 embeddable zip, then extract to <code>mozilla-build/</code> directory
# comment out [https://searchfox.org/mozilla-central/rev/152993fa346c8fd9296e4cd6622234a664f53341/python/mozboot/mozboot/bootstrap.py#444 this line]
# remove [https://searchfox.org/mozilla-central/rev/152993fa346c8fd9296e4cd6622234a664f53341/python/mozboot/mozboot/bootstrap.py#444 this line]
# download 32bit NodeJS zip and extract to .mozbuild/node
# download 32bit NodeJS zip and extract to <code>.mozbuild/node</code>
# inside mozilla-build, remove watchman directory
# inside mozilla-build, remove the directory named <code>watchman</code>
# retry ./mach bootstrap
# rerun <code>./mach bootstrap</code>
# run <code>./mach build</code>


After the artifact build succeeds, it is possible to run most suites of tests as normal:
After the artifact build succeeds, it is possible to run most suites of tests as normal:
<code>./mach mochitest <test_file></code>


./mach mochitest
= CI environment =
 
== CI environment ==


Tests that are run in Taskcluster environment against windows10-aarch64 execute using [https://github.com/taskcluster/generic-worker Taskcluster Generic-Worker]. These are installed as a service on via [https://github.com/mozilla-releng/OpenCloudConfig OpenCloudConfig].
Tests that are run in Taskcluster environment against windows10-aarch64 execute using [https://github.com/taskcluster/generic-worker Taskcluster Generic-Worker]. These are installed as a service on via [https://github.com/mozilla-releng/OpenCloudConfig OpenCloudConfig].


=== Using OpenCloudConfig ===
== Using OpenCloudConfig ==


This is the method used in production.
This is the method used in production.
Line 68: Line 69:
  Invoke-Expression (New-Object Net.WebClient).DownloadString(('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/{0}/userdata/rundsc.ps1?{1}' -f $gitBranchOrRef, [Guid]::NewGuid()))
  Invoke-Expression (New-Object Net.WebClient).DownloadString(('https://raw.githubusercontent.com/mozilla-releng/OpenCloudConfig/{0}/userdata/rundsc.ps1?{1}' -f $gitBranchOrRef, [Guid]::NewGuid()))


=== Manually install Taskcluster Generic-Worker [Not recommended] ===
== Manually install Generic-Worker [Not recommended] ==


Follow these step to install Taskcluster Generic-Worker on the hardware, and have it launch as a service.
Follow these step to install Taskcluster Generic-Worker on the hardware, and have it launch as a service.
Line 74: Line 75:
Instruction originally from [https://bugzilla.mozilla.org/show_bug.cgi?id=1522997#c2 1522997].
Instruction originally from [https://bugzilla.mozilla.org/show_bug.cgi?id=1522997#c2 1522997].


=== Prerequisites ===
'''Prerequisites'''
* disable Windows S mode
* disable Windows S mode
* disable User Account Control
* disable User Account Control
Line 84: Line 85:
* request scope `assume:project:taskcluster:generic-worker-tester`  
* request scope `assume:project:taskcluster:generic-worker-tester`  


=== Steps ===
'''Steps'''
 
# download the current 386 release of `generic-worker-windows-386.exe` from [https://github.com/taskcluster/generic-worker/releases taskcluster generic-worker].
# download the current 386 release of `generic-worker-windows-386.exe` from [https://github.com/taskcluster/generic-worker/releases taskcluster generic-worker].
# download the latest 386 version of livelog.exe and taskcluster-proxy.exe.
# download the latest 386 version of livelog.exe and taskcluster-proxy.exe.
Line 112: Line 112:
# sc query "Generic Worker"
# sc query "Generic Worker"


= Currently Running =
= Currently running on CI =
 
Currently supported list of tests include:
 
* awsy
* mochitest (all flavors)
* reftests
* xpcshell
* raptor
* marionette
 
Supported, requires non-artifact build:
 
* cppunittest
 
There is remaining work needed to get these test suites running:
 
* talos
* web-platform-tests (all flavors)
 
For an up-to-date list of tests, please refer to [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/test-platforms.yml#222 this file].
 
= Run tests Locally =
Theoretically, you can run tests locally with mach from a local build environment. However, since our aarch64 builds are usually cross-compiled in an x86 environment, you probably don't have a local build environment!
 
The recommended alternative is to use mozharness to download, install, and test a build from try or continuous integration. A handy script is provided as an attachment to {{bug|1520867}} that greatly simplifies running tests from mozharness; let's call that script 'moztest'.
 
Run moztest from a MozillaBuild shell. You need only a few parameters:
* The task-id of the Windows-aarch64 build that you want to test: Click on the aarch64 build in treeherder, and copy the "Task" shown in the treeherder detail pane; it might look like "Q-CE8DFvSAWmc08vw6bd6A".
* The name of the test suite you want to run: one of (cppunit, gtest, xpcshell, mochitest, mochitest-chrome, mochitest-clipboard, mochitest-dt, mochitest-gpu, mochitest-media, crashtest, jsreftest, reftest, jittest, web-platform, web-platform-reftest, web-platform-wdspec, raptor-speedometer, raptor-tp6, talos-g5, talos-chromez)
* Optionally, the test "chunk" number to run and the number of test chunks to split the suite into.


For example:
Currently, a limit subset of tests are running regularly on <code>mozilla-central</code> and <code>try</code>. This is to reduce the load on the windows10-aarch64 hardware, which is limited in number.
* moztest Q-CE8DFvSAWmc08vw6bd6A cppunit
* moztest Q-CE8DFvSAWmc08vw6bd6A xpcshell 1 3


= Run tests on Try =
= Run on try =


This is probably what you came to the document for. How to run tests against the windows10-aarch64 hardware currently available.  
This is probably what you came to the document for. How to run tests against the windows10-aarch64 hardware currently available.  


'''Note, the number of hardware is limited so please exercise caution when scheduling tests - a careless try can easily backlog other developers looking to test their changes.'''
'''Hardware is limited so please exercise caution when scheduling tests! A careless try will block many others. Only schedule jobs that are absolutely necessary.'''
 
== Overview ==


Follow these steps to be able to enable windows10-aarch64 tests for the try server. These steps are required as of 2019-02-25; it will become obsolete when windows10-aarch64 tests are released to the general public.
== Prerequisites ==
 
=== Prerequisites ===


* try access (commit access level 1)
* try access (commit access level 1)
* up-to-date mozilla-central codebase
* up-to-date mozilla-central codebase


=== Steps ===
== Steps ==


# open the file at taskcluster/ci/test/test-platforms.yml
Note that on <code>try</code>, windows10-aarch64 is hidden by default; please use <code>./mach try fuzzy --full</code> to schedule jobs.
# search for 'windows10-aarch64/opt'
 
# uncomment all or some of the items under 'test-sets'
# <code>./mach try fuzzy --full</code>
# make changes to the local codebase that needs testing
# ./mach try fuzzy
# select tests that need to be run (e.g. 'windows10-aarch64 xpcshell')
# select tests that need to be run (e.g. 'windows10-aarch64 xpcshell')
# enter
# enter


Tests will appear in Treeherder under the heading ''windows10-aarch64 opt''.
Tests will appear in Treeherder under the heading ''Windows 10 AArch64 opt''.


= Greening tests =
= Greening tests =
Line 191: Line 153:
Some tests provide a manifest file in the form of <test_category>.ini, such as ''mochitest.ini''.
Some tests provide a manifest file in the form of <test_category>.ini, such as ''mochitest.ini''.


For bug 1525665 it was determined to disable a certain a11y test while windows10-aarch64 a11y support was being investigated.
For [https://bugzilla.mozilla.org/show_bug.cgi?id=1525665 bug 1525665] it was determined to disable a certain a11y test while windows10-aarch64 a11y support was being investigated.


See change: https://phabricator.services.mozilla.com/D22363
See change: https://phabricator.services.mozilla.com/D22363
Line 201: Line 163:
Another example of manipulating the manifest of a category of tests, this time with ''web-platform-tests''.
Another example of manipulating the manifest of a category of tests, this time with ''web-platform-tests''.


For Bug 1533912 the manifest was modified to disable the test if it was running on aarch64 hardware.
For [https://bugzilla.mozilla.org/show_bug.cgi?id=1533912 bug 1533912], the manifest was modified to disable the test if it was running on aarch64 hardware.


See change: https://phabricator.services.mozilla.com/D23003
See change: https://phabricator.services.mozilla.com/D23003


Note that web-platform-tests use a slightly different format in order.
Note that web-platform-tests use a slightly different format in order.
== Example 4 ==
Certain test cases in reftest/crashtest/jsreftest had unexpected outcomes on windows10-aarch64.
For [https://bugzilla.mozilla.org/show_bug.cgi?id=1536365 bug 1536365] and [https://bugzilla.mozilla.org/show_bug.cgi?id=1536363 bug 1536363], the requirement was to adjust the pixel-difference values such that tests will pass.
See change: https://phabricator.services.mozilla.com/D25113


= Bugs =
= Bugs =


These are the top-level tracking bugs; the recommended view is [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1522997&hide_resolved=0 tree] (login required).
These are the top-level tracking bugs; the recommended view is [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1522997&hide_resolved=0 tree] (login required).
CI-A team will make efforts to re-test disabled tests on a semi-regular basis, or whenever fixes are committed to components that had tests disabled.


<bugzilla>
<bugzilla>
Egao
74

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1210554...1225867"