B2G/QA/BuildProccess: Difference between revisions
Jump to navigation
Jump to search
| (4 intermediate revisions by 2 users not shown) | |||
| Line 24: | Line 24: | ||
** Posted in https://mail.mozilla.org/listinfo/fxosqa-report-watchlist daily | ** Posted in https://mail.mozilla.org/listinfo/fxosqa-report-watchlist daily | ||
** Posted in QA Weekly round table weekly: https://public.etherpad-mozilla.org/p/b2g-qa-roundtable ; #QA-B2G in vidyo 5 pm PDT Wednesdays | ** Posted in QA Weekly round table weekly: https://public.etherpad-mozilla.org/p/b2g-qa-roundtable ; #QA-B2G in vidyo 5 pm PDT Wednesdays | ||
* Daily automation results are posted to "gaia-ui-automation@mozilla.org" | * Daily automation results are reviewed; Failures are manually tested and are posted to "gaia-ui-automation@mozilla.org" | ||
== Dogfood Test == | == Dogfood Test Process == | ||
=== Dogfood Test === | |||
* If smoke test passes : | * If smoke test passes : | ||
** For nightly builds, nightly-latest is pushed to nightly on [https://wiki.mozilla.org/Balrog|Balrog]. ( currently on hold until new base build/FOTA occurs) | ** For nightly builds, nightly-latest is pushed to nightly on [https://wiki.mozilla.org/Balrog|Balrog]. ( currently on hold until new base build/FOTA occurs) | ||
| Line 32: | Line 33: | ||
** Dogfood-latest build is moved to dogfood-test | ** Dogfood-latest build is moved to dogfood-test | ||
** Dogfood Tests are executed for further testing: | ** Dogfood Tests are executed for further testing: | ||
*** Dogfood Test | *** Dogfood Test Suites : https://moztrap.mozilla.org/manage/cases/?filter-suite=912&filter-suite=719 | ||
=== Dogfood Test Results === | === Dogfood Test Results === | ||
| Line 59: | Line 60: | ||
* T2M base builds follows a similar path as dogfood. | * T2M base builds follows a similar path as dogfood. | ||
* When making a build, the img files are written over top of the original 18D base build | * When making a build, the img files are written over top of the original 18D base build | ||
** https://github.com/nhirata/Nhirata_Scripts/blob/master/T2M_Flame.sh outlines what is done in the build. Note: | ** https://github.com/nhirata/Nhirata_Scripts/blob/master/T2M_Flame.sh outlines what is done in the build. | ||
** Note: It's probably better to fix/create a taskcluster script to kick off this build [to be worked on] | |||
* The build is sent to the TAM for T2M and they are asked to hosts the files | * The build is sent to the TAM for T2M and they are asked to hosts the files | ||
* Balrog will need to point to the FOTA build accordingly. | * Balrog will need to point to the FOTA build accordingly. | ||
Latest revision as of 21:16, 12 January 2016
Nightly Build Status
- please see https://wiki.mozilla.org/B2G/QA/Flame_OTA
Dogfood Build Status
- please see https://wiki.mozilla.org/B2G/QA/Spark
Processes for Smoke and Dogfood
Smoke Test Process
Smoke Test
- Builds are daily tested by QAnalysts; nightly-latest and dogfood-latest builds are tested:
- Build location for Flame nightly: https://tools.taskcluster.net/index/artifacts/#gecko.v1.mozilla-central.latest.linux.flame-kk-ota/gecko.v1.mozilla-central.latest.linux.flame-kk-ota.debug
- Build location for Aries dogfood: https://tools.taskcluster.net/index/artifacts/#gecko.v1.mozilla-central.latest.linux.aries-dogfood/gecko.v1.mozilla-central.latest.linux.aries-dogfood.debug
- Test Suite in MozTrap: https://moztrap.mozilla.org/manage/cases/?filter-suite=241
- Initial Logging : https://docs.google.com/spreadsheets/d/13BjtbYDVaiC6TvOAd1x-SuoYNNM8aH5-xdJS7TU3o-U/edit#gid=328354895
- If there is any failure, these will fail the build from being tested further for release for nightly/dogfood.
- Depending on the failure, the build may or not be used QA for feature testing.
- Daily automation results from jenkins running GIP tests are looked at, reviewed, and seeing if they manually reproduce.
- Detailed information revolving the Smoke tests are listed here : https://wiki.mozilla.org/B2G/QA/Smoke_Tests
Smoke Test Results
- Results are posted in 4 different places:
- Announced in QA Standup meeting daily : #QA-B2G in vidyo 10 am PDT Daily
- Posted in QAStandup etherpad daily : https://public.etherpad-mozilla.org/p/b2g-qa-daily-standup
- Posted in https://mail.mozilla.org/listinfo/fxosqa-report-watchlist daily
- Posted in QA Weekly round table weekly: https://public.etherpad-mozilla.org/p/b2g-qa-roundtable ; #QA-B2G in vidyo 5 pm PDT Wednesdays
- Daily automation results are reviewed; Failures are manually tested and are posted to "gaia-ui-automation@mozilla.org"
Dogfood Test Process
Dogfood Test
- If smoke test passes :
- For nightly builds, nightly-latest is pushed to nightly on [1]. ( currently on hold until new base build/FOTA occurs)
- https://wiki.mozilla.org/B2G/QA/Flame_OTA is updated when the change is made
- Dogfood-latest build is moved to dogfood-test
- Dogfood Tests are executed for further testing:
- Dogfood Test Suites : https://moztrap.mozilla.org/manage/cases/?filter-suite=912&filter-suite=719
- For nightly builds, nightly-latest is pushed to nightly on [1]. ( currently on hold until new base build/FOTA occurs)
Dogfood Test Results
- Dogfood test results are posted in https://mail.mozilla.org/listinfo/fxosqa-report-watchlist
- If passed then dogfood channel is updated on balrog with the build
- a quick sanity test to make sure people download the right build is done
- https://wiki.mozilla.org/B2G/QA/Spark is updated when the change is made
Process in event of failure
- Bugs that are smoke test failures should be investigated for regression ranges
- If a regression window is found :
- Dev is notified and discussion occurs
- If a dev cannot be notified in time, process will occur
- If a regression window is found :
- Dogfood failures may end up having the similar process. Smoke test failures is higher priority.
- If a patch is made for an issue the following day and a new failure occurs, the tested build is patched with the new patch and a new build is kicked off and tested with the dogfood test suite.
Challenges
- backouts/patching needs to occur faster
- Gecko often may have a regression that cannot necessarily be backed out easily
- This is the main reason why there are delays in a dogfood build beyond any new bug failures.
- Gecko often may have a regression that cannot necessarily be backed out easily
- more often than not, there can be multiple dogfood and smoke test failures that occur.
- Close monitoring of smoke test failures and pushing of backouts has not occurred since Bhavana (release manager) left; QA voice is not enough to convince a developer to back out faster. We need a manager to fill this role.
T2M Base Builds
- The occurrence of needing a new T2M Base build is when there has been a new gonk level change, that a new build is considered.
- T2M base builds follows a similar path as dogfood.
- When making a build, the img files are written over top of the original 18D base build
- https://github.com/nhirata/Nhirata_Scripts/blob/master/T2M_Flame.sh outlines what is done in the build.
- Note: It's probably better to fix/create a taskcluster script to kick off this build [to be worked on]
- The build is sent to the TAM for T2M and they are asked to hosts the files
- Balrog will need to point to the FOTA build accordingly.