B2G/QA/Process: Difference between revisions

From MozillaWiki
< B2G‎ | QA
Jump to navigation Jump to search
 
(10 intermediate revisions by 3 users not shown)
Line 21: Line 21:
[https://wiki.mozilla.org/B2G/QA/Device_Test_Plan/Graphics Graphics Test Plan]<br>
[https://wiki.mozilla.org/B2G/QA/Device_Test_Plan/Graphics Graphics Test Plan]<br>


'''[[Modules/FirefoxOS|List of Firefox OS Modules/Dev Owners]]''' in case you need to ask questions regarding the new features of the component.
'''[[Modules/FirefoxOS|List of Firefox OS Modules/Dev Owners (link)]]''' in case you need to ask questions regarding the new features of the component.<br>
[[Firefox_OS/functionalteams|'''Firefox OS functional teams breakdown (link)''']]<br>
[[FirefoxOS/FeatureTesting|'''QA owners for Firefox OS components (link)''']]


=== B2G Test Process ===
=== B2G Test Process ===
Line 35: Line 37:
* Execute performance/stability test suite
* Execute performance/stability test suite


=== Qanalysts Tasks ===
=== Smoke Test / Dogfood Processes ===
Following documents describe the tasks assigned to QAnalyst:<br>
* [[B2G/QA/BuildProcess|Smoke Test, Dogfood, Base Build Process]]
(If you cannot view these documents, ask ktucker for permission in #fxosqa.  You must have a valid mozilla LDAP account)
 
* [https://docs.google.com/a/mozilla.com/document/d/1pVLh0sKuC6q6d1rBmJt9umlygY9bz_2qqOudm7uoLJU/edit?usp=sharing_eid&ts=5642879f QAnalyst Daily Tasks]
=== Status Tracking ===
** [[B2G/QA/Gaia_Test_Plan/Email_Templates|Email Template for the daily report]]
* [http://mozilla-twqa.github.io/QA-Dashboard/ffos/frame_features.html Features on QA Dashboard]
* [https://docs.google.com/a/mozilla.com/document/d/19St0uLtBSDG6BILk2trEWTbq9N6E-OBOwTghMz9sztk/edit?usp=sharing_eid&ts=56426b8e Jenkins Reporting]
* [http://mozilla-twqa.github.io/QA-Dashboard/ffos/frame_bugs.html Bugs on QA Dashboard]
* [https://docs.google.com/a/mozilla.com/document/d/134G51KAVV0l_0XiEcYuk6h5vo0lnjQEIY_LyvbxEKro/edit?usp=sharing_eid&ts=56426bcd Automation Reporting]
** How to use: put [COM=NFC] in 'QA whiteboard', the bug shows up nicely to the count of NFC box, feature page filters for feature-b2g flag, bug page filters for blocking-b2g flag.
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1tvOk6Jw6beZodjpQw2e9zbVHA-tec43VpBBg4nde_Eo/edit?usp=sharing_eid&ts=56426c54 Exploratory Tests]
* [http://charts.mozilla.org/ Mozilla Charts]
* [https://docs.google.com/document/d/17_zO4ACFGblNOtV7ixenMr9f8zWEv_RzXpPcWFrEQ7Q/edit?ts=56426c94#heading=h.tejavodcjazn Exploratory Tests with Resolved Fixes]
* [[FirefoxOS/DeviceQA#Meeting_Minutes|Meeting minutes]]
* [https://docs.google.com/a/mozilla.com/document/d/1UmtV6IQzddV_1JZhGI5CboKJy2OcyrKqSOrX7-k52GI/edit?usp=sharing_eid&ts=56426d87 Getting Regression Window]
* [https://docs.google.com/a/mozilla.com/document/d/1Rf0i57kN9Fof_HgTKnnqZ3ADD0htSfploHCvGGBaL7o/edit?usp=sharing_eid&ts=56426dcd Executing Automation]
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1RhI81B01gzLYTeRVqFM8RC2_UiuGtFssGtX_pRycgqE/edit?usp=sharing_eid&ts=5642834b Application Coverage Spreadsheet]


=== [[B2G/QA/Bugzilla|Bugzilla Howtos (link)]] ===
=== [[B2G/QA/Bugzilla|Bugzilla Howtos (link)]] ===
=== Moztrap Howtos ===
* '''[https://quality.mozilla.org/docs/test-case-writing-primer Test case creation primer]'''
* '''[https://wiki.mozilla.org/QA/Execution/MozTrap Moztrap General]'''
* '''[https://wiki.mozilla.org/FirefoxOS/MoztrapGuideline Moztrap Guideline for Firefox OS]'''
* '''[https://addons.mozilla.org/EN-uS/firefox/addon/moztraphelper MozTrapHelper by Shing]''' - recommend to add this to your Firefox browser.


=== Triage ===
=== Triage ===
Line 87: Line 92:
* Non-regression bugs that we chose not to block on in the last release
* Non-regression bugs that we chose not to block on in the last release
* (though, sometimes this is because partner said "we won't block this time if you fix in the next release)
* (though, sometimes this is because partner said "we won't block this time if you fix in the next release)
=== Qanalysts Tasks ===
Following documents describe the tasks assigned to QAnalyst:<br>
(If you cannot view these documents, ask ktucker for permission in #fxosqa.  You must have a valid mozilla LDAP account)
* [https://docs.google.com/a/mozilla.com/document/d/1pVLh0sKuC6q6d1rBmJt9umlygY9bz_2qqOudm7uoLJU/edit?usp=sharing_eid&ts=5642879f QAnalyst Daily Tasks]
** [[B2G/QA/Gaia_Test_Plan/Email_Templates|Email Template for the daily report]]
* [https://docs.google.com/a/mozilla.com/document/d/19St0uLtBSDG6BILk2trEWTbq9N6E-OBOwTghMz9sztk/edit?usp=sharing_eid&ts=56426b8e Jenkins Reporting]
* [https://docs.google.com/a/mozilla.com/document/d/134G51KAVV0l_0XiEcYuk6h5vo0lnjQEIY_LyvbxEKro/edit?usp=sharing_eid&ts=56426bcd Automation Reporting]
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1tvOk6Jw6beZodjpQw2e9zbVHA-tec43VpBBg4nde_Eo/edit?usp=sharing_eid&ts=56426c54 Exploratory Tests]
* [https://docs.google.com/document/d/17_zO4ACFGblNOtV7ixenMr9f8zWEv_RzXpPcWFrEQ7Q/edit?ts=56426c94#heading=h.tejavodcjazn Exploratory Tests with Resolved Fixes]
* [https://docs.google.com/a/mozilla.com/document/d/1UmtV6IQzddV_1JZhGI5CboKJy2OcyrKqSOrX7-k52GI/edit?usp=sharing_eid&ts=56426d87 Getting Regression Window]
* [https://docs.google.com/a/mozilla.com/document/d/1Rf0i57kN9Fof_HgTKnnqZ3ADD0htSfploHCvGGBaL7o/edit?usp=sharing_eid&ts=56426dcd Executing Automation]
* [https://docs.google.com/a/mozilla.com/spreadsheets/d/1RhI81B01gzLYTeRVqFM8RC2_UiuGtFssGtX_pRycgqE/edit?usp=sharing_eid&ts=5642834b Application Coverage Spreadsheet]
=== Marigold ===
==== Overview ====
Marigold (Cheng-Du, China) is Mozilla's QA outsource team. They were in charge of executing FxOS tests, and they covers following tasks requested by Mozilla.
(You may need a valid Mozilla LDAP account to access following documents)
* Manual Test
** Daily smoke test
** Regression test
** Full test
** RAT test
** Bug work (qawanted,regressionwindow-wanted,verifyme)
* Power consumption measurement
* Test automation - Gaia-UI-test
* Patch verification
==== Management ====
* [http://mzl.la/1xHcWOp Bugzilla:Need-Info List]
* [https://goo.gl/k5STRA Project Assignment and Schedule]
* [https://goo.gl/aBtobD Resource and Devices]
* [https://goo.gl/u604ZU Meeting minutes]
* Weekly Status/Report
** [https://goo.gl/mnfAms 2015 Q1 ~ 2015 Q3]
** [https://goo.gl/dAPcEt 2015 Q4]

Latest revision as of 07:56, 17 March 2016

QA Process

This page lists the relevant information to B2G QA processes.

Mozilla QA Process Overview

Please visit MDN QA page for an overview of quality assurance activities at Mozilla.

B2G Roadmap

Here is the overview of the current Firefox OS product roadmap. Check this page for new information regarding future releases.

B2G Test Planning

UX Specification for Each Release

2.5 UX Specs
2.6 UX Specs

B2G Test Plans

Note: Some information may be out of date, in which case you can notify the QA owner of the affected component(s).
New Device Test Plan
Email Test Plan
Browser Test Plan
Crash Reporting Test Plan
Graphics Test Plan

List of Firefox OS Modules/Dev Owners (link) in case you need to ask questions regarding the new features of the component.
Firefox OS functional teams breakdown (link)
QA owners for Firefox OS components (link)

B2G Test Process

Please review the following beforehand:

Before the B2G release, following QA activities should be completed:

  • Go through the list of new features (check for feature-b2g flag in Bugzilla), and assess for the creation of new test cases and update/deprecate existing test cases as necessary
  • Create and update test cases in Moztrap
  • Execute regression test suite, incorporated with the new test cases, and verify all release blocker bugs are fixed
  • Ensure the builds are passing daily smoketests
  • Execute performance/stability test suite

Smoke Test / Dogfood Processes

Status Tracking

Bugzilla Howtos (link)

Moztrap Howtos

Triage

Refer to this page for the detailed information on how we use Bugzilla for triage process.
Refer to this page for B2G triage process.

When to Block a Bug for Release

Should Block

  • Features our product team has committed us to
  • Major issue in new feature - especially those in which a large number of users will be impacted, or a smaller number of users will be significantly impacted
  • Major identifiable regression (perf or otherwise)
  • Non-localizable strings
  • Top Crashes
  • sec-high, sec-critical Security bugs
  • Smoke-test regression (special-case insta-blocker)
  • Data loss
  • Issues that block partner certification (basically legal issues)
  • Issues getting a lot of support calls with partners or on SUMO (exceptions need to be made after FC though)
  • Issues critical around updates (especially if there's been a repro)
  • Anything critical around the first time experience
  • Major Dialer, SMS, and VM communication issues (lower bar than other blockers - it's a phone)
  • Issues that prevent automated tests in established testsuites (visible test suites on b2g integration branches on TBPL) from running green at least 90% of the time.
  • Certification Waivers

Should Not Block - any exceptions to these rules must be discussed on b2g-release-drivers@mozilla.org or with Release Management

  • Enhancements
  • New Features (Not committed for the release)
  • New perf requirements (see enhancements)
  • Non-critical string changes (think of the l10n, people)
  • Polish and other minor issues
  • Unfinished localization (except perhaps in the last 3 weeks)
  • Issues requiring the user to "zoom in and out and in and out and in and out..."
  • Issues in languages not being shipped in the version of B2G
  • Bugs without clear STR or that are not reproducible
  • Bugs that do not impact production phones or the simulator

Needs Decision

  • Non-regression bugs that we chose not to block on in the last release
  • (though, sometimes this is because partner said "we won't block this time if you fix in the next release)

Qanalysts Tasks

Following documents describe the tasks assigned to QAnalyst:
(If you cannot view these documents, ask ktucker for permission in #fxosqa. You must have a valid mozilla LDAP account)

Marigold

Overview

Marigold (Cheng-Du, China) is Mozilla's QA outsource team. They were in charge of executing FxOS tests, and they covers following tasks requested by Mozilla. (You may need a valid Mozilla LDAP account to access following documents)

  • Manual Test
    • Daily smoke test
    • Regression test
    • Full test
    • RAT test
    • Bug work (qawanted,regressionwindow-wanted,verifyme)
  • Power consumption measurement
  • Test automation - Gaia-UI-test
  • Patch verification

Management

Retrieved from "https://wiki.allizom.org/index.php?title=B2G/QA/Process&oldid=1122586"