MozillaWiki

QA/NoMore404s: Difference between revisions

Page Discussion

QA/NoMore404s (view source)

Revision as of 11:46, 21 February 2017

148 bytes removed ,  21 February 2017
General updates + Test Pilot experiment end of life updates
(General updates + Test Pilot experiment end of life updates)
 
Line 8: Line 8:
|-
|-
| 07/21/2016 || 1.0 || Ciprian Muresan || Created first draft
| 07/21/2016 || 1.0 || Ciprian Muresan || Created first draft
|-
| 07/21/2016 || 1.1 || Paul Oiegas || General and Test Pilot EOL updates
|-
|-
| || || ||
| || || ||
Line 24: Line 26:


'''QA''': <br />
'''QA''': <br />
PM for QA team - [mailto:rares.bologa@softvisioninc.eu Rares Bologa] (irc: RaresB) <br />Leading QA efforts - [mailto:paul.oiegas@softvision.ro Paul Oiegas] (irc: pauloiegasSV) <br />QA - [mailto:ciprian.muresan@softvision.ro Ciprian Muresan] (irc: cmuresan)
PM for QA team - [mailto:rares.bologa@softvisioninc.eu Rares Bologa] (irc: RaresB) <br />Leading QA efforts - [mailto:paul.oiegas@softvision.ro Paul Oiegas] (irc: pauloiegasSV) <br />QA - [mailto:ciprian.muresan@softvision.ro Ciprian Muresan] (irc: cmuresan) <br /> QA - [mailto:emil.pasca@softvision.ro Emil Pasca] (irc: emilpasca)


'''Communication channels:'''
'''Communication channels:'''
<br>IRC - TBA
<br>IRC - #txp
<br>Feature email address: TBA
<br>Feature email address: - testpilot-dev@mozilla.org


= Testing summary =
= Testing summary =
Line 77: Line 79:


== Builds ==
== Builds ==
Since NoMore404s is an add-on, you can install it from the following links:
* Add-on Development version - No longer available due to graduation from Test Pilot that occurred on 02/15/2017. You can read the [https://testpilot.firefox.com/experiments/no-more-404s graduation report] on Test Pilot experiment page.
* Add-on Release version - TBA (AMO link)
* A custom build can be made using the steps from [https://github.com/internetarchive/FirefoxNoMore404s GitHub] repo.
* A custom build can be made using the steps from [https://github.com/internetarchive/FirefoxNoMore404s GitHub] repo.
* Add-on Release version. - TBD
 
* Add-on Development version. - [http://testpilot.dev.mozaws.net/experiments/no-more-404s No More 404s Test Pilot Dev website version]
Or you can install it trough [https://testpilot.firefox.com/ Test Pilot] experiments page, when the experiment will be available.
'''Note that the add-on only works on Firefox versions 45.x and up'''.
'''Note that the add-on only works on Firefox versions 45.x and up'''.


Line 107: Line 108:
|-
|-
| QA - Nightly Testing
| QA - Nightly Testing
|style="text-align:center;" |  
|style="text-align:center;" | -
|style="text-align:center;" |
|style="text-align:center;" | -
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
|style="text-align:center;" |  
|style="text-align:center;" | -
|style="text-align:center;" |
|style="text-align:center;" | -
|-
|-
| QA - Beta Testing
| QA - Beta Testing
|style="text-align:center;" |  
|style="text-align:center;" | -
|style="text-align:center;" |
|style="text-align:center;" | -
|-
|-
| Release in Test Pilot Date
| Release in Test Pilot Date
|style="text-align:center;" |   
|style="text-align:center;" |  9/21/2016
|style="text-align:center;" |
|style="text-align:center;" | 2/15/2017
|-
|-
| Release Date
| Release Date
Line 134: Line 135:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || Testrail
| Test case creation || TestRail
|-
|-
| Test case execution || Testrail
| Test case execution || TestRail
|-
|-
| Bugs management || Github
| Bugs management || Github
Line 143: Line 144:
= Status =
= Status =
== Overview ==
== Overview ==
* Track the dates and build number where feature was released to Nightly<br />
* The NoMore404s experiment graduated from Test Pilot in 02/15/2017 ([https://testpilot.firefox.com/experiments/no-more-404s details here]). It will not reach any of the Firefox channels, the decision was to transform it into an AMO add-on.
* Track the dates and build number where feature was merged to Aurora<br />
The experimental add-on will not be uninstalled from Test Pilot users browsers since it will continue to live trough AMO. But it will no longer be available in Test Pilot website for install.
* Track the dates and build number where feature was merged to Release/Beta<br />


== Risk analysis ==
== Risk analysis ==
Line 209: Line 209:
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | Yes || Is this feature user friendly
|style="text-align:center;" | Yes || Is this feature user friendly
|-
| '''Help/Support'''
||  ||
|-
| Help/support interface required 
|style="text-align:center;" | TBD || Make sure link to support/help page exist and is easy reachable.
|-
| Support documents planned(written) 
|style="text-align:center;" | TBD || Make sure support documents are written and are correct.


|-
|-
Line 225: Line 215:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | N/A ||
|style="text-align:center;" | Yes ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
Line 238: Line 228:
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | No || We won't have partner builds available for testing in the Nightly or Aurora channels
|style="text-align:center;" | No || We won't have partner builds available for testing in the Nightly or Aurora channels
|-
| ''' Enterprise '''
||  ||  Raise up the topic to developers to see if they are expecting to work different on ESR builds
|-
| Enterprise administration 
|style="text-align:center;" |  ||
|-
| Network proxies/autoconfig 
|style="text-align:center;" | ||
|-
| ESR behavior changes 
|style="text-align:center;" |  ||
|-
| Locked preferences 
|style="text-align:center;" |  ||


|-
|-
Line 260: Line 234:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | ||
|style="text-align:center;" | - || Not on QA manual end.
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||
|style="text-align:center;" | - || Not on QA manual end.
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||
|style="text-align:center;" | - || Not on QA manual end.
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;" | - || Not on QA manual end.
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;" | - || Manual QA - Performed some minor tests with crowded web pages that were testing the "Save Page" functionality.


|-
|-
Line 279: Line 253:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | Yes ||
|style="text-align:center;" | Yes || Especially the ones from Test Pilot platform.


|-
|-
Line 295: Line 269:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ? ||
|style="text-align:center;" | - || Not on QA manual end.
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ? ||
|style="text-align:center;" | - || Not on QA manual end.
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ? ||
|style="text-align:center;" | - ||


|-
|-
Line 318: Line 292:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||
|style="text-align:center;" | - ||
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||
|style="text-align:center;" | No ||
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
Line 329: Line 303:
== Test suite ==
== Test suite ==
* Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/35&group_by=cases:section_id&group_order=asc NoMore404s Full Functional (TestRail)]
* Full Test suite - [https://testrail.stage.mozaws.net/index.php?/suites/view/35&group_by=cases:section_id&group_order=asc NoMore404s Full Functional (TestRail)]
* Smoke Test suite - TBD
* Smoke Test suite - TBD if needed
* Regression Test suite - TBD if needed
* Regression Test suite - TBD if needed


Line 340: Line 314:
Check list
Check list
* All test cases should be executed
* All test cases should be executed
* Has sufficient automated test coverage (as measured by code coverage tools) - coordinate with RelMan
* All blockers, critical issues must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering & QA)
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)


== Results ==
== Results ==
'''Nightly testing'''<br />
Link for the tests runs
 
* Full Test suite  
*Link for the tests run
**[https://testrail.stage.mozaws.net/index.php?/runs/view/1120&group_by=cases:section_id&group_order=asc Full Functional run 1]
** Daily Smoke suite - TBA
**[https://testrail.stage.mozaws.net/index.php?/runs/view/2552&group_by=cases:section_id&group_order=asc&group_id=140 Full Functional run 2]
** Full Test suite - TBA
** Regression Test suite - TBA if needed/available
 
'''Merge to Aurora Sign-off'''<br />
 
*Link for the tests run - N/A
** Full Test suite - N/A


== Checklist ==
== Checklist ==
Line 362: Line 328:
|-
|-
|  Testing Prerequisites (specs, use cases)
|  Testing Prerequisites (specs, use cases)
| style="text-align:center;" | {{mdone|}}
| style="text-align:center;" | {{mdone|}} ||
| style="text-align:center;" |
|-
|-
|  Testing Infrastructure setup
|  Testing Infrastructure setup
Line 375: Line 340:
|-
|-
|  Full Functional Tests Execution
|  Full Functional Tests Execution
|style="text-align:center;" | {{mprog}}
|style="text-align:center;" | {{mdone}} ||
|style="text-align:center;" |
|-
|-
|  Automation Coverage ||
|  Automation Coverage
|style="text-align:center;" |
|style="text-align:center;" | -
|style="text-align:center;" | Not on QA manual end.
|-
|-
|  Performance Testing
|  Performance Testing
|style="text-align:center;" |  ||
|style="text-align:center;" | -  
|-
|style="text-align:center;" | Not on QA manual end.
|  All Defects Logged || ||
|-
|  Critical/Blockers Fixed and Verified || ||
|-
|  Daily Status Report (email/etherpad statuses/ gdoc with results) || ||
|-
|  Metrics/Telemetry||
|style="text-align:center;" | N/A
|-
|-
|  QA Signoff  - Nightly Release||
|  All Defects Logged
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | {{mdone}} ||
|-
|-
|  QA Aurora - Full Testing
|  Critical/Blockers Fixed and Verified
|style="text-align:center;" | ||
|style="text-align:center;" | {{mprog}} ||
|-
|-
|  QA Signoff  - Aurora Release ||
|  Daily Status Report (email/etherpad statuses/ gdoc with results)
|style="text-align:center;"| Email to be sent
|style="text-align:center;" | - ||
|-
|-
|  QA Beta - Full Testing
|  Metrics/Telemetry
|style="text-align:center;" | ||
|style="text-align:center;" | -
|style="text-align:center;" | Not on QA manual end.
|-
|-
|  QA Signoff  - Beta Release ||
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | -
|style="text-align:center;" | No Sign off done. Graduated straight to AMO ||
|}
|}
Paul.oiegas
166

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1163393"