MozillaWiki

Releases/Firefox 47/Test Plan: Difference between revisions

Page Discussion

Releases/Firefox 47/Test Plan (view source)

Revision as of 16:39, 11 November 2016

10,571 bytes added ,  11 November 2016
→‎47.0.2 Build 2
 
(32 intermediate revisions by 4 users not shown)
Line 330: Line 330:
* Manual - [https://wiki.mozilla.org/Releases/Firefox_47/Test_Plan/Beta/RC3 Firefox 47.0 RC3]
* Manual - [https://wiki.mozilla.org/Releases/Firefox_47/Test_Plan/Beta/RC3 Firefox 47.0 RC3]
** {{mdone|}} [https://docs.google.com/spreadsheets/d/1h8PABdVzuYsIXQQz5Vj8ymkCVOBh0jhbcGShcxU9cWs/edit#gid=55133156 Smoke Testing] with focus on Video Compatibility.
** {{mdone|}} [https://docs.google.com/spreadsheets/d/1h8PABdVzuYsIXQQz5Vj8ymkCVOBh0jhbcGShcxU9cWs/edit#gid=55133156 Smoke Testing] with focus on Video Compatibility.
** {{mok|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pli=1#gid=312280130 Bug verification] with focus on the driver for this build - [https://bugzilla.mozilla.org/show_bug.cgi?id=1278005 bug 1278005].
** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit?pli=1#gid=312280130 Bug verification] with focus on the driver for this build - [https://bugzilla.mozilla.org/show_bug.cgi?id=1278005 bug 1278005].
* Automated ([[QA/Desktop_Firefox/Releases/Configs/Fx47RC3|configs]] for [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/trigger-ondemand/build?delay=0sec ondemand])  
* Automated ([[QA/Desktop_Firefox/Releases/Configs/Fx47RC3|configs]] for [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/trigger-ondemand/build?delay=0sec ondemand])  
** This is a candidate for the 47.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on release-cdntest.
** This is a candidate for the 47.0 release build. We should test updates on beta-cdntest for a preliminary sign-off (Release Management must give the "go" for pushing to beta), then on beta after the push to beta, and then on the day before release (Monday), after a full sign-off and push to that channel, run tests on release-cdntest.
** {{mok|}} Updates on ''beta-cdntest'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=b0310cb90fd0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta-cdntest( report])
** {{mdone|}} Updates on ''beta-cdntest'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=b0310cb90fd0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta-cdntest( report])
** {{mok|}} Updates on ''beta'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=b0310cb90fd0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta( report])
** {{mdone|}} Updates on ''beta'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=b0310cb90fd0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-beta( report])
** {{mdone|}} Updates on ''release-cdntest'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=b0310cb90fd0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release-cdntest( report])
** {{mdone|}} Updates on ''release'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=b0310cb90fd0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release( report])
 
=== Release 47.0 ===
==== Build Information ====
* See Release Candidate Build 3
 
==== Testing ====
* Update Testing - see Release Candidate Build 3
* Throttle Testing (see [[QA/Desktop_Firefox/Walkthroughs/Release_Coordination#Update_Throttling|Walkthrough]] for details, and detailed [[QA/Throttling_Tests|Test Instructions]] for running them)
** Used commands:
*** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
*** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/WINNT_x86_64-msvc/en-US/release/Windows_NT%2010.0/default/default/update.xml
*** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2013.4.0/default/default/update.xml
*** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/Linux_x86-gcc3/en-US/release/Linux%203.13.0-45-generic%20%28GTK%202.24.23%29/default/default/update.xml
*** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/46.0.1/20160502172042/Linux_x86_64-gcc3/en-US/release/Linux%203.16.0-30-generic%20%28GTK%202.24.25%29/default/default/update.xml
** Results
*** After release push (expected to be @ 25%), 2016-06-07: '''Win32: 24% - Win64: 23% - Mac: 25% - Linux32: 24% - Linux64: 24%'''
*** After disabling updates (expected to be @ 0%), 2016-06-08: '''Win32: 0% - Win64: 0% - Mac: 0% - Linux32: 0% - Linux64: 0%'''
*** After fully enabling updates (expected to be @ 100%): 2016-06-10: '''Win32: 100% - Win64: 100% - Mac: 100% - Linux32: 100% - Linux64: 100%'''
 
=== 47.0.1 Build 1 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/47.0.1-candidates/build1/ 47.0.1-build1] | [https://hg.mozilla.org/releases/mozilla-release/rev/7f5abf95991b 7f5abf95991b] | (ID: 20160623154057)
* Manual - [https://wiki.mozilla.org/Releases/Firefox_47/Test_Plan/Beta/47.0.1 Firefox 47.0.1]
** {{mdone|}} [https://docs.google.com/spreadsheets/d/1h8PABdVzuYsIXQQz5Vj8ymkCVOBh0jhbcGShcxU9cWs/edit#gid=843686709 Regression Testing] with focus on Web Compatibility.
** {{mdone|}} [https://docs.google.com/spreadsheets/d/12Dc8SHzJK5E4PoLzuCdsQ5DGx_oKLsajBCR3pm91RXA/edit#gid=312280130 Bug verification] with focus on the latest fixes to land in this build.
* Automated ([[QA/Desktop_Firefox/Releases/Configs/Fx47.0.1|configs]] for [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/trigger-ondemand/build?delay=0sec ondemand])
** {{mdone|}} Updates on ''release-localtest'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=7f5abf95991b&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release-localtest( report])
** {{mdone|}} Updates on ''release-cdntest'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=7f5abf95991b&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release-cdntest( report])
** {{mdone|}} Updates on ''release'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=7f5abf95991b&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release( report])
** Throttle Testing (see [[QA/Desktop_Firefox/Walkthroughs/Release_Coordination#Update_Throttling|Walkthrough]] for details)
*** Used commands:
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/WINNT_x86_64-msvc/en-US/release/Windows_NT%2010.0/default/default/update.xml
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2013.4.0/default/default/update.xml
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/Linux_x86-gcc3/en-US/release/Linux%203.13.0-45-generic%20%28GTK%202.24.23%29/default/default/update.xml
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0/20160604131506/Linux_x86_64-gcc3/en-US/release/Linux%203.16.0-30-generic%20%28GTK%202.24.25%29/default/default/update.xml
*** Results
*** After release push (expected to be @ 25%), 2016-06-28: '''Win32: % - Win64: % - Mac: % - Linux32: % - Linux64: %'''
*** After disabling updates (expected to be @ 0%), 2016-06-29: '''Win32: % - Win64: % - Mac: % - Linux32: % - Linux64: %'''
*** After fully enabling updates (expected to be @ 100%): 2016-07-01: '''Win32: % - Win64: % - Mac: % - Linux32: % - Linux64: %'''
*** '''NOTE: throttle tests do not apply to this release as we released at 100% for versions < 47.0, and 0% for 47.0 (except for forced updates) - this was confirmed by ondemand update tests on the [https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=7f5abf95991b&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release( release channel] (for forced updates), and throttling tests which were at 0% (using the throttling commands above)'''
 
=== 47.0.2 Build 1 ===
==== Build Information ====
* Build 1: [https://archive.mozilla.org/pub/firefox/candidates/47.0.2-candidates/build1/ 47.0.2-build1] | [https://hg.mozilla.org/releases/mozilla-release/rev/3dc66a6a33a0 3dc66a6a33a0] | (ID: 20161027172622)
* Manual - [https://wiki.mozilla.org/Releases/Firefox_47/Test_Plan/Beta/47.0.2 Firefox 47.0.2]
** {{mdone|}} Sanity Testing with focus on: general Web Compatibility.
*** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/reports/view/104 in TestRail] (requires access rights) or [https://drive.google.com/open?id=xxxxxxx as a pdf].
** {{mdone|}} [https://public.etherpad-mozilla.org/p/1305787 Verification] of driver [https://bugzilla.mozilla.org/show_bug.cgi?id=1305787 bug 1305787]
* Automated ([[QA/Desktop_Firefox/Releases/Configs/Fx47.0.2|configs]] for [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/trigger-ondemand/build?delay=0sec ondemand])
** {{mdone|}} Updates on ''release-localtest'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=3dc66a6a33a0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release-localtest( report])
*** Also tested manually that:
**** Fx < 47.0.2 with the hotfix installed and (websense) get updated to 47.0.2
**** Fx < 47.0.2 with the hotfix installed and (nowebsense) get updated to 48.0.2 and then 49.0.2
** {{mdone|}} Updates on ''release-cdntest'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=3dc66a6a33a0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release-cdntest( report])
** {{mnotstarted|}} Updates on ''release'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=3dc66a6a33a0&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release( report])
** Throttle Testing (see [[QA/Desktop_Firefox/Walkthroughs/Release_Coordination#Update_Throttling|Walkthrough]] for details)
*** '''N/A, as Build 2 was generated'''
 
=== 47.0.2 Build 2 ===
==== Build Information ====
* Build 2: [https://archive.mozilla.org/pub/firefox/candidates/47.0.2-candidates/build2/ 47.0.2-build2] | [https://hg.mozilla.org/releases/mozilla-release/rev/d7f98b92fdec d7f98b92fdec] | (ID: 20161031133903)
* Manual - [https://wiki.mozilla.org/Releases/Firefox_47/Test_Plan/Beta/47.0.2 Firefox 47.0.2]
** {{mdone|}} Sanity Testing with focus on: general Web Compatibility.
*** Detailed test results: [https://testrail.stage.mozaws.net/index.php?/plans/view/1136 in TestRail] (requires access rights) or [https://drive.google.com/open?id=0BzOV3ybvBASXUmVDVTRPUm9Zanc as a pdf].
* Automated ([[QA/Desktop_Firefox/Releases/Configs/Fx47.0.2|configs]] for [http://mm-ci-production.qa.scl3.mozilla.com:8080/job/trigger-ondemand/build?delay=0sec ondemand])
** {{mdone|}} Updates on ''release-localtest'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=d7f98b92fdec&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release-localtest( report])
*** Also tested manually that:
**** Fx < 47.0.2 with the hotfix installed and (websense) get updated to 47.0.2
**** Fx < 47.0.2 with the hotfix installed and (nowebsense) get updated to 48.0.2 and then 49.0.2
** {{mdone|}} Updates on ''release-cdntest'' channel ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=d7f98b92fdec&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release-cdntest( report])
** {{mdone|}} Updates on ''release'' channel
*** Automated tests failures - ([https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&revision=d7f98b92fdec&group_state=expanded&filter-tier=3&filter-searchStr=Fxup-release( report])
*** Manual tests [https://public.etherpad-mozilla.org/p/Fx47.0.2-updatetests results]
** Throttle Testing (see [[QA/Desktop_Firefox/Walkthroughs/Release_Coordination#Update_Throttling|Walkthrough]] for details)
*** Used commands:
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/WINNT_x86-msvc/en-US/release/Windows_NT%206.3/default/default/update.xml
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/WINNT_x86_64-msvc/en-US/release/Windows_NT%2010.0/default/default/update.xml
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/release/Darwin%2013.4.0/default/default/update.xml
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/Linux_x86-gcc3/en-US/release/Linux%203.13.0-45-generic%20%28GTK%202.24.23%29/default/default/update.xml
**** ./throttle.sh https://aus3.mozilla.org/update/3/Firefox/47.0.1/20160623154057/Linux_x86_64-gcc3/en-US/release/Linux%203.16.0-30-generic%20%28GTK%202.24.25%29/default/default/update.xml
*** Results
*** After release push (expected to be @ 25%), 2016-10-31: '''N/A - Releasing directly to 100%'''
*** After disabling updates (expected to be @ 0%), 2016-10-31: '''N/A - Releasing directly to 100%'''
*** After fully enabling updates (expected to be @ 100%): 2016-11-04: '''Win32: 100% - Win64: 100% - Mac: 100% - Linux32: 100% - Linux64: 100%'''


[[category:Test_Plans|Firefox]][[category:QA|Firefox Test Plans]]
[[category:Test_Plans|Firefox]][[category:QA|Firefox Test Plans]]
Florin.Mezei
Confirmed users
2,479

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1135927...1154420"