MozillaWiki

Releases/Firefox 16/Test Plan: Difference between revisions

Page Discussion

Releases/Firefox 16/Test Plan (view source)

Revision as of 09:57, 1 November 2012

10,451 bytes added ,  1 November 2012
→‎Previously Known
 
(40 intermediate revisions by 8 users not shown)
Line 2,123: Line 2,123:
; Checklist
; Checklist
* {{done|}} [[Releases/Firefox_16/Test_Plan#Automated_Testing_RC|Automated Testing]]
* {{done|}} [[Releases/Firefox_16/Test_Plan#Automated_Testing_RC|Automated Testing]]
* {{ok|}} [[Releases/Firefox_16/Test_Plan#Manual_Testing_RC|Manual Testing]]
* {{done|}} [[Releases/Firefox_16/Test_Plan#Manual_Testing_RC|Manual Testing]]
* {{ok|}} [[Releases/Firefox_16/Test_Plan#Update_Testing_RC|Update Testing]]
* {{done|}} [[Releases/Firefox_16/Test_Plan#Update_Testing_RC|Update Testing]]
* {{ok|}} [[Releases/Firefox_16/Test_Plan#Bug_Triage_RC|Bug Triage]]
* {{ok|}} [[Releases/Firefox_16/Test_Plan#Bug_Triage_RC|Bug Triage]]
* {{|}} Peer Sign-off
* {{|}} Peer Sign-off
Line 2,200: Line 2,200:
  | Mac OS X
  | Mac OS X
  | VladG
  | VladG
  | '''In progress''' - 2012-10-09
  | '''PASS''' - 2012-10-09
  | '''In progress''' - 2012-10-09
  | '''PASS''' - 2012-10-09
  | '''In progress''' - 2012-10-09
  | '''PASS''' - 2012-10-09
  |-
  |-
  | Ubuntu 12.04
  | Ubuntu 12.04
Line 2,218: Line 2,218:
  | Windows XP
  | Windows XP
  | Paul
  | Paul
  | '''in progress''' - 2012-10-09
  | '''PASS''' - 2012-10-09
  | '''N/A'''
  | '''N/A'''
  | '''in progress''' - 2012-10-09
  | '''PASS''' - 2012-10-09
  |-
  |-
  | Windows 7
  | Windows 7
Line 2,236: Line 2,236:
  | Mac OS X
  | Mac OS X
  | VladG
  | VladG
  | '''In progress''' - 2012-10-09
  | '''PASS''' - 2012-10-09
  | '''PASS''' - 2012-10-08
  | '''PASS''' - 2012-10-08
  | '''In progress''' - 2012-10-09
  | '''PASS''' - 2012-10-09
  |-
  |-
  | Ubuntu 12.04
  | Ubuntu 12.04
Line 2,316: Line 2,316:
  |-
  |-
  | class="header" | Platform
  | class="header" | Platform
  | class="header" | Betatest: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=16.0&channel=releasetest&from=2012-10-08&to=2012-10-10&target=16.0 report]
  | class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=16.0&channel=releasetest&from=2012-10-08&to=2012-10-10&target=16.0 report]
  | class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=16.0&channel=beta&from=2012-10-08&to=2012-10-10&target=16.0 report]
  | class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=16.0&channel=beta&from=2012-10-08&to=2012-10-10&target=16.0 report]
  |-
  |-
Line 2,336: Line 2,336:
  |-
  |-
  | Windows 8 32-bit
  | Windows 8 32-bit
  | '''manual'''-see below - yyyy-mm-dd
  | '''PASS'''- see below - 2012-10-09
  | '''RESULTS''' - yyyy-mm-dd
  | '''RESULTS''' - yyyy-mm-dd
  |-
  |-
Line 2,375: Line 2,375:


=== Bug Triage RC ===
=== Bug Triage RC ===
# [http://bit.ly/RquQy9 VERIFYME (11)] : verify the fix and do some exploratory testing around the issue
# [http://bit.ly/RquQy9 VERIFYME (9)] : verify the fix and do some exploratory testing around the issue
# [http://bit.ly/Ppe9E0 FIXED (180)] : triage these bugs and test to ensure the fix is effective
# [http://bit.ly/Ppe9E0 FIXED (178)] : triage these bugs and test to ensure the fix is effective
# [http://bit.ly/PMaYVx QAWANTED (6)] : triage these bugs and test to fulfill the QA request
# [http://bit.ly/PMaYVx QAWANTED (8)] : triage these bugs and test to fulfill the QA request
# [http://bit.ly/OtEqj1 UNCONFIRMED (20)] : triage these bugs for potential new regressions
# [http://bit.ly/OtEqj1 UNCONFIRMED (21)] : triage these bugs for potential new regressions


; Results
; Results
* [http://bit.ly/MXYdqS VERIFIED]: 89/304
* [http://bit.ly/MXYdqS VERIFIED]: 89/304
* [http://bit.ly/POyxuA WONTVERIFY]: 31/304
* [http://bit.ly/POyxuA WONTVERIFY]: 32/304
* [http://bit.ly/O09WSK HELPWANTED]: 4/304
* [http://bit.ly/O09WSK HELPWANTED]: 5/304
 
== 16.0.1 ==
{| class="querytable" style="width:30%"
| style="width:10%" |
| style="width:20%" |
|-
| '''Builds'''
| [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/16.0.1-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/16.0.1-candidates/build1/ HTTP]
|-
| '''ID'''
| 20121010144125
|-
| '''Changeset'''
| [https://hg.mozilla.org/releases/mozilla-release/rev/7789ab68a99d 7789ab68a99d] ([http://hg.mozilla.org/releases/mozilla-release/pushloghtml?fromchange=10fe550fadc6&tochange=7789ab68a99d pushlog])
|}
 
; Checklist
* {{done|}} [[Releases/Firefox_16/Test_Plan#Automated_Testing_7|Automated Testing]]
* {{done|}} [[Releases/Firefox_16/Test_Plan#Manual_Testing_7|Manual Testing]]
* {{done|}} [[Releases/Firefox_16/Test_Plan#Update_Testing_7|Update Testing]]
* {{ | }} Final sign-off sent and updates qualified on Release channel
 
=== Automated Testing 7 ===
The following are the detailed results of the automated Mozmill test runs.
{|class="querytable" style="width:50%"
|-
| class="header" style="width:20%" | Platform
| class="header" style="width:30%" | RESULT: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=16.0&platform=All&from=2012-10-10&to=2012-10-11 report]
|-
| Windows XP
| '''PASS''' - 2012-10-10
|-
| Windows Vista
| '''PASS''' - 2012-10-10
|-
| Windows 7 32-bit
| '''PASS''' - 2012-10-10
|-
| Windows 7 64-bit
| '''PASS''' - 2012-10-10
|-
| Windows 8 32-bit
| '''PASS''' - 2012-10-10
|-
| Windows 8 64-bit
| '''PASS''' - 2012-10-10
|-
| Mac OS X 10.6
| '''PASS''' - 2012-10-10
|-
| Mac OS X 10.7
| '''PASS''' - 2012-10-10
|-
| Mac OS X 10.8
| '''PASS''' - 2012-10-10
|-
| Ubuntu 12.04 32-bit
| '''PASS''' - 2012-10-10
|-
| Ubuntu 12.04 64-bit
| '''PASS''' - 2012-10-10
|}
 
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
 
=== Manual Testing 7 ===
There will be four bug verifications and testing around these bugs
 
; Results
{| class="querytable"
| class="header" | Platform
| class="header" | Tester
| class="header" | 16.0.1 Candidates
|-
| Windows XP
| juanb, ashughes, mwobensmith
| '''PASS''' - 2012-10-10
|-
| Windows Vista
| tester
| '''RESULT''' - 2012-10-10
|-
| Windows 7 32-bit
| tester
| '''RESULT''' - 2012-10-10
|-
| Windows 7 64-bit
| juanb, ashughes, mwobensmith
| '''PASS''' - 2012-10-10
|-
| Mac OS X 10.6
| tester
| '''RESULT''' - 2012-10-10
|-
| Mac OS X 10.7
| juanb, ashughes, mwobensmith
| '''PASS''' - 2012-10-10
|-
| Mac OS X 10.8
| tester
| '''RESULT''' - 2012-10-10
|-
| Ubuntu 12.04 32-bit
| tester
| '''RESULT''' - 2012-10-10
|-
| Ubuntu 12.04 64-bit
| juanb, ashughes, mwobensmith
| '''PASS''' - 2012-10-10
|}
 
; Issues Found
 
==== Breakpad End-to-End 7 ====
Test the end-to-end functionality in the crash reporting feature, across operating systems.
* Use the [http://ted.mielczarek.org/mozilla/crashme.html Crash Me add-on] to trigger a crash
* Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
* Repeatedly crash Firefox and submit your reports
* Verify that your session restores properly
* Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
* Try adding comments to your crash report and verify the crash-stats.com report shows the comment
 
=== Update Testing 7 ===
* Partial: 13.0.1, 14.0.1, 15.0.1
* Complete: 12.0, 13.0
 
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=16.0&channel=releasetest&from=2012-10-10&to=2012-10-10&target=16.0 report]
| class="header" | Release: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=16.0&channel=beta&from=2012-10-11&to=2012-10-11&target=16.0 report]
|-
| Windows XP
| '''PASS''' - 2012-10-11
| '''PASS''' - 2012-10-11
|-
| Windows Vista
| '''PASS''' - 2012-10-11
| '''PASS''' - 2012-10-11
|-
| Windows 7 32-bit
| '''PASS''' - 2012-10-11
| '''PASS''' - 2012-10-11
|-
| Windows 7 64-bit
| '''PASS''' - 2012-10-11
| '''PASS''' - 2012-10-11
|-
| Windows 8 32-bit
| '''Pass'''- juanb - 2012-10-11
| '''Pass''' - juanb - 2012-10-11
|-
| Windows 8 64-bit
| '''Pass'''- juanb - 2012-10-11
| '''Pass''' - 2012-10-11
|-
| Mac OS X 10.6
| '''PASS''' - 2012-10-11
| '''PASS''' - 2012-10-11
|-
| Mac OS X 10.7
| '''PASS''' - 2012-10-11
| '''PASS''' - 2012-10-11
|-
| Mac OS X 10.8
| '''PASS''' - 2012-10-11
| '''PASS''' - 2012-10-11
|-
| Ubuntu 12.04 32-bit
| '''PASS''' - 2012-10-11
| '''PASS''' - 2012-10-11
|-
| Ubuntu 12.04 64-bit
| '''PASS''' - 2012-10-11
| '''PASS''' - 2012-10-11
|}
 
''Windows 8: manual spotcheck until automation in place''
* RELEASETEST
** '''PASS''' - Firefox 16.0 en-US (partial)
** '''PASS''' - Firefox 15.0.1 de (partial)
* RELEASE
** '''PASS''' - Firefox 16.0 en-US (partial)
** '''PASS''' - Firefox 12.0 fr (complete)
 
; Issues
 
== 16.0.2 ==
{| class="querytable" style="width:30%"
| style="width:10%" |
| style="width:20%" |
|-
| '''Builds'''
| [ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/16.0.2-candidates/build1/ FTP], [http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/16.0.2-candidates/build1/ HTTP]
|-
| '''ID'''
| 20121024073032
|-
| '''Changeset'''
| [https://hg.mozilla.org/releases/mozilla-release/rev/3995ff235342 3995ff235342] ([http://hg.mozilla.org/releases/mozilla-release/pushloghtml?fromchange=7789ab68a99d&tochange=3995ff235342 pushlog])
|}
 
; Checklist
* {{done|}} [[Releases/Firefox_16/Test_Plan#Automated_Testing_8|Automated Testing]]
* {{done|}} [[Releases/Firefox_16/Test_Plan#Manual_Testing_8|Manual Testing]]
* {{ok|}} [[Releases/Firefox_16/Test_Plan#Update_Testing_8|Update Testing]]
* {{ | }} Final sign-off sent and updates qualified on Beta channel
 
=== Automated Testing 8 ===
The following are the detailed results of the automated Mozmill test runs.
{|class="querytable" style="width:50%"
|-
| class="header" style="width:20%" | Platform
| class="header" style="width:30%" | RESULT: [http://mozmill-ci.blargon7.com/#/functional/reports?branch=16.0&platform=All&from=2012-10-24&to=2012-10-24 report]
|-
| Windows XP
| '''PASS''' - 2012-10-24
|-
| Windows Vista
| '''PASS''' - 2012-10-24
|-
| Windows 7 32-bit
| '''PASS''' - 2012-10-24
|-
| Windows 7 64-bit
| '''PASS''' - 2012-10-24
|-
| Windows 8 32-bit
| '''PASS''' - 2012-10-24
|-
| Windows 8 64-bit
| '''PASS''' - 2012-10-24
|-
| Mac OS X 10.6
| '''PASS''' - 2012-10-24
|-
| Mac OS X 10.7
| '''PASS''' - 2012-10-24
|-
| Mac OS X 10.8
| '''PASS''' - 2012-10-24
|-
| Ubuntu 12.04 32-bit
| '''PASS''' - 2012-10-24
|-
| Ubuntu 12.04 64-bit
| '''PASS''' - 2012-10-24
|}
 
; Test Failures ([https://bugzilla.mozilla.org/buglist.cgi?list_id=3007328;status_whiteboard_type=allwordssubstr;query_format=advanced;status_whiteboard=mozmill-test-failure;bug_status=NEW;bug_status=ASSIGNED;component=Mozmill%20Tests;product=Mozilla%20QA known bugs])
 
=== Manual Testing 8 ===
There will be X bug verifications and testing around these bugs. Basic functionality spot checking - [https://mozqa.etherpad.mozilla.org/Firefox16-0-2 etherpad].
 
; Results
{| class="querytable"
| class="header" | Platform
| class="header" | Tester
| class="header" | 16.0.2 Candidates
|-
| Windows XP
| Simona
| '''PASS''' - 2012-10-25
|-
| Windows Vista
| Simona
| '''PASS''' - 2012-10-25
|-
| Windows 7 32-bit
| Mihaela
| '''PASS''' - 2012-10-25
|-
| Windows 7 64-bit
| Manuela
| '''PASS''' - 2012-10-25
|-
| Mac OS X 10.6
| Virgil
| '''PASS''' - 2012-10-25
|-
| Mac OS X 10.7
| Virgil
| '''PASS''' - 2012-10-25
|-
| Mac OS X 10.8
| Mihaela
| '''PASS''' - 2012-10-25
|-
| Ubuntu 12.04 32-bit
| Ioana
| '''PASS''' - 2012-10-25
|-
| Ubuntu 12.04 64-bit
| Ioana
| '''PASS''' - 2012-10-25
|}
 
; Issues Found
 
==== Breakpad End-to-End 8 ====
Test the end-to-end functionality in the crash reporting feature, across operating systems.
* Use the [http://ted.mielczarek.org/mozilla/crashme.html Crash Me add-on] to trigger a crash
* Set up a dirty profile with lots of history, bookmarks, add-ons, themes, tabs, app-tabs, plugin content loaded, etc
* Repeatedly crash Firefox and submit your reports
* Verify that your session restores properly
* Verify your crash reports are submitted properly by going through about:crashes and clicking the reports
* Try adding comments to your crash report and verify the crash-stats.com report shows the comment
 
=== Update Testing 8 ===
* Partial: 14.0.1, 15.0.1, 16.0.1
* Complete: 12.0, 13.0
 
{|class="querytable" style="width:60%"
|-
| class="header" | Platform
| class="header" | Releasetest: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=16.0&channel=releasetest&from=2012-10-24&to=2012-10-25&target=16.0 report]
| class="header" | Beta: [http://mozmill-ondemand.blargon7.com/#/update/detail?branch=16.0&channel=beta&from=2012-10-24&to=2012-10-25&target=16.0 report]
|-
| Windows XP
| '''RESULT''' - 2012-10-24
| '''RESULT''' - 2012-10-24
|-
| Windows Vista
| '''RESULT''' - 2012-10-24
| '''RESULT''' - 2012-10-24
|-
| Windows 7 32-bit
| '''RESULT''' - 2012-10-24
| '''RESULT''' - 2012-10-24
|-
| Windows 7 64-bit
| '''RESULT''' - 2012-10-24
| '''RESULT''' - 2012-10-24
|-
| Windows 8 32-bit
| '''manual'''-see below - 2012-10-02
| '''Results''' - yyyy-mm-dd
|-
| Windows 8 64-bit
| '''manual'''-see below - 2012-10-02
| '''Results''' - yyyy-mm-dd
|-
| Mac OS X 10.6
| '''RESULT''' - 2012-10-24
| '''RESULT''' - 2012-10-24
|-
| Mac OS X 10.7
| '''RESULT''' - 2012-10-24
| '''RESULT''' - 2012-10-24
|-
| Mac OS X 10.8
| '''RESULT''' - 2012-10-24
| '''RESULT''' - 2012-10-24
|-
| Ubuntu 12.04 32-bit
| '''RESULT''' - 2012-10-24
| '''RESULT''' - 2012-10-24
|-
| Ubuntu 12.04 64-bit
| '''RESULT''' - 2012-10-24
| '''RESULT''' - 2012-10-24
|}
 
''Windows 8: manual spotcheck until automation in place''
* RELEASETEST
** '''RESULT''' - Firefox 16.0.1 en-US (partial)
** '''RESULT''' - Firefox 13.0 de (complete)
* RELEASE
** '''RESULT''' - Firefox 16.0.1 en-US (partial)
** '''RESULT''' - Firefox 12.0 fr (complete)
 
; Issues


== Issues ==
== Issues ==
Line 2,403: Line 2,775:
<bugzilla>
<bugzilla>
{
{
  "id": "768375, 686782, 667399, 785754, 779492, 775082, 777051, 747909, 787056, 675645, 778819, 786655, 781190, 78414, 788423, 786237"
  "id": "768375, 686782, 667399, 785754, 779492, 775082, 777051, 747909, 787056, 675645, 778819, 786655, 781190, 78414, 788423, 786237, 791195, 717178, 794924, 692446, 792422, 797116"
}
}
</bugzilla>
</bugzilla>
Simona Badau
Confirmed users
1,333

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/477590...484320"