QA/Fennec1.0/TestPlan: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
 
(42 intermediate revisions by 2 users not shown)
Line 40: Line 40:
* Maemo Beta 3 - August 20
* Maemo Beta 3 - August 20
* WinMo Alpha 3 - August 20
* WinMo Alpha 3 - August 20
* Maemo Beta 4 - Sept 14 (l10n) - [https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/1.0Maemo_Beta4 Release Test Results]
* Maemo Beta 4 - Sept 30  - <strike>[https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/1.0Maemo_Beta4 Release Test Results], [https://wiki.mozilla.org/Mobile/Fennec/Fennec1.0/Maemo-B4#QA_Blockers QA Blockers] </strike>
* Maemo RC1 - Sept 30
* Maemo Beta 5 - Nov 9 - <strike>[https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/1.0Maemo_Beta5 Release Test Results], [https://wiki.mozilla.org/Mobile/Fennec/Fennec1.0/Maemo-B5#QA_Blockers QA Blockers]</strike>
* Winmo Beta1 - Sept 30
* Maemo RC1 - December 30 - <strike>[https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/1.0Maemo_RC1 Release Test Results] </strike>
* Maemo 1.0 - Oct 9
* Maemo RC2 - January 6 - <strike>[https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/1.0Maemo_RC2 Release Test Results] </strike>
 
* Maemo RC2 - January 7 - <strike>[https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/1.0Maemo_RC2B2 Release Test Results] </strike>
== Test Results ==
* Maemo RC3 - January 27 - <strike>[https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/1.0Maemo_RC3 Release Test Results]</strike>
 
* Maemo 1.0 - January 28
Maemo Beta3
* WinMo Alpha 4 - February 8 -  [https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/1.0WinMo_Alpha4 Release Test Results]
- litmus
* Maemo 1.0.1 - March 31 - <strike>[https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/1.0.1Maemo Release Test Results]</strike>
- automation
- testday
- installation
- l10n
 
Winmo Alpha3
- litmus
- automation
  - testday
- installation
- l10n


== Test Areas ==
== Test Areas ==
Line 385: Line 374:
Fennec 1.0 is currently built from trunk (mozilla-central) as well as the front end mobile-browser repositories.  To minimize branching and maintenance, we will target the 1.9.2 branch (splitting from mozilla-central July 2009).  All other platforms will build from the same branch.
Fennec 1.0 is currently built from trunk (mozilla-central) as well as the front end mobile-browser repositories.  To minimize branching and maintenance, we will target the 1.9.2 branch (splitting from mozilla-central July 2009).  All other platforms will build from the same branch.


Nightly builds for Maemo are located here (you need both fennec*.deb and xulrunner*.deb where the timestamps match):
Nightly builds for Maemo (you need both fennec*.deb and xulrunner*.deb where the timestamps match) and Windows Mobile (only the .cab file is needed) are located here:
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-trunk/
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2/
 
Windows mobile nightly builds are located here as well.  Download the .cab file:
ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-trunk/
 
== Banking and Financial Sites ==
 
Taken directly from the Firefox test plan.
 
The [[http://quality.mozilla.org/website-testing Website Testing Project]], will be utilized to track visited sites.
 
We will utilize this site during testdays which we plan to hold once every two months.


== Review Feedback ==
== Review Feedback ==
Line 408: Line 386:
* [http://www.samsung-omnia.org/third-party-samsung-omnia-software/ Omnia forum]
* [http://www.samsung-omnia.org/third-party-samsung-omnia-software/ Omnia forum]
* [http://forum.xda-developers.com/ xda forums] - more low level devs
* [http://forum.xda-developers.com/ xda forums] - more low level devs
== QA / Support Team Issues ==
SUMO will be a support channel for Fennec.  More details to come


== Automation ==
== Automation ==
Line 421: Line 395:
== Release Test Guidelines ==
== Release Test Guidelines ==
* QA Requirements:
* QA Requirements:
** E-mail to Drivers
** E-mail to Drivers pre/post-Release Cycle and pre/post-Respin
*** Includes:  
*** Includes:  
**** Build location link
**** Build location link
**** Bugs associated with the newest release build
**** Blocker Bugs added to the newest release build  
**** Release cycle schedule
**** Release cycle schedule
**** Dev Contact available on-hand
* List of <b>Disabled</b> Features for the Release
** Session Store
** Private Browsing
** Safebrowsing
** RSS
** Certificates Manager
** Find In Page
* Template for [https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/Template results tracking page]
* Template for [https://wiki.mozilla.org/QA/Fennec1.0/ReleaseTest/Template results tracking page]
* Updates Testing
** [http://wiki.mozilla.org/QA/Fennec1.0/TestPlan/UpdateHowTo/Maemo HowTo:Maemo]
** [http://wiki.mozilla.org/QA/Fennec1.0/TestPlan/UpdateHowTo/WinMo HowTo:WinMo]


* Betas
{| class="standard-table"
** Smoketest
|-
** BFTs (on betas 3+)
|class="header"|Release
** Bug Fix Verifications
|class="header"|Smoketest
 
|class="header"|BFT
* Major Releases
|class="header"|Blocker Verifications
** Smoketest
|class="header"|L10n
** BFTs
|class="header"|Website
** Bug Fix Verifications
|class="header"|Updates
** L10n Spotchecks
|-
** Website Spotchecks
|Early-Stage Betas
|YES
|NO
|YES
|1 RTL, 1 non-Latin, 1 Multi-L10n Switch
|NO
|YES
|-
|Late-Stage Betas (3+)
|YES
|YES
|YES
|1 RTL, 1 non-Latin, Full Multi-L10n Switch
|NO
|YES
|-
|Major Releases
|YES
|YES
|YES
|2 RTL, 2 non-Latin, Full Multi-L10n Switch
|YES
|YES
|}


== Additional References ==  
== Additional References ==  
Line 444: Line 454:


Useful Information:
Useful Information:
*Maemo Alpha1 [[http://www.mozilla.org/projects/fennec/1.0a1/releasenotes/ release notes]]
*Maemo Alpha2 [[http://www.mozilla.org/projects/fennec/1.0a2/releasenotes/ release notes]]
*Maemo Beta1 [[http://www.mozilla.org/projects/fennec/1.0b1/releasenotes/ release notes]]
*WinMo Alpha1 [[http://www.mozilla.org/projects/fennec/1.0a1-wm/releasenotes/ release notes]]
*Maemo Beta2 [[http://www.mozilla.org/projects/fennec/1.0b2/releasenotes/ release notes]]
*WiMmo Alpha2 [[http://www.mozilla.org/projects/fennec/1.0a2-wm/releasenotes/ release notes]]
*Original list of [[http://people.mozilla.com/~csejersen/fennec/req.html requirements]]
*Original list of [[http://people.mozilla.com/~csejersen/fennec/req.html requirements]]
*IRC Channels: #mobile, #wince, #symbian
*IRC Channels: #mobile, #wince, #symbian

Latest revision as of 23:09, 31 March 2010

« Mobile

Overview

Fennec 1.0 Test Strategy

The goal of this Overall Test Plan is to lay out our testing strategy, communicate it to others, and track the status.

Since Fennec is not installed or used in the same way Firefox is, the test approach needs to be modified. We will automate as much as possible. The automation we will use will be a subset (as all features are not in Fennec) of the Firefox automation with new tests developed to test specific issues of firefox. Key pieces of automation will be functionality, performance, and stability.

There will be a lot of unique tests which are device specific and Fennec specific that will be too expensive to automate. With these we will add them to litmus and run them by hand.

Items to Update

Here is a list of items that will be updated once we get more information:

Maemo 1.0 Release (RC1): 

* L10n - L10n Fennec Litmus test cases
* L10n - Spot check wiki and list of locales
* Install/Update (definition of repositories) - Update with betachannel and windows mobile
* Web compatibility Litmus test cases - update with web compatibility tool when available.
* QA/Support Team Issues
* How to scale testing to multiple devices
* How to test with wifi vs wireless carrier


Post Maemo 1.0: 

* Accessibility Litmus test cases
* Security Tools and test cases
* Stress Testing
* Performance regression plan (maybe just user performance, but probably more)

Schedule

Test Areas

General Test Areas

Feature Area and Spec Test Plan Location Testing Notes QA Contact Dev Contact Device Needed Links
Zoom gavin M
Add-ons manager mfinkle N
Awesome bar and search here jmaher enndeakin N
Multiple document Already some Firefox tests for this enndeakin N
Download Manager here jmaher mfinkle N
Running Firefox's mochitests here Status jmaher jmaher N
Updates N
Bookmarks N
Preferences N
Certificate Management johnath N
Password manager N
Popup Blocker N
Find neil Y
Browsing History gavin N

Mobile Specific Features

Feature Area and Spec Test Plan Location Testing Notes QA Contact Dev Contact Device Needed Links
Tel: uri blassey N
Navigation mfinkle, dougt M
Geolocation harthur dougt N
Panning gavin Y
Kinetic Scrolling Manual testing blassey Y
Weave dolske N

Platform Specific Features

Feature Area and Spec Test Plan Location Testing Notes QA Contact Dev Contact Device Needed Links
Windows Mobile here build notes jmaher wolfe Y
Performance Low disk/mem, large pages, startup time Y
Breakpad Y
Soft keyboard Y
Interop email, print, cut/paste, phone Dictionary, T9 christian Y
Maemo Keyboard open, close, access, hardkeys, user defined db48x, neil Y

Environments

The main environment that we will be focusing on for the 1.0 release will be the Nokia maemo platform (N8xx devices). In addition to the Maemo platform, we are working on Windows Mobile and Symbian support. These will be under concurrent development but not at the same feature complete and quality of the Maemo platform.

Platforms

Device OS Manufacturer Keyboard GPS Camera Rotate Resolution
Nokia N800 Maemo Nokia No Yes Yes No 480x800
Nokia N810 Maemo Nokia Slide Out Yes Yes No 480x800
HTC Touch Pro WinMobile Pro 6.1 HTC Slide Out Yes Yes Yes 480x640
ATT Tilt WinMobile Pro 6.1 HTC Slide Out Yes Yes Yes 240x320
Omnia WinMobile Pro 6.1 Samsung No Yes Yes Yes 400x240
Omnia 2 WinMobile Pro 6.1 Samsung No Yes Yes Yes 480x800
Treo Pro WinMobile Pro 6.1 Palm Yes Yes Yes No 320x320

Source Code and Nightly Build Locations

Fennec 1.0 is currently built from trunk (mozilla-central) as well as the front end mobile-browser repositories. To minimize branching and maintenance, we will target the 1.9.2 branch (splitting from mozilla-central July 2009). All other platforms will build from the same branch.

Nightly builds for Maemo (you need both fennec*.deb and xulrunner*.deb where the timestamps match) and Windows Mobile (only the .cab file is needed) are located here: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mobile-1.9.2/

Review Feedback

In this section, we will roll up all common feedback pulled from external sources that appears common and should be tracked by QA. Feedback sources can include:

Automation

QA is undertaking the task of porting the unit tests from Firefox to Fennec for each platform. This is a lengthy task, but our goal is to have 100% unit tests passing by the 1.0 ship.

We have created a tracking system for the unit tests which we update on a regular basis.

Release Test Guidelines

  • QA Requirements:
    • E-mail to Drivers pre/post-Release Cycle and pre/post-Respin
      • Includes:
        • Build location link
        • Blocker Bugs added to the newest release build
        • Release cycle schedule
        • Dev Contact available on-hand
  • List of Disabled Features for the Release
    • Session Store
    • Private Browsing
    • Safebrowsing
    • RSS
    • Certificates Manager
    • Find In Page
Release Smoketest BFT Blocker Verifications L10n Website Updates
Early-Stage Betas YES NO YES 1 RTL, 1 non-Latin, 1 Multi-L10n Switch NO YES
Late-Stage Betas (3+) YES YES YES 1 RTL, 1 non-Latin, Full Multi-L10n Switch NO YES
Major Releases YES YES YES 2 RTL, 2 non-Latin, Full Multi-L10n Switch YES YES

Additional References

This section is left to add any additional references relevant to the Mobile project

Useful Information:

  • Original list of [requirements]
  • IRC Channels: #mobile, #wince, #symbian

Building:

Testing:

Retrieved from "https://wiki.allizom.org/index.php?title=QA/Fennec1.0/TestPlan&oldid=212542"