QA/Fennec/Milestones/5.0b1

From MozillaWiki
Jump to navigation Jump to search

Fennec 5.0 Beta Test Plan

Overview

The goal of this Overall Test Plan is to lay out our testing strategy, determine who will own certain test areas and when they will do it and, finally, track the status. All of our releases are tested off of the "Fennec 5.0 Beta Catch-All Test Run" on the Litmus TCM. For more information about Fennec 5.0, take a look at the mobile team's Feature Planning page.

Project contacts

PM: Thomas Arend (tarend)
Dev Lead: Stuart Parmenter (stuart)
Build: John Oduinn (oduinn)
QA: Aakash Desai (aakashd)

Schedule

  • Start Date: 5/17/2011
  • End Date: 5/21/2011

Bugs in Release

Build Location and IDs

  1. Go to the nightly builds folder in our ftp directory
  2. Go to the "latest-mozilla-beta" folder
  3. Download and install the candidate onto your Android phone/device

Features Landed

Feature Status Owner
Optimized layout and interaction with some mobile web pages (improved position:fixed support) landed

SDR: N || SIR: N QA: Verified

Thomas
Add images downloaded in Firefox to the Android Gallery immediately (before: images were visible only after reboot landed

SDR: N || SIR: C - 04/27 QA: Verified

Thomas
Add a "More" button to the Android Menu when needed) landed

SDR: N || SIR: N QA: Verified

Thomas
Fix issue with color conversion for better display of images and video (Fix broken NEON Y'CbCr to RGB conversion) landed

SDR: N || SIR: P  QA: Verified

Thomas
Resolves recent breakage that would show an "undefined" error when trying to load web pages while outside of wifi or network coverave (Add remoteXUL strings for Firefox Mobile) landed

SDR: N || SIR: N QA: Verified

Thomas
Slight improvement of page load speed, especially on 3G network (e10s HTTP: Implement nsIProgressEventSink:part2) landed

SDR: N || SIR: N QA: Verified

Thomas
Add user-visible pref for Do Not Track

SDR: N || SIR: N QA: Verified/Tests Created

landed Thomas
Finish new theme styles for dialogs/prompts (prompts and confirms look like android dialogs) landed

SDR: N || SIR: C - 04-27 QA: Verified

Thomas
Smooth panning (resolves an issue where consecutive pans were a bit jumpy before) landed

SDR: N || SIR: N QA: Verified

Thomas

Test Areas

Smoketests

QA Task Person/Group How Often?
Smoketests IoanaChiorean Once

Lite verification tests (or smoketests) will be first run when the builds are ready. Tests are available on litmus under the Fennec 2.0 Test Suite. In any case of failure in any Smoketest, blockers need to be raised ASAP before proceeding to the BFTs.

Tests are available on litmus under the Fennec 2.0 Test Suite.

Results details can be found in Waverley's results wiki.

Results:

  • Device: Samsung Google Nexus S (Android 2.3)
  • Build: Mozilla /5.0 (Android;Linux armv7l;rv:5.0) Gecko/20110517 Firefox/5.0 Fennec/5.0
  • Results Fennec 5.0b2 build 1 :

Known bugs:

  • bug 628269 - Pages are zoomed in at reload
  • bug 639504 - Unable to zoom in/out a webpage on header/footer when a double tap gesture is made
  • bug 625981 - places.sqlite-wal takes too much space for Fennec

Virtual Machine (DeviceAnywhere)

QA Task Person/Group How Often?
DeviceAnywhere Studio 5.1 IoanaChiorean Once

Tests are available on litmus under the Fennec 2.0 Test Suite.

Results details can be found in Waverley's results wiki.

Results:

  • Device: Mototrola Droid X (MB810)( (Android 2.3)
  • Build: Mozilla /5.0 (Android;Linux armv7l;rv:5.0) Gecko/20110517 Firefox/5.0 Fennec/5.0
  • Results Fennec 5.0b2 build 1 :

New issues:

  • strange line in the left part (as if the right panel is overlaying the page) - screenshot - should be posted bugs also for DeviceAnywhere?
  • bug 658276 - Notification for Geolocation is not dismissed after choosing option Share

Known bugs:

  • all bugs found on Smoketest or BFTs

BFTs for 5.0 Features

QA Task Person/Group How Often?
BFTs Anna and Andreea Once

BFTs are a more comprehensive set of litmus tests that cover functional testing of features that have landed on Fennec 2.0. Tests are available on litmus under the Fennec 2.0 Test Suite.

Results details can be found in Waverley's results wiki.

  • Device: HTC Desire Z (Android 2.2) and LG Optimus 2X (Android 2.2)
  • Build: Mozilla /5.0 (Android;Linux armv7l;rv:5.0) Gecko/20110517 Firefox/5.0 Fennec/5.0
  • Results: PASS
  • Notes: Fennec 5.0b2 build 1
    • Unreproducible issues:
      • I logged in litmus using my account, I logged out and this was the result: screenshot
    • Reported bugs:
      • bug 658200: Different number of languages available on Fennec 5.0b2 build 1 from Nightly and Aurora
      • bug 658232: [HWKB] "Delete" key doesn't erase the name of a bookmark from the Bookmarks Context Menu
      • bug 658264: Right sidebar is not pannable
      • bug 658214: Improper rendering in about:config when panning to left or right and when adding a new preference
      • bug 658234: Panning outside of an Iframe causes the content of the frame to scroll
      • bug 658268: Beta branding doesn't appears on RTL
      • bug 658315: Options from "Product:" field disappear when selecting one of the items in bugzilla
      • bug 658323: Inconsistency at vkb in landscape
    • Related bugs:
      • bug 655864: Aurora: password field is not masked immediately and shows artifacts, when typing past the end of the field -> I can also see this mainly when typing in the username field from login pages, but it's not always reproducible. Meanwhile the bug was marked as RESOLVED FIXED on nightly (the 19th build) but it still reproducing on beta.
    • Installation
      • No updates from 5.0b1 to 5.0b2. Works as expected after talking with kbrosnan
    • Options (Preferences):
      • No restart notification appears after selecting Yes/No for Enable the error Console in the Feedback Panel
      • From Feedback Panel: when selecting "Go to page" button I get: "404 Error: Page Not Found"
    • General
      • Google instant blocks the "I'm Feeling Lucky" feature.
      • known bugs:
    • Location bar & autocomplete
      • access the location bar entered location list: The hardware keyboard of HTC Desire Z does not have down, up, left, right arrows
      • known bugs:
    • Tabbed Browsing
      • known bugs:
        • bug 639794: Opening a webpage from an external app should hide sidebars and panels
    • Bookmarks
      • known bugs:
        • bug 648527: Bookmark tags are not saved (Note: On a fresh installed build I can still reproduce this issue, but on a dirty profile I can save tags for a bookmarked page.)
    • Downloading
      • known bugs:
    • Layout
      • known bugs:
        • bug 580920: Acid 2 test fails in Fennec (happens only in portrait mode)
    • Form assistant
      • When typing in the search field from bugzilla, the autocomplete suggestions bar appears only after erasing the search term from the search field. Is this intended ?
    • Cookies
      • known issues:
        • bug 625981: places.sqlite-wal takes too much space for Fennec
    • Search
      • known issues:
        • bug 650817: Search engines list misinterprets tap and pan
        • bug 641876: Popup messages overlap if two different search engine links are tapped
    • Security
      • known bugs:
        • bug 341604: (framesandbox) Implement HTML5 sandbox attribute for IFRAMEs
        • bug 443564: Implement window.toStaticHTML
        • bug 446344: Implement Origin header CSRF mitigation
        • bug 471020: Add X-Content-Type-Options: nosniff support to Firefox
        • bug 528661: Heuristics to block reflected XSS (like in IE8)
        • bug 607613: HttpOnly cookies flushed by non-HttpOnly cookies (useful for session fixation)
        • bug 641876: Popup messages overlap if two different search engine links are tapped
    • Sync
      • known bugs:
        • bug 652831: Drop down menu flickers on Bugzilla on text fields.
        • bug 625981: places.sqlite-wal takes too much space for Fennec
        • bug 656594: Sync Now button is active even if the Sync is in progress
    • System pages
      • known bugs:
        • bug 641836: [Regression]The search input should get the focus after loading about:config
    • Web usage
      • known issues:
        • bug 544614: Implement touch events (I'm not able to pan the map on maps.google.com)
    • Screen navigation
      • known issues:
        • bug 639504: Unable to zoom in/out a webpage on header/footer when a double tap gesture is made
        • bug 658234: Panning outside of an Iframe causes the content of the frame to scroll
    • RTL
      • known bugs:
        • bug 634386: [RTL] Arabic pages don't render correctly

Crasher Triage

QA Task Person/Group How Often?
Crasher Triage mw22 / nhirata Daily

Socorro is Mozilla's primary web interface for triaging crash reports sent in by our user base. Due to the relatively large amount of changes coming in on a nightly basis, there is a need to check for crash reports coming in for every build. The owner of this task will triage through top crashers on a daily basis, file bugs if necessary and find steps to reproduce for those crashers if possible.

Device Compatibility

QA Task Person/Group How Often?
Device Compatibility AaronMT Once

As part of our testing strategy, we will want to verify device compatibiltiy checking for regressions or major issues in Firefox by running through our compatbility criteria spreadsheet . Mobile users can also contribute towards device qualifying by submitting feedback via our untested device survey.

Website Compatibility

QA Task Person/Group How Often?
Website Compatibility Aaronmt Once

As part of our testing cycle we want to cover and include a general testing process that focuses on website compatbility with sites viewed in mobile Firefox. We want you to use mobile Firefox to assist us in identifying any noticeably major issues found in websites. Issues can range from pages that do not look right, layout and positioning problems, interactivity issues, font issues and pages that grealy differ from browser to browser and what you would expect. You can help us by completing a website exploration form here, and following the companion instructional test-plan.

Testday

QA Task Person/Group How Often?
Testday kbrosnan Once

To continue determining better quality metrics, we'll want to work with our community to find new bugs and performance/stability metrics through the use of a Testday once per cycle per channel (i.e. nightly and beta). The QA owner for this task will send announcements, create a testplan, moderate the channel and post results for the Testday. To see when the next testday is schedule, please take a look at the Test_Days wiki.

Retrieved from "https://wiki.allizom.org/index.php?title=QA/Fennec/Milestones/5.0b1&oldid=310583"