B2G/QA/Device Test Plan/Graphics: Difference between revisions

From MozillaWiki
< B2G‎ | QA‎ | Device Test Plan
Jump to navigation Jump to search
Line 27: Line 27:


For each quarter, there are 12 work weeks, with roughly 480 work hours (per person).
For each quarter, there are 12 work weeks, with roughly 480 work hours (per person).
{| class="wikitable"
=== Communicate with Graphics team ===
|-
* Discuss with Graphics team about each active features for FxOS (from graphics features backlog) for possible areas of weakness and test strategy. When a feature becomes active and FxOS relevant, it should be discussed.  Record findings in Bugzilla and use them for testing when the feature is completed.
! Tasks !! Task Description (about what, how often, with who) !! Resource Est. !! Deliverable
   
|-
* Look for the status changes on graphics features in graphics features backlog on weekly basis, and put the testing notes in the comments of the corresponding bug.  Create a query to list all active graphics feature bugs.  Mark bugs with keywords if necessary.
| Example || Example || Example || Example
   
|-
* Monitor graphics regression bugs in Bugzilla and be up-to-date on the blocking bug status.
| Example || Example || Example || Example
* This query looks for all open graphics bugs that are blocking or nominated for blocking.
|-
* This query looks for all blocked/ nominated for blocked graphics bugs that are resolved in past week.
| Example || Example || Example || Example
* Use above queries as templates to create additional queries
|-
| Example || Example || Example || Example
|-
| Example || Example || Example || Example
|-
| Example || Example || Example || Example
|}


* Resource Estimate: 24 hrs (2 hrs per week on average)
* Deliverable: Testing notes for each feature in Bugzilla, if needed


=== Automate Graphics Smoke Test Suite ===
Communicate with Graphics team
* Automate existing graphics smoke test suite from Section 2.4, and have the test result available on web for public viewing.  Push it to master branch, if possible


    Discuss with Graphics team about each active features for FxOS (from graphics features backlog) for possible areas of weakness and test strategy. When a feature becomes active and FxOS relevant, it should be discussed.  Record findings in Bugzilla and use them for testing when the feature is completed.
* Resource Estimate: 80 hrs
    Look for the status changes on graphics features in graphics features backlog on weekly basis, and put the testing notes in the comments of the corresponding bug.  Create a query to list all active graphics feature bugs.  Mark bugs with keywords if necessary.
* Deliverable: Python Marionette test suite in Git repository, Update wiki
    Monitor graphics regression bugs in Bugzilla and be up-to-date on the blocking bug status
        This query looks for all open graphics bugs that are blocking or nominated for blocking.
        This query looks for all blocked/ nominated for blocked graphics bugs that are resolved in past week.
        Use above queries as templates to create additional queries


=== Execution of Graphics Smoke Test Suite ===
* Set up daily execution of graphics smoke test suite once all test cases are automated.


24 hrs
* Resource Estimate: 24 hrs
* Deliverable: Test log, Bugzilla entry


(2 hrs per week on average)
=== Execute manual tests ===
* Execution of exploratory time-boxed (~6 hours, one full work day) testing when each graphics feature is completed.  Testing will cover basic graphics features described in Section 2.3  After the testing is completed, log the results


    Testing notes for each feature in Bugzilla, if needed
* Resource Estimate: 40 hrs (6hrs per feature on average)
* Deliverable: Test log, Bugzilla entry


Automate Graphics Smoke Test Suite
=== Maintain Python image comparison library ===
* Update the library to handle screen capture command change in 2.1
* Update/create new test cases if needed
* Bug fixes/improvements


    Automate existing graphics smoke test suite from Section 2.4, and have the test result available on web for public viewing
* Resource Estimate: 40 hrs
    Push it to master branch, if possible
* Deliverable: Check-ins in Git repository, Update wiki


80 hrs
Total estimated hours: 208 hours (approximately 40~50% of one person workload for the quarter)
 
    Python Marionette test suite in Git repository
    Update wiki
 
Execution of Graphics Smoke Test Suite
 
    Set up daily execution of graphics smoke test suite once all test cases are automated.
 
24 hrs
 
    Test log
    Bugzilla entry
 
Execute manual tests


    Execution of exploratory time-boxed (~6 hours, one full work day) testing when each graphics feature is completed
    Testing will cover basic graphics features described in Section 2.3
    After the testing is completed, log the results
40 hrs (6hrs per feature on average)
    Test log
    Bugzilla entry
Maintain Python image comparison library
    Update the library to handle screen capture command change in 2.1
    Update/create new test cases if needed
    Bug fixes/improvements
40 hrs
    Check-ins in Git repository
    Update wiki
Total estimated hours: 208 hours (approximately 40~50% of one person workload for the quarter)
== Task Tracking ==
== Task Tracking ==
In the weekly meetings with the manager, mini-goals and progress status will be presented for feedback.  This spreadsheet will be used to track goals and work progress.
In the weekly meetings with the manager, mini-goals and progress status will be presented for feedback.  This spreadsheet will be used to track goals and work progress.

Revision as of 20:26, 24 June 2014

Objective

Graphics features on FFOS are difficult to verify due to its impact across different functional areas. We need a solid testing approach to make sure the new graphics features are implemented properly, by verifying new features and improvements as well as detecting key regression issues early.

Background Information

Graphics Features Backlog

Graphics backlog is tracked in this spreadsheet.

Features that are applicable to FxOS (marked on the 'FxOS Priority' column) and currently active or completed ('Status' column marked as active, completed or scheduled) should be subjected to FxOS QA activity.

Image comparison library to Python Marionette Framework

Please refer to this page for more information.

Graphics Regression Test Cases

Currently, test cases for Firefox 2.0 which appear relevant to the graphics performance of FxOS has a tag 'gfxregtest' attached. There are 75 test cases, and selected based on following criteria:

  • Renders the display images of application
  • Involves the changes of the displayed images via scrolling, pinching, flicking, and other UI interactions
  • Video Playback

Each features listed in above table need to be analyzed (when the feature is completed) to determine whether they need additional manual test cases defined. If so, they will contain same tag in mozTrap, as well as the tag indication the Bug ID of the feature.

Graphics Smoke Test Suite

Graphics Smoke Test Suite currently includes 9 test cases that can quickly identify graphics regressions. They can be found here.

Tasks

For each quarter, there are 12 work weeks, with roughly 480 work hours (per person).

Communicate with Graphics team

  • Discuss with Graphics team about each active features for FxOS (from graphics features backlog) for possible areas of weakness and test strategy. When a feature becomes active and FxOS relevant, it should be discussed. Record findings in Bugzilla and use them for testing when the feature is completed.
  • Look for the status changes on graphics features in graphics features backlog on weekly basis, and put the testing notes in the comments of the corresponding bug. Create a query to list all active graphics feature bugs. Mark bugs with keywords if necessary.
  • Monitor graphics regression bugs in Bugzilla and be up-to-date on the blocking bug status.
  • This query looks for all open graphics bugs that are blocking or nominated for blocking.
  • This query looks for all blocked/ nominated for blocked graphics bugs that are resolved in past week.
  • Use above queries as templates to create additional queries
  • Resource Estimate: 24 hrs (2 hrs per week on average)
  • Deliverable: Testing notes for each feature in Bugzilla, if needed

Automate Graphics Smoke Test Suite

  • Automate existing graphics smoke test suite from Section 2.4, and have the test result available on web for public viewing. Push it to master branch, if possible
  • Resource Estimate: 80 hrs
  • Deliverable: Python Marionette test suite in Git repository, Update wiki

Execution of Graphics Smoke Test Suite

  • Set up daily execution of graphics smoke test suite once all test cases are automated.
  • Resource Estimate: 24 hrs
  • Deliverable: Test log, Bugzilla entry

Execute manual tests

  • Execution of exploratory time-boxed (~6 hours, one full work day) testing when each graphics feature is completed. Testing will cover basic graphics features described in Section 2.3 After the testing is completed, log the results
  • Resource Estimate: 40 hrs (6hrs per feature on average)
  • Deliverable: Test log, Bugzilla entry

Maintain Python image comparison library

  • Update the library to handle screen capture command change in 2.1
  • Update/create new test cases if needed
  • Bug fixes/improvements
  • Resource Estimate: 40 hrs
  • Deliverable: Check-ins in Git repository, Update wiki

Total estimated hours: 208 hours (approximately 40~50% of one person workload for the quarter)

Task Tracking

In the weekly meetings with the manager, mini-goals and progress status will be presented for feedback. This spreadsheet will be used to track goals and work progress.

Retrieved from "https://wiki.allizom.org/index.php?title=B2G/QA/Device_Test_Plan/Graphics&oldid=992064"