B2G/QA/Device Test Plan/Graphics

Objective

Graphics features on FFOS are difficult to verify due to its impact across different functional areas. We need a solid testing approach to make sure the new graphics features are implemented properly, by verifying new features and improvements as well as detecting key regression issues early.

Background Information

Graphics Features Backlog

Graphics backlog is tracked in this spreadsheet.

Features that are applicable to FxOS (marked on the 'FxOS Priority' column) and currently active or completed ('Status' column marked as active, completed or scheduled) should be subjected to FxOS QA activity.

Image comparison library to Python Marionette Framework

Please refer to this page for more information.

Graphics Regression Test Cases

Currently, test cases for Firefox 2.0 which appear relevant to the graphics performance of FxOS has a tag 'gfxregtest' attached. There are 75 test cases, and selected based on following criteria:

• Renders the display images of application
• Involves the changes of the displayed images via scrolling, pinching, flicking, and other UI interactions
• Video Playback

Each features listed in above table need to be analyzed (when the feature is completed) to determine whether they need additional manual test cases defined. If so, they will contain same tag in mozTrap, as well as the tag indication the Bug ID of the feature.

Graphics Smoke Test Suite

Graphics Smoke Test Suite currently includes 9 test cases that can quickly identify graphics regressions. They can be found here.

Tasks

For each quarter, there are 12 work weeks, with roughly 480 work hours (per person).

Communicate with Graphics team

• Discuss with Graphics team about each active features for FxOS (from graphics features backlog) for possible areas of weakness and test strategy. When a feature becomes active and FxOS relevant, it should be discussed. Record findings in Bugzilla and use them for testing when the feature is completed.

• Look for the status changes on graphics features in graphics features backlog on weekly basis, and put the testing notes in the comments of the corresponding bug. Create a query to list all active graphics feature bugs. Mark bugs with keywords if necessary.

• Monitor graphics regression bugs in Bugzilla and be up-to-date on the blocking bug status.
• This query looks for all open graphics bugs that are blocking or nominated for blocking.
• This query looks for all blocked/ nominated for blocked graphics bugs that are resolved in past week.
• Use above queries as templates to create additional queries

• Resource Estimate: 24 hrs (2 hrs per week on average)
• Deliverable: Testing notes for each feature in Bugzilla, if needed

Automate Graphics Smoke Test Suite

• Automate existing graphics smoke test suite from Section 2.4, and have the test result available on web for public viewing. Push it to master branch, if possible

• Resource Estimate: 80 hrs
• Deliverable: Python Marionette test suite in Git repository, Update wiki

Execution of Graphics Smoke Test Suite

• Set up daily execution of graphics smoke test suite once all test cases are automated.

• Resource Estimate: 24 hrs
• Deliverable: Test log, Bugzilla entry

Execute manual tests

• Execution of exploratory time-boxed (~6 hours, one full work day) testing when each graphics feature is completed. Testing will cover basic graphics features described in Section 2.3 After the testing is completed, log the results

• Resource Estimate: 40 hrs (6hrs per feature on average)
• Deliverable: Test log, Bugzilla entry

Maintain Python image comparison library

• Update the library to handle screen capture command change in 2.1
• Update/create new test cases if needed
• Bug fixes/improvements

• Resource Estimate: 40 hrs
• Deliverable: Check-ins in Git repository, Update wiki

Total estimated hours: 208 hours (approximately 40~50% of one person workload for the quarter)

Task Tracking

In the weekly meetings with the manager, mini-goals and progress status will be presented for feedback. This spreadsheet will be used to track goals and work progress.