352
edits
B2G/QA/Device Test Plan/Graphics: Difference between revisions
B2G/QA/Device Test Plan/Graphics (view source)
Revision as of 20:57, 28 August 2014
, 28 August 2014no edit summary
No edit summary |
|||
| (20 intermediate revisions by 2 users not shown) | |||
| Line 12: | Line 12: | ||
Please refer to [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/Run_ImageMagick_Tool_with_Python_Marionette this page] for more information. | Please refer to [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/Run_ImageMagick_Tool_with_Python_Marionette this page] for more information. | ||
=== Graphics Smoke Test Suite === | |||
The purpose of smoke test is to detect graphics regression issues early on in the master branch. Sanity check of graphics performance will be tested on areas such as: | |||
* Scrolling of homescreen | |||
* Scrolling of web contents and rendered images | |||
* Swiping between applications and between images | |||
* Rendering of images and texts | |||
* Manipulation of images and texts in applications, including browser app | |||
* Rendering of open and closing of applications (including animations) | |||
* Video playback | |||
* Camera preview and image/video recording | |||
* Orientation change | |||
It is assumed that any new graphics feature implementation will affect one of more activities listed above. | |||
Graphics Smoke Test Suite currently includes 9 test cases that can quickly identify graphics regressions. They can be found [https://moztrap.mozilla.org/manage/cases/?filter-suite=669 here.] | |||
The smoke test suite should run daily once automated, and for the cases that cannot be automated or yet to be automated, it will be verified on weekly basis as a minimum. | |||
Examples of manual-only tests include: | |||
* Video playback | |||
* Homescreen animation (the one that follows unlocking the screen, making sure there are at least 20 extra apps installed for a total of more than 60 icons on the home screen) | |||
* Camera preview / image / video recording | |||
=== Graphics Regression Test Cases === | === Graphics Regression Test Cases === | ||
| Line 20: | Line 43: | ||
* Video Playback | * Video Playback | ||
Each | Each feature listed in Graphics Features Backlog needs to be analyzed (when the feature is completed) to determine whether they need additional manual test cases defined. If so, new test cases would also contain 'gfxregtest' tag in mozTrap, as well as the tag indicating the Bug ID of the feature. | ||
The Regression Test Suite will be executed 3 times per release cycle, as part of the full functional test run. | |||
== Tasks == | == Tasks == | ||
=== Communicate with Graphics team === | === Communicate with Graphics team === | ||
* Graphics Bug Triage: monitor incoming [https://bugzilla.mozilla.org/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=Graphics%20Blocking%20%2F%20Blocking%20Candidates&sharer_id=499933&list_id=10565678 blocking-nominated] bugs, and provide feedback during the triage session. Blocking criteria are: | |||
::* '''Feature regressing''' | |||
::* '''Blocking the successful completion of a common use case''' | |||
::* '''Risk of causing data loss, system freeze, or major user frustration''' | |||
::* '''Obvious rendering issues or performance issues such as checkerboarding or screen refresh issue''' | |||
Any graphics bugs that fall under one or more of above criteria should be nominated(with '''?''' flag) for the upcoming release in the '''b2g-release''' field. If a bug does not satisfy any of above criteria, mark it with ''''backlog'''' flag. | |||
* Discuss with Graphics team about each active features for FxOS (from [https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AnKFEBp1-VyqdFNfRlZmV0ExM0VvZGMxNThWX0d6LWc&usp=drive_web#gid=0 graphics features backlog]) for possible areas of weakness and test strategy. When a feature becomes active and FxOS relevant, it should be discussed. Record findings in Bugzilla and use them for testing when the feature is completed. | * Discuss with Graphics team about each active features for FxOS (from [https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AnKFEBp1-VyqdFNfRlZmV0ExM0VvZGMxNThWX0d6LWc&usp=drive_web#gid=0 graphics features backlog]) for possible areas of weakness and test strategy. When a feature becomes active and FxOS relevant, it should be discussed. Record findings in Bugzilla and use them for testing when the feature is completed. | ||
| Line 46: | Line 75: | ||
* Resource Estimate: 80 hrs | * Resource Estimate: 80 hrs | ||
* Deliverable: Python Marionette test suite in Git repository, Update [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/Run_ImageMagick_Tool_with_Python_Marionette wiki] | * Deliverable: [https://github.com/npark-mozilla/gaia/tree/ImageCompare/tests/python/gaia-ui-tests/gaiatest/tests/functional/imagecompare Python Marionette test suite in Git repository], Update [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/Run_ImageMagick_Tool_with_Python_Marionette wiki] | ||
=== Execution of Graphics Smoke Test Suite === | === Execution of Graphics Smoke Test Suite === | ||
| Line 58: | Line 87: | ||
* Resource Estimate: 40 hrs (6hrs per feature on average) | * Resource Estimate: 40 hrs (6hrs per feature on average) | ||
* Deliverable: Test log, Bugzilla entry | * Deliverable: [https://docs.google.com/a/mozilla.com/spreadsheet/ccc?key=0AiFZ9Y4ah19bdHBhVVVlZlgyUnFkVURHSU1DMDRsOUE#gid=1 Test log], Bugzilla entry | ||
=== Maintain Python image comparison library === | === Maintain Python image comparison library === | ||
| Line 66: | Line 95: | ||
* Resource Estimate: 40 hrs | * Resource Estimate: 40 hrs | ||
* Deliverable: Check-ins in Git repository, Update wiki | * Deliverable: [https://github.com/npark-mozilla/gaia/tree/ImageCompare/tests/python/gaia-ui-tests/ Check-ins in Git repository], Update [https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/Run_ImageMagick_Tool_with_Python_Marionette wiki] | ||
Total estimated hours: 208 hours (approximately 40~50% of one person workload for the quarter) | === Areas Not Covered === | ||
Following areas are not covered by this plan: | |||
* Graphics [https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Reftests Mochitests] [https://developer.mozilla.org/en-US/docs/Mochitest / Reftests] | |||
* Graphics [https://developer.mozilla.org/en-US/Apps/Build/Performance/Firefox_OS_performance_testing performance measurements] (e.g. APZ performance, FPS for image rendering, etc.) | |||
'''Total estimated hours: 208 hours (approximately 40~50% of one person workload for the quarter) | |||
''' | |||
== Task Tracking == | == Task Tracking == | ||
In the weekly meetings with the manager, mini-goals and progress status will be presented for feedback. This spreadsheet will be used to track goals and work progress. | In the weekly meetings with the manager, mini-goals and progress status will be presented for feedback. [https://docs.google.com/a/mozilla.com/spreadsheets/d/1uO1VPEbNupUMfcSPa5FJX-k9GxqLgkiIV-4IRcXxkL8/edit#gid=0 This spreadsheet] will be used to track goals and work progress. | ||
== Others == | |||
=== Useful sites for Browser Testing === | |||
* To test CSS/Canvas/Layout on Browser [http://people.mozilla.org/~mwargers/tests/layout/layout_demos.htm] | |||
* More Layout tests on Browser [http://people.mozilla.org/~mwargers/tests/layout/] | |||
* Panning tests on Browser [http://people.mozilla.org/~mwargers/tests/panning/] | |||