QA/Platform/Graphics: Difference between revisions
| Line 14: | Line 14: | ||
* Telemetry: [http://people.mozilla.org/~danderson/moz-gfx-telemetry/www/ GFX], [https://telemetry.mozilla.org/ All] | * Telemetry: [http://people.mozilla.org/~danderson/moz-gfx-telemetry/www/ GFX], [https://telemetry.mozilla.org/ All] | ||
* Bugzilla: [http://people.mozilla.org/~klahnakoski/modevmetrics/Bug-Counts.html#sampleInterval=week&sampleMax=2015-11-07&sampleMin=2012-10-28&productFilter=core&componentFilter=canvas%3A+2d,canvas%3A+webgl,gfx%3A+color+management,graphics,graphics%3A+layers,graphics%3A+text,image+blocking,imagelib,panning+and+zooming Bug Age] | * Bugzilla: [http://people.mozilla.org/~klahnakoski/modevmetrics/Bug-Counts.html#sampleInterval=week&sampleMax=2015-11-07&sampleMin=2012-10-28&productFilter=core&componentFilter=canvas%3A+2d,canvas%3A+webgl,gfx%3A+color+management,graphics,graphics%3A+layers,graphics%3A+text,image+blocking,imagelib,panning+and+zooming Bug Age] | ||
== The Team == | |||
In alphabetic order by last name: | |||
* Anthony Hughes, QA Engineer (aka ''ashughes'') | |||
* Jonathan Kew, Developer (aka ''jfkthame'') - primarily responsible for font/text rendering | |||
* Milan Sreckovic, Team Lead (aka ''milan'') | |||
= Get Involved = | = Get Involved = | ||
Revision as of 17:35, 26 April 2016
Summary
This page holds important information related to testing and quality assurance for Gecko's graphics code.
- About the team
- Major code changes worth tracking
- Inventory of hardware available for testing
- PCI Device ID Catalog (useful for determining hardware make/model)
- Codenames of NVIDIA chipsets
- Information about graphics driver blocklisting
- Draft guides related to Graphics QA
Metrics:
- Metrics-Graphics: staging, production
- Code Quality: Code Quality, Quality Indices
- Telemetry: GFX, All
- Bugzilla: Bug Age
The Team
In alphabetic order by last name:
- Anthony Hughes, QA Engineer (aka ashughes)
- Jonathan Kew, Developer (aka jfkthame) - primarily responsible for font/text rendering
- Milan Sreckovic, Team Lead (aka milan)
Get Involved
Here are some ways you can help the Graphics team (and Mozilla):
- Run a sanity test
- Add your details to our inventory
- Send an e-mail to introduce yourself and get further direction
Activities
One & Done Testing
DEPRECATION WARNING: One&Done is being decommissioned. In the meantime, all tasks have been archived to this wiki.
Betabreakers Lab Testing
Summary
There is a large discrepancy between the platform/hardware/driver configurations we see on Release versus what we see on our test branches. Additionally, the Graphics team and our testing community does not and can not provide sufficient test coverage to fill this gap. As a result we've historically been notified far too late about critical graphics issues triggering multiple chemspill releases and a loss of users.
In an effort to improve pre-release visibility of graphics regressions we have partnered with Betabreakers, a company that maintains a lab specifically designed to test various modern and legacy configurations of Windows versions, graphics hardware, and graphics drivers. We try to coordinate a graphics sanity testrun with them once per mozilla-aurora cycle, as detailed below.
More information about our relationship with Betabreakers can be found on mana.mozilla.org.
Test Runs
- Firefox 50: Hardware Accelerated Video
- [DONE] Firefox 48: Windows 10 Anniversary Update
- [DONE] Firefox 48: WARP disabled further
- [DONE] Firefox 46: HTML5 video with hardware acceleration disabled
- [DONE] Firefox 45: WinXP D3D9 E10S compatibility
- [DONE] Firefox 44: Country-specific top sites
- [DONE] Firefox 43: Typical sanity check plus bug cleanup
- [DONE] Firefox 42: Windows 10 AMD/NVidia stability
- [DONE] Firefox 41: WARP disabled on Windows 7 and earlier
- [DONE] Firefox 40: WebGL with e10s
- [DONE] Firefox 39: beta sanity check
- [DONE] Firefox 38: MSE stress test
Bugs
99 Total; 5 Open (5.05%); 90 Resolved (90.91%); 4 Verified (4.04%);
Triage
Get Involved
Want to help with or learn about triage? Contact me!
- E-mail: ahughes@mozilla.com
- IRC: ashughes in #gfx or #qa on irc.mozilla.org
- Dashboard
Help Wanted
Once per week, typically on Monday, we review bug reports tagged with help-wanted older than 30 days to ensure these issue are still relevant and on track to resolution.
How to Triage:
- Select a bug from this list
- Review the bug to make sure you understand the bug report and the information needed to move it forward
- Attempt to reproduce the bug
- If you can reproduce the bug, attempt to find a regression window either manually or using mozregression
- Update the bug report with any new information you can provide
- If you can't reproduce the issue or don't understand the issue, move on to the next bug report
Whenever updating a bug report be sure to add a comment explaining the change.
Incoming
Once per week, typically on Tuesday, we review bug reports that were reported more than 30 days ago but have yet to be triaged by the Graphics team.
How to Triage
- Select a bug from this list (it's best sort by most recent changed first)
- Review the information in the bug and make sure you understand the issue
- If the bug is a feature request, add the feature keyword and move on to the next bug
- Request for information from the report if there is information missing (regression window, steps to reproduce, minimized testcase, screenshots, system information, etc)
- If you can reproduce the issue as described then update the bug and escalate it to a graphics developer
- If you cannot reproduce the issue, ask a developer or the reporter if the bug is still valid
- Be sure to add the gfx-noted tag to the whiteboard field once triaged
- Move on to the next bug report if you can
Top Crashes
Once per week, typically on Wednesday, we review our top crash reports to ensure these issues have bug reports on track toward resolution.
- Explosiveness Report: http://bit.ly/1N9iR7j
- Firefox Topcrash Report: http://bit.ly/1VqmLMm
- Fennec Topcrash Report: http://bit.ly/1N9iTfG
- How to Triage
Note: Whenever updating a bug report be sure to add a comment explaining the change.
- Review the reports above
- If you see a graphics crash that doesn't have a bug report, file a bug report for it
- If you see a graphics crash with an associated bug ID, open the bug report and update the statistics for the bug
- If the crash is escalating and is now in top-crash territory, add the topcrash keyword and nominate the bug to track
- If the crash is declining and is no longer in top-crash territory, remove the topcrash keyword
- Repeat the process for each crash you find in the list
Cold Crashes
Once per week, typically on Thursday, we review crash bug reports we have on file that have not been updated in more than 30 days to ensure these issues are still relevant and on track to resolution.
How to Triage:
- Load this list of bugs and sort by most recent changed
- Open the first bug report and review the information
- Check if there are anymore crash reports by clicking the signature link
- If there are no more crash reports, close the bug
- If there are still bug reports, update the statistics in the bug report and see if you can reproduce
- If you can't reproduce, ask the reporter for new information
- If you can reproduce, ask the developer what information is needed to move the bug forward
- Update the bug with any new information and move on to the next bug report
Cold Trackers
One per week, typically on Friday, we review tracked bug reports that have not been updated in more than 30 days to ensure these issues are still relevant and on track to resolution.
How to Triage:
- Select a bug from this list (sort by most recent changed)
- Review the information in the bug report
- Try to deduce if the bug is still relevant through testing, reviewing linked reports, or asking people involved on the bug
- If the bug is still relevant, update the status/tracking flags, and needinfo flag the developer for a status update
- If the bug is no longer relevant, update the status flags and close the bug report
- Once you've updated the bug move on to the next bug report
Cold Regressions
One per week we review old regression bug reports that have not been updated in more than 42 days to ensure these issues are still relevant and on track to resolution.
- How to Triage
1. Select a bug from this list and review the information in the report
2. Make sure each bug has the following information:
- a regression window (see step 4 if it doesn't)
- [gfx-noted] in the whiteboard field
- Has Regression Window and Has STR flags set to the correct value
- 'status-firefox flags set to the correct values (old versions can be cleared, new versions can be set to ? if unknown)
3. Use the needinfo? flag and ask the bug reporter to test if the bug still occurs in the current Firefox Nightly
4. (Optional) See if you can reproduce the bug yourself and add a comment to the bug describing your test environment and result. If you're able to reproduce the bug use Mozregression to find the regression window. Be sure to use the needinfo:? flag to bring the bug to my attention (ashughes).
Retired Activities
- Fennec Skia Sanity Checks (2015)
- B2G Sanity Checks (2015)
- Toronto Lab Testing (2015)
Top Issues
Blockers
No results.
0 Total; 0 Open (0%); 0 Resolved (0%); 0 Verified (0%);
Crashes
11 Total; 11 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Features
40 Total; 40 Open (100%); 0 Resolved (0%); 0 Verified (0%);
New Issues
14 Total; 14 Open (100%); 0 Resolved (0%); 0 Verified (0%);
Understanding the Problem Space
First order of business for my transition to the Graphics team is to understand the problem space so I can understand the immediate needs of the team and make the best impact I can in the shortest amount of time.
- What are the key problems/challenges facing the Graphics team in terms of quality?
- discrepancy in environments between testers and release users
- discoverability of bugs pre-release
- ?...
- Where can QA add value/support to the Graphics team?
- improving pre-release discoverability of bugs
- closing the gap between tester and release systems
- helping with bug triage, particularly with bugs hiding in general components
- representation in crashkill
- improving code coverage and/or identifying gaps in code coverage
- identifying ways to improve participation in the graphics team (events, projects, One & Done, etc)
- documentation of tools, testing processes, etc
- building out the lab in Toronto
- continuing to drive Betabreakers testing every 6 weeks
- verifying bug fixes (what does this look like)?
- profiling areas of risk (eg. troublesome configs)
- conducting root cause analysis for regressions
- understanding problems outside of our control (eg. driver resets)
- feature testing and upcoming priorities (e10s, Windows 10, El Capitain, Android, B2G, etc)
- What does QA need to know to be effective?
- key components of an actionable Graphics bug
- fundamentals/technologies that should be learned
- how to distinguish a graphics crash from a non-graphics crash with a graphics signature
- meetings, mailing lists, bugzilla components to watch, blogs, IRC channels to join, etc
- who is each member of the team (incl. contributors) and what do they do
- where does graphics code reside in the tree?
- what role does Unified Telemetry in graphics quality?
- what are the prefs to enable/disable different functionalities?
- we need a database of known-troublesome hardware/driver configurations to inform testing, hardware acquisitions, and blocklisting
- Understanding the Stability
- How do we identify a graphics crash?
- by signature: gfx, layers, D2D, D3D, ?...
- by topmost filename: gfx, ?...
- by driver (DLL, version, ?...)
- by device/vendor ID?...
- ?...
- How do we prioritize graphics crashes?
- Overall topcrashes in release > beta > aurora > nightly
- Gfx crashes in release > beta > aurora > nightly
- Explosive crashes in release > beta > aurora > nightly
- What tools do we have at our disposal to investigate crashes?
- Bughunter for investigating crashes correlated to a URL
- KaiRo's reports for identifying crashes that are new or escalating quickly
- Socorro for getting detailed information about crash reports
- What information is needed to make a crash actionable by developers?
- Correlations to particular hardware, driver, add-on, 3rd-party software, or library
- ?...
- Participation
- Sanity checking via One & Done
- Meetups to connect testers/users with devs
- Testdays to teach people about graphics testing
- Documentation and translation of documentation
- Engaging on community spaces (Discourse, Reddit, Facebook, Twitter, etc)
- Telemetry
- COMPOSITE_TIME: time in CompositorParent::CompositeToTarget dispatching draw calls and calling SwapBuffers, but not texture upload (ie. complete composition)