MozillaWiki

QA/Content Handling: Difference between revisions

Page Discussion

QA/Content Handling (view source)

Revision as of 08:52, 24 September 2016

6,280 bytes added ,  24 September 2016
→‎Ownership
 
(36 intermediate revisions by 3 users not shown)
Line 9: Line 9:
| 05/24/2016 || 1.0 || William Hsu|| Initial draft
| 05/24/2016 || 1.0 || William Hsu|| Initial draft
|-
|-
| || || ||  
| 09/12/2016|| 1.1 || Adrian Florinescu || Updates
|-
| 09/22/2016|| 2.0 || Adrian Florinescu || Added Feature (Sub)sections chapter and table
|}
|}
<div style="float: right; margin: 0px 0px 10px 25px;">[[File:Content_Handling.png|800px]]</div>


= Overview =
= Overview =
Line 23: Line 26:
* The test schedule and major milestones
* The test schedule and major milestones
* The test deliverables
* The test deliverables
== Feature (sub)Sections==
This section is meant for features that are to be implemented in several iterations, hence landing and releasing parts of it before completion will be a reality. While the main TestPlan is to be updated and followed for the full feature, it makes more sense to have particularised TestPlans for each small iteration and the data which will be relevant for those will not be necessarily relevant for this test plan which is meant to cover the full feature.
{| class="wikitable"
|-
! Ref !! Subsection !! (sub)Metabug - if available  !! Landed Date  !! Version Landed !! Train Progress !! TestPlan !! QA Owners
|-
| 1 || Download Summary || [https://bugzilla.mozilla.org/show_bug.cgi?id=1300947 1300947 ] ||  2016-08-02 || 51.0a1 || 51.0a2|| [https://wiki.mozilla.org/QA/DownloadPanel_DropMarker DownloadPanel DropMarker Test Plan] || QA Eng Team
|-
| 2 ||  || ||  || || || ||
|}


== Scope ==
== Scope ==
Line 47: Line 64:
* [mailto:wiwang@mozilla.com Will Wang]
* [mailto:wiwang@mozilla.com Will Wang]


• QA (Manager: [mailto:atsai@mozilla.com Al Tsai])
• QA (Lead: [mailto:brindusa.tot@softvision.com Brindusa Tot])
* [mailto:whsu@mozilla.com William Hsu]
• QA
 
*[mailto:adrian.florinescu@softvision.ro Adrian Florinescu] (Feature owner)
*[mailto:ovidiu.boca@softvision.ro Ovidiu Boca]
*[mailto:roxana.leitan@softvision.ro Roxana Leitan]
*[mailto:simona.badau@softvision.ro Simona Badau]


= Testing summary =  
= Testing summary =  
== Scope of Testing ==
== Scope of Testing ==
=== In Scope ===
=== In Scope ===
In this project, we intend to improve the content handling experience of Firefox Desktop. The enhancement includes scenarios such as “open with” dialog), a file is saved (download panel), the way dealing with that file (content handling, in pref. → App.), and relevant platform side design refactoring. Therefore, the testing effort for Content Handling will be invested on the following areas:  
In this project, we intend to improve the content handling experience of Firefox Desktop. The enhancement includes scenarios such as “open with” dialog, a file is saved (download panel), the way dealing with that file (content handling, in pref. → App.), and relevant platform side design refactoring. Therefore, the testing effort for Content Handling will be invested on the following areas:  
# Release Acceptance testing
# Release Acceptance testing
# Compatibility testing - Backward compatibility
# Compatibility testing - Backward compatibility
Line 63: Line 83:
# Regression testing
# Regression testing
# Usability testing
# Usability testing


=== Out of Scope ===
=== Out of Scope ===
Line 74: Line 93:
= Requirements for testing =
= Requirements for testing =
== Environments (TBC)==
== Environments (TBC)==
* Linux  
* Linux  
* Linux x64
* OS X 10.7
* OS X 10.10
* OS X 10.10
* Windows XP
* Windows 7 x 32
* Windows 7
* Windows 10 x64
* Windows 8 x64


= Test Strategy =  
= Test Strategy =  
Line 112: Line 129:
== Builds ==
== Builds ==
This section should contain links for builds with the feature -  
This section should contain links for builds with the feature -  
* Links for Nightly builds
* Links for Nightly builds: <span style=color:red> TBC </span>
* Links for Aurora builds
* Links for Aurora builds: <span style=color:red> TBC </span>
* Links for Beta builds
* Links for Beta builds: <span style=color:red> TBC </span>


== Test Execution Schedule ==
== Test Execution Schedule ==
Line 124: Line 141:
| Start project  
| Start project  
|style="text-align:center;" | 03/28/2016  
|style="text-align:center;" | 03/28/2016  
|style="text-align:center;" | TBD
|style="text-align:center;" | <span style=color:red> TBC </span>
|-
|-
| Study documentation/specs received from developers
| Study documentation/specs received from developers
Line 139: Line 156:
|-
|-
| QA - Nightly Testing  
| QA - Nightly Testing  
|style="text-align:center;" | TBD
|style="text-align:center;" | <span style=color:red> TBC </span>
|style="text-align:center;" | TBD
|style="text-align:center;" | <span style=color:red> TBC </span>
|-
|-
| QA - Aurora Testing
| QA - Aurora Testing
|style="text-align:center;" | TBD
|style="text-align:center;" | <span style=color:red> TBC </span>
|style="text-align:center;" | TBD
|style="text-align:center;" | <span style=color:red> TBC </span>
|-
|-
| QA - Beta Testing  
| QA - Beta Testing  
|style="text-align:center;" | TBD
|style="text-align:center;" | <span style=color:red> TBC </span>
|style="text-align:center;" | TBD
|style="text-align:center;" | <span style=color:red> TBC </span>
|-
|-
| Release Date  
| Release Date  
|style="text-align:center;" | TBD
|style="text-align:center;" | <span style=color:red> TBC </span>
|style="text-align:center;" | TBD
|style="text-align:center;" | <span style=color:red> TBC </span>
|}
|}


Line 163: Line 180:
| Test plan creation || Mozilla wiki
| Test plan creation || Mozilla wiki
|-
|-
| Test case creation || Google docs
| Test case creation || Test Rail
|-
|-
| Test case execution || Google docs
| Test case execution || <span style=color:red> TBC </span>
|-
|-
| Bugs management || Bugzilla
| Bugs management || Bugzilla
Line 176: Line 193:
  Track the dates and build number where feature was merged to Release/Beta
  Track the dates and build number where feature was merged to Release/Beta


== Risk analysis ==
== Testing risks and mitigation ==
Identify the high-risk assumptions
=== TESTING RISK ===
  Identify existing bugs on the feature with high risk
Risks can be organized into these categories.
Identify if other areas are affected by the fix
* '''Test planning and scheduling''' : It may occur when there is no separate test plan, but rather highly incomplete and superficial summaries in other planning documents. Also, test plans are often ignored once we are written. Regarding the schedule, the schedule of testing is often inadequate for the amount of testing that should be performed in TDC, especially when testing is primarily manual.
 
* '''Stakeholder involvement''' : The wrong mindset would introduce wrong thought of testing, having wrong testing expectations, and having stakeholders who are inadequate committed to and supporting of the testing effort. Therefore, we must align expectations with reality between stakeholders before we kick off testing.
 
* '''Process integration''' : It often occurs when testing and engineering processes are poorly integrated. We sometimes take a "one-size-fits-all" approach taken to testing, regardless of the specific needs of the project.
 
* '''Test communication risk''' : This problems often occurs when test documents are not maintained or inadequate communication.
 
=== RISK MITIGATION ===
QA team would like to use following flow to address risk.
* '''Risk Identification''': Risks can be identified using a number of resources. E.g., project objectives, risk lists of past projects, prior knowledge, understanding of system architecture or design, prior bug reports, and complaints. For example, if certain areas of the system are unstable and those areas are being developed further in the current project, it should be listed as a risk. It is good to document the identified risks in detail so that it stays in project memory and can be clearly communicated to project stakeholders.
 
* '''Risk Prioritization''' : If a risk is fully understood, it is easy for us to prioritize a risk by two measures. (1) Risk Impact and (2) Risk Probability are applied to each risk. Risk Impact is estimated in tangible terms or on a scale (e.g., 10 to 1 or High to Low). Risk Probability is estimated somewhere between 0 (no probability of occurrence) and 1 (certain to occur) or on a scale (10 to 1 or High to Low). For each risk, the product of Risk Impact and Risk Probability gives the Risk Magnitude. Sorting the Risk Magnitude in descending order gives a list in which the risks at the top are the more serious risks and need to be managed closely.
 
* '''Risk Treatment''' : Each risk in the risk list is subject to one or more of the following Risk Treatments.
*# <small>'''Risk Avoidance''' : For example, if there is a risk related to a new feature, it is possible to postpone this feature to a later release.</small>
*# <small>'''Risk Transfer''' : For example, if the risk is insufficient security testing of a feature, it may be possible to borrow the other expertise (Engineer) to perform the security testing.</small>
*# <small>'''Risk Mitigation''' : The objective of Risk Mitigation is to reduce the Risk Impact or Risk Probability or both. For example, if the QA team is new and does not have prior system knowledge, a risk mitigation treatment may be to have a knowledgeable team member join the team to train others on-the-fly.</small>
*# <small>'''Risk Acceptance''' : This happens when there is no viable mitigation available due to reasons such as resources. For example, if all testers are at the same place, risk acceptance means no another QA resource. When holiday comes, some tests will be stopped and it may be a concern in the project.</small>
 
= References =
= References =
* List and links for specs
* UX Specification : https://mozilla.invisionapp.com/share/4Y6ZZH1E8
  List and links for available specs - documents, user stories, specifications
* MANA : https://mana.mozilla.org/wiki/display/PM/Content+Handling+Enhancement
* Meta bug
* QA test strategy: https://docs.google.com/document/d/1YK5ePlHa3QfvaVhF4eYy922CmAt1Hkw3cWbYaOZm9AI/edit?usp=sharing
* Meta: [https://bugzilla.mozilla.org/show_bug.cgi?id=1269956 Bug 1269956 [meta<nowiki>]</nowiki> Firefox Download Panel UX Redesign]
 
= Testcases =  
= Testcases =  
== Overview ==
== Overview ==
Summary of testing scenarios
* Test cases on [https://docs.google.com/spreadsheets/d/1q7BWJkeRohs9CQ3TMaPDTTHoYFlB3ircbzrLo9s-eAw/edit?pref=2&pli=1#gid=906476437 Google doc]
  Note: UX spec is not yet finalized, and test cases are waiting for review.
        Therefore, test cases will be made public later. Sorry for any inconvenience!


== Test Areas ==
== Test Areas ==
Line 196: Line 234:
|-
|-
| Private Window  
| Private Window  
|style="text-align:center;" | ||  
|style="text-align:center;" | YES  ||  
|-
|-
| Multi-Process Enabled  
| Multi-Process Enabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | YES ||  
|-
|-
| Multi-process Disabled  
| Multi-process Disabled  
|style="text-align:center;" | ||  
|style="text-align:center;" | YES ||  
|-
|-
| Theme (high contrast)  
| Theme (high contrast)  
|style="text-align:center;" | ||  
|style="text-align:center;" | NO ||  
|-
|-
| '''UI'''  
| '''UI'''  
Line 211: Line 249:
|-
|-
| Mouse-only operation   
| Mouse-only operation   
|style="text-align:center;" | ||  
|style="text-align:center;" | YES ||  
|-
|-
| Keyboard-only operation   
| Keyboard-only operation   
|style="text-align:center;" | ||
|style="text-align:center;" | NO ||
|-
|-
| Display (HiDPI)  
| Display (HiDPI)  
|style="text-align:center;" | ||  
|style="text-align:center;" | NO ||
|-
|-
| Interraction (scroll, zoom)  
| Interraction (scroll, zoom)  
|style="text-align:center;" | ||  
|style="text-align:center;" | YES ||  
|-
|-
| Usable with a screen reader   
| Usable with a screen reader   
|style="text-align:center;" | || e.g. with NVDA
|style="text-align:center;"| N/A || e.g. with NVDA
|-
|-
| Usability and/or discoverability testing   
| Usability and/or discoverability testing   
|style="text-align:center;" | || Is this feature user friendly
|style="text-align:center;" | YES || UX team will help with it


|-
|-
Line 233: Line 271:
|-
|-
| Help/support interface required   
| Help/support interface required   
|style="text-align:center;" | || Make sure link to support/help page exist and is easy reachable.
|style="text-align:center;" | TBD || Make sure link to support/help page exist and is easy reachable.
|-
|-
| Support documents planned(written)   
| Support documents planned(written)   
|style="text-align:center;" | || Make sure support documents are written and are correct.
|style="text-align:center;" | TBD || Make sure support documents are written and are correct.


|-
|-
Line 243: Line 281:
|-
|-
| Feature upgrades/downgrades data as expected   
| Feature upgrades/downgrades data as expected   
|style="text-align:center;" | ||
|style="text-align:center;" | N/A ||
|-
|-
| Does sync work across upgrades   
| Does sync work across upgrades   
|style="text-align:center;" | ||  
|style="text-align:center;"| YES ||  
|-
|-
| Requires install testing     
| Requires install testing     
|style="text-align:center;" | || separate feature/application installation needed (not only Firefox)
|style="text-align:center;" | YES || separate feature/application installation needed (not only Firefox)
|-
|-
| Affects first-run or onboarding     
| Affects first-run or onboarding     
|style="text-align:center;" | || Florin/Lawrence are investigating if there is a dedicated QA for this, or we should test? Should be an yes/no and if is yes should add in detail column the team/person assigned.
|style="text-align:center;"| N/A ||  
|-
|-
| Does this affect partner builds? Partner build testing   
| Does this affect partner builds? Partner build testing   
|style="text-align:center;" | || yes/no options, add comment with details about who will lead testing
|style="text-align:center;"| N/A ||
 
|-
|-
| ''' Enterprise '''  
| ''' Enterprise '''  
||  || Raise up the topic to developers to see if they are expecting to work different on ESR builds
||  || Raise up the topic to developers to see if they are expecting to work different on ESR builds  
|-
|-
| Enterprise administration   
| Enterprise administration   
|style="text-align:center;" | ||  
|style="text-align:center;"| N/A ||  
|-
|-
| Network proxies/autoconfig   
| Network proxies/autoconfig   
|style="text-align:center;" | ||  
|style="text-align:center;"| N/A ||  
|-
|-
| ESR behavior changes   
| ESR behavior changes   
|style="text-align:center;" | ||  
|style="text-align:center;"| N/A ||  
|-
|-
| Locked preferences   
| Locked preferences   
|style="text-align:center;" | ||
|style="text-align:center;"| N/A ||


|-
|-
Line 278: Line 315:
|-
|-
| Temporary or permanent telemetry monitoring   
| Temporary or permanent telemetry monitoring   
|style="text-align:center;" | || List of error conditions to monitor
|style="text-align:center;"| N/A || List of error conditions to monitor
|-
|-
| Telemetry correctness testing   
| Telemetry correctness testing   
|style="text-align:center;" | ||  
|style="text-align:center;"| N/A ||  
|-
|-
| Server integration testing   
| Server integration testing   
|style="text-align:center;" | ||  
|style="text-align:center;"| N/A ||  
|-
|-
| Offline and server failure testing   
| Offline and server failure testing   
|style="text-align:center;" | ||
|style="text-align:center;"| N/A ||
|-
|-
| Load testing   
| Load testing   
|style="text-align:center;" | ||
|style="text-align:center;"| N/A ||


|-
|-
Line 297: Line 334:
|-
|-
| Addon API required?   
| Addon API required?   
|style="text-align:center;" | ||  
|style="text-align:center;"| N/A ||  
|-
|-
| Comprehensive API testing   
| Comprehensive API testing   
|style="text-align:center;" | ||  
|style="text-align:center;"| N/A ||  
|-
|-
| Permissions   
| Permissions   
|style="text-align:center;" | ||  
|style="text-align:center;" | YES ||  
|-
|-
| Testing with existing/popular addons
| Testing with existing/popular addons
|style="text-align:center;" | ||  
|style="text-align:center;" | YES ||  


|-
|-
Line 313: Line 350:
|-
|-
| 3rd-party security review   
| 3rd-party security review   
|style="text-align:center;" | ||  
|style="text-align:center;" | YES || Security team will help with it
|-
|-
| Privilege escalation testing
| Privilege escalation testing
|style="text-align:center;" | ||  
|style="text-align:center;" | NO ||  
|-
|-
| Fuzzing   
| Fuzzing   
|style="text-align:center;" | ||  
|style="text-align:center;" | NO ||  


|-
|-
Line 326: Line 363:
|-
|-
| Testing against target sites   
| Testing against target sites   
|style="text-align:center;" | ||  
|style="text-align:center;" | NO ||  
|-
|-
| Survey of many sites for compatibility   
| Survey of many sites for compatibility   
|style="text-align:center;" | ||  
|style="text-align:center;" | NO ||  


|-
|-
Line 336: Line 373:
|-
|-
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
| Common protocol/data format with other software: specification available. Interop testing with other common clients or servers.   
|style="text-align:center;" | ||  
|style="text-align:center;" | NO ||  
|-
|-
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
| Coordinated testing/interop across the Firefoxes: Desktop, Android, iOS   
|style="text-align:center;" | ||  
|style="text-align:center;" | NO ||  
|-
|-
| Interaction of this feature with other browser features   
| Interaction of this feature with other browser features   
|style="text-align:center;" | ||  
|style="text-align:center;" | NO ||  
|}
|}


== Test suite ==
== Test suite ==
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
  Full Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1q7BWJkeRohs9CQ3TMaPDTTHoYFlB3ircbzrLo9s-eAw/edit#gid=906476437 link]
  Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/192BDmxxPirNZMxaH8a9bN4ak1m0sp3U1PXT-G-THpWk/edit?usp=sharing link]
  Smoke Test suite - Link with the gdoc, follow the format from [https://docs.google.com/spreadsheets/d/1q7BWJkeRohs9CQ3TMaPDTTHoYFlB3ircbzrLo9s-eAw/edit#gid=203870728 link]
  Regression Test suite - Link with the gdoc - if available/needed.
  Regression Test suite - Link with the gdoc - if available/needed.


= Bug Work =
= Bug Work =
Tracking bug – meta bug
Meta: [https://bugzilla.mozilla.org/show_bug.cgi?id=1269956 Bug 1269956 - [meta<nowiki>]</nowiki> Firefox Download Panel UX Redesign]


<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
== Logged bugs ( blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1269956 1269956 ] ) ==
 
<bugzilla>
====== Bug fix verification ======
    {
<div class="mw-collapsible-content">
        "blocks":[1269956],
{| class="wikitable" style="width:50%"
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
|-
    }
! Bug No !! Summary !! Status !! Firefox Verion
</bugzilla>
|-
|  123 || bug summary || NEW || Nighly 45
|-
| || || ||
|}
</div></div>
 
<div class="toccolours mw-collapsible mw-collapsed" style="width:auto">
 
====== Logged bugs ======
<div class="mw-collapsible-content">
Bug 111111<br/>
Bug 211111<br/>
</div>
</div>


== Bug Verification  (blocking [https://bugzilla.mozilla.org/show_bug.cgi?id=1269956 1269956 ]) ==
<bugzilla>
    {
        "blocks":[1269956],
        "resolution":"FIXED",
        "include_fields": "id, priority, component, assigned_to, summary, status, target_milestone"
    }
</bugzilla>


= Sign off =
= Sign off =
Line 404: Line 434:
|-
|-
|  Testing Prerequisites (specs, use cases)  
|  Testing Prerequisites (specs, use cases)  
| style="text-align:center;" |  
| style="text-align:center;" | {{mprog|}}
| style="text-align:center;" |  
| style="text-align:center;" |  
|-
|-
|  Testing Infrastructure setup  
|  Testing Infrastructure setup  
|style="text-align:center;" |   ||  
|style="text-align:center;" | NO ||  
|-
|-
|  Test Plan Creation  
|  Test Plan Creation  
| style="text-align:center;" |   ||  
| style="text-align:center;" | {{mprog|}} ||  
|-
|-
|  Test Cases Creation  
|  Test Cases Creation  
|style="text-align:center;" |   ||  
|style="text-align:center;" | {{mprog|}} ||  
|-
|-
|  Full Functional Tests Execution  
|  Full Functional Tests Execution  
|style="text-align:center;" |   
|style="text-align:center;" |  {{mnotstarted}}
|style="text-align:center;" |   
|style="text-align:center;" |   
|-
|-
|  Automation Coverage ||
|  Automation Coverage  
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  Performance Testing  
|  Performance Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|-
|-
|  All Defects Logged || ||  
|  All Defects Logged
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|-
|-
|  Critical/Blockers Fixed and Verified || ||  
|  Critical/Blockers Fixed and Verified
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|-
|-
|  Daily Status Report (email/etherpad statuses/ gdoc with results) || ||  
|  Daily Status Report (email/etherpad statuses/ gdoc with results)
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|-
|-
|  Metrics/Telemetry||  
|  Metrics/Telemetry
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Nightly Release||  
|  QA Signoff  - Nightly Release
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|-
|-
|  QA Aurora - Full Testing  
|  QA Aurora - Full Testing  
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Aurora Release ||  
|  QA Signoff  - Aurora Release
|style="text-align:center;"| Email to be sent
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|-
|-
|  QA Beta - Full Testing
|  QA Beta - Full Testing
|style="text-align:center;" | ||  
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|-
|-
|  QA Signoff  - Beta Release ||  
|  QA Signoff  - Beta Release
|style="text-align:center;" | Email to be sent
|style="text-align:center;" | {{mnotstarted}}
|style="text-align:center;" |  
|}
|}
Whsu
Confirmed users
452

edits

Retrieved from "https://wiki.allizom.org/Special:MobileDiff/1134309...1149022"