QA/Fennec/draft mihai 2
Jump to navigation
Jump to search
Full Query
Overview
Purpose
The purpose of this wiki is to serve as a general test plan for verifying that Presentation API is correctly implemented.
Ownership
Developer contact: Shih-Chiang Chien (:schien)
Overall Status
[pending] Nightly 53: it's going to be prefed on, on 15-11-2016
Testing Summary
Scope of testing
The testing will be focusing on:
- Verify that the manual tests made by W3C for presentation API pass
- Monitor crash rate, stability, and power consumption
- Check that no important issues are regressing from the feature work
Environments
Testing will be performed on following devices:
- Nexus Player
- Chromecast
- Phones and Tablets
- Android 4.4+
Specifications
User stories
- Use cases and requirements for Presentation API
Acceptance Criteria
- The feature does not introduce new regressions that affect performance and stability
==KPIs / Measurements Used:
- Qualitative:
- No negative feedback from users (SUMO/Google Play Store)
- Current % of RL users: we'd need to associate individual users with the telemetry events (mfinkle's script): http://cl.ly/1T36142z2X02
- Adoption: If we can't get % of users who use RL, we might be able to watch the adoption rate instead to measure this
- Monitor how many times users have seen the migration slide/info to know how many people actual use the concept of a RL
- Monitor how many times users clicked on the "offline" helper UI to get a baseline to compare it with viewed saved offline RL items
- Increase in ratio between stored RL items and loaded RL items (currently 0.75)
- Monitor how many people load RV items that are not in RL folder vs. RV items that are in the RL folder: that will help us understand if people understand the concept of read later / save for later
- Monitor how many people load RV items in offline vs. online mode to help us understand their mental model of using RV items for offline mode https://bugzilla.mozilla.org/show_bug.cgi?id=1243387
- More usage != more RL items, increased frequency of accessing RL folder now in comparison to previous panel view
- Funnel to understand how many people actually read saved RV items in offline mode: https://bugzilla.mozilla.org/show_bug.cgi?id=1246718
- User clicks on RV https://bugzilla.mozilla.org/show_bug.cgi?id=1246710
- User saves RV item to RL list (folder) https://bugzilla.mozilla.org/show_bug.cgi?id=1246712
- User loads RV item in offline vs online mode https://bugzilla.mozilla.org/show_bug.cgi?id=1243387
Testing details
Testing days
Nightly
Bug Work
Tracking bugs
- bugs to track the Presentation API feature
- https://bugzilla.mozilla.org/showdependencytree.cgi?id=1184036&hide_resolved=0
ex:
84 Total; 1 Open (1.19%); 78 Resolved (92.86%); 5 Verified (5.95%);
Signoff
Criteria
- All test cases should be executed
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed
Results
Nightly testing
Aurora testing
Beta testing