MozillaWiki

Web Apps integration/TestPlan

< Web Apps integration
Revision as of 07:42, 4 May 2012 by Jsmith (talk | contribs) (→‎Bugs Needing Verification by Severity)

Overview

Feature Release Target Dev Lead QA Lead Dev Status QA Status Health
Web Apps Integration Firefox 15 Felipe Gomes Jason Smith Landed In Progress OK

Feature Summary

This feature focuses on being able to find web applications to install on Firefox through the home tab and installing and uninstalling these applications natively different operating systems. These installable web applications are essentially websites built in web technologies (e.g. HTML, CSS, JavaScript) that users can interact with through a chromeless shell in a native environment. These web applications intend to act exactly like native applications on the operating system. For a website to become a web application, a developer creates an app manifest for their website and hosts it on an origin (e.g. www.yourhost.com) where the website is located. Then, the app developer adds this manifest to a store (e.g. Firefox store) to allow the app to be installed to users' machines.

Users can then go to this store to install this application to their native machine. Note that installable applications can be paid or free. For paid applications, users will have to pay for the application through Paypal. Upon making the application payment, a user's machine receives a receipt to prove that the application is paid for. Then a user can install this paid application to their machine on Win XP+ or Mac OS X. See the native installation flow to understand how native applications are installed a user's machine on Windows XP+ and Mac OS X. After a user selects to install an application, a confirmation should appear asking the user to confirm installation of the application to a particular file location on the user's machine. Then, upon confirmation, the application should be installed to the user's machine. At this point, the user should understand where the application was installed on their machine.

After applications are installed, users can run these applications in their native environment in a chromeless shell both online and offline. If the application is paid, validation will need to take place using the receipt for the application to ensure that the user paid for the application. On windows, applications are typically ran as shortcuts from the desktop, but may also be ran from the start menu in executables. On Mac OS X, applications are ran through the ~/Applications directory typically. When an application is launched, a chromeless window starts using Firefox's webapp mode under the hood with the web application running in the shell. In the task manager in Windows, you would see that the application is process running on the system under the name of the web application. Within the shell, users can take actions within the application, such as logging in, clicking links within the origin of the application, playing a game, and more. Within the shell itself, users have a menu to allow basic application actions such as quitting, cut/copy, and more. When a user is done using the application, they can quit using the application.

If the user no longer wants the application installed on their machine, they can uninstall the application from their machine. For windows, uninstalling can take place either directly in the directory location of the application or through add and remove programs. On Mac OS X, uninstalling occurs by moving the application to trash and deleting the application from the trash bin. Upon uninstalling an application, all locally stored app data that previously created during installation should be cleared.

Testing Scope

Major Features

The scope of testing of this feature focuses on the following major features:

  • Home tab web apps integration
  • Installing native applications
  • Launching native applications
  • Using native applications
  • Uninstalling native applications

Software Qualities

The following software qualities that need to be analyzed are (ordered by priority):

  1. Security
  2. Availability & Resilience
  3. Internationalization
  4. Location
  5. Performance & Scalability

Edge Cases

The following special edge cases need to be taken into account when testing the major parts of this feature:

  • Browser crashes
  • Loss of internet connection (e.g. offline mode)
  • App state changes (i.e. free to paid, paid to free)
  • Invalid or no receipts for paid apps
  • Local vs. roaming windows profiles
  • System restore

Testing Strategy

The following testing approaches shall be used to assess the scope of testing stated above:

Development of Manual Test Cases: Manual test cases shall be created to test the major portions of features in the testing scope. Each test case shall specify release quality requirement specifying which test cases need to pass within Firefox Aurora, Beta, or Release respectively. Test cases that are applicable to changes made in mozilla central shall be re-tested on each minor release in the Firefox release cycle (e.g. Beta 1, Beta 2). Note that running these test cases should be optimally be done on vanilla installations of the supported operating systems. Upon finishing manual testing, bugs shall be logged for functional issues.

Exploratory Testing of Tier 1 Apps: Exploratory testing of using tier 1 native applications shall be conducted to ensure that our highest priority applications are supported on our platform. When exploratory testing these applications, a checklist shall be used to test major features we would expect from these applications such as account management, migrating off the origin of the application, and more. Upon finishing exploratory testing of these applications, bugs shall be logged for functional issues and new manual and scenario tier 1 app test cases shall be developed. Also, if the bug logged affects the functionality of a top app, a whiteboard marker of '[topapps]' should be specified. Additionally, if problems with the app itself are found, then evangelism bugs shall be filed for those issues.

Development of Value-Added Automation: Automation shall be developed for manual test cases that carry the highest value add in comparison to the overhead to the build the automation. To determine the value added versus the overhead, an approach for automation shall be documented next to each manual test case to determine if the payoff to build the automation exceeds the overhead cost. Technologies that shall be used will likely be either Mochitest or Mozmill, but this could change based on what manual test cases exist for this feature.

Tier 1 App Scenario Tests: Scenario tests shall be used on this desktop feature to simulate real-use of the application. Examples being considered for scenario tests follow the model shown at the bottom of the MWC Pod Demo Script. Similar to the manual test cases, each test case shall specify a release quality requirement (e.g. Aurora, Beta, Release). Scenario test cases that are applicable to changes made in mozilla central shall be re-tested on each minor release in the Firefox release cycle (e.g. Beta 1, Beta 2). Upon finishing scenario testing, bugs shall be logged for functional issues with the whiteboard marker of '[topapps].'

TODO: Add test strategies to address software quality requirements.

Sign Off Criteria

Aurora

  • No bugs found with flagged as aurora blockers from aurora test cases
  • Aurora-level functional test cases pass
  • Aurora-level tier 1 app scenario test cases pass
  • No unresolved bugs with aurora blockers that cause aurora-level test failures
  • TODO: Add software quality requirements

Beta

  • Tier 1 app scenario aurora and beta test cases all pass
  • Aurora and beta level functional test cases pass
    • Windows XP+ and Mac OS X 10.5+ OS-specific aurora and beta test cases pass
  • No unresolved bugs with a severity level of major or higher that cause aurora or beta test cases to fail
  • No regressions in Firefox linked to the feature's code changes detected with a severity of major or higher
  • TODO: Add software quality requirements

Release

  • No bugs found with a severity level of major or higher in test cases
  • All major functional test cases pass
    • Windows XP+ and Mac OS X 10.5+ OS-specific test cases pass
  • Tier 1 app scenario test cases pass
  • No unresolved bugs with a severity level of major or higher that cause test cases to fail
  • TODO: Add software quality requirements

Infrastructure Requirements

mozqa.com's firefox test cases section shall need to be turned into a web app to allow for testing the underlying platform when using apps with desktop. If new test cases for platform are necessary, then they will need to be uploaded as test cases to the mozqa.com test server.

A test cases app directory will be need to be setup and hosted on a mozilla-based server. Each app in this app directory represents a test case application that is used during the testing of using a particular application. Currently, a apps test server can be hosted locally for app test cases. This same test server will need to be deployed to some mozilla-based server.

Test Cases and Results

Test Cases

  • Manual test cases and automation approach can be found in the Spreadsheet: Google Spreadsheet
  • Exploratory test cases of tier 1 apps can be found in the Spreadsheet: <insert link here>
  • Tier 1 app scenario tests can be found in the Spreadsheet: <insert link here>

Test Results

  • Manual test case results shall be tracked in this spreadsheet: <insert link here>
  • Exploratory test results of tier 1 apps shall be tracked in this spreadsheet: <insert link here>
  • Tier 1 app scenario test results shall be tracked in this spreadsheet: <insert link here>
  • Marketplace Beta Test Run Results are Here

Bugs

Summary

Open Bugs by Priority

Bugzilla query error

Invalid type (count) and display (table) combination1


Bugs Needing Verification by Severity

Bugzilla query error

Invalid type (count) and display (table) combination1


Top Functional Bugs

Full Query
ID Summary Priority Status
704574 Sanitize app manifest entries when attempting native installation P1 VERIFIED

1 Total; 0 Open (0%); 0 Resolved (0%); 1 Verified (100%);


Top App Bugs

Other Bugs

Full Query
ID Summary Priority Status
725533 webapps.js looks leaky -- RESOLVED
732280 Intermittent icon failures on Windows XP when installing an app natively -- RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


Important References

Open Questions

Reinstallation of Paid Apps with a Different Account Question

(jsmith) What's the difference with a different account when reinstalling the same paid app?

Install App on Two Different Firefox Profiles

(jsmith) What is the expected behavior if I install an app on two different firefox profiles? Will one app overwrite the other entirely? Partly?

Retrieved from "https://wiki.allizom.org/index.php?title=Web_Apps_integration/TestPlan&oldid=427303"