Marketplace/Reviewers/Apps/Testing

Devices required

In general, all testing is done on actual devices. For trivial testing (e.g. a minor UX change in the app or a check to make sure the manifest installs correctly) the Firefox OS Simulator may be used. The simulator is not sufficient for initial testing of device compatibility.

Don't review apps that are submitted for devices you don't have.

The minimum required version of Firefox OS that apps must work on is v1.1. If an app requires features that only work on v1.2 or greater, add the appropriate Minimum Requirements flag and request that the developer specify the minimum requirements in the app description.

Known Issues

General Runtime issues

• Desktop gecko is (obviously) a lot more mature than on Android/FxOS so rendering glitches are rare. There are nonetheless still some occasional issues with the webapp runtime implementation, especially as the development focus has been launching on Android first. If something odd happens it can sometimes help to try and run the app content in the browser directly.
• Web apps on Android and FirefoxOS regularly throw up bugs that effect either all web content, or the webapp implementation specifically. All issues should be logged as bugs on Bugzilla.

Specific Bugs

Full Query

ID	Summary	Priority	Status
749792	Web runtime in-process-plugins fail on some sites	--	RESOLVED
774727	Toma.hk app fails on mobile/tablet	--	RESOLVED
791281	Nell's Balloons App fails on Desktop	--	RESOLVED
793900	Brightcove video player using flash fallback renders white content on videos until you change the orientation of the phone	--	RESOLVED
803044	Hojoki App shows 'this app is having problems' message on startup	--	RESOLVED
810360	[oom] - http://lc.clay.io/ crashes Firefox	--	RESOLVED
816891	Unable to scroll in 3rd party app SpacePirates, works ok in browser	--	RESOLVED
818871	Fillit App doesn't work properly on Android	P5	RESOLVED
818924	Android crash in libc on call to an undefined javascript function (triggered by older version of Pacman Canvas app)	--	RESOLVED

9 Total; 0 Open (0%); 9 Resolved (100%); 0 Verified (0%);

(edit page to add more to bug_id list when they occur)

Testing Procedure - Hosted webapps

Generally we go as many steps as possible to test the app, rather stopping at the first issue and rejecting. I.e. in the steps below 'reject' doesn't mean reject immediately, just that the reason should be noted down and it shouldn't be approved after you've exhausted the steps below.

Apps that are multi-platform (more than one of Desktop, Android, FirefoxOS) should be tested on all supported platforms. (The slight exception is if the app appears to work on Desktop and Android Mobile its not always necessary to test on an Android Tablet).

See the Marketplace Review Criteria for details of what we allow and don't allow in apps for listing on Marketplace. The steps below outline the brief procedure, not the policy.

• Check the app has a sensible name, developer name, summary, description and icon. The description should be extensive enough for a user to understand what the app does (you may need to revisit this after launching the app). If not, reject.
• Have a quick look at the app manifest (the 'View' link next to the manifest url). If the manifest obviously isn't valid JSON/isn't found it won't install anyway. The manifest spec should be consulted if you aren't sure about syntax. Any issues, reject.
• Take note of any requested permissions in the manifest. There is a Security Checklist of available APIs and what they might be used/abused for, as well as in-depth Security guidelines for developers and reviewers. There are only a few APIs are available to hosted/non-privileged apps (alarms, desktop-notification, geolocation, fmradio)
• Validate that each permission description in the manifest uses plain language to accurately and clearly describe the app's use of that permission. Take extra care with every permission that seems unusual or unneccessary for the app's designated purpose.
• Press the install button. It will install natively so on Desktop you have to go find it in your Applications folder, start menu or desktop. On mobile platforms a shortcut will appear on your homescreen.
• Check the app's shortcut has an icon. The default rocketship icon is not allowed any more. If not, reject (there is a canned response). There is an occasional issue on Windows where sometimes the icon shown is cached from previous installs or appears broken at first so if it seems to be missing open the properties dialog and see if an icon is shown in the dialog.
• Launch the app. If the app doesn't launch as you would expect (just a random website, 404, directory listing, etc) then its likely that they need to specify a launch_path. Reject - there is a canned response.
• Give the app a quick try and see what experience a new user would have.
• Some apps require a login. If its straightforward you should register as a new user (to see what experience an actual user would have). If the app requires paid credentials; specific details; or isn't in a language you can understand sufficiently you can request a username & password - there is a canned response - with Request Information.
• All links load within the app unless explicitly specified otherwise so we need to check the app doesn't have any links (often in a navigation bar) that take the user out of the app 'experience' and strand them in a normal website (remember there is no back or home button in an app!). If there are such links then reject and explain they need to add the target="_blank" parameter to launch the link externally (in Firefox proper)
• A common issue on FirefoxOS is 'download halted' during installation - this is most often due to a broken appcache, either because it doesn't exist. You can check the manifest at manifest-validator.com
• If an app is Paid then check the receipt has been checked by the app - look next to the price on the review page. If not we should recommend they do that (its not a requirement). There is a canned response.
• Its important to note that we don't make any relevance or quality judgements about how the app looks in an app review, only that it functions correctly. The review criteria document should be consulted. You can make suggestions about how to improve the app though if you notice anything that would make it better.

If there is anything that requires Admin/Staff attention you can request a super review. This will remove the app from the current queue and place it in the Escalation queue.

Testing Procedure - Packaged Apps

The procedure is similar to hosted apps. Packaged app installation requires setting up the phone.

See the Marketplace Review Criteria for details of what we allow and don't allow in apps for listing on Marketplace. The steps below outline the brief procedure, not the policy.

• Check the app has a sensible name, summary, description and icon. The description should be extensive enough for a user to understand what the app does (you may need to revisit this after launching the app). If not, reject.
• The manifest url (view) link contains a copy of the manifest inside the (zip) package. Check this as you would a hosted app .
• Take note of any requested permissions in the manifest. There is a Security Checklist of available APIs and what they might be used/abused for, as well as in-depth Security guidelines for developers and reviewers. There are only a few APIs are available to hosted/non-privileged apps (alarms, desktop-notification, geolocation, fmradio)
• Validate that each permission description in the manifest uses plain language to accurately and clearly describe the app's use of that permission. Take extra care with every permission that seems unusual or unneccessary for the app's designated purpose.
• As a last check, look for the type entry. If there is no type entry in the manifest, or its 'web' the app is unprivileged. If the type is 'privileged' then see the privileged packaged app section below.
• Press the install button. On mobile platforms a shortcut will appear on your homescreen.
• Check the app's shortcut has an icon. The default rocketship icon is not allowed any more. If not, reject (there is a canned response).
• Launch the app. If the app doesn't launch as you would expect (most commonly a directory listing) then their launch_path may be incorrect.
• Give the app a quick try and see what experience a new user would have.
• Some apps require a login. If its straightforward you should register as a new user (to see what experience an actual user would have). If the app requires paid credentials; specific details; or isn't in a language you can understand sufficiently you can request a username & password - there is a canned response - with Request Information.
• All links load within the app unless explicitly specified otherwise so we need to check the app doesn't have any links (often in a navigation bar) that take the user out of the app 'experience' and strand them in a normal website (remember there is no back or home button in an app!). If there are such links then reject and explain they need to add the target="_blank" parameter to launch the link externally (in Firefox proper)
• If an app is Paid then check the receipt has been checked by the app - look next to the price on the review page. If not we should recommend they do that (its not a requirement). There is a canned response.
• Its important to note that we don't make any relevance or quality judgements about how the app looks in an app review, only that it functions correctly. The review criteria document should be consulted. You can make suggestions about how to improve the app though if you notice anything that would make it better.

If there is anything that requires Admin/Staff attention you can request a super review. This will remove the app from the current queue and place it in the Escalation queue.

Testing Procedure - *Privileged* Packaged Apps

These should only be reviewed by Marketplace Staff and Senior Reviewers who have gone through the privileged app review onboarding & training.

The procedure is similar to hosted apps. Packaged app installation requires setting up the phone.

See the Marketplace Review Criteria for details of what we allow and don't allow in apps for listing on Marketplace. The steps below outline the brief procedure, not the policy.

• Check the app has a sensible name, summary, description and icon. The description should be extensive enough for a user to understand what the app does (you may need to revisit this after launching the app). If not, reject.
• Ignore the manifest url (view) link - its in the zip package. (To download the package for offline inspection, etc, click the 'package_path' link - this shouldn't be routinely necessary.)
• In the version table at the bottom of the view load the validation report and look over any warnings/errors.
• Then inspect the app contents via the 'contents' link.
• The first file should be the manifest. Take note of any requested permissions in the manifest. There is a Security Checklist of available APIs and what they might be used/abused for, as well as in-depth Security guidelines for developers and reviewers.
• Validate that each permission description in the manifest uses plain language to accurately and clearly describe the app's use of that permission. Take extra care with every permission that seems unusual or unnecessary for the app's designated purpose.
• Read the JavaScript in all the files one by one, in particular the .js files (thankfully inline js and external files aren't allowed by the CSP), paying attention to how any permissions requested are used.
  • If the code is minified or obfuscated then readable source should be requested via info request (there is canned response)
• It may be necessary to search for and inspect different parts of the files, or other files, to establish how a particular piece of code is used. The validator is your friend as it highlights possible issues, but beware of false positives, and false negatives!
• Launch the app on the device and give the app a quick try and see what experience a new user would have.
• Some apps require a login. If its straightforward you should register as a new user (to see what experience an actual user would have). If the app requires paid credentials; specific details; or isn't in a language you can understand sufficiently you can request a username & password - there is a canned response - with Request Information.
• If an app is Paid then check the receipt has been checked by the app - look next to the price on the review page. If not we should recommend they do that (its not a requirement). There is a canned response.
• Its important to note that we don't make any relevance or quality judgements about how the app looks in an app review, only that it functions correctly. The review criteria document should be consulted. You can make suggestions about how to improve the app though if you notice anything that would make it better.

See also: https://wiki.mozilla.org/Marketplace/Reviewers/Apps/Guide/SecReviewTraining