Marketplace/Reviewers/Apps/Testing

System Setup

WebApp (aka App) installation is supported on on all release channels but Aurora or Beta are normally the best channels to routinely use. Occasionally it may be useful to test on Nightly for any recent changes.

Install both Desktop and Mobile Nightly or Aurora. Once installed you should be able to update from within the App.

For FirefoxOS the optimal test environment is an actual device. For trivial testing (e.g. a minor UX change in the app or a check to make sure the manifest installs correctly) the Firefox OS Simulator may be used. The simulator is not sufficient for initial testing of device compatibility.

Packaged App installation requires adding additional certificates to the phone.

Known Issues

General Runtime issues

• Desktop gecko is (obviously) a lot more mature than on Android/FxOS so rendering glitches are rare. There are nonetheless still some occasional issues with the webapp runtime implementation, especially as the development focus has been launching on Android first. If something odd happens it can sometimes help to try and run the app content in the browser directly.
• Web apps on Android and FirefoxOS regularly throw up bugs that effect either all web content, or the webapp implementation specifically. All issues should be logged as bugs on Bugzilla.

Specific Bugs

Full Query

ID	Summary	Priority	Status
749792	Web runtime in-process-plugins fail on some sites	--	RESOLVED
774727	Toma.hk app fails on mobile/tablet	--	RESOLVED
791281	Nell's Balloons App fails on Desktop	--	RESOLVED
793900	Brightcove video player using flash fallback renders white content on videos until you change the orientation of the phone	--	RESOLVED
803044	Hojoki App shows 'this app is having problems' message on startup	--	RESOLVED
810360	[oom] - http://lc.clay.io/ crashes Firefox	--	RESOLVED
816891	Unable to scroll in 3rd party app SpacePirates, works ok in browser	--	RESOLVED
818871	Fillit App doesn't work properly on Android	P5	RESOLVED
818924	Android crash in libc on call to an undefined javascript function (triggered by older version of Pacman Canvas app)	--	RESOLVED

9 Total; 0 Open (0%); 9 Resolved (100%); 0 Verified (0%);

(edit page to add more to bug_id list when they occur)

Apps Queue

There are four App queues (plus moderated user reviews) but this guide relates to the primary 'Apps' Queue (aka pending; nominated; submissions). The Apps queue is sorted by waiting time from the original submission, with the oldest App at the top - this means older apps will appear at the top of the queue even if they've only just been resubmitted.

The columns:

• App name. This is a link to the review page. A lock icon next to it means that another reviewer is looking at this review page.
• Flags.
  • Pending info request: a blue icon with an 'i' in the middle. A reviewer has requested additional information from the developer(s). Developers will reply to the mailing list.
  • Reviewer comments: a user icon with a speech bubble. The comment should be read carefully since it may be important.
  • Packaged App: a yellow cube. This is a packaged, rather than hosted, app which may be privileged so have more abilities than a hosted app.
  • Privileged App: a Red P. This means the (packaged) app is privileged. Currently only staff review these apps.
• Waiting time: this is from the first submission date, not the most recent update.
• Devices: The devices this App supports.
• Payments: Free, or not Free.
• Abuse Reports.

Where possible try to review Apps in queue order, prioritising the Apps at the top of the queue. (Taking into account outstanding issues and info requests)

Testing Procedure - Hosted webapps

Generally we go as many steps as possible to test the App, rather stopping at the first issue and rejecting. I.e. in the steps below 'reject' doesn't mean reject immediately, just that the reason should be noted down and it shouldn't be approved after you've exhausted the steps below.

Apps that are multi-platform (more than one of Desktop, Android, FirefoxOS) should be tested on all supported platforms. (The slight exception is if the app appears to work on Desktop and Android Mobile its not always necessary to test on an Android Tablet).

See the Marketplace Review Criteria for details of what we allow and don't allow in Apps for listing on Marketplace. The steps below outline the brief procedure, not the policy.

• Check the app has a sensible name, developer name, summary, description and icon. The description should be extensive enough for a user to understand what the app does (you may need to revisit this after launching the app). If not, reject.
• Have a quick look at the app manifest (the 'View' link next to the manifest url). If the manifest obviously isn't valid JSON/isn't found it won't install anyway. The manifest spec should be consulted if you aren't sure about syntax. Any issues, reject.
• Take note of any requested permissions in the manifest. There is a Security Checklist of available APIs and what they might be used/abused for. There are only a few APIs are available to hosted/non-privileged apps (alarms, desktop-notification, geolocation, fmradio)
• Press the install button. It will install natively so on Desktop you have to go find it in your Applications folder, start menu or desktop. On mobile platforms a shortcut will appear on your homescreen.
• Check the app's shortcut has an icon. The default rocketship icon is not allowed any more. If not, reject (there is a canned response). There is an occasional issue on Windows where sometimes the icon shown is cached from previous installs or appears broken at first so if it seems to be missing open the properties dialog and see if an icon is shown in the dialog.
• Launch the app. If the app doesn't launch as you would expect (just a random website, 404, directory listing, etc) then its likely that they need to specify a launch_path. Reject - there is a canned response.
• Give the app a quick try and see what experience a new user would have.
• Some apps require a login. If its straightforward you should register as a new user (to see what experience an actual user would have). If the app requires paid credentials; specific details; or isn't in a language you can understand sufficiently you can request a username & password - there is a canned response - with Request Information.
• All links load within the App unless explicitly specified otherwise so we need to check the app doesn't have any links (often in a navigation bar) that take the user out of the App 'experience' and strand them in a normal website (remember there is no back or home button in an app!). If there are such links then reject and explain they need to add the target="_blank" parameter to launch the link externally (in Firefox proper)
• A common issue on FirefoxOS is 'download halted' during installation - this is most often due to a broken appcache, either because it doesn't exist. You can check the manifest at manifest-validator.com
• If an app is Paid then check the receipt has been checked by the app - look next to the price on the review page. If not we should recommend they do that (its not a requirement). There is a canned response.
• Its important to note that we don't make any relevance or quality judgements about how the app looks in an App Review, only that it functions correctly. The review criteria document should be consulted. You can make suggestions about how to improve the app though if you notice anything that would make it better.

If there is anything that requires Admin/Staff attention you can request a super review. This will remove the app from the current queue and place it in the Escalation queue.

Testing Procedure - Packaged Apps

The procedure is similar to hosted apps. Currently packaged apps are only fully supported on FirefoxOS, though Android support is already in Nightly builds. Packaged App installation requires adding additional certificates to the phone.

Make sure the app is non-privileged. A privileged app is indicated by a Red P in the review queue (though not search results!), and then on the review page with "Type: Privileged Packaged App". See the privileged section for how to review those (staff only currently!)

See the Marketplace Review Criteria for details of what we allow and don't allow in Apps for listing on Marketplace. The steps below outline the brief procedure, not the policy.

• Check the app has a sensible name, summary, description and icon. The description should be extensive enough for a user to understand what the app does (you may need to revisit this after launching the app). If not, reject.
• The manifest url (view) link contains a copy of the manifest inside the (zip) package. Check this as you would a hosted app .
• Take note of any requested permissions in the manifest. There is a Security Checklist of available APIs and what they might be used/abused for. There are only a few APIs are available to hosted/non-privileged apps (alarms, desktop-notification, geolocation, fmradio)
• As a last check, look for the type entry. If there is no type entry in the manifest, or its 'web' the app is unprivileged. If the type is 'privileged' then see the privileged packaged app section below.
• Press the install button. On mobile platforms a shortcut will appear on your homescreen.
• Check the app's shortcut has an icon. The default rocketship icon is not allowed any more. If not, reject (there is a canned response).
• Launch the app. If the app doesn't launch as you would expect (most commonly a directory listing) then their launch_path may be incorrect.
• Give the app a quick try and see what experience a new user would have.
• Some apps require a login. If its straightforward you should register as a new user (to see what experience an actual user would have). If the app requires paid credentials; specific details; or isn't in a language you can understand sufficiently you can request a username & password - there is a canned response - with Request Information.
• All links load within the App unless explicitly specified otherwise so we need to check the app doesn't have any links (often in a navigation bar) that take the user out of the App 'experience' and strand them in a normal website (remember there is no back or home button in an app!). If there are such links then reject and explain they need to add the target="_blank" parameter to launch the link externally (in Firefox proper)
• If an app is Paid then check the receipt has been checked by the app - look next to the price on the review page. If not we should recommend they do that (its not a requirement). There is a canned response.
• Its important to note that we don't make any relevance or quality judgements about how the app looks in an App Review, only that it functions correctly. The review criteria document should be consulted. You can make suggestions about how to improve the app though if you notice anything that would make it better.

If there is anything that requires Admin/Staff attention you can request a super review. This will remove the app from the current queue and place it in the Escalation queue.

Testing Procedure - *Privileged* Packaged Apps

These should only be reviewed by Marketplace Staff currently.

The procedure is similar to hosted apps. Currently packaged apps are only supported on FirefoxOS. Packaged App installation requires adding additional certificates to the phone.

See the Marketplace Review Criteria for details of what we allow and don't allow in Apps for listing on Marketplace. The steps below outline the brief procedure, not the policy.

• Check the app has a sensible name, summary, description and icon. The description should be extensive enough for a user to understand what the app does (you may need to revisit this after launching the app). If not, reject.
• The manifest url (view) link only contains some details from the actual manifest, which is inside the (zip) package. To download the package for offline inspection, etc, click the 'package_path' link - this shouldn't be routinely necessary.
• In the version table at the bottom of the view load the validation report and inspect any warnings/errors.
• Then inspect the app contents via the 'contents' link.
• The first file should be the manifest.
• Take note of any requested permissions in the manifest. There is a Security Checklist of available APIs and what they might be used/abused for.
• Read the code in all the files one by one, in particular the .js files (thankfully inline js and external files aren't allowed by the CSP), paying attention to how any permissions requested are used.
• It may be necessary to search for an inspect different parts of the files, or other files, to establish how a particular piece of code is used. The validator is your friend as it highlights possible issues, but beware of false positives, and false negatives!
• Launch the app on the device and give the app a quick try and see what experience a new user would have.
• Some apps require a login. If its straightforward you should register as a new user (to see what experience an actual user would have). If the app requires paid credentials; specific details; or isn't in a language you can understand sufficiently you can request a username & password - there is a canned response - with Request Information.
• If an app is Paid then check the receipt has been checked by the app - look next to the price on the review page. If not we should recommend they do that (its not a requirement). There is a canned response.
• Its important to note that we don't make any relevance or quality judgements about how the app looks in an App Review, only that it functions correctly. The review criteria document should be consulted. You can make suggestions about how to improve the app though if you notice anything that would make it better.

Communication with other App Reviewers / Admins / Platform Developers / Marketplace Developers

IRC

• #app-reviewers - for review policy questions
• #marketplace - for Marketplace.mozilla.org technical/bug questions.
• #openwebapps - for technical questions about the App Runtime. Normally the first place to ask before logging a bug.

Email

• app-reviewers@mozilla.org - for any concerns or questions related to the review process or changing manifest URLs, and anything related to administration of the Marketplace. This list includes the Marketplace reviewers.