Apps/QA/Test Infrastructure

Test Infrastructure Requirements

The Problem

The problem we are currently facing in our testing environments is that each group has their own infrastructure. The infrastructure is built around varying concepts of dev , beta, staging environments.

The problem is those environments all have different schedules that code is pushed, and released, and it is not possible to guarantee what version of the marketplace, browser id, Apps , Sync Tests have gone through a level of integration testing.

Each group has their own infrastructure, and it's not entirely clear to each group what the benefits of moving to a shared environment are. Current Release Schedules Are the following:

Sync Service - https://intranet.mozilla.org/QA/Server_Weekly_Trains_Staging#Schedules

 Weekly Push
 Wed Push to staging 
 Monday - Release to production
 

MarketPlace - https://mail.mozilla.com/home/wclouser@mozilla.com/AMO%20Schedule.html

 Infrastructure Diagram Release Infra
 addons.allizom is staging server, but only updated as needed
 Release happens 2pm Thursday
 DBA Sheeri / MPressman

BrowserID - https://wiki.mozilla.org/QA/BrowserID/TestPlan#Weekly_Test_Schedules

 Thursdays: deployment to Production, Stage (QA), and Dev
 Thursdays/Fridays: open testing and experimentation by Dev, QA, and community
 Following week: Monday - Wednesday: QA testing and sign off of current deployment 
 DBA: petef

Apps - Web Services (HTML / JS / Dashboard)

 As needed

The Purpose

 Define an apps test environment that allows for testing javascript api's, the firefox browser, and the android Soup app. 

QA should have the ability to qualify a build, and know with certainty that those are the bits being shipped. Production releases should be done using the jenkins server to push the files to production.

Initial Hardware Requirements

 build.apps.mozilla.com
 apps-staging.apps.mozilla.com
 myapps-staging.apps.mozilla.com 

Concept

Reasoning

Multiple Build Targets

Within each build we have hardcoded values for myapps.mozillalabs, and apps.mozillalabs. Ideally we need to parameterize these inputs to the build process.

 lib/api.js:      "127.0.0.1:60172":"", "myapps.mozillalabs.com":"",
 lib/api.js:      "stage.myapps.mozillalabs.com":"", "apps.mozillalabs.com":"",
 lib/main.js:const APP_SYNC_URL = "https://myapps.mozillalabs.com";
 lib/main.js:      "https?://myapps.mozillalabs.com",
 lib/main.js:      "*.myapps.mozillalabs.com",
 lib/main.js:      "https?://apps.mozillalabs.com",
 lib/main.js:   * openwebapps@mozillalabs.com should be got from self.data
 lib/main.js:  let button = win.document.getElementById("widget:openwebapps@mozillalabs.com-" + TOOLBAR_ID);
 lib/main.js:    let dboard = "https://myapps.mozillalabs.com";
 lib/main.js: * library, hosted at https://myapps.mozillalabs.com/
 lib/main.js:      contentURL: "https://myapps.mozillalabs.com",
 lib/main.js:        if (origin == "https://myapps.mozillalabs.com") {
 lib/main.js:      if (!found) tabs.open("https://myapps.mozillalabs.com");
 lib/nativeshell.js:                                "http://apps.mozillalabs.com");
 lib/nativeshell.js:                                      "http://apps.mozillalabs.com");
 lib/nativeshell.js:        subKey.writeStringValue("Contact", "http://apps.mozillalabs.com");
 lib/nativeshell.js:        //subKey.writeStringValue("Readme", "http://apps.mozillalabs.com");
 package.json:    "id": "openwebapps@mozillalabs.com",

The same is true for android. It is important to be able to modify these parameters for QA / Release / Automation. So that when we click on the Apps icon in the lower right of the browser it launches the staging myapps instead of the production myapps.mozillalabs.com

We will probably need to be able to generate our own nightlies based upon a github repo build of firefox. This would be necessary for QA to be able to test based upon what is checked in to github vs having to wait until patches are reviewed.

Staging Servers

apps-staging.apps.mozilla.com

 This server is purely to serve apps that are staged for qa, that are served up locally.  Currently a lot of the tests we run are based upon external apps. Possibly we can use a new name for the server to indicate it is something slightly different.  That what it truly holds is our repository of test apps. apps-staging will also host a staging version of the html / js version of mozApps apis. 

myapps-staging.apps.mozilla.com

 This is a where the dashboard will be hosted. Idea is to be able to test dashboard changes before they go live.