CloudServices/Loop/Deploy

The Loop service is the server-side component of the Loop project.

It's composed of two parts: - loop-server, the Node.js service at https://github.com/mozilla/loop-server - loop-client, the "client" part composed of static files, served by our server

Both parts are currently deployed and served by the same instance servers.

See : https://wiki.mozilla.org/Loop

Contacts

• Dev Team
  • loop-server
    • Tarek Ziadé <tarek@mozilla.com>
    • Alexis Metaireau <alexis@mozilla.com>
    • Rémy Hubscher <natim@mozilla.com>
  • loop-client (main devs)
    • Mark Banner <mbanner@mozilla.com>
    • Dan Mosedale <dmose@mozilla.com>
• OPS
  • Benson Wong <mostlygeek@mozilla.com>
  • Bob Michelleto <bobm@mozilla.com>
• QA
  • James Bonacci <jbonacci@mozilla.com>

Deployment

There are three deployed environments. A fourth will be deployed later.

Dev

• Host: https://loop-dev.stage.mozaws.net/
• Maintainer: DEVs
• Tokbox mocked? NO.
• Usage: Development and integration
• Updates:
  • loop-client is updated automatically every hour and should match the latest master of the repository
  • loop-server is updated with the master branch by devs on a regular basis - or upon request. you can get the version by displaying the root URL of the server.
• Access: ec2-user@loop.dev.mozaws.net

This environment can be used to test end-to-end scenario until the service hits the Stable channel.

Load-Stage

• Host: https://loop.stage.mozaws.net/
• Maintainer: OPS
• Tokbox mocked? Yes (but can change, check the / endpoint for more info)
• Usage: Server-side QA and Loadtesting with the mock server

This environment is used by QA and dev for load tests. The goal is to measure how many connections can be handled by the server and anticipate errors that might happen on high load.

tokbox created keys are not real ones, they are collected by a fake mock server. We deployed it at http://loop-delayed.dev.mozaws.net/

Real-Stage

• Host: https://loop.stage.mozaws.net/
• Maintainer: OPS
• Mocked tokbox?: No
• Usage: Client-side QA, Server-side QA and Loadtesting with a live third-party/partner server

Not yet commissioned. This environment will be used for end-to-end testing of the service once it hits the stable channel.

This server will be a perfect mirror of the production environment, updated with the tag of the upcoming release

• (jabonacci) Correct me if I am wrong but we already have this. We are using a configurable Stage environment that can either point to a mock server (previous section) or point to a live server. So, we are able to do end-to-end testing. Host name is the same. Configuration is defined in a file on the server:
  • /data/loop-server/config/settings.json

Production

• Host: https://msisdn.services.mozilla.com/
• Maintainer: OPS
• Fake tokbox?: Of course not :)
• Usage: Production apps

This environment is used for production and is the default server for Nightly.

The prod environment provide a Mobile number validation for the following countries:

Releasing loop-server

Ops procedure

Releasing loop-client

The loop-client repository is a mirror of the necessary parts from mozilla-central. It is automatically updated via a script in the repository, every hour (if necessary).

Release process:

• Check loop-client has all the required changes
• Update the CHANGELOG file
  • Note any configuration option changes/additions
• Checkout a fresh loop-client repo, make sure the directory name is `loop-client`, then

cd loop-client

# Replace '0.4.0 with the correct version
./create_release.sh 0.4.0

• The new release tag will be automatically pushed
• File bugs for pushes to (use these templates):
  • staging
  • production.

Further info: https://etherpad.mozilla.org/loop-client-release

Release Cycle

The service is continuously pushed into the dev server where client developers can test it.

The service is released in load-stage then production every other week (or asap if we discover a security breach)

• Tuesday - end of previous cycle. tagging. pushed to load-stage
• Tuesday through Friday - load testing by James on load-stage
• Monday - push to production if no regression, if any regression backed off

The Tuesday release will be announced to the loop mailing list when the tagging is happening, so everyone has a chance to try it out.

Theorical dates for July/August:

• July 8th - tagging
• July 14th - prod push
• July 22th - tagging
• July 28th - prod push
• August 5th - tagging
• August 11th - prod push
• August 19th - tagging
• August 25th - prod push
• and so forth...

Once the service will hit the Stable channel, we will introduce the new real-stage environment

Branches and bugfix deployments

In case of a bugfix:

• A commit will with the fix will be pushed to master.
• A new branch will be created on the github repository with the versions that needs the patch, and the fixes will be applied there (backported).
• A new tag will be created with the new version (the patch version will be updated) and a deployment request will be filled.

For instance, in case the 0.9.0 release contains a bug that needs to be fixed:

1. Fix the code in master;
2. Backport (cherrypick) the commit in the 0.9.x branch (create it if needed);
3. Tag a new minor release: 0.9.1 and fill a new deployment request.

Deployment Versioning

Loop-server is backward compatible and uses version routes (e.g. /v1, /v2) to segment changes in API. Currently this is controlled by the Loop-server and does not offer the ability to update v2 route code in isolation from v1. In other words, Fx34 browsers using /v1 route will have a production code change due to a bug fix in Fx35 /v2 API. Here's a breakdown of different approaches:

Option A: /v1 and /v2 routes point to different server clusters:

• Pros:
  • Fx34 and /v1 users are not affected by any code change to /v2 api users.
  • Low risk of injecting bugs
• Cons:
  • Ops has to maintain two server clusters (dev/stage/prod)
  • Both servers need to use the same database... this gets tricky.
  • Mo' computers mo' problems (complexity).

Option B: /v1 and /v2 routes are hosted on a single server:

• Pros:
  • Simpler, less resources used and needed
  • Easier to use a single database
• Cons
  • More risk when pushing code for /v2, /v1 code will be affected.

For Fx 34/35, we are choosing Option B. till we reach a point where have a history of injecting bugs due to this architecture.