Mobile/Fennec/Android/Switchboard: Difference between revisions

From MozillaWiki
< Mobile‎ | Fennec‎ | Android
Jump to navigation Jump to search
No edit summary
(Bug 1369685 - Add a new key "region" on Switchboard)
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
[https://github.com/mozilla-services/switchboard-server Switchboard] is a service we use to segment users for A/B testing.
Switchboard is a service we use to segment users for A/B testing or staged rollout of features.


== Usage ==
An experiment can be enabled for a specified percentage of users and limited to release channels, app versions, languages, countries (locale), device names and device manufacturers. An experiment can contain additional metadata to "configure" the experiment. This can be used to change parameters of an experiment without needing to ship a new app version. Which experiments a user is enrolled in is reported to our telemetry systems via the core ping.
To use Switchboard to expose a new feature to a portion of our users, use a code snippet like this:


  if (SwitchBoard.isInExperiment(this, "yourexperimentname")) {
== History ==
    // Do something interesting.
  }


We also need to add the "yourexperimentname" name to the [https://github.com/mozilla-services/switchboard-experiments Switchboard server config].
We started with [https://github.com/KeepSafe/Switchboard KeepSafe's Switchboard library] and server component but since then the code and architecture has changed significantly. The original implementation used a special [https://github.com/mozilla-services/switchboard-server switchboard server] that did decides which experiments a client is part of and then just returns a simplified list for the client to consume. This required the client to send data (including a unique id) to the server. To avoid this and make the administration of experiments simpler we moved to using Kinto as storage and server of the experiment configuration. Now clients load the whole configuration and decide independently what experiments they are enrolled it.


== Testing Switchboard changes locally ==
== Summary ==


To test your Switchboard changes locally, follow these steps. This assumes you have already forked and cloned the [https://github.com/mozilla-services/switchboard-experiments switchboard-experiments repo], and have made the local changes you want to test.
* The experiment configuration is stored in [http://kinto.readthedocs.io/en/stable/ Kinto] and served from a CDN: https://firefox.settings.services.mozilla.com/v1/buckets/fennec/collections/experiments/records
* The client regularly loads the configuration (or diffs, see Kinto API) and stores it locally.
* Based on a UUID for bucketing the client and using the experiment filters the client decide which experiments it is part of.
* The experiment configuration can be modified using the Kinto web admin (VPN + account whitelisting required)


# Clone the [https://github.com/mozilla-services/switchboard-server switchboard-server repo]
== Adding a new experiment ==
# In the root of the server directory, run: `EXPERIMENTS_FILE=path/to/experiments.json node index.js`
# Get a URL for your local server (I used [https://localtunnel.me/ localtunnel])
# Update the Switchboard default server URLs in [http://hg.mozilla.org/mozilla-central/file/c0ba5835ca48/mobile/android/base/java/org/mozilla/gecko/BrowserApp.java#l587 BrowserApp] to match your local server URL
# Rebuild and run Fennec


== Examples ==
=== Client ===
* [http://hg.mozilla.org/mozilla-central/diff/3d97a673734c/mobile/android/base/firstrun/FirstrunPagerConfig.java#l24 First run panels experiment]
 
=== Server side ===
 
* Go to https://settings-writer.prod.mozaws.net/v1/admin (You need to be on the Mozilla network, either in an office or via VPN)
* Login with your LDAP credentials (Your account needs to be whitelisted)
* Follow the Web UI to create or modify an experiment. See below for an explanation of the fields.
* Currently saved changes go live immediately! The Kinto team is setting up a process for a mandatory review step.
 
=== Client side ===
 
* Add your experiment name to [https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/Experiments.java Experiments.java]
* Use [https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/switchboard/SwitchBoard.java SwitchBoard.isInExperiment()] to check whether the client is part of your experiment.
* Use SwitchBoard.getExperimentValuesFromJson() if you have configured additional metadata for your experiment.
 
=== Testing ===
 
* There's an add-on that you can use for overriding the local switchboard configuration: [https://addons.mozilla.org/en-US/android/addon/switchboard-experiments/ Addon Download] [https://github.com/pocmo/Addon-Switchboard-Experiments Source code]. This works great for switching experiments on/off - but doesn't let you configure additional metadata if required by the experiment.
 
== Experiment configuration ==
 
* '''Name''': A descriptive name for the experiment. This name is used in JSON that is being sent by the server and stored by the client. Examples are:  ''bookmark-history-menu'', ''malware-download-protection'', ''promote-add-to-homescreen''
* '''Description''': A more detailed description what this experiment is about. In the best case this contains a bug number for context. Currently we send this data to the clients too (Don't make it too big).
* '''Regions''': It check the country we get in the search engine manager (e.g. US, TW). If there's no value stored we just think we are not in the experiment. In the admin interface, it should be a list (e.g {'US','TW'} ).
* '''Buckets''': Every user is in one of 100 buckets (0-99). For every experiment you can set up a min and max value (0 <= min <= max <= 100). The bounds are [min, max). An experiment set to buckets min=0, max=50 means that users in buckets 0-49 will have it enabled. There is no check for out of bounds values, so "disabled" experiments can have min/max buckets keys that are either out of bounds or have min == max. A disabled experiment is usually set to min=0, max=0.
* '''Filters''' - In addition to buckets additional filters can be used to limit the experiments to a subset of users.
** appId (regex): The Android app ID (e.g. org.mozilla.fennec, org.mozilla.firefox_beta, org.mozilla.firefox)
** version (regex): The Firefox app version number (e.g. 47.0a1', 46.0)
** lang (regex): Language, pulled from the default locale (e.g. eng)
** country (regex): Country, pulled from the default locale (e.g. USA)
** device (regex): Android device name
** manufacturer (regex): Android device manufacturer
* '''values''': Every experiment can include an optional JSON object with values that are used by the experiment. For example an experiment showing a bookmark dialog after a page has been visited multiple times, could use a value that defines how many visits are needed to display the dialog. With that the experiment can change without needing to update the app.

Latest revision as of 11:28, 2 June 2017

Switchboard is a service we use to segment users for A/B testing or staged rollout of features.

An experiment can be enabled for a specified percentage of users and limited to release channels, app versions, languages, countries (locale), device names and device manufacturers. An experiment can contain additional metadata to "configure" the experiment. This can be used to change parameters of an experiment without needing to ship a new app version. Which experiments a user is enrolled in is reported to our telemetry systems via the core ping.

History

We started with KeepSafe's Switchboard library and server component but since then the code and architecture has changed significantly. The original implementation used a special switchboard server that did decides which experiments a client is part of and then just returns a simplified list for the client to consume. This required the client to send data (including a unique id) to the server. To avoid this and make the administration of experiments simpler we moved to using Kinto as storage and server of the experiment configuration. Now clients load the whole configuration and decide independently what experiments they are enrolled it.

Summary

Adding a new experiment

Client

Server side

  • Go to https://settings-writer.prod.mozaws.net/v1/admin (You need to be on the Mozilla network, either in an office or via VPN)
  • Login with your LDAP credentials (Your account needs to be whitelisted)
  • Follow the Web UI to create or modify an experiment. See below for an explanation of the fields.
  • Currently saved changes go live immediately! The Kinto team is setting up a process for a mandatory review step.

Client side

  • Add your experiment name to Experiments.java
  • Use SwitchBoard.isInExperiment() to check whether the client is part of your experiment.
  • Use SwitchBoard.getExperimentValuesFromJson() if you have configured additional metadata for your experiment.

Testing

  • There's an add-on that you can use for overriding the local switchboard configuration: Addon Download Source code. This works great for switching experiments on/off - but doesn't let you configure additional metadata if required by the experiment.

Experiment configuration

  • Name: A descriptive name for the experiment. This name is used in JSON that is being sent by the server and stored by the client. Examples are: bookmark-history-menu, malware-download-protection, promote-add-to-homescreen
  • Description: A more detailed description what this experiment is about. In the best case this contains a bug number for context. Currently we send this data to the clients too (Don't make it too big).
  • Regions: It check the country we get in the search engine manager (e.g. US, TW). If there's no value stored we just think we are not in the experiment. In the admin interface, it should be a list (e.g {'US','TW'} ).
  • Buckets: Every user is in one of 100 buckets (0-99). For every experiment you can set up a min and max value (0 <= min <= max <= 100). The bounds are [min, max). An experiment set to buckets min=0, max=50 means that users in buckets 0-49 will have it enabled. There is no check for out of bounds values, so "disabled" experiments can have min/max buckets keys that are either out of bounds or have min == max. A disabled experiment is usually set to min=0, max=0.
  • Filters - In addition to buckets additional filters can be used to limit the experiments to a subset of users.
    • appId (regex): The Android app ID (e.g. org.mozilla.fennec, org.mozilla.firefox_beta, org.mozilla.firefox)
    • version (regex): The Firefox app version number (e.g. 47.0a1', 46.0)
    • lang (regex): Language, pulled from the default locale (e.g. eng)
    • country (regex): Country, pulled from the default locale (e.g. USA)
    • device (regex): Android device name
    • manufacturer (regex): Android device manufacturer
  • values: Every experiment can include an optional JSON object with values that are used by the experiment. For example an experiment showing a bookmark dialog after a page has been visited multiple times, could use a value that defines how many visits are needed to display the dialog. With that the experiment can change without needing to update the app.
Retrieved from "https://wiki.allizom.org/index.php?title=Mobile/Fennec/Android/Switchboard&oldid=1172510"