Mobile/Fennec/Android/Switchboard
Switchboard is a service we use to segment users for A/B testing or staged rollout of features.
An experiment can be enabled for a specified percentage of users and limited to release channels, app versions, languages, countries (locale), device names and device manufacturers. An experiment can contain additional metadata to "configure" the experiment. This can be used to change parameters of an experiment without needing to ship a new app version. Which experiments a user is enrolled in is reported to our telemetry systems via the core ping.
History
We started with KeepSafe's Switchboard library and server component but since then the code and architecture has changed significantly. The original implementation used a special switchboard server that did decides which experiments a client is part of and then just returns a simplified list for the client to consume. This required the client to send data (including a unique id) to the server. To avoid this and make the administration of experiments simpler we moved to using Kinto as storage and server of the experiment configuration. Now clients load the whole configuration and decide independently what experiments they are enrolled it.
Summary
- The experiment configuration is stored in Kinto and served from a CDN: https://firefox.settings.services.mozilla.com/v1/buckets/fennec/collections/experiments/records
- The client regularly loads the configuration (or diffs, see Kinto API) and stores it locally.
- Based on a UUID for bucketing the client and using the experiment filters the client decide which experiments it is part of.
- The experiment configuration can be modified using the Kinto web admin (VPN + account whitelisting required)
Adding a new experiment
Client
Server side
- Go to https://settings-writer.prod.mozaws.net/v1/admin (You need to be on the Mozilla network, either in an office or via VPN)
- Login with your LDAP credentials (Your account needs to be whitelisted)
- Follow the Web UI to create or modify an experiment. See below for an explanation of the fields.
- Currently saved changes go live immediately! The Kinto team is setting up a process for a mandatory review step.
Client side
- Add your experiment name to Experiments.java
- Use SwitchBoard.isInExperiment() to check whether the client is part of your experiment.
- Use SwitchBoard.getExperimentValuesFromJson() if you have configured additional metadata for your experiment.
Testing
- There's an add-on that you can use for overriding the local switchboard configuration: [Addon Download https://addons.mozilla.org/en-US/android/addon/switchboard-experiments/] Source code. This works great for switching experiments on/off - but doesn't let you configure additional metadata if required by the experiment.