Engagement/Mozilla.org Durable Team/Testing Playbook

1. Testing: Why
   1. Why test?
      1. Testing gives us data to optimize user experience, leading to increased conversion (downloads, Accounts sign ups, newsletter sign ups, etc) supporting key performance indicators.
   2. Backlog (link to come)
2. Planning. Define the following in bugzilla/google docs linked from the test tracker:
   1. Hypothesis
   2. Test Plan
   3. Measurement requirements
3. Implementation
   1. Choose testing tool(s)
      1. what tool do we use to split traffic?
         1. Optimizely offers the most detailed targeting options
         2. Custom js keeps the page weight lighter and doesn’t depend on third party tools
         3. GA
      2. what tool do we use to run the test?
         1. When do we use GA?
            1. More control over the code changes
            2. More complex changes in Design and Page functionality
            3. Pages change based off of information in the Browser (eg. Welcome page - changes based off whether your browser is set as default
            4. Segmenting results
            5. Multiple Pages
         2. When do we use Optimizely?
            1. Simple Changes
               1. Copy - testing a lot of different versions
               2. Design - basic changes
            2. Can use Optimizely for directing traffic to any page.
            3. Basic user-agent
         3. When do we use funnel cakes?
            1. 1. Funnel cakes are special Firefox builds that are used to measure the impact of changes to the onboarding flow on user retention, (primary use case for this team).
               2. Funnel cake set up process (link to come)
   2. Review
      1. Checklist for reviewing Optimizely set up
         1. Does test look and work as expected on demo server?
         2. Are correct measurements being reported in GA?
4. Reporting
   1. Tests run in Optimizely: use simple Optimizely reports
   2. Tests run in GA: work with Analytics team to pull/build more complex reports
5. Next steps
   1. Review results
      1. Newsletter conversion
      2. Participation tasks
   2. Deploy winning tests globally with L10N team
   3. Define additional hypotheses and tests based on test data