Engagement/Mozilla.org Durable Team/Testing Playbook

Testing: Why
1. Why test?
  1. Testing gives us data to optimize user experience, leading to increased conversion (downloads, Accounts sign ups, newsletter sign ups, etc)
Test tracker
1. Planning. Define the following in bugzilla/google docs linked from wiki page:
2. Hypothesis
3. Test Plan
4. Measurement requirements
Implementation
1. Choose testing tool(s)
  1. what tool do we use to split traffic?
    1. Optimizely offers the most detailed targeting options
    2. Custom js keeps the page weight lighter and doesn’t depend on third party tools
    3. GA
  2. what tool do we use to run the test?
    1. When do we use GA?
      1. More control over the code changes
      2. More complex changes in Design and Page functionality
      3. Pages change based off of information in the Browser (eg. Welcome page - changes based off whether your browser is set as default
      4. Segmenting results
      5. Multiple Pages
    2. When do we use Optimizely?

Simple Changes Copy - testing a lot of different versions Design - basic changes Can use Optimizely for directing traffic to any page. Basic user-agent Review Checklist for reviewing Optimizely set up Does test look and work as expected on demo server? Are correct measurements being reported in GA? Reporting Tests run in Optimizely: use simple Optimizely reports Tests run in GA: work with Analytics team to pull/build more complex reports Next steps Review results Data studio links on wiki page Deploy winning tests globally with L10N team Define additional hypotheses and tests based on test data