QA/Sync/Test Plan/grinder tests

Overview

We would like to have a test framework that tests the sync servers for load and functionality. For this, we will use Grinder in order to mimic the action of many firefox clients simultaneously.

We are trying to accomplish the following goals

• Create a continuous baseline testing framework (tied to hudson)
• Be able to answer questions about current and future growth.
  • What types of services can we support now and what changes can we support in the future?

We are trying to answer the following questions

• How does the app fail under high load conditions? How does it fail?
  • From a large data input?
  • From a lot of connections?
• Which functions and potential use cases create a particularly high amount of load?
  • High amounts of registration? (and corresponding initial sync?)
  • Lots of empty requests? (as generated by instant sync?)
• Are any functions effected sooner by high load?
• How do these services scale?

Test Cases

• Sustained Load - Long periods of load until read time becomes unbearably large because we are no longer caching data
• Spike Load - How do we deal with a ton of requests at once or in a very short time (say >15 min)
• Generating Data (and verification)
  • Bookmarks
  • History
  • Bookmarks and History
  • Account creation
  • Account deletion
  • Tabs syncing (later priority)
  • Passwords Syncing (later priority)

These test cases need to mimic the actions of firefox as closely as possible. We will use grinder to simulate the actions of a firefox client.

Issues

• In order to validate data we need to store that data. In order to validate a sync servers worth of data we will need a lot of storage. (Some optimizations may be able to take place).
• We need to parallelize the load generation over multiple machines to match the capability of the server
  • Grinder supports multiple hosts
  • We need help to get the adequate hardware for this task

Priorities

First priority is Bookmarks and history - this effects the most users.