Buildbot/Talos/Misc
Adding a new test
Everybody wants moar tests, there is a lot that goes into adding a talos test:
Steps to add a test
- File a bug to add appropriate rows to graphserver [example].
- File an additional IT bug to deploy the sql changes on the production and staging graph servers [example].
- If this is adding new pages, ensure these sql changes include page definitions. NOTE: this one detail is usually forgotten since it is so rare
- MOBILE ONLY: if we are changing/adding pages which mobile tests use or need, you need to update bm-remote [example]
- create a talos.zip file and file a releng bug [example] to upload it to the build network
- add the test to the running configs (as seen on tbpl)
- for aurora/central on desktop only - we update testing/talos/talos.json with a revision to the talos tree [example]
- NOTE: one exception is adding an entirely new suite- this requires a buildbot patch to add the new scheduled job [example]
- for other branches or mobile - we still need a buildbot patch, many times we need to update both buildbot and talos.json
- ensure the test follows the trains
- if we are dealing with buildbot configs, then we will need bugs for each merge cycle to turn on/off appropriate tests
- if we are just updating talos.json, this will be automatic
- if this is an update to an existing test that could change the numbers, this needs to be treated as a new test and run side by side for a week to get a new baseline for the numbers.
- file a bug to get TBPL updated with a new letter to track this test [example]
- document the test on Buildbot/Talos and Buildbot/Talos/Tests
- NOTE: this should be updated every time the branch moves closer to release. Once on release we don't need to update
While that is a laundry list of items to do, if you are developer of a component just talk to the a*team (jhammel or jmaher) and they will handle the majority of the steps above.
What we need to know about tests
When adding a new test, we really need to understand what we are doing. Here are some questions that you should know the answer to before adding a new test:
- What does this test measure?
- Does this test overlap with any existing test?
- What is the unit of measurement that we are recording?
- What would constitute a regression?
- What is the expected range in the results over time?
- Are there variables or conditions which would affect this test?
- browser configuration (prefs, environment variables)?
- OS, resources, time of day, etc... ?
- Indepenedent of Observation? Will this test produce the same number regardless of what was run before it?
- What considerations are there for how this test should be run and what tools are required?
Background Information
Other usage of Talos
- https://wiki.mozilla.org/Auto-tools/Projects/JetPerf
- compare-talos : https://bitbucket.org/mconnor/compare-talos deployed at http://perf.snarkfest.net/compare-talos/
Happy Talos testing!
Naming convention
't' is pre-pended to the names to represent 'test'. Thus, ts = 'test startup', tp = 'test pageload', tdhtml = 'test dhtml'.
History of tp Tests
tp
The original tp test created by Mozilla to test browser page load time. Cycled through 40 pages. The pages were copied from the live web during November, 2000. Pages were cycled by loading them within the main browser window from a script that lived in content.
tp2/tp_js
The same tp test but loading the individual pages into a frame instead of the main browser window. Still used the old 40 page, year 2000 web page test set.
tp3
An update to both the page set and the method by which pages are cycled. The page set is now 393 pages from December, 2006. The pageloader is re-built as an extension that is pre-loaded into the browser chrome/components directories.
tp4
Updated web page test set to 100 pages from February 2009.
tp4m
This is a smaller pageset (21 pages) designed for mobile Firefox. This is a blend of regular and mobile friendly pages.
We landed on this on April 18th, 2011 in bug 648307. This runs for Android and Maemo mobile builds only.
tp5
Updated web page test set to 100 pages from April 8th, 2011. Effort was made for the pages to no longer be splash screens/login pages/home pages but to be pages that better reflect the actual content of the site in question.