Buildbot/Talos/Misc: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(Replaced content with "#REDIRECT TestEngineering/Performance/Talos/Misc")
 
(21 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 
#REDIRECT [[TestEngineering/Performance/Talos/Misc]]
= Adding a new test =
 
Adding a new performance test or modifying an existing test is much easier than most people think.  I general we need to create a patch for talos which has:
* determine if this is a [http://hg.mozilla.org/build/talos/file/tip/talos/startup_test startup test] or a [http://hg.mozilla.org/build/talos/file/tip/talos/page_load_test page load test] and create the appropriate folder for your test in talos
* add all files and resources for the test (make sure there is no external network accesses) to that folder
* for [http://hg.mozilla.org/build/talos/file/tip/talos/page_load_test page load tests] make sure you have a manifest file defined and that you either report your own numbers using tpRecordTime, or let talos measure it.
* for startup tests, make sure you have the proper log format of "__start_report<result>__end_report"
* add a definition of your test to [http://hg.mozilla.org/build/talos/file/tip/talos/test.py test.py] in the talos repo
* file a bug in the [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos testing:talos] component with your patch
 
== Steps to add a test to production ==
* [[https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos File a bug]] to add tests to [[http://hg.mozilla.org/build/talos/file/tip/talos/test.py talos]] [[https://bugzilla.mozilla.org/show_bug.cgi?id=893071 example]].
 
* [[https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Webdev File a bug]] to add appropriate rows to graphserver's [[http://hg.mozilla.org/graphs/file/tip/sql/data.sql data.sql]] file. Example: [[https://bugzilla.mozilla.org/show_bug.cgi?id=907876 Bug 907876 - add tscrollr to graph server]].
** [[https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org&component=Server%20Operations%3A%20Database File an additional IT bug]] to deploy the sql changes on the production and staging graph servers. Example: [[https://bugzilla.mozilla.org/show_bug.cgi?id=907255 Bug 907255 - update graphs.m.o and graphs.a.o with sql statements]].
** If this is adding new pages, ensure these sql changes include [http://hg.mozilla.org/graphs/file/5dd6f4fb9c27/sql/pages_table.sql page definitions].
** MOBILE ONLY: if we are changing/adding pages which mobile tests use or need, you need to update bm-remote Example: [[https://bugzilla.mozilla.org/show_bug.cgi?id=883896 Bug 883896 - update bm-remote to get tsvgx files]]
 
* add the test to the running [[http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/config.py#l176 buildbot configs]] (as seen on [[https://tbpl.mozilla.org/ tbpl]])
** for aurora/central on desktop only - Update [[http://hg.mozilla.org/mozilla-central/tip/testing/talos/talos.json talos.json]] with a revision of the [[http://hg.mozilla.org/build/talos talos tree]]. Example [[https://bugzilla.mozilla.org/show_bug.cgi?id=903681 Bug 903681 - turn on ASAP tests by default]]
** NOTE: one exception is adding an entirely new suite- this requires a [[http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/config.py#l176 buildbot]] patch to add the new scheduled job. Example: [[https://bugzilla.mozilla.org/show_bug.cgi?id=886288 Bug 886288 -  deploy new tsvgx and tscrollx talos tests to m-c only]]
** for other branches or mobile - we still need a [[http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla-tests/config.py#l176 buildbot]] patch, many times we need to update both buildbot and talos.json
 
* Update talos
** for aurora/central on desktop only - we update [[http://hg.mozilla.org/mozilla-central/tip/testing/talos/talos.json testing/talos/talos.json]] with a revision of the [[http://hg.mozilla.org/build/talos talos tree]]. Example: [[https://bugzilla.mozilla.org/show_bug.cgi?id=900913 Bug 900913 - update talos to latest bits to capture xperf fix and other cleanup]]
** for other branches or mobile - we still need an updated talos.zip, many times we need to update both. example: [[https://bugzilla.mozilla.org/show_bug.cgi?id=906825 Bug 906825 - upload talos.zip to capture many talos fixes which are not being run on android]]
 
* Ensure the test follows the branch through the [[http://mozilla.github.io/process-releases/draft/development_overview/ release stages]]
** if we are dealing with buildbot configs, then we will need bugs for each merge cycle to turn on/off appropriate tests
** if we are just updating talos.json, this will be automatic
 
* if this is an update to an existing test that could change the numbers, this needs to be treated as a new test and run side by side for a week to get a new baseline for the numbers.
 
* [[https://bugzilla.mozilla.org/enter_bug.cgi?product=Webtools&component=Tinderboxpushlog File a bug]] to get TBPL updated with a new letter to track this test. Example: [[https://bugzilla.mozilla.org/show_bug.cgi?id=897415 Bug 897415 -  Add TBPL support for the new tsvgx, rafx & otherx talos jobs]]
 
* document the test on [[Buildbot/Talos]] and [[Buildbot/Talos/Tests]]
** NOTE: this should be updated every time the branch moves closer to release. Once on release we don't need to update
 
While that is a laundry list of items to do, if you are developer of a component just talk to the a*team ([[https://mozillians.org/en-US/u/jhammel/ jhammel]] or [[https://mozillians.org/en-US/u/jmaher/ jmaher]]) and they will handle the majority of the steps above.
 
== What we need to know about tests ==
 
When adding a new test, we really need to understand what we are doing.  Here are some questions that you should know the answer to before adding a new test:
* What does this test measure?
* Does this test overlap with any existing test?
* What is the unit of measurement that we are recording?
* What would constitute a regression? 
* What is the expected range in the results over time?
* Are there variables or conditions which would affect this test?
** browser configuration (prefs, environment variables)?
** OS, resources, time of day, etc... ?
* Indepenedent of Observation?  Will this test produce the same number regardless of what was run before it?
* What considerations are there for how this test should be run and what tools are required?
 
Please document the answers to these questions on '''[[Buildbot/Talos/Tests]]'''.
 
= Background Information =
 
== Hardware Profile of machines used in automation ==
 
* linux32, linxu64, winXP, win7, win8x64:
iX21X4 2U Neutron "Gemini" Series Four Node Hot-Pluggable Server (4 nodes per 2U)
920W High-Efficiency redundant (1+1) Power Supply
1 Intel X3450 CPU per node
8GB Total: 2 x 4GB DDR3 1333Mhz ECC/REG RAM per node
1 WD5003ABYX hard drive per node
1 NVIDIA GPU GeForce GT 610 per node
 
* OSX 10.6 (snow), OSX 10.7 (lion):
Model Name: Mac mini
Model Identifier: Macmini4,1
Processor Name: Intel Core 2 Duo
Processor Speed: 2.4 GHz
Number of Processors: 1
Total Number of Cores: 2
L2 Cache: 3 MB
Memory: 8 GB
Disk: 320G SATA
 
* OSX 10.8 (mtnlion):
Model Name: Mac mini
Model Identifier: Macmini5,3
Processor Name: Intel Core i7
Processor Speed: 2 GHz
Number of Processors: 1
Total Number of Cores: 4
L2 Cache (per Core): 256 KB
L3 Cache: 6 MB
Memory: 8 GB
Disk: 2 x 500G SATA drives (RAID)
 
** All Mac Minis have EDID devices attached that set the resolution at 1600x1200.
 
 
== Other usage of Talos ==
* https://wiki.mozilla.org/Auto-tools/Projects/JetPerf
* compare-talos : https://bitbucket.org/mconnor/compare-talos deployed at http://perf.snarkfest.net/compare-talos/
 
Happy Talos testing!
 
== Naming convention ==
't' is pre-pended to the names to represent 'test'.  Thus, ts = 'test startup', tp = 'test pageload', tdhtml = 'test dhtml'.
 
== History of tp Tests ==
 
=== tp ===
 
The original tp test created by Mozilla to test browser page load time. Cycled through 40 pages. The pages were copied from the live web during November, 2000. Pages were cycled by loading them within the main browser window from a script that lived in content.
 
=== tp2/tp_js ===
 
The same tp test but loading the individual pages into a frame instead of the main browser window. Still used the old 40 page, year 2000 web page test set.
 
=== tp3 ===
 
An update to both the page set and the method by which pages are cycled. The page set is now 393 pages from December, 2006. The pageloader is re-built as an extension that is pre-loaded into the browser chrome/components directories.
 
=== tp4 ===
 
Updated web page test set to 100 pages from February 2009.
 
=== tp4m ===
This is a smaller pageset (21 pages) designed for mobile Firefox.  This is a blend of regular and mobile friendly pages.
 
We landed on this on April 18th, 2011 in {{bug|648307}}.  This runs for Android and Maemo mobile builds only.
 
=== tp5 ===
 
Updated web page test set to 100 pages from April 8th, 2011.  Effort was made for the pages to no longer be splash screens/login pages/home pages but to be pages that better reflect the actual content of the site in question.

Latest revision as of 11:42, 16 March 2021

Redirect to:

Retrieved from "https://wiki.allizom.org/index.php?title=Buildbot/Talos/Misc&oldid=1234440"