TestEngineering/Performance/Talos: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(update sheriff list)
No edit summary
 
(26 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Image:Talos.jpg]]  
__NOTOC__[[Image:Talos.jpg|frameless|right]]Talos is a versatile cross-platform Python performance testing framework. It was created to serve as a test runner for the performance tests that Mozilla was running back in 2007, as well as providing an extensible framework for new tests as they were created.  
 
 
== Talos ==
 
Talos is a Python performance testing framework that is usable on Windows, Mac and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.  


So, why Talos? Talos is the bronze automaton of Greek myth. Talos protected the island of Crete, throwing giant boulders at unwary seamen. He's also purported to have heated himself glowing hot and then embraced his enemies. '''Basically, he was awesome.'''
So, why Talos? Talos is the bronze automaton of Greek myth. Talos protected the island of Crete, throwing giant boulders at unwary seamen. He's also purported to have heated himself glowing hot and then embraced his enemies. '''Basically, he was awesome.'''


Talos code lives at https://dxr.mozilla.org/mozilla-central/source/testing/talos
= Documentation =
* [https://firefox-source-docs.mozilla.org/testing/perfdocs/talos.html#test-definitions Talos tests]
* [[/Platforms/|Talos platforms]]
* [[/Running|Running Talos tests]]
* [[/Adding tests/|Adding Talos tests]]
* [[/Development|Talos development]]
* [[/Data|Talos data format]]


== Talos Tests ==
= Reporting issues =
Information about each test is available in [[Performance_sheriffing/Talos/Tests]]
* Talos bugs, such as requests for new tests or repairs to the talos code itself can be filed in {{enterbug|Testing|Talos}}
* Perfherder bugs can be filed in {{enterbug|Tree Management|Perfherder}}
* Talos machine maintenance bugs are filed under {{enterbug|Infrastructure & Operations|RelOps: General}}, such as bugs having to do with the hardware that Talos is run on or requests to run extra talos tests against a given build


== Running Talos Tests ==
= Contributing =
We welcome contributions. Please see [https://codetribute.mozilla.org/projects/automation?project%3DTalos codetribute] for some good first bugs to get you started!


[[Performance sheriffing/Talos/Running|See the page on running Talos tests]].
= Further reading =
 
* http://armenzg.blogspot.com/2011/12/taloszip-talosjson-and-you.html
== Lifecycle of a Performance Test, from push to graph server ==
 
* Buildbot schedules Talos jobs
* Buildbot runs a talos job on a slave when one is available
** Buildbot downloads the build and talos.zip (found in [http://hg.mozilla.org/mozilla-central/tip/testing/talos/talos.json talos.json])
** Buildbot [https://wiki.mozilla.org/Buildbot/Talos/Running#How_Talos_is_Run_in_Production configures the test and runs it]
** After the test is completed the data is uploaded to the [http://graphs.mozilla.org/ Graph Server] (and to [https://treeherder.mozilla.org/perf.html#/graphs PerfHerder])
** a link to the Graph Server data (ex. [http://graphs.mozilla.org/graph.html#tests=[[206,63,22]] tp5]) is posted in the log file
* Treeherder displays a green (all OK) status and has a link to the graph server
* 13 pushes later, [http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py analyze_talos.py] is ran which compares your push to the previous 12 pushes and next 12 pushes to look for a [https://wiki.mozilla.org/Buildbot/Talos/Data#Regressions regression]
** if a regression is found, it will be posted on [http://groups.google.com/group/mozilla.dev.tree-management/topics?lnk=srg&pli=1 dev.tree-management]
 
== Resources / How-to ==
 
=== Adding a new test ===
To create a new test you ideally need a webpage(s) that performs the test(s) desired. The results can be a simple alert.
 
Once you've got that, just talk to folks in #perf (:jmaher, :rwood, :igoldan) and they will handle the majority of the [https://wiki.mozilla.org/Buildbot/Talos/Misc#Adding_a_new_test steps required].
 
=== Bugs ===
* [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Talos&product=Testing Talos bugs] are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos&op_sys=All&rep_platorm=All Testing/Talos], such as requests for new tests or repairs to the talos code itself.
* Perfherder bugs are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management&component=Perfherder].
* Talos machine maintenance bugs are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org mozilla.org/Release Engineering], such as bugs having to do with the hardware that talos is run on or requests to run extra talos tests against a given build.
 
=== Subpages ===
* Definition of [https://wiki.mozilla.org/Buildbot/Talos/Tests tests]
* How to [https://wiki.mozilla.org/Buildbot/Talos/Profiling use Talos profiling].
* How to [https://wiki.mozilla.org/Buildbot/Talos/Running Run Talos], and how it is run in production.
** How to run Talos on [[Mobile/Fennec/Android#talos mobile]]
* [https://wiki.mozilla.org/Buildbot/Talos/Internals Internals] of Talos
** [[Buildbot/Talos/Counters]] : Talos counters and auxiliary data '''(STUB)'''
* What our raw  [https://wiki.mozilla.org/Buildbot/Talos/Data data] looks like and what our formulas for filtering and summarization.
* History and other uses of [https://wiki.mozilla.org/Buildbot/Talos/Misc Talos]
 
=== Blog posts about Talos ===
* https://blog.mozilla.org/nfroyd/2012/09/19/looking-at-talos-differently/
* https://blog.mozilla.org/nfroyd/2012/09/19/looking-at-talos-differently/
* https://blog.mozilla.org/nfroyd/2012/10/05/looking-at-talos-differently-part-2/
* https://blog.mozilla.org/nfroyd/2012/10/05/looking-at-talos-differently-part-2/
* http://armenzg.blogspot.com/2011/12/taloszip-talosjson-and-you.html
* https://elvis314.wordpress.com/2012/03/12/reducing-the-noise-in-talos/
* http://ask.mozilla.org/question/45/how-can-i-know-if-my-push-to-try-regresses-talos-results/
* http://limpet.net/mbrubeck/2013/11/10/improving-regression-detection.html
* http://limpet.net/mbrubeck/2013/11/10/improving-regression-detection.html
* [https://wiki.mozilla.org/Buildbot/Talos duplicated & updated from old page]

Latest revision as of 11:45, 16 March 2021

Talos.jpg

Talos is a versatile cross-platform Python performance testing framework. It was created to serve as a test runner for the performance tests that Mozilla was running back in 2007, as well as providing an extensible framework for new tests as they were created.

So, why Talos? Talos is the bronze automaton of Greek myth. Talos protected the island of Crete, throwing giant boulders at unwary seamen. He's also purported to have heated himself glowing hot and then embraced his enemies. Basically, he was awesome.

Documentation

Reporting issues

Contributing

We welcome contributions. Please see codetribute for some good first bugs to get you started!

Further reading

Retrieved from "https://wiki.allizom.org/index.php?title=TestEngineering/Performance/Talos&oldid=1234443"