StandaloneTalos

How to set up Talos for testing at home

StandaloneTalos is deprecated: https://bugzilla.mozilla.org/show_bug.cgi?id=714659 and see https://wiki.mozilla.org/Buildbot/Talos#Running_locally_-_Source_Code for how to run with modern Talos

Prerequisites

1. Python >= 2.5
2. Pyyaml
3. PyWin32 Extensions (if this is to be run on a windows box)
4. Standalone V2.1
5. Nightly firefox builds (grab the appropriate type for your OS)

[dmandelin] When I installed PyWin32 the installer put things in my Windows installation of Python, not my mozilla-build copy. I copied the PyWin32 files from the one site-packages directory to the other and it worked.

pyyaml and pywin32 can be installed with:

easy_install pyyaml easy_install pywin32

(jmaher: "easy_install pywin32" didn't work and gave "No local packages or download links found for pywin32" so I used:

easy_install http://superb-sea2.dl.sourceforge.net/project/pywin32/pywin32/Build216/pywin32-216.win32-py2.6.exe

Note that this URL is python 2.6 specific.)

Set Up

1. unzip standaloneV2.1.zip
2. unzip/install your nightly firefox build
3. in the talos directory
   • edit sample.config, change the path to firefox to be the appropriate path to the firefox executable of the nightly build that you just installed

Seriously, really use the command-line unzip instead of the Ubuntu GUI Archive Manager. The latter doesn't extract the archive properly.

TP

If running tp you will need to extract the tp pageset package, pageloader.xpi to talos/page_load_test. You will want to install a webserver such as apache to serve the content, using file:// is strongly discourage as we have seen it run significantly slower. We are not sure exactly why this is the case however beware that using file:// to serve the pageset can mask regression. I've seen scores of 300 with a webserver and 500 without [BenWa].

Testing

1. in the talos directory run 'python run_tests.py sample.config'
   • for more ouput run 'python run_tests.py --noisy sample.config'
   • for debug output run 'python run_tests.py --debug sample.config'
   • to dump all results directly to stdout run 'python run_tests.py --screen sample.config'

Results

1. results from all the tests should be found in talos/output directory.

(On a MacBook Pro, the default test configuration takes about 35 minutes to run with a nightly Firefox 3 build.)

N900 notes

• Python does not get installed on your N900 by default.
  • Add "deb http://repository.maemo.org/extras/ freemantle free non-free" to your /etc/apt/sources.list file
  • "apt-get upgrade"
  • "apt-get install python"
• You will need to download Pyyaml and run "python setup.py install"

Release Notes

V2.1 (April 18th, 2011)

In This Version

• includes dirty profiles for dirty profile tests
• includes new tpaint tests
• updated to latest talos code

V2 (October 7th, 2010)

In This Version

• now using the mercurial Talos code base. We are free from CVS!

V1.9 (September 24th, 2010)

In This Version

• addition of permissions.sqlite to base_profiles to allow talos to bypass certain security restrictions to run tests

V1.8 (August 5th, 2010)

In This Version

• new accessibility test
• switched to pageloader as bundle (https://bugzilla.mozilla.org/show_bug.cgi?id=580698 Switch Talos pageloader to ship in distribution/bundles instead of dropping into components/ and chrome/)

V1.7.2 (June 28th, 2010)

In This Version

• another fix to how pages are loaded so that this works without installing a local web server

V1.7.1 (June 18th, 2010)

In This Version

• changed so that pages are loaded through file:// - no longer requires installation of a local web server

V1.7 (June 11th, 2010)

In This Version

• new dromaeo tests
• new --screen option (for sending all results directly to stdout
• new tscroll test
• talos code stability updates

V1.6 (September 16th, 2009)

In This Version

• talos code stability updates
• inclusion of dirty profile tests, though they are off by default

V1.5 (May 7th, 2009)

In This Version =

• ability to run tshutdown tests
• ability to run firefox-bin (instead of firefox) on mac/linux

V1.4 (January 15th, 2009)

In This Version

• updated pageloader extension to latest (includes fix to how median is calculated)
• tests are now run in the order that they appear in the sample.config
• csv output now written out post each individual test

V1.3.1 (September 26th, 2008

In This Version

• updated pageloader extension to latest
• fixed errors in sample.config introduced in last version

V1.3 (September 25th, 2008)

In This Version

• removed some failing web pages from the tp test
• bug 454999 (Standalone talos kills all running Firefox processes)
  • instead of killing off processes warns the user that existing Firefox processes are running and need to be shut down before testing can begin
  • not a full fix, but more of a stop gap to stop unexpected browser killing
• updates to talos code
  • better process management, better ability to kill frozen/crashed browsers

V1.2 (May 8th, 2008)

In This Version

• compliance with bug 432321 (Need to add the browser.EULA.override pref to all Talos profiles)

V1.1 (April 17th, 2008)

In This Version

• compliance with bug 424594 ( have talos comply with new file:// security restrictions)
• new & better process management bug 419492 ( talos needs to ensure that browser is closed at end of testing)

V1.0 (February 14th, 2008)

In This Version

• tsvg/tgfx/twinopen/ts/tdhtml/tjss/tsunspider/tp
• includes a small, distributable page set for page load time testing