Buildbot/Talos/Profiling

How to Run Talos in Profiling Mode

Warning: We're currently working on some major improvements to Talos profiling in bug 1121571. You'll want to use the in-progress patches from there (or just clone http://hg.mozilla.org/users/mstange_themasta.com/talos-sps-profiling at revision 1a737105cc43), otherwise the results won't be very useful. This work will hopefully land within the next few days.

When profiling is enabled, we use the Gecko Profiler to capture profiles that can then be displayed by Cleopatra. When running in profiling mode, captured profiles are copied to the upload directory.

On TryServer

0. Apply this patch to your Gecko repository in order to use the good stuff from bug 1121571 before it lands (which should happen on Jan 22 2015 at the latest):

--- a/testing/talos/talos.json
+++ b/testing/talos/talos.json
@@ -1,16 +1,16 @@
 {
     "talos.zip": {
         "url": "http://talos-bundles.pvt.build.mozilla.org/zips/talos.f3179facd945.zip",
         "path": ""
     },
     "global": {
-        "talos_repo": "https://hg.mozilla.org/build/talos",
-        "talos_revision": "ebc4327b8cb8"
+        "talos_repo": "http://hg.mozilla.org/users/mstange_themasta.com/talos-sps-profiling",
+        "talos_revision": "1a737105cc43"
     },
     "extra_options": {
         "android": [ "--apkPath=%(apk_path)s" ]
     },
     "suites": {
         "chromez": {
             "tests": ["tresize", "tcanvasmark"]
         },

1. When you push to try, add "mozharness: --spsProfile" after "try: ..." to your commit message. Example:

try: -b o -p linux,linux64,macosx64,win32,win64 -u none -t all[Ubuntu,10.6,10.8,Windows XP,Windows 7,Windows 8] mozharness: --spsProfile

2. When the tests finishes and you view the log, you will see something like

TinderboxPrint: Uploaded profile_tcanvasmark.sps.zip to http://mozilla-releng-blobs.s3.amazonaws.com/blobs/...

3. Download the zip file from this URL, extract it and see this page for info about how to open profiles in cleopatra.

When running Talos locally

You need to set an upload directory and use the --spsProfile command line parameter with Talos to capture profiles and have them copied into the upload folder.

1. Set the upload folder (make sure it exists):

export MOZ_UPLOAD_DIR=/home/username/talos/upload

2. Run talos with --spsProfile

talos -n -d --develop --executablePath ~/work/gecko-dev/obj-x86_64-unknown-linux-gnu/dist/bin/firefox --activeTests ts --spsProfile

3. You will have one zip file per test in your upload folder. See this page for info about how to open profiles in cleopatra.

FAQ

The Talos results with profiling are always higher, so how can I even use this?

Yes, profiling usually has non-trivial overhead. This is why you might want to try one of these approaches:

• Use Talos profiling only when you already know that there is a problem.
• Do a Talos run with profiling on before and after your change and compare those numbers.
• Do a Talos run with profiling off to check if the issue still exists and at the same time do a run with profiling on to get the data from it.

Why are the Windows tests sometimes hanging with profiling on?

There's a known issue with Talos profiling on Windows. See this bug.

I need to download all the profiles! Help!

Try this python script:

import urllib2
import json
from pprint import pprint
import re
from StringIO import StringIO
import gzip
import os
import sys

if len(sys.argv) < 2:
  print "usage:" , sys.argv[0] , "<try revision>"
  exit(1)

builds = json.loads(urllib2.urlopen('https://tbpl.mozilla.org/php/getRevisionBuilds.php?branch=try&rev='+ sys.argv[1]).read())

if len(builds) == 0:
  print "No bulids found for this revision"
  exit(1)

i = 0
for build in builds:
  name = build['buildername']
  print i, '/', len(builds), name
  i += 1
  name = name.replace(' ', '_')
  log = gzip.GzipFile(fileobj=StringIO(urllib2.urlopen(build['log']).read())).read()
  zips = re.findall("TinderboxPrint: Uploaded (profile_[a-zA-Z0-9-_]+\.zip) to ([^\n]+)", log)
  for zipp in zips:
    print ' ', zipp[0]
    try:
      os.mkdir(name)
    except OSError:
      pass
    with open(os.path.join(name, zipp[0]), 'w') as out:
      out.write(urllib2.urlopen(zipp[1]).read())
  print