Auto-tools/Projects/MozmillBuildbot

This page documents the work from https://bugzilla.mozilla.org/show_bug.cgi?id=516984 : Run Mozmill Tests with buildbot

This is an Auto-tools project and a Q2 goal

Strategy

mozmill and the related python packages will be added to mozilla-central following the manner of existing test harnesses.

Accomplished Work

• reworked patch from bug 516984 so that it would cleanly apply: https://bugzilla.mozilla.org/attachment.cgi?id=434790 (March 24, 2010)

• added patch to disable notification: http://k0s.org/mozilla/hg/buildbotcustom/.hg/patches/rev/8ba008fdce67 ; i'm not sure if this is complete or not, but staging instances of buildbot should not email tinderbox or other live recipients (this patch no longer exists after several restarts on patching buildbotcustom; the patch should be redone and ticketed for inclusion)

• added patch to allow copying buildbot configuration: http://k0s.org/mozilla/hg/buildbot-configs/rev/aa3ff093fa4b ; useful for making new buildmasters as it clones versus symlinks to a versioned master

• added patch to raise a more verbose exception to diagnose download error: http://k0s.org/mozilla/hg/buildbotcustom/.hg/patches/rev/71b8c045d03a

• wrote script to print URL from latest build (for forcing builds): http://k0s.org/mozilla/hg/GetLatestTinderbox/file/tip/getlatesttinderbox/get_latest.py

• investigated various hosting and workflow options for python packages (e.g. mozmill)

• created a program that will "grab" the tarballs of a package and its dependencies from the cheeseshop: http://k0s.org/hg/stampit ; this is really rough, as it uses the pip command line interface which is at this moment ill-suited for this purpose; maybe using the pip API directly is better? This should probably be moved to http://k0s.org/mozilla/hg/stampit or to a mozilla.org website

• created a script, http://k0s.org/mozilla/package-it.sh , that with stampit can create a bare-bones cheeseshop

• fixed virtualenv to not bomb if PYTHONHOME is set. Formerly, if PYTHONHOME was set, sys.prefix would not resolve correctly and virtualenv would bomb with exit code 100. This actually obfuscates the true bug -- if PYTHONHOME is set, regardless of whether or not the virtualenv already exists, packages will not be retrieved from $VIRTUAL_ENV/lib but instead from $PYTHONHOME/lib ; workaround this by fixing PYTHONHOME in bin/activate scripts. A higher-level workaround is not possible (at least easily), as PYTHONHOME is honored at the (C) binary level. See (e.g.) http://bitbucket.org/ianb/virtualenv/changeset/60eec6127aa0 and http://pypi.python.org/pypi/virtualenv/1.4.8

How Buildbot Works

notes from integrating the MozmillFactory

Resources

Buildbot:

• buildbot configs
• buildbotcustom
• Buildbot wiki page

Mozmill:

• MDC wiki page
• pypi page

Related Bugs

• bug 557336 : Allow UnittestPackagedBuildFactory to be a useful base class
• bug 500142 : Create mozmill and mozmill-one make targets for build automation to use
• bug 562107 : Provide mechanism to run l10n mozmill tests from m-c without a build
• bug 562106 : Land some set of Mozmill tests into mozilla-central

Upgrading Mozmill

TODO: udpate this now that we're not using tarballs

Since mozmill and the tests are mirrored into mozilla-central, when mozmill is upgraded the tests need to be repackaged into the tree.

1. Wipe Existing Tarballs: from the root of the mozilla-central tree:

rm testing/mozmill/*.tar.gz

2. Generate New Tarballs: stampit may be used to generate new tarballs from http://pypi.python.org . If you don't want the released copies, skip this step and do this by hand. It is better to install stampit and use it interactively, but it may also be used from the web:

curl http://k0s.org/hg/stampit/raw-file/tip/stampit/main.py | python - -o testing/mozmill mozmill simplejson

A similar procedure should be followed when virtualenv or the mozmill tests are updated.

TODO: add a script into m-c to do this

Tips

• buildbotcustom is not a python package. On the VM, it is installed in /tools/buildbotcustom/buildbotcustom which is added on cltbld login to PYTHONPATH. On a local machine, I use a [1] to isolate python development. I download buildbotcustom to $VIRTUAL_ENV/buildbotcustom/buildbotcustom and then add $VIRTUAL_ENV/buildbotcustom to the PYTHONPATH in the $VIRTUAL_ENV/bin/activate file so that the code is available within the virtualenv.

• forcing builds through the web does not work. This is due to the fact that the build and the crashsymbols are downloaded must be downloaded from the tinderbox builds so the location of the desired build is unknown as they are indexed by time stamps (seconds since epoch). buildbot sendchange should instead be used to trigger a build for testing purposes.

• debugging buildbot by running directly with twistd through the command line. buildbot, in general, is difficult to debug, as the log does not contain good summaries of the python errors encountered. As pointed out in http://osdir.com/ml/python.buildbot.devel/2007-10/msg00033.html , twistd may be directly on the master's buildbot.tac file for debugging: twistd -by master/buildbot.tac

Outstanding Work

• Add a README and a test-runner (e.g. run-mozmill.{sh,py}) to testing/mozmill

• Add corrections to https://wiki.mozilla.org/Buildbot and linked pages. First this page should be finished (presumedly near project completion) and then the both pages audited to figure out what needs to be done.

• Decide what to *really* do about the mozmill-tests. Currently, the tests from http://hg.mozilla.org/qa/mozmill-tests are exported to the mozilla-central tree. However, this gives no clear upstream path for pushing committed changes to the m-c tests to those at hg.mozilla.org/qa/mozmill-tests . It is questionable if these are the same resource. This is mostly a policy decision of what we want to do. There are several possibilities and no clear correct answer.

Long-term Recommendations

These recommendations are hereby recorded for future use. Their need should be assessed before filed as bugs and made actionable. These are not blockers for this body of work and will not be worked on as part of it.

• Use templates for buildbot-configs: the configurations in http://hg.mozilla.org/build/buildbot-configs are hard-coded for the setup for VMs. Instead, they could be made templates for easy deployment. Cleanup should also be done so that the configurations are more universal and adding new configurations is easy.

• Make buildbotcustom a python package: reason: there's no reason not to (except legacy reasons, continue reading) and having the python be installable saves the trouble with messing with the PYTHONPATH. There is a considerable downside in practical terms, however; http://hg.mozilla.org/build/buildbotcustom is a top level repository; all files therein would have to be moved to http://hg.mozilla.org/build/buildbotcustom/buildbotcustom with the top level directory containing a setup.py file. While the effort is minimal, it may cause issues for existing installations which, while easily resolvable, might involve some work.

• Allow force build through the web: several factories, including the MozmillFactory, use a buildstep like:

       def get_fileURL(build):
           fileURL = build.source.changes[-1].files[0]
           build.setProperty('fileURL', fileURL, 'DownloadFile')
           return fileURL
       self.addStep(DownloadFile(
        url_fn=get_fileURL,
        filename_property='build_filename',
        haltOnFailure=True,
        name="download_build",
       ))

However, for the case of TTW force builds, build.source.changes is an empty tuple, so the step fails with an exception. In general, if build.source.changes is empty, a URL should be constructed from the platform and other variables appropriate to the build

• Make a paster template for buildbot testing: As a generalization, the production buildbot configs are less appropriate for testing and developing Mozilla's buildbot. A template could be created to make it easier to make testing master and slave instances. Files should include master.cfg, a forcebuild script for the buildslave, and a script for (re)starting both master and slave and forcing a build. The template additionally serves as instructions for future generations.