Projects/ManifestDestiny/Integration

mozinfo, writemozinfo, and manifestparser

Work has begun to solidify a few tools that will be integrated with our test harnesses but could have more general usage at Mozilla.

- writemozinfo.py: writes information for a build into a JSON blob; see http://hg.mozilla.org/mozilla-central/rev/6d19baaa339f and http://mxr.mozilla.org/mozilla-central/source/config/writemozinfo.py

- mozinfo: presents a unified interface for accessing system information via python (though extensible to other languages as needed); see http://k0s.org/mozilla/hg/mozinfo or bug 606524

- manifestparser: parses test manifests; http://hg.mozilla.org/automation/ManifestDestiny

How they work together

mozinfo is an interface layer for exposing system information according to a unified convention. For instance, instead of:

if sys.platform.startswith('linux')

you can do:

if mozinfo.os == 'linux'

or:

if mozinfo.isLinux:

writemozinfo.py produces a JSON blob detailing the information about a particular build during the ``make`` process:

│cat mozinfo.json
{'debug': false, 'os': 'linux', 'bits': 32, 'processor': 'x86'}

This information may be consumed by mozinfo, among other consumers. Since mozinfo and writemozinfo share the same keys for the same items, mozinfo's introspected ``dict`` may be updated according to a particular build independent of identified information from the system you are currently on (using ``mozinfo.update()``).

The manifestparser (python package name: ManifestDestiny) architecture is detailed at http://hg.mozilla.org/automation/ManifestDestinty/file/tip/README.txt . manifestparser contains two classes for handling .ini manifests:

• ManifestParser contains the basic logic for parsing ordered

 and nested manifests as well as querying them based on their
 tags. ManifestParser is not specific to tests

• TestManifest inherits from ManifestParser and is the integration

 layer for test harnesses.

The tests are returned as a list of dicts, with some keys, such as the test name and path, provided by the parser class, with the other keys taken from the .ini manifest section keys, as eachg section corresponds to one test.

TestManifest has a method, ``active_tests``, which formats and filters tests based on criteria. Tests can be filtered by (``os.path``) existence or according by variables passed into the method and some special keys in the test dict. Currently, there are:

• skip-if: mark the test as ``disabled`` if the condition given as the

 value of the key is true.  The variables passed in are used with
 basic javascript-syntax boolean logic (e.g.)::

  skip-if = os == 'linux' || (toolkit == 'cocoa' && debug)

 For this example, the variables ``os``, ``toolkit``, and ``debug``
 should be passed into the ``active_tests`` method.

• run-if: similar to skip-if, but mark the test as ``disabled`` if the

 condition is not true. skip-if and run-if can be used together, but
 if either disables the test, the test is disabled.

• fail-if: the ``expected`` value of tests returned from

 ``active_tests`` is ``pass`` unless specified otherwise in the .ini
 file. If the test for fail-if is true, however, the value of
 ``expected`` will be ``fail``.

It is up to the harness to treat the tests appropriately based on their metadata. Unless ``disabled=False`` is passed to ``active_tests()``, all tests will be returned but the disabled (i.e. skipped) tests will have the key ``disabled`` with, by convention, the reason as the value. (You can also mark a test as ``disabled`` directly in the manifest if desired.) Similarly with ``expected``. It is presumed that the harness will want to deal with the handling and reporting of skipped and failed tests.

While ``active_tests`` can take an arbitrary ``**values`` dict, the entire system is set up such that passing in a dict as produced by mozinfo or writemozinfo forms the standard convention for test harnesses. Test harnesses may augment the dictionary produced by mozinfo, or they may introduce an additional and specialized integration layer augmenting TestManifest. The degrees of freedom in both of these directions gives a flexible system but one reducing to a simple case for the common use case.

For harnesses running inside the mozilla-central tree and wishing to access ``${OBJDIR}/mozinfo.json``, pseudocode for calling ``active_tests`` is as follows::

import json
import manifestparser
import mozinfo
import os
mozinfo.update(json.loads(os.path.join(objdir, 'mozinfo.json')))
manifest = manifestparser.TestManifest(manifests=('mymanifest.ini,'))
tests = manifest.active_tests(**mozinfo.info)

For harnesses running outside of mozilla-central where ``mozinfo.json`` is not available, or otherwise not applicable, the above pseudocode may be used sans the ``mozinfo.update`` line.

To recap, the standard pattern is:

• mozinfo introspects system information

• the mozinfo information may be augmented or overridden from the

 build information as output by writemozinfo

• the mozinfo information may be fed into

 ``TestManifest.active_tests`` to retrieve a properly marked-up list
 of tests

• test harnesses and other consumers have several degrees of freedom

 to customize how things fit together