View source for Projects/ManifestDestiny/Integration

= mozinfo, writemozinfo, and manifestparser =

Work has begun to solidify a few tools that will be integrated with
our test harnesses but could have more general usage at Mozilla.

- writemozinfo.py: writes information for a build into a JSON blob; see
http://hg.mozilla.org/mozilla-central/rev/6d19baaa339f and
http://mxr.mozilla.org/mozilla-central/source/config/writemozinfo.py

- mozinfo: presents a unified interface for accessing system
information via python (though extensible to other languages as
needed); see http://k0s.org/mozilla/hg/mozinfo or bug 606524

- manifestparser: parses test manifests; https://github.com/mozilla/gecko-dev/tree/master/testing/mozbase/manifestparser

== How they work together ==

mozinfo is an interface layer for exposing system information
according to a unified convention.  For instance, instead of:

 if sys.platform.startswith('linux')

you can do:

 if mozinfo.os == 'linux'

or:

 if mozinfo.isLinux:

writemozinfo.py produces a JSON blob detailing the information about a
particular build during the ``make`` process:

 │cat mozinfo.json
 {'debug': false, 'os': 'linux', 'bits': 32, 'processor': 'x86'}

This information may be consumed by mozinfo, among other consumers.
Since mozinfo and writemozinfo share the same keys for the same items,
mozinfo's introspected ``dict`` may be updated according to a
particular build independent of identified information from the system
you are currently on (using ``mozinfo.update()``).

The manifestparser (python package name: ManifestDestiny)
architecture is detailed at
http://hg.mozilla.org/automation/ManifestDestinty/file/tip/README.txt
. manifestparser contains two classes for handling .ini manifests:

* ManifestParser contains the basic logic for parsing ordered and nested manifests as well as querying them based on their tags. ManifestParser is not specific to tests

* TestManifest inherits from ManifestParser and is the integration layer for test harnesses.

The tests are returned as a list of dicts, with some keys, such as the
test name and path, provided by the parser class, with the
other keys taken from the .ini manifest section keys, as eachg section
corresponds to one test.

TestManifest has a method, ``active_tests``, which formats and filters
tests based on criteria.  Tests can be filtered by (``os.path``)
existence or according by variables passed into the method and some
special keys in the test dict.  Currently, there are:

* skip-if: mark the test as ``disabled`` if the condition given as the value of the key is true.  The variables passed in are used with basic javascript-syntax boolean logic (e.g.)::

   skip-if = os == 'linux' || (toolkit == 'cocoa' && debug)

For this example, the variables ``os``, ``toolkit``, and ``debug``
should be passed into the ``active_tests`` method.

* run-if: similar to skip-if, but mark the test as ``disabled`` if the condition is not true. skip-if and run-if can be used together, but if either disables the test, the test is disabled.

* fail-if: the ``expected`` value of tests returned from ``active_tests`` is ``pass`` unless specified otherwise in the .ini file. If the test for fail-if is true, however, the value of ``expected`` will be ``fail``.

It is up to the harness to treat the tests appropriately based on
their metadata.  Unless ``disabled=False`` is passed to
``active_tests()``, all tests will be returned but the disabled
(i.e. skipped) tests will have the key ``disabled`` with, by
convention, the reason as the value.  (You can also mark a test as
``disabled`` directly in the manifest if desired.)  Similarly with
``expected``. It is presumed that the harness will want to deal with
the handling and reporting of skipped and failed tests.

While ``active_tests`` can take an arbitrary ``**values`` dict, the
entire system is set up such that passing in a dict as produced by
mozinfo or writemozinfo forms the standard convention for test
harnesses.  Test harnesses may augment the dictionary produced by
mozinfo, or they may introduce an additional and specialized
integration layer augmenting TestManifest.  The degrees of freedom in
both of these directions gives a flexible system but one reducing to a
simple case for the common use case.

For harnesses running inside the mozilla-central tree and wishing to
access ``${OBJDIR}/mozinfo.json``, pseudocode for calling ``active_tests``
is as follows::

 import json
 import manifestparser
 import mozinfo
 import os
 mozinfo.update(json.loads(os.path.join(objdir, 'mozinfo.json')))
 manifest = manifestparser.TestManifest(manifests=('mymanifest.ini,'))
 tests = manifest.active_tests(**mozinfo.info)

For harnesses running outside of mozilla-central where
``mozinfo.json`` is not available, or otherwise not applicable, the
above pseudocode may be used sans the ``mozinfo.update`` line.

To recap, the standard pattern is:

* mozinfo introspects system information

* the mozinfo information may be augmented or overridden from the build information as output by writemozinfo

* the mozinfo information may be fed into ``TestManifest.active_tests`` to retrieve a properly marked-up list of tests

* test harnesses and other consumers have several degrees of freedom to customize how things fit together