Gaia/Email/LoggestTestFramework

Overview

gaia-email-libs-and-more currently uses a largely custom testing framework right now that was built for the deuxdrop project based on experiences trying to test Thunderbird.

Concepts

Asynchronous Steps

Each test is organized into one or more asynchronous steps. Each step runs until all of the asynchronous operations in the step are completed. Each step has a timeout; if the timeout fires before the asynchronous operations complete, the step fails and the test case fails and the test driver framework just tries to run the cleanup code for a test.

There are a few different types of test step; the delineation was just made to try and provide meta-data that could help automated tooling or those reading the tests. The TestContext class defines the following methods:

• setup(): A test step that is just trying to set up the test; these kinds of steps should usually pass unless something is very broken in the system.
• convenienceSetup(): A setup step that is automatically generated by testing support code rather than explicitly written to be part of the test. For example, when our test involves talking to an IMAP server, setting up the account happens automatically.
• action(): A test step that's trying to do something. Currently there is no concept of a 'convenienceAction', although action steps can be marked as "boring" so the UI displayed them with less contrast.
• check(): A test step that's just trying to check the state of the system and not perform any actions. Currently there is no concept of a 'convenienceCheck', although check steps can be marked as "boring".
• cleanup(): Performing shutdown operations where we either care to make sure that the shutdown happens for correctness reasons or where we are trying to clean-up after resources that won't be automatically destroyed. These steps currently will be performed if we abort the test to provide them a chance to perform necessary cleanup, but we might want to stop doing that.
• convenienceCleanup(): like convenienceSetup, a cleanup step that is automatically generated.
• convenienceDeferredCleanup(): a convenienceCleanup that is added to the list of steps after the defining function finishes running. Usually a function that calls convenienceSetup() will also call convenienceDeferredCleanup() to ensure its cleanup function is invoked at the right time.

Test steps are defined/declared as 'test definition' time. Let's look at the boilerplate for a test:

// (this is global code that is run when the JS file is evaluated; but we use
// AMD-style modules, so the body of our module is not actually evaluated
// until our module is require()d)
define(['rdcommon/testcontext', './resources/th_main',
        'exports'],
       function($tc, $th_main, exports) {
// This is module-level code that is run when the module is evaluated
var TD = exports.TD = $tc.defineTestsFor(
  { id: 'test_blah' }, null, [$th_main.TESTHELPER], ['app']);

TD.commonCase('I am a test', function(RT) {
  // This is test case definition code, it is run when we want to set up our
  // test.  Our calls to T.check/T.action/etc. produce test steps as a
  // byproduct of our being run.
  T.action('action 1', function() {
    // I run asynchronously.  I will not run until after the defining function
    // has fully completed running.
  });
  T.action('action 2', function() {
    // I do not run until 'action 1' has completed in its entirety, including
    // any asynchronous operations it was waiting for.
  });
  // actions can also be created in a loop, or by other functions.  The calls
  // to T.action() just need to complete before this function returns.
}); // (end 'I am a test')

}); // (end module 'define' call)

Code is run like so:

• The 'I am a test' function runs, defining one or more test steps. In this case, it defines 'action 1' and 'action 2'.
• 'action 1' is run. We do not advance to the next test step until all tracked asynchronous operations complete or we time out.
• 'action 2' runs after 'action 1' and all of its asynchronous operations have completed.
• the test is over!

Loggers, Expectations, and Actors

The idea at the heart of this test framework is that logging helps us understand complicated asynchronous processes that are operating in parallel.

The e-mail back-end is instrumented throughout with named loggers. Typically, each notable class in the system has its own structured logging definition. A logger instance is instantiated for each instance of the class and has the same lifetime as that class. Whenever something notable happens, we log a named log entry along with data that provides context.

The testing framework is built around specifying expectations about the log messages that our system (and our test infrastructure) will log while it is run.

"Actors" in the test framework are the term given to the test-framework's counterpart to a logger.

In a unit test, the following code creats an actor that we have dubbed 'A' for UI display purposes:

 var protoActor = T.actor('FooProtocol', 'A');

The actor will correspond to a logger that would be created something like this:

 var protoLogger = LOGFAB.FooProtocol(this, parentLogger, 'name');

The definition for the logger would look something like this:

var LOGFAB = exports.LOGFAB = $log.register($module, {
  FooProtocol: {
    type: $log.CONNECTION,
    subtype: $log.CLIENT,
    events: {
      connected: { hostname: true }
      disconnected: {}
    },
    TEST_ONLY_events: {
      connected: { username: false }
    },
}); // end LOGFAB

The key life-cycle issues related to actors and expectations are:

• How do we know when an actor should be created? This is especially important when multiple loggers of the same type are going to be created.
• How do we know