NSS:StateOfContinuousIntegration

This page documents the state and needs of NSS continous integration (CI) and other automated testing.

Introduction

As of today, the primary contributors to the NSS CI are Mozilla (MOZ) and Red Hat (RH).

Both parties have different requirements, on various dimensions, which are documented on this page. During the past years, the CI configuration that we've used (contributed by both parties, but mostly administrated by Red Hat) was our attempt to accomodate everyone's needs.

As multiple different computer architectures were required, we used computers that were contributed by both parties, and used a buildbot configuration that grouped the output of all architectures into a common view. (See: https://test.nss-crypto.org )

Because neither of Mozilla nor Red Hat has access to the complete set of required hardware architectures and operating system, nor or they interested to fully administrate the systems that they don't require for their own needs, collaboration has been necessary.

For this reason, it seems difficult for either party to completely assume responsibility for the set union of required environments, because of the risk that some configurations might get neglected.

Mozilla's requirements

Test NSS on the platforms that Mozilla supports for the oldest still supported version of:

• Firefox Desktop (currently Firefox 45 ESR)
  • Mac OS X 10.6 +
  • Windows XP +
  • Linux x86 and x86_64, using a rather modern C/C++ compiler
• Firefox Android:
  • Android

Other preferences are:

• focus on mainstream hardware
• prefers performance over FIPS, therefore disables certain startup tests
• prefers performance over full PKCS#11 support, therefore uses the PKCS#11-bypass code paths in the NSS SSL/TLS code

Red Hat's requirements

Test NSS on various modern and older, still supported versions of the RHEL distribution, as of today these are:

• RHEL 5 with gcc 4.1
• RHEL 6 with gcc 4.4
• RHEL 7 with gcc 4.8
• currently supported Fedora versions

Test multiple hardware architectures:

• x86_64 and x86
• s390x and s390
• ppc64 and ppc

Other preferences are:

• support server hardware architectures that customers deploy
• prefer full FIPS support, therefore enable all required startup tests
• prefer full integration of PKCS#11, therefore the PKCS#11-bypass code paths aren't used

Kinds of NSS tests and their constraints

The NSS project offers several tests. Some are routinely executed for every change made to NSS, executed on every system configuration that we have configured as part our shared NSS buildbot setup. However, some tests are unsuitable for being executed for every single change made by developersm, because of the amount of time and space required to execute them.

Tests that are executed for most commits:

• the regular NSS test suite
• the new ssl_gtests test suite, on all systems that provide a sufficiently modern compiler (C++11)

Executing the PKITS subset of tests requires special test data to be available in the environment (3 MB, 1600 files).

The above says "for most commits", not "for each commits". The reason is that we are constrained by the amount of computers that we have available for executing tests. Instead of executing a full set of tests for each commit, only after an available machine has completed a previous task, it will test the most recently used snapshot of NSS, which might test several recent changes at once.

Some of those configurations might only get tested at most a few times per day, because they might share a single computer/VM environment, where one machine cycles through multiple configurations one after the other, or the machines executing them are rather slow, or even both.

In addition to the above constraints for testing certain configurations, we also need to execute other tests, which take a very long time. Those include:

• memory leak tests on Linux, which require between 4 and 10 hours, depending on the system
• a special fuzzing testsuite, which uses NISCC test data, which is slow and has additional requirements

Here are the additional requirements of the above NISCC tests, which require them to be executed on a specially prepared system:

• they take a very long time, so they currently being executed just once a week
• they involve building two different variations of NSS, where one is the regular to be tested, and the other build deliberately allows sending/production of invalid protocol data. This is being used to test that NSS correctly rejects invalid inputs.
• the input data for the NSS test suite is very large, and cannot be processed on the fly. It requires a system that has been prepared to have a copy of the input test data already (2 million files, 19 gigabytes of space, which can be reduced to 550 MB, if used as a compressed sqsh filesystem that can be mounted readonly).

Test Feedback requirements for developers

When adding a change to NSS, developers wish to see if their change has worked, or has caused any test failures (regressions).

Ideally, developers would have to visit only one information web page, which displays the status of all tests.

The buildbot waterfall page at https://test.nss-crypto.org has been the attempt to implement that, by combining the test results of machines from both Mozilla and Red Hat.

This has been recently made more difficult, as additional initiatives have been started to perform additional tests, on different locations, using different CI tools, using only subsets of platforms, etc.

We should strive to make the combined status easily accessible. For the time being, it might mean to have a collection of links to the various systems on the NSS:BuildBotStatus page.

Branches

As of today, we don't have automated support for watching and building additional branches, all testing is currently limited to the "default" branch (trunk).

However, buildbot offers a web interface, where a developer can request testing of specific code revisions. This may be used when testing release candidates on release branches.

This is further inconvinent, because the buildbot web interface doesn't show those branch builds clearly separate, the developer must pay attention to specifically identify and watch the builds that were started manually.

Experimental ("try") commits

As of today, we don't have any support to perform testing of experimental commits on all available build configurations.

It would require the use of a separate code branch for experimental commits (which also means, they are being recorded), and the manual use of the buildbot web interface to start a branch build.

Release trains for NSS

It has been suggested to adopt a regular release schedule for NSS, closely aligned with the release schedule used by the Mozilla Firefox project.

The NSS developers have expressed support for this idea, however, not without mechanisms to maintain and test these branches in an easy way.

In short, the idea is to start a new release branch of NSS every 6 weeks, at the same time as Firefox development is branching. The motivation is to simplify the management of security issues that need to be backported to multiple Firefox versions, which require separate NSS releases for each of them, in order to help Linux distributions that ship NSS separately from Firefox. (Those distributions need cleanly labeled versions of NSS, not arbitrary snapshots.) The amount of supported branches of NSS would be equivalent to the number of supported versions of Firefox, which are the latest released version, the beta and alpha/aurora branches, the extended support branch (for Firefox ESR), in addition to the default development branch (trunk).

In order to fully support such a branched release approach for NSS, we ideally should have separate CI setups for each supported branch of NSS. This seems very difficult to achieve with the limited resources we have at this time.

This is difficult, because the current administrator of the NSS buildbot, due to limited experience with buildbot and limited time to learn more, hasn't been able to implement a buildbot configuration, that supports multiple code repositories (NSPR plus NSS) and multiple branches at the same time, with a single buildbot. Because we have an limited amount of build/test slave machines (for several architectures/environments we have only one), and because a single slave machine cannot be shared by more than one buildbot master machine, having just one buildbot master for all branches is a strict requirement at this time.

(There is an idea for a transition period, which would be cumbersome, but doable, if all involved developers agreed to comply. If we wanted to start early with the branched NSS release schedule, which requires commits to release branches that aren't automatically tested by our current CI, we could require that every developer, who backports a fix to a release branch, is responsible to manually use the buildbot web interface to trigger tests for that specific branch revision, and watch for the results.)

Resources and work required going forward

In order to implement the Release trains mentioned before, we should have:

• get help to enhance the NSS buildbot master configuration, in order to handle multiple branches of NSS+NSPR
• quadruple (?) the amount of build slave machines for most configurations, to allow for faster execution of tests
• figure out how to collaborate to ensure the needs of all actively contributing parties are addressed