ReleaseEngineering/Applications/RelengAPI

Warning: This is still mostly vaporware!

As ReleaseEngineering/Applications attests, releng has a lot of apps, and many of them have APIs. But these APIs are all bespoke implementations, and are not tied to a single endpoint, authentication scheme, etc. RelengAPI aims to be the glue that binds all of these apps together, makes it easy to add new ones, and makes it easy for others to build tools that consume releng's APIs.

Goals

• Simple implementation process for new apps
  • Common requirements such as authentication, database access, scheduled tasks, configuration handling are already satisfied
  • All apps use the same technologies (language, web framework, DB framework, etc.), so the learning curve from one app to the next is small
  • Tailored for easy local development - minimal requirements, minimal installed components, etc.
• Simple usage for consumers
  • Industry-standard access mechanisms (REST, oAuth, etc.) that require no client-side custom libraries
  • One or very few endpoints (e.g., https://api.pub.build.mozilla.org)
  • Self-documenting tools
  • Semantic versioning
• Operations
  • Horizontally scalable using normal webops techniques
  • Easily deployed in multiple environments with normal devops processes
  • Resilient to failure: no in-memory state

Implementation

The releng API is composed of a Flask application and a number of blueprints installed as separate Python packages. The base application registers any blueprints installed in the python environment, and provides access to all of the common services. Development can take place with as few or as many blueprints installed as desired.

Configuration

Configuration comes from a typical Flask configuration file: a single Python file defining a number of variables.

Databases

RelengAPI handles accessing multiple databases simultaneously, identified by short names like 'scheduler'. Each blueprint, as well as the RelengAPI base, can specify and access tables in any database. Database access information is specified in the configuration. A command-line subcommand is provided to automatically create all attached tables.

Support for Alembic is planned, enabling automatic, graceful upgrades and downgrades of databases.

Authentication

TBD

Scheduled Tasks

TBD

Job Workers

TBD

Message Broker

TBD

Documentation

Documentation is generated automatically from the URLs defined by each blueprint, and by other, additional information provided in the blueprint (e.g., on data types or usage patterns).

Command Line

Support is included for adding subcommands to the 'relengapi' command for specific blueprints.

Proxies

In many cases, especially for transitioning new services into relengapi, we'll want to proxy from relengapi to other HTTP APIs.

TBD

Deployment

The plan is to deploy this on the releng web cluster, a group of VMs running Apache behind a load-balancer in scl3. This is simplest in terms of securing access to critical resources like databases and other hosts on the releng network. However, nothing in the design of the system precludes hosting on services like Elastic Beanstalk, Heroku, or PaaS.