ReleaseEngineering/Applications/RelengAPI: Difference between revisions
No edit summary |
|||
| Line 1: | Line 1: | ||
{{warning|This is still | {{warning|This is still under development!}} | ||
As [[ReleaseEngineering/Applications]] attests, releng has a lot of apps, and many of them have APIs. | As [[ReleaseEngineering/Applications]] attests, releng has a lot of apps, and many of them have APIs. | ||
But these APIs are all bespoke implementations, and are not tied to a single endpoint, authentication scheme, etc. | But these APIs are all bespoke implementations, and are not tied to a single endpoint, authentication scheme, etc. | ||
| Line 6: | Line 6: | ||
= Goals = | = Goals = | ||
* Simple implementation | * Simple self-service usage for consumers | ||
** Industry-standard access mechanisms (REST, oAuth2, etc.) that require no client-side custom libraries | |||
** One or very few endpoints (e.g., https://api.pub.build.mozilla.org) | |||
** Self-documenting tools | |||
** Semantic versioning | |||
* Simple, rapid implementation of new apps | |||
** Common requirements such as authentication, database access, scheduled tasks, configuration handling are already satisfied | ** Common requirements such as authentication, database access, scheduled tasks, configuration handling are already satisfied | ||
** All apps use the same technologies (language, web framework, DB framework, etc.), so the learning curve from one app to the next is small | ** All apps use the same technologies (language, web framework, DB framework, etc.), so the learning curve from one app to the next is small | ||
** Tailored for easy local development - minimal requirements, minimal installed components, etc. | ** Tailored for easy local development - minimal requirements, minimal installed components, etc. | ||
* | * Operations-friendly | ||
** Horizontally scalable using normal webops techniques | ** Horizontally scalable using normal webops techniques | ||
** Easily deployed in multiple environments with normal devops processes | ** Easily deployed in multiple environments with normal devops processes | ||
| Line 21: | Line 21: | ||
= Implementation = | = Implementation = | ||
* Code: https://github.com/djmitche/relengapi | |||
* Implementation: https://api-pub-build.allizom.org (staging) | |||
The releng API is composed of a [http://flask.pocoo.org/ Flask] application and a number of [http://flask.pocoo.org/docs/blueprints/ blueprints] installed as separate Python packages. | The releng API is composed of a [http://flask.pocoo.org/ Flask] application and a number of [http://flask.pocoo.org/docs/blueprints/ blueprints] installed as separate Python packages. | ||
| Line 41: | Line 44: | ||
== Authentication == | == Authentication == | ||
RelengAPI currently supports BrowserID for user authentication, and oAuth2 for authentication of client applications. | |||
== Job Workers == | == Job Workers == | ||
RelengAPI integrates with Celery to support workers. | |||
== Documentation == | == Documentation == | ||
General documentation is in Sphinx format and is automatically served by the API itself. | |||
API endpoints, database tables, and so on will be automatically documented as well. | |||
== Command Line == | == Command Line == | ||
| Line 67: | Line 63: | ||
In many cases, especially for transitioning new services into relengapi, we'll want to proxy from relengapi to other HTTP APIs. | In many cases, especially for transitioning new services into relengapi, we'll want to proxy from relengapi to other HTTP APIs. | ||
RelengAPI provides a simple mechanism to configure an API endpoint to proxy to another HTTP endpoint. | |||
= Deployment = | = Deployment = | ||
Revision as of 20:00, 7 March 2014
Warning: This is still under development!As ReleaseEngineering/Applications attests, releng has a lot of apps, and many of them have APIs. But these APIs are all bespoke implementations, and are not tied to a single endpoint, authentication scheme, etc. RelengAPI aims to be the glue that binds all of these apps together, makes it easy to add new ones, and makes it easy for others to build tools that consume releng's APIs.
Goals
- Simple self-service usage for consumers
- Industry-standard access mechanisms (REST, oAuth2, etc.) that require no client-side custom libraries
- One or very few endpoints (e.g., https://api.pub.build.mozilla.org)
- Self-documenting tools
- Semantic versioning
- Simple, rapid implementation of new apps
- Common requirements such as authentication, database access, scheduled tasks, configuration handling are already satisfied
- All apps use the same technologies (language, web framework, DB framework, etc.), so the learning curve from one app to the next is small
- Tailored for easy local development - minimal requirements, minimal installed components, etc.
- Operations-friendly
- Horizontally scalable using normal webops techniques
- Easily deployed in multiple environments with normal devops processes
- Resilient to failure: no in-memory state
Implementation
- Code: https://github.com/djmitche/relengapi
- Implementation: https://api-pub-build.allizom.org (staging)
The releng API is composed of a Flask application and a number of blueprints installed as separate Python packages. The base application registers any blueprints installed in the python environment, and provides access to all of the common services. Development can take place with as few or as many blueprints installed as desired.
Configuration
Configuration comes from a typical Flask configuration file: a single Python file defining a number of variables.
Databases
RelengAPI handles accessing multiple databases simultaneously, identified by short names like 'scheduler'. Each blueprint, as well as the RelengAPI base, can specify and access tables in any database. Database access information is specified in the configuration. A command-line subcommand is provided to automatically create all attached tables.
Support for Alembic is planned, enabling automatic, graceful upgrades and downgrades of databases.
Authentication
RelengAPI currently supports BrowserID for user authentication, and oAuth2 for authentication of client applications.
Job Workers
RelengAPI integrates with Celery to support workers.
Documentation
General documentation is in Sphinx format and is automatically served by the API itself. API endpoints, database tables, and so on will be automatically documented as well.
Command Line
Support is included for adding subcommands to the 'relengapi' command for specific blueprints.
Proxies
In many cases, especially for transitioning new services into relengapi, we'll want to proxy from relengapi to other HTTP APIs.
RelengAPI provides a simple mechanism to configure an API endpoint to proxy to another HTTP endpoint.
Deployment
The plan is to deploy this on the releng web cluster, a group of VMs running Apache behind a load-balancer in scl3. This is simplest in terms of securing access to critical resources like databases and other hosts on the releng network. However, nothing in the design of the system precludes hosting on services like Elastic Beanstalk, Heroku, or PaaS.