CloudServices/Sagrada/Roadmaps/2011Q3

Introduction

Services has passed several milestones since its separation as a team from Labs. It has launched multiple products, successfully transitioned the flagship product (Sync) to a new codebase, worked with ops to build a robust operational infrastructure, and supported the rapid user growth that comes from being integrated into Firefox.

Up to this point (mid-2011), the focus has been on launching these products. It's been done well; things are generally modular and focused on code reuse, but the present allows us an opportunity to step back and focus on building a platform for further applications. By analyzing these as generic components, rather than code pieces working towards a goal, we can make them robust enough to support anticipated future work and flexible enough to accommodate reasonable unplanned feature needs. These components should be available to all of Mozilla (as well as the open-source community) to enable rapid development of services. The fact that this is a framework focused on REST-like *services* is what sets us apart from the other frameworks out there.

Q3 will be focused on cleanup and getting these tools into shape, as detailed below.

Core Components

Baseapp

It should be a goal of all services projects (with the possible, though unlikely, exception of AP) to make use of the baseapp with as little overriding as possible. This means that we need to have the ability to load arbitrary classes defined in the config file into the app namespace. How can we make building webservices easier?

Bugs:

• Bug 672664 - Allow configuration to load modules (including auth) into env.modules dictionary
• Turn authentication off by default
• Script to generate project framework
• Bug 625183 - Configuration to bail on bad XFF header
• Bug 670630 - Support for HEAD __heartbeat__

Configuration

The general structure of configuration is good, but there appear to be two approaches (generally matching .ini and .cfg suffixes) that have different semantics, and some of the differences are being compensated for outside of the config libraries. We need to figure out the source of differences, whether it is necessary, and encapsuate everything into the config libraries such that app.config = Config(filename) gives us a standard object to work with (one that largely shares the current hierarchy semantics, which are good).

Bugs:

• Enforce [0-9A-Za-z_] as legal characters in section header names
• Get rid of ini and just use cfg everywhere. Drop support for file: extension syntax
• migrate GlobalConfFile wiki page to docs.services
• get rid of config post-processing (e.g. services.tests.support lines 142-145) and move it into config.py if needed

Dynamic Loading

We currently have two approaches for dynamically generating interfaces - one for classes that implement a defined abstract interface, and one for more generic class loading. Looking generally good as is.

Bugs:

• Add interface support to load_and_configure

Errors

It should be easy to find out whether a relevant exception has been defined in the core library.

Bugs:

• Merge errors and response codes into a single exceptions file
• Bug 630299 - Add more information about the source of the error into BackendError
• Add better error responses throughout all the codebases

User Management

This has been recently rewritten to support a user-information-caching object to cut down on the number of queries into LDAP. Node assignment and reset codes have been spun off into their own library.

Bugs:

• Bug 659156 - change ldap to support getting new userids from there
• Rewrite reg to support the changes to user management (a-p and sreg done)

Auth

Technically, this is part of user management. However, it's worth calling out separately, because it is the user management piece that all our products tend to use.

Auth eventually needs to encapsulate a permissions model. This can be done through a token based permissions system that is generated once logged in. We'll use whatever the favored identification system to come out of browserID.

Bugs:

• Bug 666237 - Cache authentication credentials for a period of time

Test Support

We have the start of a test framework library, but it isn't yet to the point where it's easy to fire up a test and have access to the app. What utility functions would help in here?

Bugs:

• Clean up initenv and make it more generic - should just generate the app.
• Get rid of TestEnv class
• Have a central tests/support.py rather than one in each project

Utility Functions

There are 30+ functions (and growing) in util.py. Time to curate!

Bugs:

• Create services.responses
• Create services.crypto
• Create services.proxy
• Create services.mail
• Move extract_username and extract_node into services.user
• Delete generate_reset_code and check_reset_code
• Delete filter_params and convert_config

Reset Codes

This is newly separated from User Management so that it can be used anywhere else it is needed.

Bugs:

• Pull reset codes out into their own package

Node Assignment

This is new service that has not yet been deployed. Once it has, zeus traffic to the User API call will be redirected to this service

Bugs:

• Bug 659157 - package and deploy new node-assignment
• Clean node assignment out of server-core

Logging

What do we need to have a robust logging interface that doesn't get bogged down in complexity?

Bugs:

• Add support for metrics logging

Auxiliary Support

Packaging

We need to make it as clean and easy to deploy packages as possible. The new makefiles have been a big help there. Much of the work can be done by the framework setup script, and one goal should be to minimize the number of places that need changing on an update.

Bugs:

• Bug 672655 - Set everything up so that a version bump only needs to be done in one location
• Set everything up so that dependencies only need to be in one location

Loadtesting

QA and ops have taken the lead here recently and this is probably on hold until we see the new version they're working on.

Abuse Management

The Big Lebowski's scope has been drastically reduced. It is now spec'd as a RESTful API to manage IP white and black lists with TTLs. The API will accept commands from a single client: scripts run via InfraSec's monitoring appliance. Their appliance will manage all threshold violation detection and issue commands to the Big Lebowski which will manage the list of banned IPs on the Zeus instance(s).

Python Clients

There are bits and pieces of python sync clients littered throughout our repos. We should try to unify them into a complete, easy-to-use client interface that implements the entire* API.

(* I think we can ignore some of the deprecated options!)

Bugs:

• get crypto support for sync encryption/decryption into services.crypto
• move delete and quota functions from account-portal to python-clients
• work with hanson and stefan to see what's built and fill in the gaps.

Jenkins

Jenkins is the automated unit-testing tool.

Bugs:

• Add documentation to docs.services release process that requires Jenkins to be green before a launch
• Fix up false-positives in Jenkins

Metrics

Bugs:

• Define a structure for metrics notifications
• Write a parser to consume them over a network stream (generated by the logger component)

Mercurial

Bugs:

• Add a services/deprecated folder
• Clean up the services hg repo directory.