CloudServices/Roadmaps/Server

Introduction

Services has passed several milestones since its separation as a team from Labs. It has launched multiple products, successfully transitioned the flagship product (Sync) to a new codebase, worked with ops to build a robust operational infrastructure, and supported the rapid user growth that comes from being integrated into Firefox.

Up to this point (mid-2011), the focus has been on launching these products. It's been done well; things are generally modular and focused on code reuse, but the present allows us an opportunity to step back and focus on building a platform for further applications. By analyzing these as generic components, rather than code pieces working towards a goal, we can make them robust enough to support anticipated future work and flexible enough to accommodate reasonable unplanned feature needs. These components should be available to all of Mozilla (as well as the open-source community) to enable rapid development of services. The fact that this is a framework focused on REST-like *services* is what sets us apart from the other frameworks out there.

Q3 will be focused on cleanup and getting these tools into shape, as detailed below.

Core Components

Baseapp

It should be a goal of all services projects (with the possible, though unlikely, exception of AP) to make use of the baseapp with as little overriding as possible. This means that we need to have the ability to load arbitrary classes defined in the config file into the app namespace. How can we make building webservices easier?

Bugs:

• Allow configuration to load modules (including auth) into env.modules dictionary
• Turn authentication off by default
• Script to generate project framework

Configuration

The general structure of configuration is good, but there appear to be two approaches (generally matching .ini and .cfg suffixes) that have different semantics, and some of the differences are being compensated for outside of the config libraries. We need to figure out the source of differences, whether it is necessary, and encapsuate everything into the config libraries such that app.config = Config(filename) gives us a standard object to work with (one that largely shares the current hierarchy semantics, which are good).

Bugs:

• Remove filter_params and convert_config from util.py
• Enforce [0-9A-Za-z_] as legal characters in section header names
• Get rid of ini and just use cfg everywhere. Drop support for file: extension syntax
• migrate GlobalConfFile wiki page to docs.services
• get rid of config post-processing (e.g. services.tests.support lines 142-145) and move it into config.py if needed

Dynamic Loading

We currently have two approaches for dynamically generating interfaces - one for classes that implement a defined abstract interface, and one for more generic class loading. Looking generally good as is.

Bugs:

• Add interface support to load_and_configure

Errors

It should be easy to find out whether a relevant exception has been defined in the core library.

Bugs:

• Merge errors and response codes into a single error.py file
• Add more information about the source of the error into BackendError

User Management

This has been recently rewritten to support a user-information-caching object to cut down on the number of queries into LDAP. Node assignment and reset codes have been spun off into their own library.

Bugs:

• Bug 659156 - change ldap to support getting new userids from there
• Rewrite reg to support the changes to user management (a-p and sreg done)

Auth

Technically, this is part of user management. However, it's worth calling out separately, because it is the user management piece that all our products tend to use.

Auth eventually needs to encapsulate a permissions model. This can be done through a token based permissions system that is generated once logged in. We'll use whatever the favored identification system to come out of browserID.

Bugs:

• Cache authentication credentials for a period of time

Test Support

We have the start of a test framework library, but it isn't yet to the point where it's easy to fire up a test and have access to the app. What utility functions would help in here?

Bugs:

• Clean up initenv and make it more generic - should just generate the app.
• Get rid of TestEnv class
• Have a central tests/support.py rather than one in each project

Abuse Detection

Some parts of this - CEF - are working well. Other parts are just getting ready to launch. What else do we need here? This is an area where we've only begun to scratch the surface, and future apps that have less-encrypted data may need more automated detection.

How is Big Lebowski going? How can we make it easy to incorporate into apps?

Utility Functions

There are 30+ functions (and growing) in util.py. Time to curate!

Bugs:

• Create services.responses
• Create services.crypto
• Create services.proxy
• Create services.mail
• move extract_username and extract_node into services.user
• delete generate_reset_code and check_reset_code

Reset Codes

This is newly separated from User Management so that it can be used anywhere else it is needed. Does it belong in core, or is it its own library? How might we use reset codes elsewhere?

Node Assignment

This is new service that has not yet been deployed. Once it has, zeus traffic to the User API call will need to be redirected to this service, and once that's done with, we can look into cleaning it out of server-core (and the implications this has for external server-builders).

Logging

What do we need to have a robust logging interface that doesn't get bogged down in complexity?

Other

What other core services do we need?

Auxilary Support

Packaging

This seems to be in pretty good shape. Do people have feature requests? I'd like to see if set up so that it only requires us to specify a version once, but that's pretty minor.

Loadtesting

QA and ops have taken the lead here recently and this is probably on hold until we see the new version they're working on.

Python Clients

There are bits and pieces of python sync clients littered throughout our repos. Should we try to unify them into a complete, easy-to-use client interface that implements the entire* API? Is this something we need on more than an ad hoc basis?

(* I think we can ignore some of the deprecated options!)

Operations Support

Operations requires periodic management scripts that often cross projects. Where should these live in our repos? Can we standardize such that loading configs into these scripts is easy (fixed location)?

Hudson and Jenkins

How do we make it so that Jenkins is an essential part of our development and deployment process? What is the process for getting a project into Jenkins, and at what point in the project life cycle should it be expected to be in there?

Deployment

Ability to deploy without operational intervention. Ability to rollback quickly. Easy installation.