CloudServices/Roadmaps/Server

Introduction

Services has passed several milestones since its separation as a team from Labs. It has launched multiple products, successfully transitioned the flagship product (Sync) to a new codebase, worked with ops to build a robust operational infrastructure, and supported the rapid user growth that comes from being integrated into Firefox.

Up to this point (mid-2011), the focus has been on launching these products. It's been done well; things are generally modular and focused on code reuse, but the present allows us an opportunity to step back and focus on building a platform for further applications. By analyzing these as generic components, rather than code pieces working towards a goal, we can make them robust enough to support anticipated future work and flexible enough to accommodate reasonable unplanned feature needs. These components should be available to all of Mozilla (as well as the open-source community) to enable rapid development of services. The fact that this is a framework focused on REST-like *services* is what sets us apart from the other frameworks out there.

Core Components

Baseapp

It should be a goal of all services projects (with the possible, though unlikely, exception of AP) to make use of the baseapp with as little overriding as possible. This means that we need to have the ability to load arbitrary classes defined in the config file into the app namespace. What else? How can we make building webservices easier?

Configuration

The general structure of configuration is good, but there appear to be two approaches (generally matching .ini and .cfg suffixes) that have different semantics, and some of the differences are being compensated for outside of the config libraries. We need to figure out the source of differences, whether it is necessary, and encapsuate everything into the config libraries such that app.config = Config(filename) gives us a standard object to work with (one that largely shares the current hierarchy semantics, which are good).

Config should take an optional filename and should also have a way to pass in a dict and have it be populated.

Dynamic Loading

We currently have two approaches for dynamically generating interfaces - one for classes that implement a defined abstract interface, and one for more generic class loading. These were created as needed for projects and have undergone several revisions as requirements changed. Do they meet our needs, or are there further requirements? Are there refinements to be done to the methods or underlying code? Can (should?) we extend this to dynamically load functions?

Looking generally good as is.

Errors

It should be easy to find out whether a relevant exception has been defined in the core library. Which ones belong generically in the core?

User Management

This has been recently rewritten to support a user-information-caching object. Is this the right approach? Are there optimizations that can be made to reduce the number of queries?

Auth

Technically, this is part of user management. However, it's worth calling out separately, because it is the user management piece that all our products tend to use. Should we tear it out into its own space to make it easier to extend?

The usual profile of a user interaction with the Sync server is to request info/collections, followed sometimes by a series of futher requests. Each request does a pair of LDAP calls (one to get the DN, one to bind). Can we do some sort of short-term caching that will let us only talk to LDAP once for each of these 'sessions'? How do we handle password changes? Can we abstract this out to something more generic?

Auth needs to encapsulate a permissions model. This can be done through a token based permissions system that is generated once logged in. We ll use whatever the favored identification system to come out of browserID.

Test Support

We have the start of a test framework library, but it isn't yet to the point where it's easy to fire up a test and have access to the app. What utility functions would help in here?

Abuse Detection

Some parts of this - CEF - are working well. Other parts are just getting ready to launch. What else do we need here? This is an area where we've only begun to scratch the surface, and future apps that have less-encrypted data may need more automated detection.

How is Big Lebowski going? How can we make it easy to incorporate into apps?

Utility Functions

There are 30+ functions (and growing) in util.py. How can we make discoverability of what's in there easier? are thematic libraries the answer? It could definitely use some curating.

Reset Codes

This is newly separated from User Management so that it can be used anywhere else it is needed. Does it belong in core, or is it its own library? How might we use reset codes elsewhere?

Node Assignment

This is new service that has not yet been deployed. Once it has, zeus traffic to the User API call will need to be redirected to this service, and once that's done with, we can look into cleaning it out of server-core (and the implications this has for external server-builders).

Logging

What do we need to have a robust logging interface that doesn't get bogged down in complexity?

Other

What other core services do we need?

Auxilary Support

Packaging

This seems to be in pretty good shape. Do people have feature requests? I'd like to see if set up so that it only requires us to specify a version once, but that's pretty minor.

Loadtesting

QA and ops have taken the lead here recently and this is probably on hold until we see the new version they're working on.

Python Clients

There are bits and pieces of python sync clients littered throughout our repos. Should we try to unify them into a complete, easy-to-use client interface that implements the entire* API? Is this something we need on more than an ad hoc basis?

(* I think we can ignore some of the deprecated options!)

Operations Support

Operations requires periodic management scripts that often cross projects. Where should these live in our repos? Can we standardize such that loading configs into these scripts is easy (fixed location)?

Hudson and Jenkins

How do we make it so that Jenkins is an essential part of our development and deployment process? What is the process for getting a project into Jenkins, and at what point in the project life cycle should it be expected to be in there?

Deployment

Ability to deploy without operational intervention. Ability to rollback quickly. Easy installation.