View source for CloudServices/Roadmaps/Server

= Introduction =

Services has passed several milestones since its separation as a team from Labs. It has launched multiple products, successfully transitioned the flagship product (Sync) to a new codebase, worked with ops to build a robust operational infrastructure, and supported the rapid user growth that comes from being integrated into Firefox.

Up to this point (mid-2011), the focus has been on launching these products. It's been done well; things are generally modular and focused on code reuse, but the present allows us an opportunity to step back and focus on building a platform for further applications. By analyzing these as generic components, rather than code pieces working towards a goal, we can make them robust enough to support anticipated future work and flexible enough to accommodate reasonable unplanned feature needs. These components should be available to all of Mozilla (as well as the open-source community) to enable rapid development of services. The fact that this is a framework focused on REST-like *services* is what sets us apart from the other frameworks out there.

Q3 will be focused on cleanup and getting these tools into shape, as detailed below.

= Core Components =

=== Baseapp ===

It should be a goal of all services projects (with the possible, though unlikely, exception of AP) to make use of the baseapp with as little overriding as possible. This means that we need to have the ability to load arbitrary classes defined in the config file into the app namespace. How can we make building webservices easier?

Bugs:

* Allow configuration to load modules (including auth) into env.modules dictionary
* Turn authentication off by default
* Script to generate project framework

=== Configuration ===

The general structure of configuration is good, but there appear to be two approaches (generally matching .ini and .cfg suffixes) that have different semantics, and some of the differences are being compensated for outside of the config libraries. We need to figure out the source of differences, whether it is necessary, and encapsuate everything into the config libraries such that app.config = Config(filename) gives us a standard object to work with (one that largely shares the current hierarchy semantics, which are good).

Bugs:

* Remove filter params from util.py
* Enforce [0-9A-Za-z_] as legal characters in section header names
* Get rid of ini and just use cfg everywhere. Drop support for file: extension syntax
* migrate GlobalConfFile wiki page to docs.services
* get rid of config post-processing (e.g. services.tests.support lines 142-145) and move it into config.py if needed

=== Dynamic Loading ===

We currently have two approaches for dynamically generating interfaces - one for classes that implement a defined abstract interface, and one for more generic class loading. Looking generally good as is.

Bugs:

* Add interface support to load_and_configure

=== Errors ===

It should be easy to find out whether a relevant exception has been defined in the core library. Which ones belong generically in the core?

=== User Management ===

This has been recently rewritten to support a user-information-caching object. Is this the right approach? Are there optimizations that can be made to reduce the number of queries?

=== Auth ===

Technically, this is part of user management. However, it's worth calling out separately, because it is the user management piece that all our products tend to use. Should we tear it out into its own space to make it easier to extend?

The usual profile of a user interaction with the Sync server is to request info/collections, followed sometimes by a series of futher requests. Each request does a pair of LDAP calls (one to get the DN, one to bind). Can we do some sort of short-term caching that will let us only talk to LDAP once for each of these 'sessions'? How do we handle password changes? Can we abstract this out to something more generic?

Auth needs to encapsulate a permissions model. This can be done through a token based permissions system that is generated once logged in. We
ll use whatever the favored identification system to come out of browserID.

=== Test Support ===

We have the start of a test framework library, but it isn't yet to the point where it's easy to fire up a test and have access to the app. What utility functions would help in here?

=== Abuse Detection ===

Some parts of this - CEF - are working well. Other parts are just getting ready to launch. What else do we need here? This is an area where we've only begun to scratch the surface, and future apps that have less-encrypted data may need more automated detection.

How is Big Lebowski going? How can we make it easy to incorporate into apps?

=== Utility Functions ===

There are 30+ functions (and growing) in util.py. How can we make discoverability of what's in there easier? are thematic libraries the answer? It could definitely use some curating.

=== Reset Codes ===

This is newly separated from User Management so that it can be used anywhere else it is needed. Does it belong in core, or is it its own library? How might we use reset codes elsewhere?

=== Node Assignment ===

This is new service that has not yet been deployed. Once it has, zeus traffic to the User API call will need to be redirected to this service, and once that's done with, we can look into cleaning it out of server-core (and the implications this has for external server-builders). 

=== Logging ===

What do we need to have a robust logging interface that doesn't get bogged down in complexity?

=== Other ===

What other core services do we need?

= Auxilary Support =

=== Packaging ===

This seems to be in pretty good shape. Do people have feature requests? I'd like to see if set up so that it only requires us to specify a version once, but that's pretty minor.

=== Loadtesting ===

QA and ops have taken the lead here recently and this is probably on hold until we see the new version they're working on.

=== Python Clients ===

There are bits and pieces of python sync clients littered throughout our repos. Should we try to unify them into a complete, easy-to-use client interface that implements the entire* API? Is this something we need on more than an ad hoc basis?

(* I think we can ignore some of the deprecated options!)

=== Operations Support ===

Operations requires periodic management scripts that often cross projects. Where should these live in our repos? Can we standardize such that loading configs into these scripts is easy (fixed location)?

=== Hudson and Jenkins ===

How do we make it so that Jenkins is an essential part of our development and deployment process? What is the process for getting a project into Jenkins, and at what point in the project life cycle should it be expected to be in there?

=== Deployment ===

Ability to deploy without operational intervention. Ability to rollback quickly. Easy installation.

=== Bucketing ===

=== Metrics ===

=== Monitoring ===