Firefox Home Server Architecture

Huh What Is This!?

In short: we are turning Firefox Home into a web application.

Home will become a Web App. In case of iOS it will be likely be shipped inside a small native wrapper for the iPhone and iPad but its core will be a portable Mobile Web App.

So this means that we have to build something server side. To put it simple, we are basically building a web site that acts as a sync client. When the web app has your data, we can present it in all kinds of interesting ways.

For recent devices (Android, iOS, PlayBook) we will expose a modern HTML5 web site that is mostly build on top of JSON web services that we expose. For older devices, or less capable devices, we plan to serve simple static HTML4.

Simple plan for Firefox Home 2.0

Home becomes Web App with native iPhone and iPad wrapper

Focus on iOS: iPhone and iPad
Integration with Mozilla’s new identity service for secure and easy single-signon with your iPhone or iPad
Same functionality as (native app) 1.0: access history, bookmarks?, tabs, awesomebar-style searching
New visual design
Optional: Personas

This is work in progress. We want to keep the first release simple.

Players

(Missing: memcache where appropriate)

Zeus

Zeus is the SSL frontend and the load balancer. Ideally Zeus will be able to look at the web service URLs from Firefox Home and based on some value in the request, send the request to the right Web Server.

(Home can of course also ask services to what server to go, just like is done for sync clients now.)

Web Server

The Web Server runs the Firefox Home web application. The web application will be written in Python. Currently it is a Pyramid/SQLAlchemy type application. Ideally it is hosted on the same kind of platform that Services uses.

The Web Server also periodically pushes sync requests to the Sync Queue. For example every 15 minutes.

Ideally web servers do not do this kind of work but I think that a simple cron job to queue messages will not cause a major load on these systems. If it does, or when there is a need for more periodic tasks, then we can move those to an extra machine.

Each Web Server has a database dedicated to it. The Web Server talks directly to this database. It will need to do real queries. It will of course try to cache as much as it can in for example memcache.

This application differs from the sync services that it will actually need to run real queries and searches on people's sync data. This will be a very different load/usage pattern.

Database

Description

Initially the databases will mainly contain the following:

Bookmarks
History Items
Tabs

We will normalize or extend the data that we take from sync as much as needed so that it is possible to run queries on the data.

All data is stored unencrypted in the database.

Software Requirements

There is currently no preference for any specific type of database. MySQL, Drizzle, PostgreSQL or anything that allows queries and joins will be fine.

Uptime

Data on this server is a *copy* of what already exists in the Sync Service. I would therefore say that there is a much lower requirement to keep this running. In case of problems we can always sync the data again. This will be a relatively expensive operation but it should be pretty fast.

(This changes as soon as we are also going to generate data based on sync data, usage or when we receive new data from the app)

Future

Sync Queue

The Sync Queue exists to coordinate sync operations. The Web Servers periodically push requests to the queue, which the Sync Workers will pick up.

I understand that we do not currently use any specific queue system. Anything is really fine, as long as it is relatively simple to talk to it from Python.

I do not think queue persistence is needed for this application. If the queue gets lost then we simply schedule a request to sync within the next 15 minutes or whatever the sync interval is.

Sync Worker

A Sync Worker is a machine that will sync a user's account.

To make this part of the infrastructure more reliable and distributed, the workers listen to a work queue. Each worker can sync N accounts in parallel.

The worker takes a job from the queue and then talks to the database where that user's data lives and to the Public Sync Services and sync data.

The frequency and scheduling of sync operations is not in control of the Sync Workers. They simply handle sync requests.

This is an important part of the infrastructure but there is no need to get extremely high uptime on this part. In case of failures we would simply not sync, which is not something the end user would notice in form of data loss. The data would simply not be up to date.

Public Sync Services

Simply the sync services as they exist now. Ideally no changes are needed and Firefox Home is simply another client that accesses sync using the defines APIs that all other clients use.

There is no special need for shortcuts to the Sync Services on an infrastructure level. As long as the Firefox Home Sync Workers can talk HTTPS to the Sync Services, all is good.