67
edits
MDN/Development/CompatibilityTables/Infrastructure: Difference between revisions
Jump to navigation
Jump to search
MDN/Development/CompatibilityTables/Infrastructure (view source)
Revision as of 23:12, 5 February 2016
, 5 February 2016Add a sample request load
(Add production services) |
(Add a sample request load) |
||
| Line 85: | Line 85: | ||
== On The Way To Production == | == On The Way To Production == | ||
'' | Several changes will be made in the months leading up to production. | ||
The API will switch to UUIDs. This will support data import and export, and reduce or eliminate the need for production database dumps for development or staging. | |||
The '''C&M GUI''' will continue to be developed, and make the current '''Data Explorer''' obsolete. | |||
A caching load balancer will be added in front of Data API web workers, along with the code changes to support it. This will allow the API to efficiency server many similar requests. | |||
A CDN will be used to host assets, including possible the entire '''C&M GUI'''. | |||
'''MDN''' will add an OAuth2 provider interface, to allow BrowserCompat users to use their MDN account on BrowserCompat sites. This may become the only way to authenticate with BrowserCompat. | |||
Load testing will be developed, to test the infrastructure under expected production loads. | |||
MDN contribution data will be cleaned up and imported into the API. The number of MDN pages displaying API-backed data will increase, until all pages are backed by the API. The API backed tables will be turned on for all users, and the legacy wiki tables removed. | |||
== Production Services == | == Production Services == | ||
| Line 91: | Line 105: | ||
'''MDN''' is planning to move to an AWS-based infrastructure in Q1/Q2 2016. | '''MDN''' is planning to move to an AWS-based infrastructure in Q1/Q2 2016. | ||
BrowserCompat will be deployed as a set of small services hosted on subdomains of browsercompat.org: | BrowserCompat will be deployed as a set of small services hosted on subdomains of browsercompat.org: | ||
| Line 150: | Line 162: | ||
as well as backends appropriate for the load test, and as many load testing servers as needed. | as well as backends appropriate for the load test, and as many load testing servers as needed. | ||
== Sample Request Load == | |||
MDN had 13 million page views in December 2016. The BrowserCompat infrastructure needs to be able to handle a similar load. 1 million daily requests is a good launch target. Here's a walkthrough of a "cold cache" request to BrowserCompat. | |||
A user loads https://developer.mozilla.org/en-US/docs/Web/CSS/display. This contains specification and compatibility data that is sourced from the API, and was current at the time it was saved. In-page JavaScript requests up-to-date data from the '''Table Renderer''' service. | |||
The request to the '''Table Renderer''' includes a locale ('en-US'), the target feature (a UUID for "Web/CSS/Display"), a cache header (<code>ETag</code>), and the format type (for display on MDN). In the "warm cache" case, the load balancer knows the content is up to date, and returns a <code>304 Not Modified</code> response. If there is a cache miss, a ''Table Renderer'' web worker is called. | |||
The Table Render makes a similar call to the '''Data API''', asking for the table data for the UUID for Web/CSS/Display. Again, there is a chance the caching load balancer has this data, and can serve it without communicating with the backend '''Data API''' workers. | |||
The compatibility and specification data for "Web/CSS/Display" is spread over 100+ resources of 8 types (Browsers, Versions, Features, etc.). The '''Data API''' maintains a resource cache to avoid requesting this data from the database. If the instance cache is warm, then there are no database requests to load this data. For the cold cache case, each resource is loaded from the database and stored in the instance cache. | |||
The instances are combined into a single JSON response, which is hashed with an <code>ETag</code>. This response has all the known localizations for the data. This is cached by the load balancer and returned to the '''Table Renderer'''. | |||
The '''Table Renderer''' constructs HTML fragments of a table representing the compatibility data, localized as requested. This is cached by the load balancer and returned to the in-page JavaScript on MDN. The in-page JavaScript replaces the stale compatibility table with the new table. | |||
The '''Data Explorer''' also uses the '''Data API''', and the caching load balancer and instance cache also protect against repetitive data requests. | |||
A minority of users also contribute data to MDN. There are a few hundred MDN changes in a day, and the BrowserCompat interfaces should support a similar number. | |||
These users are directed to the '''C&M GUI''', which talks directly to the '''Data API''' to read, update, add, and delete resources. The '''C&M GUI''' is in in-page JavaScript application, and all resources can be stored on a CDN. Changing a resource invalidates the instance cache, so that in-page JS will eventually see the new data. A forced cache refresh is used to ensure the contributor sees their contribution on MDN. | |||
It will also be possible for users to access the '''Data API''' directly, for example to programmatically update many items. We don't expect much load from this usage in the first few years after launch. | |||