Confirmed users
632
edits
Loop/Data Collection: Difference between revisions
Jump to navigation
Jump to search
→User Issue Reports
| Line 245: | Line 245: | ||
No existing Mozilla data ingestion systems provide a perfect fit for the data that these user reports will contain. Instead, we will create a very simple system that has the ability to grow more complex as needs evolve. | No existing Mozilla data ingestion systems provide a perfect fit for the data that these user reports will contain. Instead, we will create a very simple system that has the ability to grow more complex as needs evolve. | ||
To that end, we will be using Microsoft Azure blob storage for the report upload and storage. The Loop client will perform | To that end, we will be using Microsoft Azure blob storage for the report upload and storage. The Loop client will perform PUT requests directly against into a blob container. Azure includes an access control mechanism that allows servers to hand out time-limited signed URLs that can then be used to access the indicated resource. | ||
When a user indicates an issue, the Loop client selects a unique issue ID, and contacts the Loop server asking for a new URL to store the issue report in (including the date on which the report was generated): | When a user indicates an issue, the Loop client selects a unique issue ID, and contacts the Loop server asking for a new URL to store the issue report in (including the date on which the report was generated): | ||
| Line 262: | Line 262: | ||
* The host is the Azure instance assigned to Mozilla's account | * The host is the Azure instance assigned to Mozilla's account | ||
* The container is "loop-" followed by a four-digit year and two-digit month (e.g., if the report date sent by the client in its | * The container is "loop-" followed by a four-digit year and two-digit month (e.g., if the report date sent by the client in its POST request falls in August of 2014, UTC, then the container would be named "loop-201408"). | ||
* The filename is {issueID}.zip, using the issueID field provided by the client. | * The filename is {issueID}.zip, using the issueID field provided by the client. | ||
* The "signedversion" field (sv) is the Azure API version we're currently using | * The "signedversion" field (sv) is the Azure API version we're currently using | ||
| Line 298: | Line 298: | ||
This means that, upon startup, the Loop client code needs to check for outstanding (not-yet-upoaded) reports, and attempt to send them. If a report is over 30 days old and has not been successfully uploaded, clients will delete the report. The Loop server will similarly check that the issueDate field is no older than 30 days, and will reject the request for an upload URL. | This means that, upon startup, the Loop client code needs to check for outstanding (not-yet-upoaded) reports, and attempt to send them. If a report is over 30 days old and has not been successfully uploaded, clients will delete the report. The Loop server will similarly check that the issueDate field is no older than 30 days, and will reject the request for an upload URL. | ||
Once it aqcuires an issue upload URL, the loop client then performs a | Once it aqcuires an issue upload URL, the loop client then performs a PUT to the supplied URL to upload the report zipfile. The [http://msdn.microsoft.com/en-us/library/azure/dd135733.aspx Azure REST API documentation] contains more detailed information about this operation. | ||
=== Data Indexing === | === Data Indexing === | ||