|
|
(12 intermediate revisions by 6 users not shown) |
Line 1: |
Line 1: |
| = Kinto Integration in Firefox = | | = Kinto Integration in Firefox = |
|
| |
|
| [[http://www.kinto-storage.org Kinto]] is a simple JSON storage service that is used in Firefox for remote settings and the <code>storage.sync</code> API.
| | [http://www.kinto-storage.org Kinto] is a simple JSON storage service that was integrated into Firefox, and currently used for RemoteSettings. |
|
| |
|
| = Key features = | | = Key features = |
Line 10: |
Line 10: |
| * Peer review to publish data changes | | * Peer review to publish data changes |
| * Admin panel UI | | * Admin panel UI |
| | * Built-in Telemetry |
| * Firefox Accounts Integration | | * Firefox Accounts Integration |
| * ''Built-in telemetry (coming soon)''
| |
|
| |
|
| = Use Cases = | | = Use Cases = |
Line 20: |
Line 20: |
| ! Contact | | ! Contact |
| |- | | |- |
| |OneCRL | | |RemoteSettings |
| |Certificates revocation blocklist | | |[[Firefox/RemoteSettings]] allows to ship data and files to Firefox. It is used for blocklists, security state, Activity Stream, Fennec assets and experiments etc. |
| |Mark Goodwin — mgoodwin | | |Mathieu Leplatre — :leplatrem |
| |- | | |- |
| |HPKP
| |
| |HTTP Public Key Pinning
| |
| |Mark Goodwin — mgoodwin
| |
| |-
| |
| |Addons/Plugins/Gfx [[Blocklisting]]
| |
| |Black list for unsecure/instable/malicious addons, plugins or graphical drivers
| |
| |Mathieu Leplatre — leplatrem
| |
| |-
| |
| |<code>storage.sync</code>
| |
| |[[WebExtensions]]) API for user data storage
| |
| |Ethan Glasser Camp — glasserc
| |
| |-
| |
| |Fennec assets catalog
| |
| |Catalog of assets to be downloaded after installation
| |
| |Sebastian Kaspari — sebastian
| |
| |-
| |
| |Fennec experiments
| |
| |Data for A/B testing in Fennec
| |
| |Sebastian Kaspari — sebastian
| |
| |-
| |
| |iOS experiments
| |
| |Data for A/B testing in Firefox iOS
| |
| |Stephan Leroux — sleroux
| |
| |} | | |} |
|
| |
|
| Ideas/WIP
| | It used be supporting more use-cases in the past. See also [https://github.com/mozilla-services/servicedenuages.fr/blob/master/content/2017.12.kinto-at-mozilla.rst Kinto at Mozilla] |
| | |
| * Password manager recipes ''(contact: Matthew Noorenberghe — MattN)''
| |
| | |
| Feel free to come and discuss on <code>#storage</code> :)
| |
|
| |
|
| = Usage = | | = Usage = |
|
| |
|
| Two client libraries are embedded in Firefox: | | Two client libraries are vendored in Firefox: |
|
| |
|
| * [https://github.com/Kinto/kinto-http.js Kinto HTTP client]: for direct interactions with the Kinto HTTP API | | * [https://github.com/Kinto/kinto-http.js Kinto HTTP client]: for direct interactions with the Kinto HTTP API |
| * [https://github.com/Kinto/kinto.js Kinto offline client]: for offline persistence in internal SQLite | | * [https://github.com/Kinto/kinto.js Kinto offline client]: for offline persistence in internal SQLite or IndexedDB |
|
| |
|
| == HTTP client == | | == HTTP client == |
Line 68: |
Line 41: |
|
| |
|
| const remote = "https://kinto.dev.mozaws.net/v1"; | | const remote = "https://kinto.dev.mozaws.net/v1"; |
| const headers = {Authorization: "Basic " + btoa("user:pass")}; | | const headers = { Authorization: "Basic " + btoa("user:pass") }; |
|
| |
|
| const client = new KintoHttpClient(remote, {headers}); | | const client = new KintoHttpClient(remote, {headers}); |
| const records = yield client.bucket("a-bucket") | | const records = await client.bucket("a-bucket") |
| .collection("a-collection") | | .collection("a-collection") |
| .listRecords(); | | .listRecords(); |
Line 80: |
Line 53: |
| <pre> | | <pre> |
| const { Kinto } = Cu.import("resource://services-common/kinto-offline-client.js", {}); | | const { Kinto } = Cu.import("resource://services-common/kinto-offline-client.js", {}); |
| const { FirefoxAdapter } = Cu.import("resource://services-common/kinto-storage-adapter.js", {});
| |
|
| |
|
| const remote = "https://kinto.dev.mozaws.net/v1"; | | const remote = "https://kinto.dev.mozaws.net/v1"; |
| const headers = {Authorization: "Basic " + btoa("user:pass")}; | | const headers = { Authorization: "Basic " + btoa("user:pass") }; |
|
| |
|
| const kinto = new Kinto({adapter: FirefoxAdapter}); | | const kinto = new Kinto(); |
| | const bucket = "a-bucket"; |
| const collection = kinto.collection("a-collection"); | | const collection = kinto.collection("a-collection"); |
|
| |
|
| // Fetch/Publish changes. | | // Fetch/Publish changes. |
| const {ok} = yield collection.sync({ | | const { ok } = await collection.sync({ bucket, remote, headers }); |
| bucket: "a-bucket",
| |
| remote,
| |
| headers});
| |
|
| |
|
| // Read local collection of records. | | // Read local collection of records. |
| const records = yield collection.list(); | | const records = await collection.list(); |
| </pre> | | </pre> |
|
| |
|
| == Servers == | | = Resources = |
| | |
| {|class="wikitable"
| |
| ! Server
| |
| ! API
| |
| ! Admin
| |
| ! Contact
| |
| |-
| |
| |Public development instance (flushed every day) with many plugins
| |
| |https://kinto.dev.mozaws.net/v1/
| |
| |[https://kinto.dev.mozaws.net/v1/admin/ Admin UI]
| |
| |[https://mail.mozilla.org/listinfo/kinto Kinto mailing-list]
| |
| |-
| |
| |Remote settings STAGE
| |
| |https://kinto.stage.mozaws.net/v1/ (read-only)
| |
| |[https://kinto-writer.stage.mozaws.net/v1/admin/ Admin UI] '''VPN only'''
| |
| |OPS via Bugzilla
| |
| |-
| |
| |Remote settings PROD
| |
| |https://firefox.settings.services.mozilla.com/v1/ (read-only)
| |
| |[https://settings.prod.mozaws.net/v1/admin/ Admin UI] '''VPN only'''
| |
| |OPS via Bugzilla
| |
| |-
| |
| |storage.sync
| |
| |https://webextensions.settings.services.mozilla.com/v1/
| |
| |N/A
| |
| |OPS via Bugzilla
| |
| |}
| |
| | |
| == How to add a new official dataset ? ==
| |
| | |
| The only persons allowed to create buckets/collections in stage/production is the OPs team, so you have to create a Bugzilla ticket and specify everything you need to be created.
| |
| | |
| * The bucket name, and its permissions (who can read/write/create collections)
| |
| * A collection name, and its permissions (who can read/write/create records)
| |
| | |
| For use-cases where a few administrators manage some data downloaded by all Firefox clients, you have to enable peer reviewing. You will have 3 buckets: a ''staging'' where editors make changes, a ''preview'' bucket where changes are published when a review is requested, and the final bucket where changes are published when the review is approved. You must specify:
| |
| | |
| * Who can edit (members of the editors group)?
| |
| * Who can review/approve data (members of the reviewers group)?
| |
| * Who can read data (members of the observers group, eg. QA)? ''Optional''
| |
| * A new key for the digital signatures (Autograph) will have to be configured on the server side for your use-case.
| |
| | |
| For use-cases where a lot of users write data (like storage.sync), the [https://kinto.readthedocs.io/en/latest/api/1.x/quotas.html quota plugin] may have to be configured.
| |
| | |
| | |
| = Specifications =
| |
| | |
| The distribution package of Kinto that is deployed is [https://github.com/mozilla-services/kinto-dist/ kinto-dist].
| |
| There are several instances, depending on the use-case — mainly if clients have write access or not.
| |
| | |
| == Blocklists ==
| |
| | |
| The goal is to replace the current system — based on a single XML file downloaded everyday — by several Kinto collections on the '''remote settings instance'''.
| |
| | |
| | |
| {|class="wikitable"
| |
| ! List
| |
| ! Bucket
| |
| ! Collection
| |
| |-
| |
| |OneCRL
| |
| |<code>blocklists</code>
| |
| |<code>certificates</code>
| |
| |-
| |
| |Add-ons
| |
| |<code>blocklists</code>
| |
| |<code>addons</code>
| |
| |-
| |
| |Plugins
| |
| |<code>blocklists</code>
| |
| |<code>plugins</code>
| |
| |-
| |
| |Gfx
| |
| |<code>blocklists</code>
| |
| |<code>gfx</code>
| |
| |-
| |
| |HPKP
| |
| |<code>pinnings</code>
| |
| |<code>pings</code>
| |
| |}
| |
| | |
| See https://bugzilla.mozilla.org/show_bug.cgi?id=1197707
| |
| | |
| === Transition ===
| |
| | |
| Currently the blocklist system relies on a big XML file that is downloaded every day. It contains block entries for certificates to be revoked, addons and plugins to be disabled, and gfx environments that cause problems or crashes. Everything is managed via the Addons server.
| |
| | |
| Firefox (and derivatives like Thunderbird, Seamonkey, ...) downloads it on an URL that contains client information (eg. https://blocklist.addons.mozilla.org/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/44.0a1/)
| |
| | |
| Using the same XPCOM notification callback, the new mechanism will [https://dxr.mozilla.org/mozilla-central/rev/fc15477ce628599519cb0055f52cc195d640dc94/toolkit/mozapps/extensions/nsBlocklistService.js#632-642 synchronize the local copy of each collection from the remote server].
| |
| | |
| If changes are available, the local copy of the data will be updated and content signature verified.
| |
| | |
| * '''phase 1: DONE''': Both mechanisms run in parallel but only the legacy one will be used.
| |
| * '''phase 2: DONE''': The source of truth for blocklist is Kinto but produces the same XML as the legacy one.
| |
| * '''phase 3''': Eventually, [https://bugzilla.mozilla.org/show_bug.cgi?id=1257565 blocking mechanism will rely on the data managed via JSON], and the old XML client will be decommissioned.
| |
| | |
| | |
| == Fennec assets catalog ==
| |
| | |
| The goal is to remove the static assets (fonts, hyphenation dicts, etc.) from the distribution package and download them asynchronously using an online Kinto catalog, on the '''remote settings instance'''.
| |
| | |
| {|class="wikitable"
| |
| ! List
| |
| ! Bucket
| |
| ! Collection
| |
| |-
| |
| |Download catalog
| |
| |<code>fennec</code>
| |
| |<code>catalog</code>
| |
| |}
| |
| | |
| See https://bugzilla.mozilla.org/show_bug.cgi?id=1201059
| |
| | |
| | |
| == WebExtensions storage.sync ==
| |
| | |
| * Users are authenticated
| |
| * Every users have their own bucket (using the ''default bucket plugin'')
| |
| * The amount of data that users are allowed to store is limited
| |
| * Data is encrypted using Firefox Accounts keys
| |
| | |
|
| |
|
| | * [https://remote-settings.readthedocs.io Remote Settings docs] |
| | * [https://blog.mathieu-leplatre.info/the-history-of-firefox-remote-settings.html The History of Remote Settings] |
| | * [https://github.com/mozilla-services/servicedenuages.fr/blob/master/content/2017.12.kinto-at-mozilla.rst Kinto at Mozilla] (blog post) -- see also [https://web.archive.org/web/20180111085958/https://www.servicedenuages.fr/en/kinto-at-mozilla the published version, no longer available at its initial location] |
|
| |
|
| = Contribute = | | = Contribute = |
Line 260: |
Line 111: |
| <pre> | | <pre> |
| $ cp dist/moz-kinto-http-client.js ../mozilla-central/services/common/kinto-http-client.js | | $ cp dist/moz-kinto-http-client.js ../mozilla-central/services/common/kinto-http-client.js |
| </pre>
| |
|
| |
| === Run the tests ===
| |
|
| |
| First, follow [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions the instructions to build Firefox].
| |
|
| |
| In order to speed up the build and being able to run tests properly, create a **mozconfig** file at the root of the repository:
| |
|
| |
| <pre>
| |
| ac_add_options --enable-debug
| |
| ac_add_options --disable-optimize
| |
| ac_add_options --disable-crashreporter
| |
| ac_add_options --with-ccache=/usr/bin/ccache
| |
| </pre>
| |
|
| |
| For JavaScript updates only, have a look at [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Artifact_builds Artifacts Builds], trading bandwidth for compilation time.
| |
|
| |
| <pre>
| |
| $ ./mach build faster
| |
| $ ./mach xpcshell-test services/common/tests/unit/test_kinto.js
| |
| $ ./mach xpcshell-test services/common/tests/unit/test_storage_adapter.js
| |
| </pre>
| |
|
| |
| Or both at once:
| |
|
| |
| <pre>
| |
| $ ./mach xpcshell-test services/common/tests/unit/test_kinto.js services/common/tests/unit/test_storage_adapter.js
| |
| </pre>
| |
|
| |
| There are also tests relying on Kinto in services/common/tests/unit/test_blocklist_* .
| |
|
| |
| ==== Gfx blocklist tests ====
| |
|
| |
| The Gfx test suite requires the debug mode to be enabled. Add this to <code>mozconfig</code> file, rebuild and run the tests:
| |
|
| |
| <pre>
| |
| ac_add_options --enable-debug
| |
| </pre>
| |
|
| |
| ==== Debug content signature ====
| |
|
| |
| You can get tests (or Firefox) to give you more information on what the content signature verifier is doing by setting the NSPR_LOG_MODULES environment variable. For example:
| |
|
| |
| <pre>
| |
| EXPORT NSPR_LOG_MODULES=ContentSignatureVerifier:5,CSTrustDomain:5
| |
| </pre>
| |
|
| |
| ==== TDD mode ====
| |
|
| |
| Using ''inotify'', we will detect a file change in the <code>dist/</code> folder and run a series of commands to execute the tests automatically.
| |
|
| |
| First, install ''inotify-tools'':
| |
|
| |
| <pre>
| |
| sudo apt-get install inotify-tools
| |
| </pre>
| |
|
| |
| Then start an infinite loop with <code>inotify-wait</code>:
| |
|
| |
| <pre>
| |
| while true; do
| |
| # Wait for a change
| |
| inotifywait -q -e create,modify,delete -r ~/Code/Mozilla/kinto.js/dist
| |
| # Execute these commands
| |
| cp ~/Code/Mozilla/kinto.js/dist/moz-kinto-offline-client.js services/common/kinto-offline-client.js
| |
| ./mach xpcshell-test services/common/tests/unit/test_storage_adapter.js
| |
| ./mach xpcshell-test services/common/tests/unit/test_kinto.js
| |
| done
| |
| </pre>
| |
|
| |
| [http://makina-corpus.com/blog/metier/2013/utiliser-inotify-pour-tester-en-continu-son-code Source: Antoine Cezar]
| |
|
| |
| === Submit patch ===
| |
|
| |
| > Patch are contributed to kinto.js and kinto-http.js, which are first released on NPM.
| |
|
| |
| '''DO NOT''' land files that are not tagged officially on upstream repositories.
| |
|
| |
| ==== Become a contributor ====
| |
|
| |
| * Generate a SSH key
| |
| * Open a bugzilla ticket to request Level-1 access
| |
| * https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/
| |
| * Get a vouch (*optional for level1*)
| |
|
| |
| Configure SSH key for hg:
| |
|
| |
| <pre>
| |
| Host hg.mozilla.org
| |
| User user@server.com
| |
| IdentityFile ~/.ssh/contrib_moz
| |
| </pre>
| |
|
| |
| ==== Run integration tests: «Try» ====
| |
|
| |
| See https://wiki.mozilla.org/ReleaseEngineering/TryServer
| |
|
| |
| With Mercurial, push a patch to MozReview (see below) and trigger a Try build from the UI (*Automation > Trigger a Try build*).
| |
|
| |
| Or with git, use a ''gecko-dev'' fork from Github, and with install [https://github.com/mozilla/moz-git-tools moz git tools]
| |
|
| |
| <pre>
| |
| git push-to-try -t --rev master..HEAD ~/hg/mozilla-central/ -b do -p linux,linux64,macosx64,win32,win64 -u xpcshell,mochitests -t none
| |
| </pre>
| |
|
| |
| ==== Submit for review ====
| |
|
| |
| See http://mozilla-version-control-tools.readthedocs.org
| |
|
| |
| <pre>
| |
| # Keep a bookmark of your branch to address review.
| |
| # (equivalent of git branches)
| |
| hg bookmark bug/XXXXX
| |
|
| |
| # Commit with link to Bugzilla
| |
| hg commit -m "Bug XXXXX - Upgrade <lib> to X.Y.Z"
| |
|
| |
| # Submit to MozReview
| |
| hg push review
| |
|
| |
| # Go back to «master»
| |
| hg update -C central
| |
| </pre>
| |
|
| |
| Examples:
| |
|
| |
| * https://reviewboard.mozilla.org/r/45445/
| |
|
| |
|
| |
| To adjust previously submitted patch:
| |
|
| |
| <pre>
| |
| # Go to bookmark
| |
| hg update -C bug/XXXXX
| |
|
| |
| # Address comments
| |
| ...
| |
|
| |
| # Amend commit
| |
| hg commit --amend
| |
|
| |
| # Inspect history
| |
| hg log -f -G --rev bug/XXXXX | less
| |
|
| |
| # Update review
| |
| hg push review
| |
| </pre>
| |
|
| |
| When a patch has several commits:
| |
|
| |
| <pre>
| |
| # Inspect current branch
| |
| hg log -f -G --rev bug/XXXXX | less
| |
|
| |
| # Squash/Reword commits since rev c59308877f9a
| |
| hg histedit c59308877f9a
| |
| </pre>
| |
|
| |
| To rebase revision c59308877f9a and descendants on last «master»:
| |
|
| |
| <pre>
| |
| hg pull
| |
|
| |
| hg rebase -d central -s c59308877f9a
| |
| </pre> | | </pre> |