Security/Safe Browsing/V4 Implementation: Difference between revisions

← Older edit

Security/Safe Browsing/V4 Implementation (view source)

Revision as of 19:48, 17 August 2017

3,986 bytes added , 17 August 2017

Split Milestone 9 into two (i.e. move the Fennec work to 58)

Fmarier

Confirmed users

908

edits

@@ Line 1: / Line 1: @@
+* [[Security/Safe_Browsing/V4_Implementation/Dashboard|Dashboard]]
+* [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1167038&hide_resolved=1 Bug Dependency Tree]
 = Introduction =
 Safe Browsing v4 is designed to reduce network bandwidth and disk storage (mainly for mobile devices). The usage of partial URL hashes (aka prefixes) and complete hashes remains the same but the way we get those prefixes and complete hashes is changed. The update and hash completion API will be based on protobuf and content compression (in additional to HTTP compression) is introduced. For further information, see https://developers.google.com/safe-browsing/v4 (The public specification is not complete. For example, the protobuf is not mentioned.)
+= Timeline =
+Given that ESR 52.8 will be the last supported version of Firefox using the V2 servers, we have the following '''deadlines''':
+* Firefox 59 (Desktop): merge on 2017-11-13, release on 2018-03-06
+* Firefox 61 (Android): merge on 2018-03-05, release on 2018-06-26
+We will therefore '''target''' the following releases:
+* Firefox 58 (Desktop): merge on 2017-09-20, release on 2018-01-16
+* Firefox 60 (Android): merge on 2018-01-15, release on 2018-05-01
 = Design and Implementation =
@@ Line 6: / Line 22: @@
 When '''ListManager''' and '''HashCompleter''' (and any other related components like ProtocolParser) sees table names suffixed by "-proto", they would behave differently. For example, in '''listmanager.js''', while making update request for table '''goog-phish-proto''', ''nsIUrlClassifier.makeUpdateRequestV4'' will be called to build a v4 specific request. (See {{Bug|1264885}} and {{Bug|1275507}} for more information.)
+== Using custom API key ==
+We are required to have a API key to access Google services like geolocation and Safe Browsing. The nightly/beta/aurora/release Firefox builds will have a proper API key associated with Mozilla. However, if you build your own Firefox, the API key will be substituted with the string <tt>"no-google-api-key"</tt>, which is not a valid key.
+Here's how to get your own key:
+# Obtain a Safe Browsing Google API key from https://console.developers.google.com and put it in <tt>~/.gapi.data</tt>
+# Add <tt>ac_add_options --with-google-api-keyfile=/home/foobar/.gapi.data</tt> to your <tt>.mozconfig</tt>
+# Rebuild and check if <tt>config.status</tt> has something like <tt>'MOZ_GOOGLE_API_KEY': b'AIzaSooxxxxoxoxoxoxooxx'</tt>
+== Try Google Safe Browsing V4! ==
+To use Google Safe Browsing V4, set the following preferences:
+# <tt>urlclassifier.downloadAllowTable</tt> to <tt>goog-downloadwhite-proto</tt>
+# <tt>urlclassifier.downloadBlockTable</tt> to <tt>goog-badbinurl-proto</tt>
+# <tt>urlclassifier.phishTable</tt> to <tt>goog-phish-proto,test-phish-simple</tt>
+# <tt>urlclassifier.malwareTable</tt> to <tt>goog-malware-proto,goog-unwanted-proto,test-malware-simple,test-unwanted-simple</tt>
+To use both V4 and V2 at the same time:
+# <tt>urlclassifier.downloadAllowTable</tt> to <tt>goog-downloadwhite-proto,goog-downloadwhite-digest256</tt>
+# <tt>urlclassifier.downloadBlockTable</tt> to <tt>goog-badbinurl-proto,goog-badbinurl-shavar</tt>
+# <tt>urlclassifier.phishTable</tt> to <tt>goog-phish-proto,goog-phish-shavar,test-phish-simple</tt>
+# <tt>urlclassifier.malwareTable</tt> to <tt>goog-malware-proto,goog-unwanted-proto,goog-malware-shavar,goog-unwanted-shavar,test-malware-simple,test-unwanted-simple</tt>
 = Milestones =
-All of the V4 bugs can be seen in this [https://bugzilla.mozilla.org/showdependencytree.cgi?id=1167038&hide_resolved=1 dependency tree].<br>
-And they are tracked by this [https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard dashboard]
 == M0 (2016/7/31) ==
@@ Line 38: / Line 76: @@
 [https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M1 Dashboard for M1 bugs]
+=== Workweek in Vancouver ===
+# Etherpad: https://public.etherpad-mozilla.org/p/sbv4yvr
+== M2 (TBC, maybe 10/15) ==
+The intention of M2 is to push the date where we "enable v4 list update on nightly by default" to two weeks later because we have identified some issues and lacks in M0 and M1 in Vancouver SBv4 work week. The target date is the end of second week in October.
+=== Deliverables ===
+# {{Bug|1305486}} to enable v4 table update on nightly by default (The most important!)
+# {{Bug|1305567}} so that we won't get 400 error while doing list update. (landed)
+# {{Bug|1305581}} to examine checksum against the updated lists.
+# {{Bug|1285848}} to support compressed list update. (under review)
+# {{Bug|1305801}} to store v4 list to disk. (landed)
+# {{Bug|1305484}} to store v4 list states to disk rather than preference.
+=== Bugs ===
+[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M2 Dashboard for M2 bugs]
-== M2 (Right before Hawaii Workweek) ==
+== M3 (Right before Hawaii Workweek) ==
 === Deliverables ===
 # Check v4 prefixes (in addition to v4) but ignore the result
@@ Line 46: / Line 100: @@
 === Bugs ===
-[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M2 Dashboard for M2 bugs]
+[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M3 Dashboard for M3 bugs]
+== M4 (2017/1/23) ==
+=== Deliverables ===
+# Enable update by default
+# Fix crashes and regressions
-== M3 (Deadline TBD) ==
+=== Bugs ===
+[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M4 Dashboard for M4 bugs]
+== M5 (2017/3/6) ==
 === Deliverables ===
-# Do and use v4 complete hashes (in addition to v2) but ignore the result
+# Enable v4 completion on nightly but ignore the result
+# Add V4 table for Download protection.
 # v2/v4 URL hash matching consistency telemetry
 === Bugs ===
-[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M3 Dashboard for M3 bugs]
+[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M5 Dashboard for M5 bugs]
+== M6 (2017/4/18) ==
+=== Deliverables ===
+# Caching
+# Enable both V4 and V2 on Nightly 55
+=== Bugs ===
+[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M6 Dashboard for M6 bugs]
+== M7 (2017/6/12) ==
+=== Deliverables ===
+# V4 is feature complete
+=== Bugs ===
+[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M7 Dashboard for M7 bugs]
-== M4 (Deadline TBD) ==
+== M8 (2017/8/7) ==
 === Deliverables ===
-# Cache
+# V4 only in Nightly 56
-# Anything else
+# Initial Soft Vision tests
 === Bugs ===
-[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M4 Dashboard for M4 bugs]
+[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M8 Dashboard for M8 bugs]
+== M9 (2017/09/20) ==
+=== Deliverables ===
+# Shipping to release
+# Crash comparison on beta
+# Gradual roll-out on release
-== M5 (Deadline TBD) ==
+=== Bugs ===
+[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M9 Dashboard for M9 bugs]
+== M10 (2017/11/13) ==
 === Deliverables ===
-# Enable v4 by default!
+# V4 working on Fennec
+# Follow-ups completed
 === Bugs ===
-[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M5 Dashboard for M5 bugs]
+[https://wiki.mozilla.org/Security/Safe_Browsing/V4_Implementation/Dashboard#M10 Dashboard for M10 bugs]