Security/Safe Browsing/V4 Implementation: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(fix inversion malwareTable / phishTable)
(→‎Try it on!: improve explanation on how to activate v4)
Line 20: Line 20:
# Rebuild and check if config.status has something like 'MOZ_GOOGLE_API_KEY': b'AIzaSooxxxxoxoxoxoxooxx'
# Rebuild and check if config.status has something like 'MOZ_GOOGLE_API_KEY': b'AIzaSooxxxxoxoxoxoxooxx'


= Try it on! =
= Try Google Safe Browsing v4! =
V4 has been function ready (but pref'ed off) since Bug 1312339 was resolved. By changing preferences
v4 tables are suffixed by "-proto", to indicate that the table should be updated and completed via protobuf: goog-phish-proto, goog-unwanted-proto and goog-malware-proto.  
# '''urlclassifier.phishTable''' to '''goog-phish-shavar,goog-phish-proto,test-phish-simple'''  
To use Google Safe Browsing v4, change preferences as follows:
# '''urlclassifier.malwareTable''' to '''goog-malware-shavar,goog-malware-proto,goog-unwanted-shavar,goog-unwanted-proto,test-malware-simple,test-unwanted-simple'''
 
# '''urlclassifier.phishTable''' to '''goog-phish-proto'''  
# '''urlclassifier.malwareTable''' to '''goog-malware-proto'''
# '''browser.safebrowsing.provider.google4.gethashURL''' to '''https://safebrowsing.googleapis.com/v4/fullHashes:find?$ct=application/x-protobuf&key=%GOOGLE_API_KEY%'''
# '''browser.safebrowsing.provider.google4.gethashURL''' to '''https://safebrowsing.googleapis.com/v4/fullHashes:find?$ct=application/x-protobuf&key=%GOOGLE_API_KEY%'''
# '''browser.safebrowsing.temporary.take_v4_completion_result''' to '''true'''
# '''browser.safebrowsing.temporary.take_v4_completion_result''' to '''true'''
you can turn on v4 feature (while v2 is still up and running).


If you want to get rid of v2 and use v4 only, you could also set the following preferences to empty string
To use v4 while v2 is still up and running, set the names of both v2 and v4 tables:
 
# '''urlclassifier.phishTable''' to '''goog-phish-proto,goog-phish-shavar'''
# '''urlclassifier.malwareTable''' to '''goog-malware-proto,goog-malware-shavar'''
 
If you want to completely get rid of v2, you could also set the following preferences to empty string
 
# '''browser.safebrowsing.provider.google.lists''' to '''""'''
# '''browser.safebrowsing.provider.google.lists''' to '''""'''
then you are ready to go!
then you are ready to go!

Revision as of 21:30, 22 June 2017

Quick Links

Introduction

Safe Browsing v4 is designed to reduce network bandwidth and disk storage (mainly for mobile devices). The usage of partial URL hashes (aka prefixes) and complete hashes remains the same but the way we get those prefixes and complete hashes is changed. The update and hash completion API will be based on protobuf and content compression (in additional to HTTP compression) is introduced. For further information, see https://developers.google.com/safe-browsing/v4 (The public specification is not complete. For example, the protobuf is not mentioned.)

Design and Implementation

To have v2 and v4 run in parallel, we must carefully refactor some core components like ListManager and HashCompleter. Instead of having a master preference to switch between v2 and v4, we take the "table name driven" approach. The plan is to add a separate provider called google4, which owns v4 tables: goog-phish-proto, goog-unwanted-proto and goog-malware-proto. The suffix "-proto" indicates that the table should be updated and completed via protobuf. Besides, the new provider google4 has its own updateURL and gethashURL. You can consider google4 yet another provider like mozilla.

When ListManager and HashCompleter (and any other related components like ProtocolParser) sees table names suffixed by "-proto", they would behave differently. For example, in listmanager.js, while making update request for table goog-phish-proto, nsIUrlClassifier.makeUpdateRequestV4 will be called to build a v4 specific request. (See bug 1264885 and bug 1275507 for more information.)

Using custom API key

We are required to have a API key to access google services like geolocation and safe browsing. The nightly/beta/aurora/release Firefox builds will have a proper API key associated with mozilla. However, if you are using the Firefox built on your own, the API key will be substituted with "no-google-api-key", which is apparently not available. There are a couple of ways to use a working API key for Safe Browsing. The most recommended way is to prepare a key file and add the path to mozconfig:

  1. Obtain a Safe Browsing google API key from https://console.developers.google.com
    • If you don't know how to get a API key, buy hchang@mozilla.com a beer and ask him :p
  2. Create a key file which only contains the API key.
  3. Add "ac_add_options --with-google-api-keyfile=/path/to/your/keyfile" to mozconfig
  4. Rebuild and check if config.status has something like 'MOZ_GOOGLE_API_KEY': b'AIzaSooxxxxoxoxoxoxooxx'

Try Google Safe Browsing v4!

v4 tables are suffixed by "-proto", to indicate that the table should be updated and completed via protobuf: goog-phish-proto, goog-unwanted-proto and goog-malware-proto. To use Google Safe Browsing v4, change preferences as follows:

  1. urlclassifier.phishTable to goog-phish-proto
  2. urlclassifier.malwareTable to goog-malware-proto
  3. browser.safebrowsing.provider.google4.gethashURL to https://safebrowsing.googleapis.com/v4/fullHashes:find?$ct=application/x-protobuf&key=%GOOGLE_API_KEY%
  4. browser.safebrowsing.temporary.take_v4_completion_result to true

To use v4 while v2 is still up and running, set the names of both v2 and v4 tables:

  1. urlclassifier.phishTable to goog-phish-proto,goog-phish-shavar
  2. urlclassifier.malwareTable to goog-malware-proto,goog-malware-shavar

If you want to completely get rid of v2, you could also set the following preferences to empty string

  1. browser.safebrowsing.provider.google.lists to ""

then you are ready to go!

Milestones

M0 (2016/7/31)

Deliverables

  1. Send v4 update request on time
  2. Parse v4 update response but not store to disk
  3. Use v4 request backoff settings
  4. v2 will still be up and running

Notes

  1. v4 table download/update will be opt-in.
  2. To test M0 features, modify the following preferences:
    • urlclassifier.malwareTable ==> Add goog-malware-proto and goog-unwanted-proto
    • urlclassifier.phishTable ==> Add goog-phish-proto

Bugs

Dashboard for M0 bugs


M1 (2016/9/30)

Deliverables

  1. Store v4 tables to disk (including fixed and variable length prefixes)
  2. Store table states
  3. Split v4 tables to different directory per provider

Bugs

Dashboard for M1 bugs

Workweek in Vancouver

  1. Etherpad: https://public.etherpad-mozilla.org/p/sbv4yvr

M2 (TBC, maybe 10/15)

The intention of M2 is to push the date where we "enable v4 list update on nightly by default" to two weeks later because we have identified some issues and lacks in M0 and M1 in Vancouver SBv4 work week. The target date is the end of second week in October.

Deliverables

  1. bug 1305486 to enable v4 table update on nightly by default (The most important!)
  2. bug 1305567 so that we won't get 400 error while doing list update. (landed)
  3. bug 1305581 to examine checksum against the updated lists.
  4. bug 1285848 to support compressed list update. (under review)
  5. bug 1305801 to store v4 list to disk. (landed)
  6. bug 1305484 to store v4 list states to disk rather than preference.

Bugs

Dashboard for M2 bugs

M3 (Right before Hawaii Workweek)

Deliverables

  1. Check v4 prefixes (in addition to v4) but ignore the result
  2. v2/v4 prefix matching consistency telemetry (e.g. v2/v4 should both 'have' or 'not have' certain URL hash)
    1. Be careful of the variable length prefixes: it's possible to get a 32-bit prefix match for foo.com in V2 and no match on V4 because that entry uses a 48-bit prefix instead

Bugs

Dashboard for M3 bugs

M4 (2017/1/23)

Deliverables

  1. Enable update by default
  2. Fix crashes and regressions

Bugs

Dashboard for M4 bugs

M5 (2017/3/6)

Deliverables

  1. Enable v4 completion on nightly but ignore the result
  2. Add V4 table for Download protection.
  3. v2/v4 URL hash matching consistency telemetry

Bugs

Dashboard for M5 bugs

M6 (2017/4/18)

Deliverables

  1. Caching
  2. Enable both V4 and V2 on Nightly 55

Bugs

Dashboard for M6 bugs

M7 (2017/6/12)

Deliverables

  1. V4 is feature complete

Bugs

Dashboard for M7 bugs

M8 (2017/8/7)

Deliverables

  1. V4 only in Nightly 56
  2. Initial Soft Vision tests

Bugs

Dashboard for M8 bugs

M9 (2017/09/25)

Deliverables

  1. Follow-ups completed

Bugs

Dashboard for M9 bugs

Retrieved from "https://wiki.allizom.org/index.php?title=Security/Safe_Browsing/V4_Implementation&oldid=1174446"