Security/Safe Browsing/V4 Implementation

From MozillaWiki
< Security‎ | Safe Browsing
Revision as of 20:22, 22 July 2016 by Fmarier (talk | contribs) (→‎Milestones: link to the V4 dependency tree)
Jump to navigation Jump to search

Introduction

Safe Browsing v4 is designed to reduce network bandwidth and disk storage (mainly for mobile devices). The usage of partial URL hashes (aka prefixes) and complete hashes remains the same but the way we get those prefixes and complete hashes is changed. The update and hash completion API will be based on protobuf and content compression (in additional to HTTP compression) is introduced. For further information, see https://developers.google.com/safe-browsing/v4 (The public specification is not complete. For example, the protobuf is not mentioned.)

Design and Implementation

To have v2 and v4 run in parallel, we must carefully refactor some core components like ListManager and HashCompleter. Instead of having a master preference to switch between v2 and v4, we take the "table name driven" approach. The plan is to add a separate provider called google4, which owns v4 tables: goog-phish-proto, goog-unwanted-proto and goog-malware-proto. The suffix "-proto" indicates that the table should be updated and completed via protobuf. Besides, the new provider google4 has its own updateURL and gethashURL. You can consider google4 yet another provider like mozilla.

When ListManager and HashCompleter (and any other related components like ProtocolParser) sees table names suffixed by "-proto", they would behave differently. For example, in listmanager.js, while making update request for table goog-phish-proto, nsIUrlClassifier.makeUpdateRequestV4 will be called to build a v4 specific request. (See bug 1264885 and bug 1275507 for more information.)

Milestones

All of the V4 bugs can be seen in this dependency tree.

M0 (2016/7/31)

Deliverables

  1. Send v4 update request on time
  2. Parse v4 update response but not store to disk
  3. Use v4 request backoff settings
  4. v2 will still be up and running

Notes

  1. v4 table download/update will be opt-in.
  2. To test M0 features, modify the following preferences:
    • urlclassifier.malwareTable ==> Add goog-malware-proto and goog-unwanted-proto
    • urlclassifier.phishTable ==> Add goog-phish-proto

Bugs

Full Query
ID Summary Status Assigned to Resolution
1254766 Stop caching Safe Browsing completions to disk RESOLVED Dimi Lee [:dimi] FIXED No
1264885 Refactor the listmanager to add support for both V2 an V4 of the protocol RESOLVED Henry Chang [:hchang] FIXED No
1272239 Support completion for test database RESOLVED Dimi Lee [:dimi] FIXED No
1273398 Implement RequestBackoff for Safe Browsing v4 RESOLVED Henry Chang [:hchang] FIXED No
1273410 "Table Name" (used by v2) to "Threat Type" conversion RESOLVED INVALID No
1273412 "Table/List name" (v2) to "Threat type" (v4) conversion RESOLVED Henry Chang [:hchang] DUPLICATE No
1274112 Implement Safe Browsing v4 update request VERIFIED Henry Chang [:hchang] FIXED No
1275507 XPCOM API to create SafeBrowsing v4 update request RESOLVED Henry Chang [:hchang] FIXED No
1276595 Parse SafeBrowsing v4 update response RESOLVED Henry Chang [:hchang] DUPLICATE No
1281083 Changing the urlclassifier.*Table prefs doesn't take effect before the next browser restart RESOLVED Dimi Lee [:dimi] FIXED No
1287059 Keep track of the Safe Browsing V4 state in one pref per table RESOLVED Henry Chang [:hchang] FIXED No
1305567 V4 updates always fail with a 400 status code RESOLVED Henry Chang [:hchang] FIXED No
1307541 V4 updates are not scheduled at the right time RESOLVED Henry Chang [:hchang] FIXED No

13 Total; 0 Open (0%); 12 Resolved (92.31%); 1 Verified (7.69%);


M1 (2016/9/30)

Deliverables

  1. Store v4 tables to disk (including fixed and variable length prefixes)
  2. Store table states
  3. Split v4 tables to different directory per provider

Bugs

Full Query
ID Summary Status Assigned to Resolution
1037560 Safebrowsing pleasereset resets all tables VERIFIED Dimi Lee [:dimi] FIXED No
1179301 Latent buffer overrun bug in SafebrowsingHash RESOLVED Henry Chang [:hchang] FIXED No
1254763 Split Safe Browsing directory in per-provider sub-directories for V4 providers RESOLVED Henry Chang [:hchang] FIXED No
1276042 Intermittent test_classify_track.html | Test timed out RESOLVED Dimi Lee [:dimi] DUPLICATE No
1283007 Implement variable length PrefixSet class for Safe Browsing v4 RESOLVED Dimi Lee [:dimi] DUPLICATE No
1283009 Store variable-length prefix to disk RESOLVED Dimi Lee [:dimi] DUPLICATE No
1284178 Implement HashStore for v4 RESOLVED Dimi Lee [:dimi] WONTFIX No
1284204 Parse complete Safe Browsing V4 updates into a new TableUpdate class RESOLVED Henry Chang [:hchang] FIXED No
1285103 Refactor TableUpdate to support V2 and V4 RESOLVED Henry Chang [:hchang] DUPLICATE No
1285848 Supports Rice-encoded table update for v4 RESOLVED Henry Chang [:hchang] FIXED No
1287058 Supports SafeBrowsing v4 partial update RESOLVED Dimi Lee [:dimi] DUPLICATE No
1288833 Ensure that full hashes received in updates aren't used before we call gethash on them RESOLVED Thomas Nguyen (:tnguyen) WONTFIX No
1291024 Intermittent toolkit/components/url-classifier/tests/mochitest/test_gethash.html | Should not import bad css - didn't expect "hidden", but got it RESOLVED Dimi Lee [:dimi] FIXED No
1292789 Intermittent toolkit/components/url-classifier/tests/mochitest/test_gethash.html | Should not load bad javascript - got "loaded malware javascript!", expected "untouched" RESOLVED Dimi Lee [:dimi] DUPLICATE No
1296201 Intermittent toolkit/components/url-classifier/tests/mochitest/test_gethash.html | Test timed out. RESOLVED Dimi Lee [:dimi] FIXED No
1296820 Enabling Safe Browsing V4 updates breaks all list updates RESOLVED Henry Chang [:hchang] FIXED No
1297518 Intermittent toolkit/components/url-classifier/tests/mochitest/test_bug1254766.html | Should not import bad css - didn't expect "hidden", but got it RESOLVED Dimi Lee [:dimi] FIXED No
1301008 v4 list states cannot be correctly sent if it includes '\0' RESOLVED Henry Chang [:hchang] FIXED No
1302044 Disabled v4 tables would still be updated RESOLVED Henry Chang [:hchang] FIXED No
1305478 Use 0-1 min as the initial update delay for both V2 and V4 RESOLVED Henry Chang [:hchang] FIXED No
1305801 Store V4 update data to disk RESOLVED Dimi Lee [:dimi] FIXED No
1308606 Crash in mozilla::safebrowsing::Classifier::UpdateHashStore RESOLVED Dimi Lee [:dimi] FIXED No
1364611 Add telemetry to track complete matches per page load NEW No
1370753 Google API key missing from official Firefox (release and beta) for Android RESOLVED Gian-Carlo Pascutto [:gcp] FIXED No
1375277 Add support for the POTENTIALLY_HARMFUL_APPLICATION threat type RESOLVED Henry Chang [:hchang] FIXED No
1384326 Add Google API key on Android Try builds RESOLVED DUPLICATE No
1385609 Backoff seems to be interfering with updates RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1388494 Undefined string for PHA threat type and broken advisory text VERIFIED Henry Chang [:hchang] FIXED No
1388501 PHA warning pages not working on Fennec RESOLVED Henry Chang [:hchang] FIXED No
1388582 The goog-harmful-proto list doesn't appear to be working RESOLVED Henry Chang [:hchang] FIXED No
1389315 Noise entries and negative cache should be restricted to their own provider RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1392204 Failure to update safe browsing v4 DB on Android device RESOLVED Dimi Lee [:dimi] FIXED No
1394017 Enable Safe Browsing V4 on Fennec Nightly 58 RESOLVED François Marier [:francois] FIXED No
1394031 Intermittent test_platform_specific_threats.js,test_pref.js ,test_safebrowsing_protobuf.js | application crashed [@ nsNSSShutDownObject::shutdown(nsNSSShutDownObject::ShutdownCalledFrom)] RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1397544 Use the IP malware Safe Browsing list RESOLVED Dimi Lee [:dimi] WONTFIX No
1397599 Intermittent toolkit/components/url-classifier/tests/mochitest/test_donottrack.html | application crashed [@ mozilla::detail::MutexImpl::lock] RESOLVED DUPLICATE No
1397930 Official builds of Fennec need to use the same Safe Browsing client ID as desktop RESOLVED François Marier [:francois] FIXED No
1397938 Download protection whitelist and blacklist are not available for the Android platform RESOLVED Ethan Tseng [:ethan] FIXED No
1400816 SafeBrowsing v4 for Fennec - Failed to ignore the warning in an iframe RESOLVED DUPLICATE No
1408396 Failing Updating Safebrowser DB will trigger a frozen browser VERIFIED Dimi Lee [:dimi] FIXED No
1408631 Crash in shutdownhang | nsThread::Shutdown | nsUrlClassifierDBService::Shutdown RESOLVED Dimi Lee [:dimi] FIXED No

41 Total; 1 Open (2.44%); 37 Resolved (90.24%); 3 Verified (7.32%);


M2 (Right before Hawaii Workweek)

Deliverables

  1. Check v4 prefixes (in addition to v4) but ignore the result
  2. v2/v4 prefix matching consistency telemetry (e.g. v2/v4 should both 'have' or 'not have' certain URL hash)
    1. Be careful of the variable length prefixes: it's possible to get a 32-bit prefix match for foo.com in V2 and no match on V4 because that entry uses a 48-bit prefix instead

Bugs

Full Query
ID Summary Status Assigned to Resolution
1305484 Store state in the file instead of preference RESOLVED Henry Chang [:hchang] FIXED No
1305581 Verify that V4 updates were applied correctly by computing a checksum on the final result RESOLVED Dimi Lee [:dimi] FIXED No
1305780 Implement the update fail scheme for v4 RESOLVED Dimi Lee [:dimi] FIXED No
1310142 Move backup databases and raw table update data to a "update wreck" directory RESOLVED Henry Chang [:hchang] FIXED No
1312323 Single encoded value (either prefix or removal index) is not handled well RESOLVED Henry Chang [:hchang] FIXED No

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);


M3 (Deadline TBD)

Deliverables

  1. Do and use v4 complete hashes (in addition to v2) but ignore the result
  2. v2/v4 URL hash matching consistency telemetry

Bugs

Full Query
ID Summary Status Assigned to Resolution
1296802 Telemetry pings are run through the URL Classifier RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1298257 Implement url matching for variable-length prefix set RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1311910 Add telemetry to measure update error rate for V2 and V4 RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1315893 Add telemetry to measure update time for V2 and V4 RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1319286 Cache v4 table states in memory RESOLVED Henry Chang [:hchang] FIXED No

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);


M4 (Deadline TBD)

Deliverables

  1. Cache
  2. Anything else

Bugs

Full Query
ID Summary Status Assigned to Resolution
1305486 Enable V4 update by default on Nightly only RESOLVED Henry Chang [:hchang] FIXED No
1313629 Version-aware (v2/v4) hash completer RESOLVED Henry Chang [:hchang] DUPLICATE No

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);


M5 (Deadline TBD)

Deliverables

  1. Enable v4 by default!

Bugs

Full Query
ID Summary Status Assigned to Resolution
1276826 Implement Safe Browsing v4 hash completion request RESOLVED Henry Chang [:hchang] FIXED No
1297962 Support adding noise when send v4 gethash request RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1311926 Add telemetry to measure gethash error and gethash timeout rate for V2 and V4 RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1311931 Add telemetry to measure full match rate for v2 and v4 RESOLVED Dimi Lee [:dimi] FIXED No
1312339 Return length in LookupCache.Has and support VariableLengthPrefix in LookupResultArray RESOLVED Henry Chang [:hchang] FIXED No
1328821 hash completion request for v4 should not depend on table freshness RESOLVED Dimi Lee [:dimi] FIXED No
1329558 Implement Minimum wait duration for V4 gethash RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1331139 Update download protection for V4 RESOLVED Thomas Nguyen (:tnguyen) FIXED No
1331881 Minimum wait duration and negative cache duration should be passed even if there is no match RESOLVED Dimi Lee [:dimi] FIXED No
1332767 4% of V4 updates return a 400 RESOLVED Dimi Lee [:dimi] DUPLICATE No
1332780 Telemetry probes not recognizing 4xx and 5xx server status codes RESOLVED Dimi Lee [:dimi] FIXED No
1335974 URLCLASSIFIER_UPDATE_ERROR shows a number of unexpected values coming from the server RESOLVED Dimi Lee [:dimi] FIXED No
1336865 Add telemetry to measure time spent on constructing variable-length prefix set RESOLVED Dimi Lee [:dimi] FIXED No
1338082 Add telemetry probes to track the positive and negative cache durations in V4 RESOLVED Dimi Lee [:dimi] FIXED No

14 Total; 0 Open (0%); 14 Resolved (100%); 0 Verified (0%);