Security/Safe Browsing/V4 Implementation

Milestones

M0 (2016/7/31)

Deliverables

1. Send v4 update request on time
2. Parse v4 update response but not store to disk
3. Use v4 request backoff settings
4. v2 will still be up and running

Bugs

Full Query

ID	Summary	Status	Assigned to	Resolution
1254766	Stop caching Safe Browsing completions to disk	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1264885	Refactor the listmanager to add support for both V2 an V4 of the protocol	RESOLVED	Henry Chang [:hchang]	FIXED	No
1272239	Support completion for test database	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1273398	Implement RequestBackoff for Safe Browsing v4	RESOLVED	Henry Chang [:hchang]	FIXED	No
1273410	"Table Name" (used by v2) to "Threat Type" conversion	RESOLVED		INVALID	No
1273412	"Table/List name" (v2) to "Threat type" (v4) conversion	RESOLVED	Henry Chang [:hchang]	DUPLICATE	No
1274112	Implement Safe Browsing v4 update request	VERIFIED	Henry Chang [:hchang]	FIXED	No
1275507	XPCOM API to create SafeBrowsing v4 update request	RESOLVED	Henry Chang [:hchang]	FIXED	No
1276595	Parse SafeBrowsing v4 update response	RESOLVED	Henry Chang [:hchang]	DUPLICATE	No
1281083	Changing the urlclassifier.*Table prefs doesn't take effect before the next browser restart	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1287059	Keep track of the Safe Browsing V4 state in one pref per table	RESOLVED	Henry Chang [:hchang]	FIXED	No
1305567	V4 updates always fail with a 400 status code	RESOLVED	Henry Chang [:hchang]	FIXED	No
1307541	V4 updates are not scheduled at the right time	RESOLVED	Henry Chang [:hchang]	FIXED	No

13 Total; 0 Open (0%); 12 Resolved (92.31%); 1 Verified (7.69%);

M1 (2016/9/30)

Deliverables

1. Store v4 tables to disk (including fixed and variable length prefixes)
2. Store table states
3. Split v4 tables to different directory per provider

Bugs

Full Query

ID	Summary	Status	Assigned to	Resolution
1037560	Safebrowsing pleasereset resets all tables	VERIFIED	Dimi Lee [:dimi]	FIXED	No
1179301	Latent buffer overrun bug in SafebrowsingHash	RESOLVED	Henry Chang [:hchang]	FIXED	No
1254763	Split Safe Browsing directory in per-provider sub-directories for V4 providers	RESOLVED	Henry Chang [:hchang]	FIXED	No
1276042	Intermittent test_classify_track.html | Test timed out	RESOLVED	Dimi Lee [:dimi]	DUPLICATE	No
1283007	Implement variable length PrefixSet class for Safe Browsing v4	RESOLVED	Dimi Lee [:dimi]	DUPLICATE	No
1283009	Store variable-length prefix to disk	RESOLVED	Dimi Lee [:dimi]	DUPLICATE	No
1284178	Implement HashStore for v4	RESOLVED	Dimi Lee [:dimi]	WONTFIX	No
1284204	Parse complete Safe Browsing V4 updates into a new TableUpdate class	RESOLVED	Henry Chang [:hchang]	FIXED	No
1285103	Refactor TableUpdate to support V2 and V4	RESOLVED	Henry Chang [:hchang]	DUPLICATE	No
1285848	Supports Rice-encoded table update for v4	RESOLVED	Henry Chang [:hchang]	FIXED	No
1287058	Supports SafeBrowsing v4 partial update	RESOLVED	Dimi Lee [:dimi]	DUPLICATE	No
1288833	Ensure that full hashes received in updates aren't used before we call gethash on them	RESOLVED	Thomas Nguyen (:tnguyen)	WONTFIX	No
1291024	Intermittent toolkit/components/url-classifier/tests/mochitest/test_gethash.html | Should not import bad css - didn't expect "hidden", but got it	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1292789	Intermittent toolkit/components/url-classifier/tests/mochitest/test_gethash.html | Should not load bad javascript - got "loaded malware javascript!", expected "untouched"	RESOLVED	Dimi Lee [:dimi]	DUPLICATE	No
1296201	Intermittent toolkit/components/url-classifier/tests/mochitest/test_gethash.html | Test timed out.	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1296820	Enabling Safe Browsing V4 updates breaks all list updates	RESOLVED	Henry Chang [:hchang]	FIXED	No
1297518	Intermittent toolkit/components/url-classifier/tests/mochitest/test_bug1254766.html | Should not import bad css - didn't expect "hidden", but got it	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1301008	v4 list states cannot be correctly sent if it includes '\0'	RESOLVED	Henry Chang [:hchang]	FIXED	No
1302044	Disabled v4 tables would still be updated	RESOLVED	Henry Chang [:hchang]	FIXED	No
1305478	Use 0-1 min as the initial update delay for both V2 and V4	RESOLVED	Henry Chang [:hchang]	FIXED	No
1305801	Store V4 update data to disk	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1308606	Crash in mozilla::safebrowsing::Classifier::UpdateHashStore	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1364611	Add telemetry to track complete matches per page load	NEW			No
1370753	Google API key missing from official Firefox (release and beta) for Android	RESOLVED	Gian-Carlo Pascutto [:gcp]	FIXED	No
1375277	Add support for the POTENTIALLY_HARMFUL_APPLICATION threat type	RESOLVED	Henry Chang [:hchang]	FIXED	No
1384326	Add Google API key on Android Try builds	RESOLVED		DUPLICATE	No
1385609	Backoff seems to be interfering with updates	RESOLVED	Thomas Nguyen (:tnguyen)	FIXED	No
1388494	Undefined string for PHA threat type and broken advisory text	VERIFIED	Henry Chang [:hchang]	FIXED	No
1388501	PHA warning pages not working on Fennec	RESOLVED	Henry Chang [:hchang]	FIXED	No
1388582	The goog-harmful-proto list doesn't appear to be working	RESOLVED	Henry Chang [:hchang]	FIXED	No
1389315	Noise entries and negative cache should be restricted to their own provider	RESOLVED	Thomas Nguyen (:tnguyen)	FIXED	No
1392204	Failure to update safe browsing v4 DB on Android device	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1394017	Enable Safe Browsing V4 on Fennec Nightly 58	RESOLVED	François Marier [:francois]	FIXED	No
1394031	Intermittent test_platform_specific_threats.js,test_pref.js ,test_safebrowsing_protobuf.js | application crashed [@ nsNSSShutDownObject::shutdown(nsNSSShutDownObject::ShutdownCalledFrom)]	RESOLVED	Thomas Nguyen (:tnguyen)	FIXED	No
1397544	Use the IP malware Safe Browsing list	RESOLVED	Dimi Lee [:dimi]	WONTFIX	No
1397599	Intermittent toolkit/components/url-classifier/tests/mochitest/test_donottrack.html | application crashed [@ mozilla::detail::MutexImpl::lock]	RESOLVED		DUPLICATE	No
1397930	Official builds of Fennec need to use the same Safe Browsing client ID as desktop	RESOLVED	François Marier [:francois]	FIXED	No
1397938	Download protection whitelist and blacklist are not available for the Android platform	RESOLVED	Ethan Tseng [:ethan]	FIXED	No
1400816	SafeBrowsing v4 for Fennec - Failed to ignore the warning in an iframe	RESOLVED		DUPLICATE	No
1408396	Failing Updating Safebrowser DB will trigger a frozen browser	VERIFIED	Dimi Lee [:dimi]	FIXED	No
1408631	Crash in shutdownhang | nsThread::Shutdown | nsUrlClassifierDBService::Shutdown	RESOLVED	Dimi Lee [:dimi]	FIXED	No

41 Total; 1 Open (2.44%); 37 Resolved (90.24%); 3 Verified (7.32%);

M2 (Right before Hawaii Workweek)

Deliverables

1. Check v4 prefixes (in addition to v4) but ignore the result
2. v2/v4 prefix matching consistency telemetry (e.g. v2/v4 should both 'have' or 'not have' certain URL hash)
   1. Be careful of the variable length prefixes

Bugs

Full Query

ID	Summary	Status	Assigned to	Resolution
1305484	Store state in the file instead of preference	RESOLVED	Henry Chang [:hchang]	FIXED	No
1305581	Verify that V4 updates were applied correctly by computing a checksum on the final result	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1305780	Implement the update fail scheme for v4	RESOLVED	Dimi Lee [:dimi]	FIXED	No
1310142	Move backup databases and raw table update data to a "update wreck" directory	RESOLVED	Henry Chang [:hchang]	FIXED	No
1312323	Single encoded value (either prefix or removal index) is not handled well	RESOLVED	Henry Chang [:hchang]	FIXED	No

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);