Security/Tracking protection: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
(→‎Tests: Link to the Safe Browsing tests.)
(Add documentation for new lists and test pages.)
 
(6 intermediate revisions by 2 users not shown)
Line 8: Line 8:


* <tt>browser.contentblocking.enabled</tt>: master switch for all content blocking features (includes tracking protection, but excludes tracking annotations)
* <tt>browser.contentblocking.enabled</tt>: master switch for all content blocking features (includes tracking protection, but excludes tracking annotations)
* <tt>browser.safebrowsing.debug</tt>: show debugging info from the JavaScript list update code on the command line as long as browser.dom.window.dump.enabled is also enabled
* <tt>browser.safebrowsing.provider.mozilla.lists</tt>: list of tables coming from the Mozilla shavar service
* <tt>browser.safebrowsing.provider.mozilla.lists</tt>: list of tables coming from the Mozilla shavar service
* <tt>browser.safebrowsing.provider.mozilla.updateURL</tt>: server endpoint for downloading list updates
* <tt>browser.safebrowsing.provider.mozilla.updateURL</tt>: server endpoint for downloading list updates
Line 32: Line 33:
* [[Security/Tracking_protection/Client_implementation|Client implementation]]
* [[Security/Tracking_protection/Client_implementation|Client implementation]]
* [[Services/TrackingProtection|Server implementation]]
* [[Services/TrackingProtection|Server implementation]]
* [https://people.mozilla.org/~fmarier/tracking-dashboard/ Telemetry dashboard]
* [https://github.com/mozilla/itisatrap itisatrap.org] test page
* [https://github.com/mozilla/itisatrap itisatrap.org] test page
* [https://docs.google.com/document/d/1TxKCWn0qTgo0hnsKQINka-Jhm69nlbjbClgVWUHt22Y/edit Sandboxing trackers] (as an alternative to blocking)
* [https://docs.google.com/document/d/1TxKCWn0qTgo0hnsKQINka-Jhm69nlbjbClgVWUHt22Y/edit Sandboxing trackers] (as an alternative to blocking)
Line 56: Line 56:
Tracking annotations are used in a few different places:
Tracking annotations are used in a few different places:


* devtools label requests from tracking domain as such
* control center indicates the presence of trackers based on annotations
* [https://searchfox.org/mozilla-central/rev/d4b9e50875ad7e5d20f2fee6a53418315f6dfcc0/netwerk/base/nsChannelClassifier.cpp#1028-1030 lower the priority of tracker loads] in necko
* [https://searchfox.org/mozilla-central/rev/d4b9e50875ad7e5d20f2fee6a53418315f6dfcc0/netwerk/base/nsChannelClassifier.cpp#1028-1030 lower the priority of tracker loads] in necko
* [https://searchfox.org/mozilla-central/rev/5dbfd833bbb114afe758db4d4bdbc5b13bcc33ef/netwerk/base/nsChannelClassifier.cpp#244-263 enable throttling of these requests] in necko (flag added in [https://bugzilla.mozilla.org/show_bug.cgi?id=1360580 bug 1360580])
* [https://searchfox.org/mozilla-central/rev/5dbfd833bbb114afe758db4d4bdbc5b13bcc33ef/netwerk/base/nsChannelClassifier.cpp#244-263 enable throttling of these requests] in necko (flag added in [https://bugzilla.mozilla.org/show_bug.cgi?id=1360580 bug 1360580])
Line 96: Line 98:
*** Same upstream source as the base list.
*** Same upstream source as the base list.
*** '''Only includes''' the <tt>Content</tt> category.
*** '''Only includes''' the <tt>Content</tt> category.
** Category-specific lists (currently only used by Focus for Android):
*** <tt>ads-track-digest256</tt>: trackers in the Advertising category
*** <tt>analytics-track-digest256</tt>: trackers in the Analytics category
*** <tt>social-track-digest256</tt>: trackers in the Social category
** Fingerprinting lists:
*** <tt>base-fingerprinting-track-digest256</tt>: domains in both the Fingerprinting category AND in one of the tracking categories (Advertising, Analytics, Social, or Content)
*** <tt>content-fingerprinting-track-digest256</tt>L domains in the Fingerprinting category that are NOT in one of the tracking categories
** Cryptomining lists:
*** <tt>base-cryptomining-track-digest256</tt>: domains in the Cryptomining category
*** <tt>content-cryptomining-track-digest256</tt>: placeholder list, currently empty. Intended to include cryptomining domains that we don't want to block by default (for some reason).
* Legacy blacklists (Firefox 42 to 49):
* Legacy blacklists (Firefox 42 to 49):
** Blacklist (<tt>mozstd-track-digest256</tt>)
** Blacklist (<tt>mozstd-track-digest256</tt>)
Line 102: Line 114:
*** Same as the union of all of the base '''and''' content lists.
*** Same as the union of all of the base '''and''' content lists.
* Entity whitelist (<tt>mozstd-trackwhite-digest256</tt>)
* Entity whitelist (<tt>mozstd-trackwhite-digest256</tt>)
** [https://s3.amazonaws.com/lists.disconnect.me/entitylist.json Upstream source]
** [https://github.com/mozilla-services/shavar-prod-lists/blob/master/disconnect-entitylist.json source list] (i.e. what we ship to clients in binary form)
** [https://github.com/mozilla-services/shavar-prod-lists/blob/master/disconnect-entitylist.json Our copy] (i.e. what we ship to clients in binary form)
** Implemented in [https://bugzilla.mozilla.org/show_bug.cgi?id=1141352 bug 1141352]
** Implemented in [https://bugzilla.mozilla.org/show_bug.cgi?id=1141352 bug 1141352]
* List creation script
* List creation script
Line 136: Line 147:
** <tt>itisatrap.com</tt>: blocked by <tt>base-track-digest256</tt> (and included in the [https://github.com/mozilla-services/shavar-prod-lists/blob/f16248d7f33367bb3c48d72fb32fdb239dbe0c8e/disconnect-blacklist.json#L8204-L8210 Disconnect list])
** <tt>itisatrap.com</tt>: blocked by <tt>base-track-digest256</tt> (and included in the [https://github.com/mozilla-services/shavar-prod-lists/blob/f16248d7f33367bb3c48d72fb32fdb239dbe0c8e/disconnect-blacklist.json#L8204-L8210 Disconnect list])
** <tt>itisatrap.org/?resource=itisatracker.org</tt>: whitelisted in <tt>[https://searchfox.org/mozilla-central/rev/fd5c37f1dd9a0d1e327a6c6b4d81ea92f52c4330/toolkit/components/url-classifier/SafeBrowsing.jsm#479 test-trackwhite-simple]</tt>
** <tt>itisatrap.org/?resource=itisatracker.org</tt>: whitelisted in <tt>[https://searchfox.org/mozilla-central/rev/fd5c37f1dd9a0d1e327a6c6b4d81ea92f52c4330/toolkit/components/url-classifier/SafeBrowsing.jsm#479 test-trackwhite-simple]</tt>
** <tt>*.dummytracker.org</tt>: a test domain added to all lists using the pattern <tt><list_name>.dummytracker.org</tt>
*** For example, <tt>base-track-digest256.dummytracker.org</tt> is on the <tt>base-track-digest256</tt> list and <tt>base-fingerprinting-track-digest256.dummytracker.org</tt> is on the <tt>base-fingerprinting-track-digest256</tt> list.
*** [https://github.com/mozilla/dummytracker Github repo for test page].
* [[QA/Polaris/Tracking protection|Test plan for Fx42]]
* [[QA/Polaris/Tracking protection|Test plan for Fx42]]
* [[Services/TrackingProtection/Shavar_Server_-_Testing|Shavar test plan]] (includes end-to-end tests)
* [[Services/TrackingProtection/Shavar_Server_-_Testing|Shavar test plan]] (includes end-to-end tests)
* [https://github.com/fmarier/sbdbdump/blob/master/dump.py Script to dump the contents of <tt>mozpub-track-digest256*</tt> files]
* [https://github.com/fmarier/sbdbdump/blob/master/dump.py Script to dump the contents of <tt>mozpub-track-digest256*</tt> files]
* [https://github.com/mozilla/trackingprotection-tools Tools and utility files for analyzing Tracking Protection]


To turn on debugging output, export the following environment variable:
To turn on debugging output, export the following environment variable:

Latest revision as of 00:59, 21 March 2019

Description

Tracking Protection is a new platform-level technology that blocks HTTP loads at the network level. It is based on the Safe Browsing technology that powers our phishing and malware protection.

This feature was part of the Polaris initiative.

Prefs

  • browser.contentblocking.enabled: master switch for all content blocking features (includes tracking protection, but excludes tracking annotations)
  • browser.safebrowsing.debug: show debugging info from the JavaScript list update code on the command line as long as browser.dom.window.dump.enabled is also enabled
  • browser.safebrowsing.provider.mozilla.lists: list of tables coming from the Mozilla shavar service
  • browser.safebrowsing.provider.mozilla.updateURL: server endpoint for downloading list updates
  • browser.safebrowsing.provider.mozilla.gethashURL: server endpoint for completions
  • browser.safebrowsing.provider.mozilla.lastupdatetime: timestamp (in ms) of when the last list update happened.
  • browser.safebrowsing.provider.mozilla.nextupdatetime: timestamp (in ms) of when the list should next be downloaded.
  • privacy.trackingprotection.annotate_channels: flag network channels loading resources on the tracking list (see how that information can be used)
  • privacy.trackingprotection.enabled: to enable TP globally
  • privacy.trackingprotection.lower_network_priority: lower the priority of channels loading tracking resources
  • privacy.trackingprotection.pbmode.enabled: to enable TP in Private Browsing mode (not needed if the global pref is enabled)
  • privacy.trackingprotection.introCount
  • privacy.trackingprotection.introURL: URL that kicks off the UI tour (target of the "See how this works" button in about:privatebrowsing)
  • urlclassifier.disallow_completions: list of tables for which we never call gethash
  • list of tables to use when looking for trackers (they need to be named *-track-*):
    • urlclassifier.trackingAnnotationTable: for tracking annotations
    • urlclassifier.trackingTable: for tracking protection
  • list of tables to use when checking whether or not a tracker is part of the same entity as the page (they need to be named *-trackwhite-*):
    • urlclassifier.trackingAnnotationWhitelistTable: for tracking annotations
    • urlclassifier.trackingWhitelistTable: for tracking protection

Engineering

Code walkthrough

The classification for tracking protection, separate from the full Safe Browsing classification, is kicked off in nsHttpChannel::BeginConnectContinue() and goes like this:

  1. we asynchronously check the blacklist
  2. if there's a match, we then check the entity whitelist
  3. if it doesn't match the whitelist, we treat it as a tracker
  4. we either cancel the channel (for full tracking protection) or set a tracking flag (for tracking annotations only)

Note that only eligible resources are run through the classifier:

Tracking annotations

Tracking annotations are used in a few different places:

Tests

In addition to the Safe Browsing tests, here are all of the tests which are relevant to tracking protection:

./mach test browser/base/content/test/trackingUI/
./mach test netwerk/test/unit/test_trackingProtection_annotateChannels.js
./mach test netwerk/test/unit_ipc/test_trackingProtection_annotateChannels_wrap1.js
./mach test netwerk/test/unit_ipc/test_trackingProtection_annotateChannels_wrap2.js

Lists

  • Current blacklists (Firefox 50 and later):
    • Base lists:
      • base-track-digest256: non-DNT-compliant trackers
      • baseeff-track-digest256: DNT-compliant trackers (EFF definition)
      • basew3c-track-digest256: DNT-compliant trackers (W3C definition)
      • Upstream source
      • Our copy (i.e. what we ship to clients in binary form)
      • Submit feedback and track changes
      • Excludes the Content category.
    • Content lists:
      • content-track-digest256: non-DNT-compliant content trackers
      • contenteff-track-digest256: DNT-compliant content trackers (EFF definition)
      • contentw3c-track-digest256: DNT-compliant content trackers (W3C definition)
      • Same upstream source as the base list.
      • Only includes the Content category.
    • Category-specific lists (currently only used by Focus for Android):
      • ads-track-digest256: trackers in the Advertising category
      • analytics-track-digest256: trackers in the Analytics category
      • social-track-digest256: trackers in the Social category
    • Fingerprinting lists:
      • base-fingerprinting-track-digest256: domains in both the Fingerprinting category AND in one of the tracking categories (Advertising, Analytics, Social, or Content)
      • content-fingerprinting-track-digest256L domains in the Fingerprinting category that are NOT in one of the tracking categories
    • Cryptomining lists:
      • base-cryptomining-track-digest256: domains in the Cryptomining category
      • content-cryptomining-track-digest256: placeholder list, currently empty. Intended to include cryptomining domains that we don't want to block by default (for some reason).
  • Legacy blacklists (Firefox 42 to 49):
    • Blacklist (mozstd-track-digest256)
      • Same as the union of all of the base lists.
    • "Strict" blacklist (mozfull-track-digest256)
      • Same as the union of all of the base and content lists.
  • Entity whitelist (mozstd-trackwhite-digest256)
  • List creation script
  • The lists are stored in these files:
    • ~/.cache/mozilla/firefox/XXXX/safebrowsing/mozstd-track* on Linux
    • ~/Library/Caches/Firefox/Profiles/XXXX/safebrowsing/mozstd-track* on Mac
    • C:\Users\XXXX\AppData\Local\mozilla\firefox\profiles\XXXX\safebrowsing\mozstd-track* on Windows

QA

  • Bugzilla:
    • Firefox::Tracking Protection for UI and general feature requests/bugs
    • Toolkit::Safe Browsing for list updates and the actual blocking in necko
    • Cloud Services::Server: Shavar for server-side bugs
    • Breakage bugs
    • Bug triage

To turn on debugging output, export the following environment variable:

MOZ_LOG_FILE=/tmp/trackingprotection.log
MOZ_LOG="UrlClassifierDbService:5,nsChannelClassifier:5"

To produce the "digest256" hash that sbdbdump -v will contain for example.com:

echo -n "example.com/" | sha256sum 
7fc983ea552f7c8d153fc308d621eb4f52e84aa63ecccf3a735698a11a2a4a8d

Telemetry

Alerts are sent to safebrowsing-telemetry@mozilla.org.

Notes:

  • no telemetry pings are sent while in Private Browsing mode
  • we only have telemetry when TP is enabled in the session

Documentation