CA/Responding To An Incident

From MozillaWiki
< CA
Revision as of 23:33, 15 March 2019 by Wthayer (talk | contribs) (→‎Revocation: tweaks based on serial number entropy issue)
Jump to navigation Jump to search

The page gives guidance to CAs as to how Mozilla expects them to react to reported incidents such as misissuances, and what the best practices are.

For the purposes of this page, a "misissuance" is defined as any certificate issued in contravention of any applicable standard, process or document - so it could be RFC non-compliant, BR non-compliant, issued contrary to the CA's CP/CPS, or have some other flaw or problem. Researchers who report CA incidents such as misissuances are welcome to include a link to this page in their report to the CA, reminding the CA that Mozilla has the following expectations. This document is framed in terms of misissuance of certificates; it will need to be adapted as necessary for incidents of a different nature, respecting the spirit of the information requests contained therein.

Other examples of incidents include misconfigured OCSP responders, un-revocations, and any other event affecting trust in the WebPKI which does not involve the actual contents of certificates.

While some forms of incident may be seen as less serious than others, opinions vary on which these are. Mozilla sees all incidents as good opportunities for the CA to test that their incident response processes are working well, and so we expect a similar level of timeliness of response and quality of reporting for all incidents, whatever their adjudged severity.

To be clear on the status of this document: this is a best practices document, not an official policy, and does not use normative language. Therefore, failure to follow one or more of the recommendations here is not by itself sanctionable. However, failure to do so without good reason may affect Mozilla's general opinion of the CA. Our confidence in a CA is in part affected by the number and severity of incidents, but it is also significantly affected by the speed and quality of incident response.

Immediate Actions

In misussuance cases, a CA should almost always immediately cease issuance from the affected part of your PKI until you have diagnosed the source of the problem, or explain why this has not been done.

Once the problem is diagnosed, you may restart issuance even if a full fix is not rolled out, if you are able to put in place temporary or manual procedures to prevent the problem from re-occurring. You should not restart issuance until you are confident that the problem will not re-occur.

Revocation

It is normal practice for CAs to revoke misissued certificates. But that leaves the question about when this should be done, particularly if it's not possible to contact the customer immediately, or if they are unable to replace their certificate quickly. Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements currently states (version 1.6.3):

“The CA SHOULD revoke a Certificate within 24 hours and MUST revoke a Certificate with 5 days if one or more of the following occurs: …
7. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement;
8. The CA determines or is made aware that any of the information appearing in the Certificate is inaccurate; …
10. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or
11. The CA is made aware of a demonstrated or proven method that exposes the Subscriber's Private Key to compromise, methods have been developed that can easily calculate it based on the Public Key (such as a Debian weak key, see http://wiki.debian.org/SSLkeys), or if there is clear evidence that the specific method used to generate the Private Key was flawed."

This means that, in most cases of misissuance, the CA has an obligation under the BRs to revoke the certificates concerned within 5 days.

Mozilla recognizes that in some exceptional circumstances, revoking misissued certificates within the prescribed deadline may cause significant harm, such as when the certificate is used in critical infrastructure and cannot be safely replaced prior to the revocation deadline, or when a defect affects a massive number of Subscribers and certificates. However, Mozilla does not grant exceptions to the BR revocation requirements. It is our position that your CA is ultimately responsible for deciding if the harm caused by following the requirements of BR section 4.9.1 outweighs the risks that are passed on to individuals who rely on the web PKI by choosing not to meet this requirement.

If your CA will not be revoking the certificates within the time period required by the BRs, our expectations are that:

  • The decision and rationale for delaying revocation will be disclosed to Mozilla in the form of a preliminary incident report immediately; preferably before the BR mandated revocation deadline. The rationale must include an explanation for why the situation is exceptional. Responses similar to “we deem this misissuance not to be a security risk” are generally not acceptable, and must be discussed on the mozilla.dev.security.policy list. When revocation is delayed at the request of specific Subscribers, the rationale should be provided on a per-Subscriber basis.
  • Any decision to not comply with the timeline specified in the Baseline Requirements must also be accompanied by a clear timeline describing if and when the problematic certificates will be revoked and supported by the rationale to delay revocation.
  • The issue will need to be listed as a finding in your CA’s next BR audit statement.
  • Your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable.
  • That you will perform an analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays.

If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.

Follow-Up Actions

  • Work out how the bug or problem was introduced. For a code bug, were the code review processes sufficient? Does your code have automated tests, and if so, why did they not catch this case?
  • Work out why the problem was not detected earlier. Were these certificates missed by your self-audits? Or is the code or process you use for such audits insufficiently frequent or rigorous?
  • If the problem is lack of compliance to an RFC, Baseline Requirement or Mozilla Policy requirement: were you aware of this requirement? If not, why not? If so, was an attempt made to meet it? If not, why not? If so, why was that attempt flawed? Do any processes need updating for making sure your CA complies with the latest version of the various requirements placed upon it?
  • Scan your corpus of certificates to look for others with the same issue. It does not look good for a CA to claim they have revoked all affected certificates and resolved the issue, and then for a researcher to discover another set of certificates with the same or a similar problem.
  • Examine whether there are potential related problems which you can also remediate at the same time. For example, if the problem was bad data in a particular field, consider improving the validation of all fields in the certificate prior to issuance. You should be proactively looking for ways to harden your issuance pipeline against further problems.
  • If, as happens in a regrettably large number of cases, a problem report was sent to your CA but action in accordance with BR section 9.4.5 was not taken within 24 hours, investigate what happened to that report and whether your report handling processes are adequate.

Incident Report

The purpose of incident reporting is to help all of us work together to build a more secure web. Therefore, the incident report should share lessons learned that could be helpful to all CAs to build better systems. The incident report should explain how the systems failed, how was the mis-issuance or incident possible, and why the problem was not detected earlier. In addition to the timeline of responding to and resolving the incident, the incident report should explain how the CA's systems will be made more robust, and how other CAs may learn from the incident.

Each incident should result in an incident report, written as soon as the problem is fully diagnosed and (temporary or permanent) measures have been put in place to make sure it will not re-occur. If the permanent fix is going to take significant time to implement, you should not wait until this is done before issuing the report. We expect to see incident reports as soon as possible, and certainly within two weeks of the initial issue report. While remediation work may still be ongoing, a satisfactory incident report will serve to resolve the issue from a Mozilla perspective.

The incident report may well repeat things which have been said previously in discussions or bug comments. This is entirely expected. The report should be a summary of previous findings. The existence of data in discussions or bug comments does not excuse a CA from the task of compiling a proper incident report.

Your CA may submit an incident report by creating a bug in Bugzilla under the NSS:CA Certificate Compliance component, or by posting the report to the mozilla.dev.security.policy mailing list. If an incident report is sent to the list without a corresponding bug, a new one will be created to track the incident.

The incident report should cover at least the following topics:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
  3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
  4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
  5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
  6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
  7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Keeping Us Informed

Once the report is posted, you should respond promptly to questions that are asked, and in no circumstances should a question linger without a response for more than one week, even if the response is only to acknowledge the question and provide a later date when an answer will be delivered. You should also provide updates at least every week giving your progress, and confirm when the remediation steps have been completed - unless Mozilla representatives agree to a different schedule by setting a “Next Update” date in the “Whiteboard” field of the bug. Such updates should be posted to the m.d.s.p. thread, if there is one, and the Bugzilla bug. The bug will be closed when remediation is completed.

Examples of Good Practice

Here are some examples of good practice, where a CA did most or all of the things recommended above.

Let's Encrypt Unicode Normalization Compliance Incident

In this case, the CA managed to diagnose the problem, remediate it, and deploy the fix to production within 24 hours.

PKIOverheid Short Serial Number Incident

While the CA could have provided interim updates, and the final report was a little delayed, the contents of it were excellent.

Retrieved from "https://wiki.allizom.org/index.php?title=CA/Responding_To_An_Incident&oldid=1209031"