CA/Symantec Issues

	THIS PAGE IS A WORKING DRAFT
	The page may be difficult to navigate, and some information on its subject might be incomplete and/or evolving rapidly. If you have any questions or ideas, please add them as a new topic on the discussion page.

This page is having updates from Symantec integrated, and so is not currently complete. Check back tomorrow.

This page lists all confirmed or suspected issues involving the CA "Symantec". It will be updated by Mozilla as more information becomes available. Please do not edit this page yourself; if you have proposed changes, email Gerv.

Information here is correct to the best of Mozilla's knowledge and belief. Note that Symantec has not yet had a chance to respond to some of the issues presented here, and so we expect this information will be updated over time.

The issues are in broadly chronological order by end date.

Issue B: 1024-bit Certificate Issued Directly From Root (Dec 2013 - Jan 2014)

Symantec issued a cert to one of its customers, Pitney Bowes, that did not comply with at least two provisions of both the CA/Browser Forum Baseline Requirements and Mozilla policy. Firstly, it was issued directly from a root, and secondly it was a 1024-bit cert which expired after the end of 2013. Symantec believed this was the only technical way to ensure continuity of service for the customer concerned.

This cert was backdated, but that is not a BR or Mozilla policy violation, as long as it was not done to evade a technical control. It also has a short serial number. Entropy in the serial number is a SHOULD in the relevant version of the BRs (version 1.1.6). 20 bits of entropy is a MUST in the Mozilla policy (version 2.2), but it doesn't say it has to be in the serial number - it could be that they randomised the notBefore time. I am told Microsoft removed the allowance for doing entropy in the Date field on 11th November 2013, so this was a violation of their policies.

Symantec did not request permission to issue in advance, they disclosed the issuance at least a month after it had happened, and the replacement certificate (unlike the original) asserted a "BR Compliant" OID.

The incident is recorded in bug 966350.

Symantec Response

Symantec self-reported this issuance to Mozilla via Bugzilla at the end of January 2014. Even though a question was asked about the BR Compliance OID that the new cert asserted, they did not comment on the bug beyond the initial report.

Further Comments and Conclusion

The lack of discussion in advance, the delayed disclosure and the inclusion of a BR-compliant OID in a certificate Symantec knew was not BR-compliant are all disappointing.

Issue D: Test Certificate Misissuance (April 2009 - September 2015)

Between 2009 and 2015, Symantec issued a large number of test certificates in their publicly trusted hierarchies. These contained domains that Symantec did not own or control, and for which domain validation was not performed. Some of these domains were unregistered, and others were owned by other organizations. Issuing test certificates for unregistered domains was not a BR violation before April 2014, but Symantec continued the practice after it had been forbidden. The registered domains used included those belonging to Google and Opera Software. Given the numbers involved, this sort of test certificate issuance appears to have been common practice at Symantec. Some of the test certificates (including one for www.google.com) left Symantec's network because they were logged in CT; Symantec claims no others did. However, Symantec personnel would have had access to the public and private keys of the certs.

Some details of this incident are recorded in bug 1214321.

Symantec Response

It took quite a while for the full scale of the problem to emerge, and Symantec had to publish numerous updates to their "Final Report". Symantec released a number of documents relating to this - Final Report, Final Report v2, Final Report v3, list of certs containing owned domains, list of certs containing unregistered domains, further update. They also issued a blog post, "A Tough Day As Leaders", which is no longer on their website but has been archived. It indicates that 3 employees were fired as a result of the misissuance. They posted another page with "complete investigation results".

In their report, Symantec wrote:

We have identified three root causes underlying the mis-issuance of these test certificates. First, we continued to issue internal test certificates to unregistered domains after the April 2014 change in the Baseline Requirements that removed authorization to do so. The overwhelming majority of the mis-issued test certificates fall into this first category. Second, certain Symantec Quality Assurance (QA) personnel had systems access, including the ability to use certain legacy tools, which enabled them to request a limited number of test certificates that were issued without review by authentication personnel. Third, authentication personnel did not consistently follow all verification steps when they received test certificate requests from their Symantec colleagues, or requested test certificates themselves.

Symantec took a number of remediation steps, as outlined in the report.

Issue E: Domain Validation Vulnerability (October 2015)

In October 2015, it was discovered that Symantec's DV certificate products (e.g. RapidSSL, QuickSSL) had a flaw when parsing email address data from WHOIS, in that they did not parse + and = characters correctly. This meant that, in cases where the domain owner had used these characters, the address as parsed by Symantec was not the one the domain owner had inserted, and if other email addresses of a particular form could be registered at that domain, there was a possibility that an attacker could get a cert for the domain.

Symantec Response

Symantec fixed the issue within a week, audited their logs to make sure the flaw had not been abused, and issued a security advisory. They have said they have no further comment.

Further Comments and Conclusion

The set of circumstances which would have allowed this issue to be exploited (+ or = character in WHOIS, domain where arbitrary email address registration by 3rd parties is possible, necessary email address still available to register) are relatively rare, and Symantec fixed the issue quickly and performed appropriate remediation.

Issue F: Symantec Audit Issues 2015 (December 2014 - November 2015)

Symantec's 2015 audit reports can be found in their legal repository. Symantec's standard audit period is from December 1st to November 31st.

The 2015 Baseline Requirements audits for Symantec's GeoTrust roots and their Symantec and Thawte roots run from December 1st, 2014 to November 30th, 2015. In those audits, the management assertions (and thereby the auditors) call out the following violations of the Baseline Requirements or Network Security Guidelines:

1. Issuance of Internal Server Names past the deadline date
2. Test certificates issued for domains Symantec did not own or control (see above)
3. No audit report, or invalid audit report, obtained for 3 of 5 external partner sub-CAs (GeoTrust only)
4. Failure to maintain physical security records for an appropriate period of time
5. Unauthorized employees with access to certificate issuance capability
6. Failure to review application and system logs

The 2015 WebTrust for CAs audits for Symantec's Verisign and own-brand roots, their Thawte roots and their GeoTrust roots run from December 1st, 2014 to November 30th, 2015. In those audits, the management assertions (and thereby the auditors) call out the following violations:

1. Background checks not renewed for trusted personnel
2. Unauthorized employees with access to certificate issuance capability
3. Failure to maintain physical security records for an appropriate period of time (GeoTrust only)
4. Test certificates issued for domains Symantec did not own or control (see above)

Of these, only the 'background checks' issue is not a repeat of an issue raised in the BR audits.

The 2015 Extended Validation audits for Symantec's Verisign and own-brand roots, their Thawte roots and their GeoTrust roots run from December 1st, 2014 to November 30th, 2015. In those audits, the management assertions (and thereby the auditors) call out the 'test certificates' and the 'physical security records' issues which are noted above.

Symantec Response

Each of the documents contains, in a following table, Symantec's comments on the qualifications and what they have done or are doing to remedy them. They have said that they have no further comment.

Further Comments and Conclusion

Mozilla did not object to these qualifications in Symantec's audits at the time the audit documentation was submitted to us. Because of this, it is not reasonable for us to take action based on the mere existence of these qualifications. They are listed here because they are one part of the general picture of Symantec's compliance or otherwise with the BRs.

Issue H: SHA-1 Issuance After Deadline (January 2016)

Symantec issued five SHA-1 certificates after the SHA-1 issuance deadline. These were certificates requested ("enrolled") in 2015 but issued in 2016. Therefore, they passed through the point in the process where the presence of SHA-1 was checked for before the check was enabled.

Symantec Response

Symantec self-reported this issue to the CAB Forum public mailing list, and recently stated that they have no further comment.

Further Comments and Conclusion

Symantec were not the only CA to have similar issues with accidental SHA-1 issuance. This issue is listed here because it was not the only time it happened (see below).

Issue J: SHA-1 Issuance After Deadline, Again (February 2016)

Symantec issued a SHA-1 CT precertificate in February 2016. They then seemed to claim that misissuance of a pre-certificate is not misissuance (The CT RFC states that issuance of a pre-certificate is considered equivalent to issuance of the certificate, and so Mozilla considers that pre-certificate misissuance is misissuance) although they now acknowledge that it was.

Symantec Response

Their explanation for the incident was as follows:

Following discussions with the customer who initiated this order, we have identified a technical deficiency in our system that allowed for hash algorithm modifications by a subset of customers to existing enrollments in limited circumstances, and only when pending administrator review prior to issuance. We released a patch today to add this case to our system-wide SHA1 blocking mechanisms. In addition, as an added precaution, we are evaluating an update to actively change any SHA1 orders encountered in our system to SHA256.

They have also produced a more detailed explanation of the events, including details of the remediation steps taken.

Further Comments and Conclusion

It is concerning that their first experience with SHA-1 misissuance did not cause them to analyse their systems and find this potential problem, or to put in place SHA-1 blocks in enough places to catch this.

Issue L: Cross-Signing the US Federal Bridge (February 2011 - July 2016)

The US Government has an extremely complicated PKI called the Federal PKI. It has applied for inclusion in the Mozilla root store but that application seemed unlikely ever to be successful due to the difficulty of bringing the entire FPKI in line with Mozilla's policies. At the time of this incident, it had a number of non-audited subordinate CAs.

Since February 2011, Symantec has regularly had a valid cross-sign for one or both of "Federal Bridge CA" and "Federal Bridge CA 2013", which are both part of the FPKI, thereby making certificates below those roots in the chain of trust be publicly trusted, and technically making Symantec responsible to Mozilla for all certificates issued in the covered part of the FPKI, including any BR violations. The intermediate CA certificate(s) concerned were not disclosed in the CCADB, as Mozilla practice at the time required. This was reported in m.d.s.policy.

Symantec is not the only CA to have done this; IdenTrust also did it on multiple occasions from 2011-01-14 onwards. I don't believe there are any unexpired unrevoked (by OneCRL) links between the FPKI and the Mozilla trust store any more, via any CA.

Symantec appear not to have been aware that this would lead to certificates from the FPKI being trusted in browsers until around the time it was drawn to their attention by Eric Mill in February 2016.

Symantec Response

When this was drawn to their attention, Symantec did not revoke the cross-sign certificate under discussion, instead allowing it to expire (less than a month later).

Symantec claim that the problem is with browsers not processing certificate policy extensions which are used within the FPKI. When they realised the problem, they negotiated with the FPKI to allow the relevant cross-cert to expire rather than renewing it.

Issue N: Premature Manual Signing Using SHA-1 (July 2016)

Symantec applied to the root programs for leave to issue SHA-1 certificates for a customer. As part of that process, they planned to prepare tbsCertificates ("To Be Signed Certificates" - all the data except for the signature) for CAB Forum review. However, they accidentally issued real certificates - i.e. they signed the tbsCertificate data using SHA-1. This was supposedly a manual process but it seems like the relevant instructions were either deficient (in that they specified too many steps) or not followed.

Symantec Response

Symantec self-reported this issue to Mozilla. Their remediation was to revoke all of those certificates, and issue new ones after permission was granted.

Symantec state:

This matter represents the first time any CA attempted to follow the exception process which was developed over the course of weeks, beginning at the Bilbao CABF face-to-face meeting in May 2016, and with the input of our partners. Google initially proposed this exception process, which was later modified following input from other CABF members. Our internal process did not clearly specify to our PKI Operations team to stop at the point of TBS creation, which subsequently resulted in the creation of signed certificates instead of TBS Certificates. Importantly, our audit process promptly identified the error, and Symantec never released the certificates. We also promptly improved our internal process.

Issue P: UniCredit Sub CA Failing To Follow BRs (April - October 2016)

Symantec issued an unconstrained sub-CA to a company called UniCredit. This company persistently issued certificates which were BR-noncompliant (e.g. missing SANs, missing OCSP URIs). During this time, they did not have the appropriate audits and Symantec were aware of this. Symantec claimed, in response to the CA Communication of March 2016, that they had disclosed to Mozilla all their intermediate CAs along with audits, but this was not true in the case of UniCredit. The UniCredit CPS was also BR-noncompliant, in that it said that SANs were optional.

This incident is recorded in bug 1261919.

Symantec Response

Symantec's initial response was to get UniCredit to put in place controls to fix the violations found, and to review and replace any affected certificates. However, they continued to be without an audit. Symantec eventually asked them for one, and when they were unable to produce (presumably, pass) one, ordered them to stop issuing. However they continued, in violation of that agreement. Symantec then finally revoked their intermediate.

Symantec admit that the omission of UniCredit from their CA Communication response was an oversight. They further state:

We worked with UniCredit over a long period of time to enforce their compliance with audit requirements. In July 2016, we received an assessment that did not meet WebTrust audit standards. We then took action, helping UniCredit transition to a managed PKI solution for their certificate needs that did not require an audit. In parallel, we notified them of termination of their subordinate CA.

Because our customers are our top priority, we attempted to minimize business disruption while they transitioned by permitting UniCredit to operate only its CRL service until November 30, 2016, at which point we would revoke the UniCredit subordinate CA. In October 2016, UniCredit issued one new certificate in violation of the terms of that transition plan. Following that, Symantec promptly revoked the UniCredit subordinate CA on October 18, 2016.

Issue Q: Symantec Audit Issues 2016 (December 2015 - November 2016)

The Baseline Requirements section 8.6 says that CAs SHOULD provide audits within 90 days of the end of the audit period; this SHOULD was not followed by Symantec for both the 2014/15 and 2015/16 audit cycles. However, Symantec is not the only CA which regularly supplies its audits late.

Symantec's 2016 audit reports can be found in their legal repository. Symantec's standard audit period is from December 1st to November 31st. However, for 2016, they have split the audits into two roughly six-month periods, and had separate audit opinions issued for each.

As detailed in their covering letter, the audits for the second period, June 16th to November 30th 2016, are mostly unqualified. The BR audits have a total of five qualifications, two of which relate to previously disclosed incidents which are not of concern, and three other qualifications which seem to be only of minor concern.

The audits for the first period contain many or all of the same issues as the 2015 audits (Issue F). One can surmise that Symantec chose to split the audits in order that they would not have a qualified audit covering the entirety of 2016. This does raise questions about how long the issues which led to these qualifications persisted.

Symantec Response

Each of the documents contains, in a following table, Symantec's comments on the qualifications and what they have done or are doing to remedy them.

Symantec recently stated: "In our 2014-2015 audits, certain issues were identified that we promptly took action on, including addressing the test certificate incident. We continued these efforts until the Point in Time audit was conducted." There are questions outstanding regarding the timeline here.

Issue R: Insecure Issuance API (2013 or earlier - November 2016)

According to a report, it is alleged that for several years Symantec operated an issuance API which was insufficiently secure, such that URL parameter substitution attacks would allow one customer to view, reissue, revoke and otherwise control certificates (including non-server certs) belonging to another customer. It is further alleged that, when made aware of these issues, Symantec took a very long time to fix them, and they may not have been fully fixed at the time Symantec terminated its RA program entirely.

Symantec Response

Symantec has not yet been formally asked by Mozilla to respond to this issue. However, they commented to the press:

"We have looked into Chris Byrne’s research claim and could not recreate the problem. We would welcome the proof of concept from the original research in 2015 as well as the most recent research. In addition, we are unaware of any real-world scenario of harm or evidence of the problem. However, we can confirm that no private keys were accessed, as that is not technically feasible.

In addition, Tarah from Symantec has posted a detailed comment which suggests that the issue is or was substantially less serious than the initial write-up made it sound. A discussion has ensued which I believe includes the original reporter, so we will wait to see if additional information emerges.

Issue T: RA Program Misissuances (January 2010 - January 2017)

For several years, Symantec operated an RA (Registration Authority) program. The companies in this program had independent authority to issue certificates under Symantec intermediates, and those certificates were not specifically marked to say that a particular RA had issued them instead of Symantec directly. This by itself is not against any existing policy.

However, In January 2017, it was discovered that a small number of certificates had been misissued using Symantec intermediates by an RA called CrossCert in Korea. The investigation of these matters led to increasing further discoveries of misissuances (totalling 127 confirmed cases) and potential misissuances by CrossCert, together with poor CA practice by them and other companies in Symantec's RA network. This issue deals with the certificate misissuances; the following issue deals with the audit-related problems uncovered.

Types of misissuance by CrossCert:

• Use of unvalidated domain names not owned by either Symantec or CrossCert
• Typos in domain names
• Bogus ST, L, O and OU fields: numbers, "test" or similar
• Violation of CPS (use of non-KR country code)

Some of these misissuance were caused by employees of CrossCert overriding compliance flags in Symantec's issuance system. Symantec had no process in place to review the logs of overridden flags. For some of the certs, they contained domains neither Symantec nor CrossCert own or control, and CrossCert did not complete the appropriate domain validations for them.

This incident is recorded in bug 1334377.

Symantec Response

Symantec made a number of comments on this issue - 0, 1, 2, 3, 4.

The Baseline Requirements, in section 4.9.1.1 item 9, state that the CA SHALL revoke a certificate if "The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Poliy or Certification Practice Statement". However, Symantec did not revoke all the certificates.

Instead, Symantec decided to shut down the RA program entirely and re-assess every certificate issued under it. Symantec committed to revalidating all of the CrossCert-issued certificates (10,000+) and any of the 20,000+ certificates issued by their other RAs if deficient validation was discovered. However, the determination of deficient validation was made based on the RAs own logs of activity, which may themselves be suspect given some of the audit deficiencies found at these RAs.

Symantec has produced a further, long account of the situation. It confirms that, while there were multiple systems designed to teach the RAs what to do, the only system which checked that they were actually doing it (and which Symantec paid attention to) were the WebTrust audits. They disagree that full revocation of all certs is a proportionate response.

Further Comments and Conclusion

Symantec's reaction to the discovery of these problems was unarguably swift and comprehensive. Their case is that WebTrust audit monitoring should have been sufficient, but that they were let down by their auditor, who failed to notice some of the problems, or in other cases it just so happened that the issues were either a long time ago or too recent to be caught by audit.

Issue V: GeoRoot Program Audit Issues (2013 or earlier - January 2017)

Symantec runs a program called GeoRoot, where intermediate CAs have been created for the sole use and independent operation by specific customers at premises under their control. These customers appear to have had a history of poor compliance with the BRs and other audit requirements, facts which were known to Symantec but not disclosed to Mozilla or dealt with in appropriately comprehensive ways.

Over multiple years (2013-12-01 to 2014-11-30, 2014-12-01 to 2015-11-30), Symantec's "GeoTrust" audits were qualified to say that they did not have proper audit information for some of these GeoRoot customers. This information was in their management assertions, and repeated in the audit findings. So the poor audit situation was ongoing and known.

There are five customers mentioned in the audits, and Symantec say they are Intel, Aetna, UniCredit, Google and Apple. They also say:

Separately, Symantec operates two subordinate CAs solely for NTT DoCoMo in an enterprise PKI application. These subordinate CAs had been considered part of the "GeoRoot" program as well, and we had therefore excluded them (similar to the above externally operated ones) from the list of Symantec CAs in our audits. After reviewing our approach, our compliance team determined that they should be included going forward. As such, for the 2016-2017 Period in Time, these subordinate CAs are included in the GeoTrust WebTrust for CA and BR audits.

Further Comments and Conclusion

Does this mean the NTT DoCoMo intermediates were entirely unaudited for a period of years?

Issue W: RA Program Audit Issues (2013 or earlier - January 2017)

We currently know of four RAs who were in Symantec's program - CrossCert, Certisign, Certsuperior, and Certisur.

Certsuperior's audit is particularly dreadful:

• There was no legible CPS;
• inadequately-trained personnel were doing issuance;
• the network was an insecure mess; and
• non-trusted staff had access to issuance.

CrossCert's audit does not list or cover the full number of Symantec roots under which they had issuance capability. Symantec's investigation discovered that CrossCert had the scope of the audit reduced for cost reasons.

Certisign's audit and Certisur's audit are only WebTrust for CAs audits - neither CA appears to have a Baseline Requirements audit. The WebTrust audit criteria require that such a CA has a BR audit. In addition, Mozilla policy requires "CA operations and issuance of certificates to be used for SSL-enabled servers" to conform to the Baseline Requirements. As Symantec has stated that audit was their only mechanism for monitoring their RAs, they can have had no assurance that RAs without a BR audit were actually following the BRs.

Symantec Response

Symantec required the issues at CertSuperior to be fixed and a 90-day action plan was executed to fix them. However, until they decided to shut down the RA program, no certificates issued during the period of suspect operations were checked to see if the poor practice had caused misissuance.

Despite the clear warning signs shown on the Certsuperior audit, Symantec did not put in place any monitoring of their RAs, other than audit, to check that they were correctly performing the tasks delegated to them under the BRs. There were some - overridable - technical checks on certificate issuance.

Symantec appears to have taken no action to deal with that fact that Certisign and Certisur did not have BR audits.

Symantec did not notice that CrossCert's audits did not cover all the relevant roots until they did the RA investigation in early 2017.

Issue X: Incomplete RA Program Remediation (February - March 2017)

At the time Symantec shut down their RA program, they had four RAs - CrossCert, Certisign, Certsuperior, and Certisur. Symantec committed to revalidating all certificates issued by those RAs. Independent of the rightness or otherwise of this course of action, it should have been applied consistently. However, the program seems to have previously had additional RAs, and Symantec has as yet taken no action to revalidate the certificates that they issued, despite some still being valid.

Those RAs include at least E-Sign, for which we have found audits from March - June 2014, April - July 2015 and July - September 2016. This party does not appear to have an unbroken series of audits; it is unclear what that means.

Another possible RA is MSC TrustGate, which appears to have been operating as a sub-CA based on the scope of their audit. However this audit, like those for some other Symantec RAs, is only to the WebTrust for CAs criteria and does not cover the Baseline Requirements. The WebTrust audit criteria require that such a CA has a BR audit.

Symantec Response

Symantec state:

The only Symantec RAs capable of authorizing and issuing publicly trusted SSL/TLS certificates are: CrossCert, Certisign, Certsuperior and Certisur. Symantec continues to maintain a partner program for non-TLS certificates. E-Sign SA and MSC Trustgate are amongst these partners.