CA/Symantec Issues

This page lists all confirmed or suspected issues involving the CA "Symantec". It will be updated by Mozilla as more information becomes available. Please do not edit this page yourself; if you have proposed changes, email Gerv.

Information here is correct to the best of Mozilla's knowledge and belief. Note that Symantec has not yet had a chance to respond to some of the issues presented here, and so we expect this information will be updated over time.

The issues are in broadly chronological order by end date.

Issue B: 1024-bit Certificate Issued Directly From Root (Dec 2013 - Jan 2014)

Symantec issued a cert to one of its customers, Pitney Bowes, that did not comply with at least two provisions of both the CA/Browser Forum Baseline Requirements and Mozilla policy. Firstly, it was issued directly from a root, and secondly it was a 1024-bit cert which expired after the end of 2013. Symantec believed this was the only technical way to ensure continuity of service for the customer concerned.

This cert was backdated, but that is not a BR or Mozilla policy violation, as long as it was not done to evade a technical control. It also has a short serial number. Entropy in the serial number is a SHOULD in the relevant version of the BRs (version 1.1.6). 20 bits of entropy is a MUST in the Mozilla policy (version 2.2), but it doesn't say it has to be in the serial number - it could be that they randomised the notBefore time. I am told Microsoft removed the allowance for doing entropy in the Date field on 11th November 2013, so this was a violation of their policies.

Symantec did not request permission to issue in advance, they disclosed the issuance at least a month after it had happened, and the replacement certificate (unlike the original) asserted a "BR Compliant" OID.

The incident is recorded in bug 966350.

Symantec Response

Symantec self-reported this issuance to Mozilla via Bugzilla at the end of January 2014. Even though a question was asked about the BR Compliance OID that the new cert asserted, they did not comment on the bug beyond the initial report.

Further Comments and Conclusion

The lack of discussion in advance, the delayed disclosure and the inclusion of a BR-compliant OID in a certificate Symantec knew was not BR-compliant are all disappointing.

Issue D: Test Certificate Misissuance (April 2009 - September 2015)

Between 2009 and 2015, Symantec issued a large number of test certificates in their publicly trusted hierarchies. These contained domains that Symantec did not own or control, and for which domain validation was not performed. Some of these domains were unregistered, and others were owned by other organizations. Issuing test certificates for unregistered domains was not a BR violation before April 2014, but Symantec continued the practice after it had been forbidden. The registered domains used included those belonging to Google and Opera Software. Given the numbers involved, this sort of test certificate issuance appears to have been common practice at Symantec. Some of the test certificates (including one for www.google.com) left Symantec's network because they were logged in CT; Symantec claims no others did. However, Symantec personnel would have had access to the public and private keys of the certs.

Some details of this incident are recorded in bug 1214321.

Symantec Response

It took quite a while for the full scale of the problem to emerge, and Symantec had to publish numerous updates to their "Final Report". Symantec released a number of documents relating to this - Final Report, Final Report v2, Final Report v3, list of certs containing owned domains, list of certs containing unregistered domains, further update. They also issued a blog post, "A Tough Day As Leaders", which is no longer on their website but has been archived. It indicates that 3 employees were fired as a result of the misissuance.

In their report, Symantec wrote:

We have identified three root causes underlying the mis-issuance of these test certificates. First, we continued to issue internal test certificates to unregistered domains after the April 2014 change in the Baseline Requirements that removed authorization to do so. The overwhelming majority of the mis-issued test certificates fall into this first category. Second, certain Symantec Quality Assurance (QA) personnel had systems access, including the ability to use certain legacy tools, which enabled them to request a limited number of test certificates that were issued without review by authentication personnel. Third, authentication personnel did not consistently follow all verification steps when they received test certificate requests from their Symantec colleagues, or requested test certificates themselves.

Symantec took a number of remediation steps, as outlined in the report.

Issue E: Domain Validation Vulnerability (October 2015)

In October 2015, it was discovered that Symantec's DV certificate products (e.g. RapidSSL, QuickSSL) had a flaw when parsing email address data from WHOIS, in that they did not parse + and = characters correctly. This meant that, in cases where the domain owner had used these characters, the address as parsed by Symantec was not the one the domain owner had inserted, and if other email addresses of a particular form could be registered at that domain, there was a possibility that an attacker could get a cert for the domain.

Symantec Response

Symantec fixed the issue within a week, audited their logs to make sure the flaw had not been abused, and issued a security advisory.

Further Comments and Conclusion

The set of circumstances which would have allowed this issue to be exploited (+ or = character in WHOIS, domain where arbitrary email address registration by 3rd parties is possible, necessary email address still available to register) are relatively rare, and Symantec fixed the issue quickly and performed appropriate remediation.

Issue F: Audit Issues For Symantec Itself (December 2014 - November 2015)

All of Symantec's current audit reports can be found in their legal repository. I don't believe they provide links to historic versions. Symantec's standard audit period is from December 1st to November 31st. We would therefore expect their 2016 audit to be available by now. However Symantec regularly only supplies their audit reports more than 180 days after the audit has been completed. The Baseline Requirements section 8.6 says that CAs SHOULD provide them in 90 days or fewer. Symantec is not the only CA which regularly supplies its audits late.

The most recent available Baseline Requirements audits for Symantec's GeoTrust roots and their Symantec and Thawte roots run from December 1st, 2014 to November 30th, 2015. In those audits, the management assertions (and thereby the auditors) call out the following violations of the Baseline Requirements or Network Security Guidelines:

1. Issuance of Internal Server Names past the deadline date
2. Test certificates issued for domains Symantec did not own or control (see above)
3. No audit report, or invalid audit report, obtained for 3 of 5 external partner sub-CAs (GeoTrust only)
4. Failure to maintain physical security records for an appropriate period of time
5. Unauthorized employees with access to certificate issuance capability
6. Failure to review application and system logs

The most recent available WebTrust for CAs audits for Symantec's Verisign and own-brand roots, their Thawte roots and their GeoTrust roots run from December 1st, 2014 to November 30th, 2015. In those audits, the management assertions (and thereby the auditors) call out the following violations:

1. Background checks not renewed for trusted personnel
2. Unauthorized employees with access to certificate issuance capability
3. Failure to maintain physical security records for an appropriate period of time (GeoTrust only)
4. Test certificates issued for domains Symantec did not own or control (see above)

Of these, only the 'background checks' issue is not a repeat of an issue raised in the BR audits.

The most-recently available Extended Validation audits for Symantec's Verisign and own-brand roots, their Thawte roots and their GeoTrust roots run from December 1st, 2014 to November 30th, 2015. In those audits, the management assertions (and thereby the auditors) call out the 'test certificates' and the 'physical security records' issues which are noted above.

Symantec Response

Each of the documents contains, in a following table, Symantec's comments on the qualifications and what they have done or are doing to remedy them.

Further Comments and Conclusion

Mozilla did not object to these qualifications in Symantec's audits at the time the audit documentation was submitted to us. Because of this, it is not reasonable for us to take action based on the mere existence of these qualifications. They are listed here because they are one part of the general picture of Symantec's compliance or otherwise with the BRs.

Issue H: SHA-1 Issuance After Deadline (January 2016)

Symantec issued five SHA-1 certificates after the SHA-1 issuance deadline. These were certificates requested ("enrolled") in 2015 but issued in 2016. Therefore, they passed through the point in the process where the presence of SHA-1 was checked for before the check was enabled.

Symantec Response

Symantec self-reported this issue to the CAB Forum public mailing list.

Further Comments and Conclusion

Symantec were not the only CA to have similar issues with accidental SHA-1 issuance. This issue is listed here because it was not the only time it happened (see below).

Issue J: SHA-1 Issuance After Deadline, Again (February 2016)

Symantec issued a SHA-1 CT precertificate in February 2016. They then seemed to claim that misissuance of a pre-certificate is not misissuance. (The CT RFC states that issuance of a pre-certificate is considered equivalent to issuance of the certificate, and so Mozilla considers that pre-certificate misissuance is misissuance.)

Symantec Response

Their explanation for the incident was as follows:

Following discussions with the customer who initiated this order, we have identified a technical deficiency in our system that allowed for hash algorithm modifications by a subset of customers to existing enrollments in limited circumstances, and only when pending administrator review prior to issuance. We released a patch today to add this case to our system-wide SHA1 blocking mechanisms. In addition, as an added precaution, we are evaluating an update to actively change any SHA1 orders encountered in our system to SHA256.

Further Comments and Conclusion

It is concerning that their first experience with SHA-1 misissuance did not cause them to analyse their systems and find this potential problem, or to put in place SHA-1 blocks in enough places to catch this.

Issue L: Cross-Signing the US Federal Bridge (December 2015 - July 2016)

The US Government has an extremely complicated PKI called the Federal PKI. It has applied for inclusion in the Mozilla root store but that application seemed unlikely ever to be successful due to the difficulty of bringing the entire FPKI in line with Mozilla's policies. At the time of this incident, it had a number of non-audited subordinate CAs.

Presumably in November or December of 2015, Symantec cross-signed one of the root CAs in the FPKI ("Federal Bridge CA 2013"), thereby making certificates below that root in the chain of trust be publicly trusted, and technically making Symantec responsible to Mozilla for all certificates issued in the covered part of the FPKI, including any BR violations. The intermediate CA certificate concerned was not disclosed in the CCADB, as Mozilla practice at the time required. This was reported in m.d.s.policy.

Symantec is not the only CA to have done this; IdenTrust also did it. Their cross-sign's notBefore is in January 2015 and they revoked it on 17th February 2016, two years before it was due to expire.

Symantec Response

When this was drawn to their attention, Symantec did not revoke the cross-sign, instead allowing it to expire (less than a month later).

Issue N: Premature Manual Signing Using SHA-1 (July 2016)

Symantec applied to the root programs for leave to issue SHA-1 certificates for a customer. As part of that process, they planned to prepare tbsCertificates ("To Be Signed Certificates" - all the data except for the signature) for CAB Forum review. However, they accidentally issued real certificates - i.e. they signed the tbsCertificate data using SHA-1. This was supposedly a manual process but it seems like the relevant instructions were either deficient (in that they specified too many steps) or not followed.

Symantec Response

Symantec self-reported this issue to Mozilla. Their remediation was to revoke all of those certificates, and issue new ones after permission was granted.

Issue P: UniCredit Sub CA Failing To Follow BRs (April - October 2016)

Symantec issued an unconstrained sub-CA to a company called UniCredit. This company persistently issued certificates which were BR-noncompliant (e.g. missing SANs, missing OCSP URIs). During this time, they did not have the appropriate audits and Symantec were aware of this. Symantec claimed, in response to the CA Communication of March 2016, that they had disclosed to Mozilla all their intermediate CAs along with audits, but this was not true in the case of UniCredit. The UniCredit CPS was also BR-noncompliant, in that it said that SANs were optional.

This incident is recorded in bug 1261919.

Symantec Response

Symantec's initial response was to get UniCredit to put in place controls to fix the violations found, and to review and replace any affected certificates. However, they continued to be without an audit. Symantec eventually asked them for one, and when they were unable to produce (presumably, pass) one, ordered them to stop issuing. However they continued, in violation of that agreement. Symantec then finally revoked their intermediate.

Issue R: Insecure Issuance API (2013 or earlier - November 2016)

According to a report, it is alleged that for several years Symantec operated an issuance API which was insufficiently secure, such that URL parameter substitution attacks would allow one customer to view, reissue, revoke and otherwise control certificates (including non-server certs) belonging to another customer. It is further alleged that, when made aware of these issues, Symantec took a very long time to fix them, and they may not have been fully fixed at the time Symantec terminated its RA program entirely.

Symantec Response

Symantec has not yet been formally asked by Mozilla to respond to this issue. However, they commented to the press:

"We have looked into Chris Byrne’s research claim and could not recreate the problem. We would welcome the proof of concept from the original research in 2015 as well as the most recent research. In addition, we are unaware of any real-world scenario of harm or evidence of the problem. However, we can confirm that no private keys were accessed, as that is not technically feasible.

In addition, Tarah from Symantec has posted a detailed comment which suggests that the issue is or was substantially less serious than the initial write-up made it sound. A discussion has ensued which I believe includes the original reporter, so we will wait to see if additional information emerges.

Issue T: RA Program Misissuances (January 2010 - January 2017)

For several years, Symantec operated an RA (Registration Authority) program. The companies in this program had independent authority to issue certificates under Symantec intermediates, and those certificates were not specifically marked to say that a particular RA had issued them instead of Symantec directly. This by itself is not against any existing policy.

However, In January 2017, it was discovered that a small number of certificates had been misissued using Symantec intermediates by an RA called CrossCert in Korea. The investigation of these matters led to increasing further discoveries of misissuances (totalling 127 confirmed cases) and potential misissuances by CrossCert, together with poor CA practice by them and other companies in Symantec's RA network. This issue deals with the certificate misissuances; the following issue deals with the audit-related problems uncovered.

Types of misissuance by CrossCert:

• Use of unvalidated domain names not owned by either Symantec or CrossCert
• Typos in domain names
• Bogus ST, L, O and OU fields: numbers, "test" or similar
• Violation of CPS (use of non-KR country code)

Some of these misissuance were caused by employees of CrossCert overriding compliance flags in Symantec's issuance system. Symantec had no process in place to review the logs of overridden flags. For some of the certs, they contained domains neither Symantec nor CrossCert own or control, and CrossCert did not complete the appropriate domain validations for them.

This incident is recorded in bug 1334377.

Symantec Response

Symantec made a number of comments on this issue - 0, 1, 2, 3, 4.

The Baseline Requirements, in section 4.9.1.1 item 9, state that the CA SHALL revoke a certificate if "The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Poliy or Certification Practice Statement". However, Symantec did not revoke all the certificates.

Instead, Symantec decided to shut down the RA program entirely and re-assess every certificate issued under it. Symantec committed to revalidating all of the CrossCert-issued certificates (10,000+) and any of the 20,000+ certificates issued by their other RAs if deficient validation was discovered. However, the determination of deficient validation was made based on the RAs own logs of activity, which may themselves be suspect given some of the audit deficiencies found at these RAs.

Further Comments and Conclusion

When Symantec put various controls and restrictions in place following the previous "test cert" incident, those controls, checks and restrictions did not extend to their RA network. Symantec say that this is because the test tool used in the previous incident was not available to RAs; however, it does not seem to be a great leap to have looked for similar capabilities and problems elsewhere in their issuance process.

Issue V: RA Program Audit Issues (2013 or earlier - January 2017)

Symantec's RAs appear to have had a history of poor compliance with the BRs and other audit requirements, facts which were known to Symantec but not disclosed to Mozilla or dealt with in appropriately comprehensive ways.

Over multiple years (2013-12-01 to 2014-11-30, 2014-12-01 to 2015-11-30), Symantec's "GeoTrust" audits were qualified to say that they did not have proper audit information for some of these RAs. This information was in their management assertions, and repeated in the audit findings. So the poor audit situation was ongoing and known. Also, other audit reports, despite being in hierarchies accessible for issuance by the same RAs, did not have similar qualifications (Symantec Trust Network, 2014-12-01 to 2015-11-30).

We currently know of four RAs who were in Symantec's program - CrossCert, Certisign, Certsuperior, and Certisur.

Certsuperior's audit is particularly dreadful:

• There was no legible CPS;
• inadequately-trained personnel were doing issuance;
• the network was an insecure mess; and
• non-trusted staff had access to issuance.

CrossCert's audit does not list or cover the full number of Symantec roots under which they had issuance capability. Symantec's investigation discovered that CrossCert had the scope of the audit reduced for cost reasons.

Certisign's audit and Certisur's audit are only WebTrust for CAs audits - neither CA appears to have a Baseline Requirements audit. The WebTrust audit criteria require that such a CA has a BR audit. In addition, Mozilla policy requires "CA operations and issuance of certificates to be used for SSL-enabled servers" to conform to the Baseline Requirements. As Symantec has stated that audit was their only mechanism for monitoring their RAs, they can have had no assurance that RAs without a BR audit were actually following the BRs.

Symantec Response

Symantec required the issues at CertSuperior to be fixed and a 90-day action plan was executed to fix them. However, until they decided to shut down the RA program, no certificates issued during the period of suspect operations were checked to see if the poor practice had caused misissuance.

Despite the clear warning signs shown on the Certsuperior audit, Symantec did not put in place any monitoring of their RAs, other than audit, to check that they were correctly performing the tasks delegated to them under the BRs. There were some - overridable - technical checks on certificate issuance.

Symantec appears to have taken no action to deal with that fact that Certisign and Certisur did not have BR audits.

Symantec did not notice that CrossCert's audits did not cover all the relevant roots until they did the RA investigation in early 2017.

Issue X: Incomplete RA Program Remediation (February - March 2017)

At the time Symantec shut down their RA program, they had four RAs - CrossCert, Certisign, Certsuperior, and Certisur. Symantec committed to revalidating all certificates issued by those RAs. Independent of the rightness or otherwise of this course of action, it should have been applied consistently. However, the program seems to have previously had additional RAs, and Symantec has as yet taken no action to revalidate the certificates that they issued, despite some still being valid.

Those RAs include at least E-Sign, for which we have found audits from March - June 2014, April - July 2015 and July - September 2016. This party does not appear to have an unbroken series of audits; it is unclear what that means.

Another possible RA is MSC TrustGate, which appears to have been operating as a sub-CA based on the scope of their audit. However this audit, like those for some other Symantec RAs, is only to the WebTrust for CAs criteria and does not cover the Baseline Requirements. The WebTrust audit criteria require that such a CA has a BR audit.

Symantec Response

Symantec has not yet been officially requested to respond to this issue.