The ICSI Certificate Notary

The Matter of Heartbleed

Nov 7, 2014

Today, a paper titled The Matter of Heartbleed which examines the aftermath of the Heartbleed bug will be presented at the ACM Internet Measurement Conference (IMC) 2014. The paper was authored by Zakir Durumeric, Frank Li, James Kasten, Johanna Amann, Jethro Beekman, Mathias Payer, Nicholas Weaver, David Adrian, Vern Paxson, Michael Bailey, and J. Alex Halderman.

Among other active and passive measurements, data from the ICSI SSL Notary was used for a few measurements in the paper.

Examining data from the Notary allowed us to witness pre-disclosure patching activity. The Notary stores information about the TLS extensions that are sent by the client as well as the server for each of the connections. This enabled us to go back into our data and check if a server supported the Heartbeat TLS extension. Notary data shows that Google started disabling Heartbeat support for its servers at least 12 days prior to public disclosure. Akamai started disabling the extension at least 4 days prior to disclosure.

In a second measurement, we examined how often servers changed certificates in the month directly proceeding Heartbleed, as well as in April 2014, when Heartbleed happened. To be more exact, we start our measurement at the 6th of March and the 6th of April, the day before the Heartbleed bug was published. We then track the number of servers that still serve the same certificate X days after the 6th. The resulting plot can be seen in the following figure.


In total, for April, 47% of sites in the Notary still sent the same certificate that they sent before disclosure of the Heartbleed bug. In contrast, for the month of March, 85% of sites still sent the same certificate at the end of the month. This shows that a significant percentage of the sites which are accessed by the Notary user took action because of Heartbleed, especially in the light that a lot of the servers that did not change their certificates might not have been vulnerable.

The SSLv3 POODLE Attack

Oct 16, 2014

On Tuesday, October 14th 2014, Google published details about a vulnerability in the design of SSLv3 that was found by Bodo Möller, Thai Duong and Krzysztof Kotowicz (all Google). The vulnerability allows a Man-in-the-middle (MITM) attacker to recover the plaintext from an encrypted network connection.

Adam Langley posted an in-depth explanation of the bug on his blog. In a nutshell, an attacker can extract one byte of plaintext from the network traffic using about 256 requests. This makes it feasible to recover text which always is at the same fixed locations in the datastream, like e.g. cookies, from encrypted SSLv3 connections. For an attack like this, the attacker has to coerce a client into creating a steady stream of new connections to the same URL. This can easily be achieved by a MITM attacker, for example, by injecting a javascript that performs the requests on an unrelated page that uses unencrypted HTTP.

If the client and the server both support higher versions than SSLv3, the attacker also has to perform a protocol downgrade attack. This attack uses the fact that, when a connection fails, SSL/TLS clients try re-establishing an encrypted connection with a lower protocol version. This behavior was added to deal with faulty servers which cannot deal with recent TLS versions and misbehave by, e.g., just closing the connection when a client tries to use a TLS version they do not support.

Current TLS version use

To judge the impact that this vulnerability has, we use connection data from the ICSI SSL Notary to estimate how SSLv3 is still being used in the wild at the moment.

For our analysis, we looked at the 12,795,585,451 connections that were observed by the Notary in the last 2 months. For each of the connections, we examine which SSL/TLS version was used. The results are summarized in the following table:

VersionConnectionsServer IPs
TLS 1.27,801,854,918489,743
TLS 1.1159,124,190115,932
TLS 1.04,760,762,827629,654
SSL 373,843,19780,454
SSL 231944

As we can see, during the 2 month period 73,843,197 (0.6%) of the observed connections use SSL version 3. The connections are established to 80,454 different servers, which is 9% of the total servers we saw during these 2 months. We also saw TLS connections to most of these servers. There were just 6,532 servers where we only saw SSLv3 and no TLS connections being established. These servers thus probably do not support TLS at all.

Mitigation and TLS_FALLBACK_SCSV

There are two ways to mitigate this issue. The first, probably preferable way, is to completely disable SSLv3 support on clients and on servers. The second method is to support TLS_FALLBACK_SCSV.

This proposal prevents the aforementioned protocol downgrade attacks by defining a Fallback Signaling Cipher Suite Value (SCSV). This value is sent by clients in their client hello message when retrying a connection with a lower protocol version after a previous connection attempt failed. When a server supports the proposal, it will reject any connection containing the SCSV when the protocol version is lower than the highest protocol version that it supports.

To illustrate this in an example: Browser A tries to connect to server S using TLS 1.2. The server resets the connection and A tries to connect again, this time using TLS 1.1 sending TLS_FALLBACK_SCSV in its client hello. This time, the connection succeeds. Since S only supports TLS up to 1.1 it accepts the connection. If S would also support TLS 1.2 it would assume that a protocol downgrade attack is in progress and reject the connection.

Since the SCSV is only sent when a client already downgraded the protocol version, we cannot use our data to determine how widespread adoption of it is. However, Google Chrome, which has quite a bit market share at the moment has supported TLS_FALLBACK_SCSV since February, we can use our data for an educated guess of how often this fallback is currently needed in the wild.

Looking at our 2 months of data revealed just 334,075 connections that set the signaling value. This equals to less than 500 connections per hour over the more than 100,000 users of our data providers. Hence, since it is probable that Firefox and other big browser vendors will add support for SCSV in the near future, monitoring your network for high connection rates that use it might expose attacks that are in progress.

Dissecting the GnuTLS Bug

Mar 5, 2014

The recent GnuTLS certificate verification bug made it possible to craft an arbitrary certificate in a way that GnuTLS would validate correctly against a given CA root certificate store.

Since we started our SSL Notary service about a year ago, whenever a big SSL or TLS security incident happens, we scan all the certificates we have seen so far to see if they could have been used for an attack. For this bug, we dived into the GnuTLS source, examined the conditions that trigger the bug, and figured how an attacker would exploit the bug. Based on our analysis, we believe that there exist multiple methods to trigger the bug. In the following, we describe one that worked.

Looking at the patch that fixes this bug, we observe that the return codes for error cases were not set properly in a few examples. For example, the function _gnutls_verify_certificate2 performs the actual certificate verification starting at line 465. GnuTLS first attempts to extract the part of the certificate that is signed, the signature, and the signature algorithm. If any of these lookups fail, the verification still succeeds due to missing adjustments of the variable result, which denotes the function return value. On success, the function should return a non-zero value, but the code lacked checks to set the value to zero.

From an attacker perspective, making the signature lookup fail is probably the easiest method to exploit the bug. According to the function _gnutls_x509_get_signature, the signature verification fails if the number of bits in the certificate signature is not divisible by 8. Certificate signatures have an ASN.1 bit string encoding, which theoretically may exhibit an arbitrary number of zeros and ones, and may not necessarily be divisible by 8. But the GnuTLS function assumes a byte-aligned signature, hence the check. In the ASN.1 DER encoding, used in X509 certificates, a bit string begins with a single byte which denotes the number of missing bits in the last byte. For all normal certificates, this value is always zero, and the function continues. But if we change this byte to a non-zero value, the function fails the divisibility check and returns with wrong error code, causing the certificate verification to succeed.

Moreover, we also looked at a few other entry points with the goal to make the signature algorithm verification fail. However, none of them seemed to be as easy to exploit as the signature algorithm.

To trigger the bug, we patched the certificate for, exchanged the private key, and tried to validate it with GnuTLS. Indeed, the verification succeeds with gnutls-certtool and we could establish a secure connection to the server using gnutls-cli.

You can test the vulnerability of your GnuTLS installation by trying to connect to our test server:

gnutls-cli -p 443 --x509cafile [root store]

You also can test it by downloading the following certificate chain and verifying it using

gnutls-certtool --verify --infile exploit.pem --load-ca-certificate [root store]

Usually, you can find your operating system root store in /etc/ssl, or you can use our root-store copy. Note that the output of gnutls-certtool is slightly truncated when using a chain that exploits the bug, nevertheless certificate verification (including hostname verification) works.

At the moment we are scanning all the certificates contained in the ICSI SSL Notary to see if there is any certificate that specifies a non 8-bit-divisible signature. Due to the size of our data set - at the moment it consists or more than 1.8 million certificates extracted from more than 50 billion connections - the scan is still running; so far there have been no hits.

Searching the ICSI Notary for Rogue Turktrust Intermediate Certificates

Jan 4, 2013

Turktrust, a Certificate Authority (CA) that is trusted by all major browsers and systems accidentally issued intermediate CA certificates instead of end-host certificates to two of its clients. Both of these intermediates were valid and signed by the Turktrust root, and hence they could be used to sign certificates for any site on the Internet. Having access to an intermediate CA certificate also makes it possible to mount difficult to detect man-in-the-middle attacks on SSL connections.

According to Turktrust, both of the certificates were created on the 8th of August, 2011.

At least one of the certificates was used to create a rogue * certificate that was used for a man-in-the-middle attack. Google became aware of the certificate because Chrome detected it, blocked the access and sent a report back to Google.

We examined the data of the ICSI Certificate Notary: since the start of our data collection effort in February no user at any of the sites we monitor did encounter either of the two intermediate certificates. That’s good news and suggests that the intermediate certificates were indeed not used in a wide-scale attack, but only on a local gateway interface like stated by Turktrust.

Updates to distrust the rogue intermediate CA certificates have been pushed by all major Browser and Operating System vendors.

The Tree of Trust

Dec 12, 2012

The Tree of Trust

To better understand the relationships between root and intermediate CAs, we extracted from each certificate chain the path of intermediate CAs up to the root and then merged all paths into one single graph—the tree of trust.

In this graph, each node represents a CA, where red nodes correspond to root CAs and green nodes to intermediates. The node diameter scales logarithmically with the number of certificates signed by the node. Similarly, the color of the green nodes scales proportional to the diameter. For example, large and deep green nodes represent intermediate CAs which have signed a large number of certificates, as opposed to small and light green nodes.

We provide an interactive version of the tree of trust for easier navigation. Hovering over a node highlights all intermediate CAs for the corresponding root CA. Clicking on a node displays additional information in a side bar, including the exact number of certificates issued by that CA, the value of the issuer field, the validity period, key size and signature algorithm of the associated public key. Moreover, the search bar allows for quick location of a CA by name.