Skip to the letters
Machine Learning

eMudhra's Weak CA Keys: What Live Traffic Shows Weeks After Disclosure

from Maxime Guerreiro 856 reads

Traffic data from Cloudflare shows forged certificates appearing within hours of disclosure, and more than half of probes from large networks still accepting the compromised chain.

eMudhra's Weak CA Keys: What Live Traffic Shows Weeks After Disclosure

The eMudhra problem started with a key generation error that looks almost absurd on paper. A certificate authority meant to protect internet trust had intermediate keys with values as low as 4. The full list of weak values was 4, 56, 66, 115, 155, 156, 230 and 237. Anyone who could read those values from public certificates could recover the signing keys. That is what makes this incident different from earlier certificate authority failures like DigiNotar, where access to the keys stayed contained to a limited set of parties.

Cloudflare has now published a follow-up on two things: whether it has seen the weak keys being used, and how many visitors still trust them. The short answer on the first is that forged certificates have appeared, but there's no sign yet that anyone used them to intercept Cloudflare's own connections. The second answer is less comfortable.

Patching the high-impact systems first

Cloudflare started patching on October 19, beginning with origin-facing proxies built on Pingora, which handle most of its outbound connections. Those systems were patched within hours to reject the weak keys. In parallel, the team ran an AI-assisted analysis of its source code to find every other system that trusted them. That analysis finished within four hours. Each finding got a ticket with a preliminary impact assessment, and engineers confirmed or adjusted it the same day. High-impact items were patched immediately, and lower-impact work was scheduled for the end of the following week.

As of writing, every high-priority system is patched, and a small number of low-impact tickets remain open.

What the outbound logging found

To see whether anything was presenting these certificates, Cloudflare added logging to Pingora, which carries the vast majority of outgoing TLS connections from its machines. That logging finished rolling out on October 21. Over the following week, the company recorded about 22,000 handshakes that presented certificates chained to the weak keys.

Roughly two-thirds were fetches of test domains, including the public test site that eMudhra's exposure prompted developers to check. Nearly all of the remainder went to origin servers that appeared to have issued certificates to themselves under the weak intermediate. Every handshake configured to verify the chain was refused.

About 160 handshakes looked different. Their certificates listed Subject Alternative Names belonging to other organizations, none of which matched the hostname the client requested. Some of those names belonged to consumer IoT products. Several of these certificates were issued after the keys became public, and some were valid for more than two years, which is longer than any public CA is allowed to issue.

Still, the pattern doesn't look like interception. The names didn't match the requested hosts, so these connections would have failed hostname verification anyway. In several cases, a single server presented the same certificate from many Cloudflare locations, which suggests the certificate came from the server itself and not from a device sitting on a network path. One forged certificate was dated October 20 at 18:19 UTC, less than an hour after Mozilla's bug report went public. That timing shows how quickly the weakness was being exploited in practice.

Cloudflare is careful about what this rules out. Carefully targeted interception would leave few traces in this data, and detecting it would depend on the usual indirect signs of a compromised system. Those signs haven't appeared so far.

How many visitors still trust the keys

Checking visitor devices is harder, because a direct test would break the connections being tested. Cloudflare instead used its challenge pages. A fraction of pages sent two probe connections: one to a host presenting an eMudhra-signed certificate, and one to a control host with a normally signed certificate. If the probe succeeded, the visitor's client trusted the weak chain.

Across about 50 million probes in a week, the results were mixed, and the measurement changed during the period. Cloudflare altered the test endpoints several times, removed certificate timestamps from one endpoint, rolled back a challenge platform version for an unrelated issue, and later added a missing cross-sign to both endpoints. Those changes produced jumps in the data that aren't about patching. Cloudflare also flagged a large influx of bot traffic claiming to be Chrome 150 through 154 on macOS, which started appearing on October 29. That traffic inflated apparent acceptance rates, so the bot signal had to be read separately.

The broad picture holds up after those adjustments. Current desktop browsers are mostly patched. Bots and older browsers mostly aren't. Embedded browsers such as Android WebView lag badly, with 30 to 40 percent still accepting the chain. Chrome 150 through 154 on Linux is the one cohort whose acceptance rate keeps falling, which fits real patching, though the traffic mix makes that evidence suggestive rather than conclusive.

Breaking results down by autonomous system tells a similar story. Across large networks, more than half of probes still accepted the compromised chain, whether or not bot signals were present. Some patching showed up over the first weekend, after which the trend went flat.

That's the uncomfortable part. Bots are software too, and they need security updates just as browsers do. If their patch rate is any guide to software updating in general, the exposure window will be long.

What to do now

The fix is patching. The community-maintained crowdsourced guide from Dennis Jackson lists platform-specific steps, and developers on macOS should follow the Apple-specific instructions, since that platform appeared prominently in the bot traffic. Cloudflare customers using WARP or Tunnel have separate steps to follow, which were covered in its earlier guidance.

eMudhra has said it is exiting the public CA business and attributed the error to a mistake in a key generation script. It hasn't published that script yet, and Cloudflare notes that questions remain about how the error passed through audits and cross-signing. Those questions matter for anyone who builds trust decisions on top of certificate chains.

The lasting lesson for trust stores

This incident also exposes a design assumption. A trust store that behaves as a static list of root authorities can't respond fast enough when one of them fails. Cloudflare points to projects like upki, which treat a trust store as something that must receive revocation and Certificate Transparency updates quickly. Clients that don't enforce Certificate Transparency were the ones accepting forged certificates here, and that gap is where the next version of this problem will land. Vulnerabilities are being found faster than they used to be, so the update path for trust decisions will matter as much as the keys themselves.

Source: Maxime Guerreiro · blog.cloudflare.com

Same correspondent