Cloudflare's June 2024 Incident: Cookies, Tail Calls, and Runaway CPU
A careful postmortem of how one new DDoS rule, two interacting bugs, and a Lua tail-call loop burned CPU across Cloudflare's network for 114 minutes.
Cloudflare’s traffic-shifting systems performed exactly as designed during the June 20, 2024 incident — and that’s what spread the damage. Every failover that moved requests away from struggling servers carried the poison requests along with them.
Source
Cloudflare incident on June 20, 2024
By: Cloudflare (Lloyd Wallis, Julien Desgats, and Manish Arora)
Published: Jun 26 2024
Type: Engineering article
I saved this because it’s a rare postmortem where the escalation path runs through the safety systems themselves: a latent bug, a new rule that found it, and load balancers that faithfully distributed the failure worldwide. It also features an unusual villain — a language-level optimization, the tail call, sitting at the center of a global incident.
How a DDoS rule poisoned Cloudflare’s request handlers
On Thursday, June 20, 2024, two independent problems hit Cloudflare within minutes of each other, degrading latency and error rates for 114 minutes. During the worst 30 minutes, 1.4–2.1% of HTTP requests to the CDN got a generic error page, and 99th-percentile time-to-first-byte latency roughly tripled. The first problem — network monitoring rerouting traffic suboptimally, congesting the backbone from 17:33 to 17:50 UTC — was already being fixed with capacity expansion. The postmortem concentrates on the second; so will this read.
Two pieces of background make the failure followable. First, rate limiting: Cloudflare caps how many requests a visitor can make in a time window, either through customer configuration or when DDoS rules flag suspicious activity. These limits usually key on IP address — a broad brush, since whole institutions and ISPs share single IPs.
Second, traffic balancing. Unimog, the edge load balancer, touches every packet entering the anycast network and hands it to a server with available capacity — possibly elsewhere — while keeping CPU even within each data center. Above it, Traffic Manager watches CPU, latency, and bandwidth across all locations and rebalances traffic between data centers, with built-in safety limits against outsized shifts.
Between 14:14 and 17:06 UTC that day, Cloudflare gradually activated a new DDoS rule built on a new mitigation approach: combine rate limits with cookies, so that legitimate clients falsely caught up in attack mitigation can proceed anyway. A flagged request is first checked for a valid cookie and blocked without one; with a valid cookie, a rate-limit rule keyed on the cookie value is added for evaluation later, after all other mitigations run. Deferring the rate-limit work is deliberate — blocking early is cheaper, and it leaves room for other rule types to act.
The failure sits in how that deferred work builds its key. Rate limiting needs a per-client key — normally the IP, here the cookie value — and the engineers reused existing key-generation logic for it. Two flaws met there. The DDoS rules used internal APIs in ways nobody had anticipated, which left the “parent” key generator pointing at the very function calling it — an infinite self-call. And the reused code re-validated the cookie with a second, different validation function, one that can disagree with the first when a client sends multiple cookies, some valid and some not. The broken validator reports the cookie invalid, execution takes the branch that calls the parent generator, and the function chases its own tail.
Here’s the punchline. The logic is written in Lua, and Lua implements proper tail calls: when a function’s final action is to call another function, the runtime replaces the current stack frame instead of adding a new one. The recursion-depth protection that would normally turn this into a stack-overflow error never engages. The loop runs at 100% CPU, forever, without growing the stack or throwing — a poisoned process that simply never comes back.
Each server runs dozens of these processes, so one poisoned process is survivable; the balancers turned it into an outage. Rising CPU makes Unimog steer new traffic toward other servers; rising data-center-wide CPU makes Traffic Manager redirect whole locations. The redirected traffic includes the requests doing the poisoning, so destinations start failing the same way. Within minutes many data centers carried many poisoned processes, and Traffic Manager had moved all it could within its safety limits and the shrinking pool of healthy capacity. By 18:09 UTC it was rerouting heavily out of Europe, where the incident had eaten 10% of capacity in the West and 4% in the East during those regions’ peak hours.
The response fought through two misdirections. The incident auto-declared at 18:04 UTC on sustained global CPU, while many responders were already working the backbone congestion. Stranger still, CPU was highest exactly where traffic was lowest — the rerouting had inverted the normal overload signature and pulled investigators the wrong way. The work then split: prove restarts recover poisoned processes, and find the trigger. Restarts were validated on a sample server 25 minutes after declaration; mass restarts followed, first by data center, then targeted at the worst servers. At 19:29 UTC the DDoS rule was identified as the likely cause and disabled shortly after. Error rates peaked at 2.1% service-unavailable (3.45% counting origin errors) at 19:05 UTC, halved by 19:11, back to baseline at 19:27.
Traffic Manager had its own bad day. The incident conditions tripped a latent bug that made it attempt a graceful restart and go quiet — first at 18:17 UTC, then repeatedly between 18:35 and 18:57 — leaving two windows where it issued no routing actions at all while recovered data centers still had traffic routed away. The Traffic team had a fix written, tested, and deployed by 19:05 UTC.
The rule stays fully rolled back until the cookie-validation check is fixed. The follow-ups name the structural issues: the rate-limiting code inside the DDoS module is a legacy component (customers’ own rate-limit rules already run a newer engine); Cloudflare is exploring ways to bound tail-call loops, and longer term replacing the service with one that enforces limits on uninterrupted and total execution time per request; and staging will tighten further. That last point is worth dwelling on — the rule was staged in a few data centers before global activation, and it didn’t help, because geographic staging can’t catch a failure that needs one specific request shape, a client sending a mix of valid and invalid cookies, to detonate.
My take
The reframe this post quietly hands you: the safety systems were the distribution network. Health-based rerouting assumes traffic is innocent — that moving requests away from a struggling server moves them toward health. When the failure travels inside the request itself, every layer of failover becomes a courier for it. Unimog and Traffic Manager didn’t malfunction; they industrialized the spread. Any architecture with automatic, health-driven traffic shifting should ask what happens when the traffic is the pathogen.
Below that sits a quieter lesson about reuse. Neither bug was harmful alone: a parent pointer aimed wrong by unanticipated API usage, and two cookie validators that only disagree on mixed-cookie requests. The new DDoS feature inherited the first flaw and supplied the trigger for the second. Reusing internal interfaces in new contexts re-tests nothing — the contracts were never written down, so nothing flagged the new usage pattern.
And the tail-call detail earns its place in the title. An optimization designed to make recursion cheap silently deleted the failure mode engineers rely on to catch runaway recursion. Protections you assume the language provides are worth verifying in the language you actually run.
Where this is useful
- Running Lua — or any language with proper tail calls — in a request path: assume infinite self-recursion won’t trip a stack limit, and enforce wall-clock or CPU-time budgets per request instead.
- Reusing internal libraries and APIs in new features: this is the case study for contract tests, and for reviewing the call patterns new code introduces rather than just the code itself.
- Designing health-based load balancing: rerouting spreads request-borne failures; pair failover with per-request circuit breakers so a poisoned handler dies loudly instead of spinning forever.
- Planning staged rollouts: location-based staging misses conditions that depend on rare request shapes — replaying real traffic mixes in staging is the complement the post implies.
Apply this: Pick one request-handling codepath in your system and trace it for loops that never grow the stack. If you can’t rule them out by reading, add a per-request execution-time limit — a wall-clock circuit breaker — so the next latent loop costs you an error, not a data center.
What I want to remember
Remember: Tail calls make infinite recursion stack-invisible — a language optimization can quietly remove a safety net you’ve never tested. And failover reroutes requests, not harm: when the failure travels inside the request, your load balancers become its distribution network.