Cloudflare Outage on December 5, 2025
A 25-minute outage affecting 28% of Cloudflare's HTTP traffic traced to a nil dereference in a Lua killswitch, exposed when two individually tested changes intersected at a code path no one had ever hit before.
Testing passes, gradual rollout is running, and the killswitch is well-practiced operational tooling — yet on December 5, 2025, all three of those assurances failed to prevent a 25-minute outage. The failure came not from a broken system, but from a code path that had never been exercised in the years since it was written.
Source
Cloudflare outage on December 5, 2025
By: Cloudflare (Dane Knecht)
Published: Dec 05 2025
Type: Engineering article
This is one of the cleaner examples of emergent failure I have read — two changes, each individually tested and justified, producing a catastrophe by intersecting at a latent nil access bug. The postmortem’s transparency about what had not yet finished deploying from the previous incident makes it especially instructive.
How Two Correct Changes Produced One Broken System
On December 5, 2025, at 08:47 UTC, Cloudflare’s network began returning HTTP 500 errors to roughly 28% of HTTP traffic passing through the service. The incident lasted 25 minutes. It was resolved not by a fix but by reverting a configuration change — a revert that itself completed in about 60 seconds once initiated.
The chain started with a legitimate security response. A critical vulnerability, CVE-2025-55182, had been disclosed that week in React Server Components. To protect customers, Cloudflare needed its WAF to inspect larger request bodies — specifically, to increase the body parsing buffer from its default 128KB to 1MB, matching the default ceiling of Next.js applications. This change was deployed through the gradual rollout system, which propagates changes slowly across the fleet while validating health metrics at each stage. It was working correctly.
During that rollout, engineers noticed a problem: an internal WAF testing tool — used to evaluate new firewall rules before public release — didn’t support the new 1MB buffer. Since the tool wasn’t needed at that moment and had no effect on customer traffic, the decision was made to disable it. The mechanism for that was a killswitch, a standard operational tool within Cloudflare’s rulesets system for quickly disabling misbehaving rules. The killswitch had been used before, had a well-documented SOP, and had always worked. The second change — disabling the test tool — was deployed through a completely different path: the global configuration system, which propagates changes to every server in the fleet within seconds, with no gradual rollout guard.
The catch was that the killswitch was being applied to a rule with an action type of "execute" — a type that triggers evaluation of a nested sub-ruleset rather than applying a simple block, log, or skip. The killswitch correctly skipped evaluation of the sub-ruleset. But the Lua code that processed the results of the parent ruleset assumed that any rule with action == "execute" would have a populated rule_result.execute object. After the killswitch, that object was nil. Lua doesn’t fail at compile time for nil access — it fails at runtime. The exception propagated, and every request handled by the FL1 proxy with the Cloudflare Managed Ruleset deployed returned an HTTP 500 error. The full fleet received the change within one minute of it being applied.
What made this a latent bug rather than a recent regression is important to understand. The code had existed in the codebase for years. It survived undetected because using a killswitch on an execute action type had never happened before. This is a recognizable class of infrastructure failure: code that is stable not because it handles all cases correctly, but because the operational envelope has never reached the uncovered ones. The December 5 changes stepped outside that envelope for the first time.
The FL1/FL2 split adds another dimension worth examining. Cloudflare’s newer FL2 proxy, written in Rust, was not affected. The same error would have been a compile-time failure in Rust — accessing a field on a potentially-absent value is either handled explicitly by the type system or rejected at build time. The Lua code had no such protection, and the organization had no process requiring that killswitch behavior be tested against each distinct rule action type. Both conditions needed to be true for the bug to remain hidden across years of operation.
There is a second thread running through this incident: Cloudflare had been here before. Two weeks earlier, on November 18, 2025, a structurally similar failure — a fast-propagating config change deployed to address a security issue, no gradual rollout, widespread errors — had caused a larger and longer outage. The response to November 18 had produced three reliability initiatives: enhanced rollouts with health validation, fail-open error handling, and streamlined break-glass capabilities. Those projects were in progress but hadn’t fully shipped by December 5. The urgency of responding to CVE-2025-55182 moved faster than the protective tooling being built to prevent exactly this class of event.
That gap — between the team’s intent and what had actually deployed — is the honest answer to how this happened. The killswitch bug was the mechanism. The incomplete Code Orange work was the context that allowed the mechanism to operate at full fleet scale. Understanding both layers is what makes this postmortem useful beyond Cloudflare’s specific architecture.
My take
This incident reveals that operational tooling carries implicit preconditions that are rarely documented and almost never tested against all possible inputs. The killswitch was not a general-purpose tool — it had an untested assumption baked into the calling code downstream of it. At scale, every gap in testing coverage becomes a latent precondition, and latent preconditions surface at the worst possible moment: when two systems that have never interacted before are combined under pressure in a live security response.
Where this is useful
- Auditing emergency override or killswitch systems to verify they have been tested against all action types, not only the common ones
- Reviewing fast-propagating configuration paths for the absence of health-gated rollout controls
- Investigating whether consecutive incidents share structural patterns — not just surface-level causes — before treating them as unrelated
- Not when: the fast config path cannot affect customer traffic; for internal tooling with no production-path exposure, instant propagation is fine
Apply this: Take inventory of every fast-propagating configuration system in your stack and ask what happens if it is applied to an input type it has never seen before.
What I want to remember
Remember: “This tool has worked before” is evidence of correctness only for the input types it has already seen — not for all inputs.