A Doubled Config File and Cloudflare's Worst Outage Since 2019
Cloudflare's postmortem of its worst outage since 2019: a database permissions change doubled a bot-management feature file and panicked the core proxy.
The change behind Cloudflare’s worst outage since 2019 was a security hardening step: making database permissions more explicit. And while it played out, the network failed in five-minute waves — the broken file was regenerated on a schedule, sometimes by an updated database node, sometimes not.
Source
Cloudflare outage on November 18, 2025
By: Cloudflare (Matthew Prince)
Published: Nov 18 2025
Type: Engineering article
I saved this because it’s a same-day postmortem that traces a global outage to one missing database filter in a SQL query — and because it’s honest about the wrong turns, including a coincidence that made responders suspect an attack. The failure chain is long, but every link is small and recognizable.
How a metadata change stopped Cloudflare’s core proxy
At 11:20 UTC on 18 November 2025, Cloudflare’s network began failing to deliver core traffic, and visitors to customer sites saw error pages pointing at failures inside Cloudflare. The cause was not an attack — the post says so in bold in its opening summary. The chain: a database permissions change made a query emit duplicated rows into a “feature file” used by Bot Management; the file doubled in size and propagated to every machine in the network; and the traffic-routing software enforced a size limit below the file’s new doubled size. It failed. Cloudflare first misread the symptoms as a hyper-scale DDoS attack, then stopped the bad file and restored an earlier version — core traffic was largely normal by 14:30 UTC, everything by 17:06 UTC.
The strangest symptom: the system would fail, recover, and fail again. The feature file is regenerated every five minutes by a query running on a ClickHouse cluster that was being upgraded gradually, node by node. Bad output only came from nodes that had already taken the permissions change — so every five minutes, the network got either a good file or a bad one, propagated network-wide within that cycle. To an on-call engineer, a system that flaps like that looks like an attack, not an internal error — until every node took the change and the failure settled into its steady, broken state. Recovery meant stopping generation and propagation, manually inserting a known-good file into the distribution queue, and forcing a restart of the core proxy.
Why does a bot-management file matter this much? Every request entering Cloudflare terminates at the HTTP/TLS layer, flows into the core proxy (internally FL, “Frontline”), then through Pingora, which serves cache or fetches from the origin. The proxy applies each customer’s settings — WAF rules, DDoS protection, routing to the Developer Platform and R2 — through domain-specific modules. One, Bot Management, runs a machine learning model that assigns every request a bot score; customers use scores to decide which automated traffic to allow. The model consumes a feature configuration file: each “feature” is one input trait used to classify a request as automated or not. That file refreshes every few minutes network-wide so protection can track new bots quickly — a speed that cut both ways here.
Now the database layer. Cloudflare queries its sharded ClickHouse clusters through “distributed” tables in a database called default, sitting atop underlying tables in a database called r0, historically under a shared system account. The change in progress: run distributed queries under each original user account, so grants and limits apply per user and one bad subquery can’t affect others. Before the change, users only saw default-database tables when querying metadata. At 11:05 UTC, Cloudflare made the users’ implicit access to the r0 tables explicit — and its metadata became visible.
That visibility collided with an assumption buried in the feature-file generator, whose query looked like this:
SELECT name, type
FROM system.columns
WHERE table = 'http_requests_features'
ORDER BY nameNotice what isn’t there: any filter on the database. The query had only ever returned default columns, so nobody needed one. Once r0’s metadata appeared, every column came back twice — more than doubling the rows, which became features in the generated file.
The proxy was the next link. Modules in the proxy preallocate memory and enforce limits to keep consumption bounded; Bot Management’s runtime limit is 200 features, comfortably above the roughly 60 in real use. The doubled file blew past 200. In the Rust code of Cloudflare’s newer proxy, FL2, the limit check hit an unhandled error — a panic on a Result::unwrap() — which took down request-handling threads and returned 5xx for any traffic depending on the bots module. A migration detail made it two-faced: FL2 customers saw hard errors; customers on the older FL proxy got no errors but silently wrong bot scores of zero, so bot-blocking rules produced false positives — and only customers not using bot scores at all escaped impact.
The misdirections piled up. The status page — hosted entirely off Cloudflare’s infrastructure, with no Cloudflare dependencies — went down at the same time by pure coincidence, making some responders think an attacker was hitting both. The flapping also resembled the recent high-volume Aisuru DDoS campaigns. And the elevated latency wasn’t only the outage: debugging and observability systems were consuming large CPU enriching the flood of uncaught errors — the diagnostics amplified what they were diagnosing.
The blast radius ran through everything downstream of the core proxy: Workers KV threw elevated 5xx rates (its front-end gateway sat behind the proxy); Turnstile failed to load; the Dashboard stayed up but couldn’t log people in without Turnstile; Access authentication failed widely until a 13:05 UTC bypass, existing sessions unaffected; Email Security briefly lost an IP-reputation source, denting spam-detection accuracy with no critical customer impact. Patching Workers KV to bypass the proxy just after 13:04 UTC relieved the chain. After the main fix, a second dashboard window (14:40–15:30 UTC) came from a backlog of retried logins, absorbed by scaling control-plane concurrency.
The timeline: automated detection at 11:31 UTC, an incident call at 11:35, confidence that the Bot Management configuration file was the trigger by 13:37, new file generation halted at 14:24, a known-good file deployed globally at 14:30, all services restored by 17:06. The follow-ups are pointed: harden ingestion of Cloudflare-generated configuration files like user-generated input; more global kill switches for features; stop core dumps and error reports from overwhelming system resources; review failure modes across every core proxy module. Matthew Prince closes plainly — Cloudflare’s worst outage since 2019, the first in over six years to stop the majority of core traffic.
My take
What the source reveals is that “trusted producer” is a validation category, not a safety property. Every gate that file passed, it passed because of where it came from, not what was in it — and Cloudflare’s own remediation list admits exactly that, committing to harden ingestion of internally generated config “in the same way we would for user-generated input.” The untested assumption wasn’t even in the file; it was in a query that assumed a filtered world.
The second reveal is about gradual rollouts — usually the safe answer. Because the permissions change rolled out node by node behind a five-minute generation job, the network didn’t fail cleanly; it flapped, in a signature that resembles an attack and defeats the first question every responder asks: what changed? Partial states behind recurring generators need their own monitoring — validate the derived artifact, not just the database.
And the FL versus FL2 split is the detail worth carrying into any migration: the same bad input produced loud failure in the new proxy and silent wrong answers in the old one. The silent one is worse — it mislabeled real traffic as bots and would have gone unnoticed as an outage entirely if no customer had blocking rules.
Where this is useful
- Generating config or artifacts from database queries for privileged consumers: validate size, schema, and sanity at ingestion, regardless of who produced the file.
- Rolling out database permissions or metadata changes: making access explicit changes what queries return. Audit consumers of metadata tables like
system.columnsfor missing database filters before broadening visibility. - Running periodic generation jobs behind gradual rollouts: expect flapping, and monitor the generated output, not only the system being changed.
- Feature pipelines feeding models with preallocated, fixed-size buffers: the size limit is a contract with every upstream query, including ones people have forgotten about.
Apply this: Pick one internally generated config file your services ingest and ask: if it doubled in size tonight, would the consumer panic, silently misbehave, or fall back to the last good copy? If the answer requires reading the code to find out, that’s the finding.
What I want to remember
Remember: A trusted producer is just an unvalidated one with a good reputation. Treat internally generated configuration with the same checks as external input — size and schema validation, plus a last-known-good fallback.