Introducing Meerkat: An Experiment in Global Consensus
Cloudflare is building Meerkat, a consensus service powered by the QuePaxa algorithm, to provide linearizable reads and writes across 330+ global data centers without the availability holes that leader-based protocols create.
Raft is designed to be understandable. It is also designed around a leader — and in a wide-area network, that leader is the single point of failure that every distributed systems textbook says you were supposed to eliminate.
Source
Introducing Meerkat: an experiment in global consensus
By: James Larisch, Bob Halley, João Pedro Leite
Published: Jul 08 2026
Type: Engineering article
Cloudflare’s wide-area network is adversarial enough that it exposed Raft’s leader timeout problem in production, repeatedly. This post names the exact failure modes they hit and describes the algorithmic properties that make QuePaxa a better fit for their network specifically.
The tyranny of timeouts and what comes after
Cloudflare operates over 330 data centers connected by the public internet. Many internal services read and write the same control-plane data from machines scattered across all of them — which database replica is currently the authoritative writer, where a particular AI model instance is placed. This data must be linearizable: any read after a write must see that write, even when the read and write arrive at different replicas in different parts of the world. The system must also keep accepting writes when some machines or links fail.
Linearizability at global scale is only achievable through distributed consensus: a group of machines runs a protocol that forces them to agree on the same sequence of operations even when they cannot all communicate simultaneously. As long as more than half of the replicas can reach each other, they can make progress. This majority requirement is the hard minimum — it’s where every classical consensus algorithm, including Raft, starts.
Raft adds one more requirement on top: a leader. The leader is the only replica allowed to drive consensus. All writes, regardless of which replica they arrive at, get forwarded to the leader. This makes Raft tractable to implement and reason about — the leader is the authoritative source, and linearizability follows almost automatically. But it creates a single point of temporary failure. When the leader crashes or becomes unreachable, the system stops accepting writes until a new leader is elected.
The election mechanism is where Raft meets wide-area networks and starts to break down. Other replicas detect a dead leader by timeout: if they haven’t heard from the leader in some configured interval, they nominate themselves. The problem is calibrating that timeout across a network where round-trip latencies range from 5ms to 500ms or more depending on path conditions. Set the timeout too short and replicas trigger elections for a leader that’s merely slow — elections that interfere with each other as competing candidates split votes, blocking writes while the campaigns cancel each other out. Set it too long and the system genuinely stalls after a real leader failure until the timeout expires. Cloudflare saw both failure modes in production. Their network makes unpredictable latency a feature, not an exception, and tuning a timeout-based protocol for it is close to impossible.
QuePaxa, published in 2023 by Tennage and Băsescu, starts from a different premise: the leader should be an optimization, not a requirement. Any replica in the cluster can drive consensus for the latest log slot. The leader, when present, needs only one round trip between proposer and the majority (plus a broadcast to notify stragglers of the decision). A non-leader needs three round trips. The difference is real but bounded — when the leader is down or slow, consensus slows slightly rather than becoming unavailable entirely.
The more important property is how concurrent proposals behave. In Raft, multiple replicas proposing themselves as leader create competing campaigns that can cycle indefinitely: each campaign voids the others’ progress, and the cluster can thrash through repeated elections while writes block. In QuePaxa, replicas proposing values for the same log slot work cooperatively. They exchange enough information to converge on one of the proposals without blocking each other. A replica that can contact any majority of the cluster can make progress regardless of what any other replica is doing at the same moment.
Meerkat layers applications on top of this consensus log. The log is a sequence of numbered slots; once a slot’s value is decided, every live replica will eventually agree on that value. Applications like the key-value store treat log entries as operations — put k v, get k — and construct their state by replaying the log in sequence. The non-obvious design choice that makes reads linearizable is that get requests also go through the log as entries rather than being served from local state. If replica Y hasn’t yet seen a recent write that was decided at slot 3, and a client submits a get to Y targeting slot 3, the majority will refuse to decide Y’s proposal at slot 3 — that slot is already decided with the prior write. Y is forced to record the write first, then propose the read at slot 4. A stale replica cannot short-circuit this; it either drives consensus correctly or the read doesn’t complete.
The performance constraints are honest. Proposal decision latency is proportional to the round-trip time between a proposer and a majority of replicas — there’s no escaping that cost. This makes Meerkat unsuitable for high-throughput data systems, and the post doesn’t claim otherwise. It’s designed for control-plane state: resource placement information, database leadership elections, distributed leases. These values change infrequently but must be consistent when they do change. Meerkat manages the metadata that databases need to operate correctly, not the databases themselves. In Cloudflare’s proof-of-concept with up to 50 globally distributed replicas, leaders fail continuously — and the cluster keeps processing with no measurable increase in error rate.
My take
The reframe worth keeping is that QuePaxa doesn’t remove leaders — it makes them optional. Raft’s simplicity comes from making the leader mandatory, which gives you a clean authority model and essentially free linearizability. QuePaxa achieves the same consistency guarantee by ensuring that any replica driving consensus for a slot will reach the same result a leader would. The leader is kept because it’s cheaper (fewer round trips), not because the correctness model requires it. This is the design principle: optimize for the common case without making the system unavailable when that optimization fails. For a wide-area network where “the common case” can become a 500ms timeout on a degraded cross-continental link, an optional leader changes your availability story completely — not theoretically, but in the production incidents Cloudflare has already experienced.
Where this is useful
- Building control-plane systems that require linearizable reads and writes across geographically distributed replicas with unpredictable inter-node latency
- Diagnosing availability gaps in Raft-based systems where timeout tuning can’t keep up with network latency variance
- Not applicable for high-throughput data workloads — consensus algorithms add round-trip overhead per operation that rules them out as general-purpose storage backends
Apply this: If you’re designing a distributed system with control-plane state that must stay consistent across wide-area replicas, ask what happens to writes during the leader election window — and whether your timeout values are actually tunable given the latency distribution of your network.
What I want to remember
Remember: Leader-based consensus algorithms like Raft trade simplicity for availability during leader failure. In wide-area networks with variable latency, that tradeoff produces availability holes that are hard to tune away. Leaderless algorithms like QuePaxa degrade gracefully — more round trips when the leader is absent, but no writes blocked waiting for an election to resolve.