ZGateway: Learnings from Putting a Proxy in Front of ZippyDB
Why Meta put a stateless proxy in front of its most-used key-value store, and how that one tier absorbed connection storms, hot keys, and noisy neighbors.
The last thing a strained data path sounds like it needs is another network hop. Meta added a proxy in front of its most-used key-value store anyway — because the real problem was a mesh of tens of thousands of connections per host that neither the clients nor the database could fix from their own side.
Source
ZGateway: Learnings from Putting a Proxy in Front of ZippyDB
By: Rittik Banik and Yunhao Cao, Meta
Published: Sep 03 2026
Type: Engineering article
This is worth saving because it states the general case for proxy tiers better than most design documents: what a proxy buys, what it costs, and when the trade flips in your favor. The ZippyDB numbers make the argument concrete, and the operational sections show how much careful work the trade actually takes.
What a proxy buys that no client can
ZippyDB is Meta’s most widely used key-value store — data addressed by a key and fetched with simple get and put operations — backing product metadata, counters, and configuration at billions of operations per second across a global fleet. ZGateway is the stateless proxy layer Meta is putting in front of it to unify client traffic.
The problem it solves is client sprawl: more than a million hosts, owned by hundreds of teams, talk to ZippyDB and can’t be changed quickly. In the direct-access model, every client connects to every database host it might need. A single client can touch tens of thousands of distinct shards — slices of the keyspace on different hosts — spread across hundreds of thousands of machines. The result is a dense many-to-many mesh of encrypted connections: a typical client holds tens of thousands of outbound connections, a typical database host accepts as many inbound, and each one costs memory, CPU, and a file descriptor on both ends, mostly while idle. Inbound connections grow with the client population, so every new client cohort makes every database host worse. When a cohort restarts or a deploy rolls and connection reuse drops, the fleet gets hit with a storm of fresh connections; Meta has traced host crashes from file-descriptor exhaustion and out-of-memory kills to exactly this. In one incident, a routing bug made every client open a connection per shard; hosts breached their file-descriptor limits and the fleet fell into a reboot loop.
You can’t fix that from the clients, because both fleets keep moving — client teams change pooling policies while the database consolidates on its own schedule, which the article likens to jumping between two moving cars. A proxy decouples them, and the post makes the general case: interposing between many callers and a shared backend bounds the problem (the database sees a fleet its operators control, not the client population), gives shared work like pooling, retries, routing, and caching a single home, and creates a control point — the only place the whole workload is visible and behavior can change in minutes instead of a fleet-wide rollout. The cost is one more hop and one more tier to operate; the trade pays off when the client population is large, diverse, and not yours to change.
ZGateway itself handles more than 1 billion operations per second, carries about 40% of all ZippyDB traffic (projected past 60%), and adds roughly 6% computational overhead to an average use case. It runs as regional tiers discovered through ServiceRouter, Meta’s service mesh (the layer that routes requests between services), in two flavors sharing one pipeline: a pure proxy and a read-through cache. A request arrives on a sticky regional connection, gets its TLS (the connection’s encryption) terminated, is authorized against the use case’s access lists, passes per-tenant admission control, is resolved to a shard, checked against the local cache, batched, and sent to the right replicas; responses return with per-use-case metrics, traces, and quota accounting. The key property is asymmetric connection counts: each client needs only a small sticky pool to its regional gateways, and each database host sees connections only from the gateway fleet, whose size Meta controls. Some responsibilities deliberately stay put — key-to-shard mapping, replica selection — so the gateway owns traffic management, not a reimplementation of the database client.
The arithmetic of the collapse is modeled as balls-into-bins: throw the shards a host touches at the fleet and count how many distinct hosts get hit. With round mock figures — 20 regions, 500,000 database hosts, 30,000 proxy hosts, 1,000,000 clients, 50,000 shards per client — per-host connection counts fall by about 97–98%, and total persistent connections drop roughly 19x. But the one-time reduction isn’t the real point; the scaling behavior is. Direct-access fan-in is linear in the client population. With the gateway in the path, the client population drops out of the formula entirely. “An unbounded number driven by everyone else becomes a bounded number we control.”
Because the proxy sees many clients at once, it can do what no client library can: combine work across unrelated callers. A shared batcher on each host groups requests headed for the same destination — keyed by use case and physical shard — into one backend request, and coalescing goes further: several callers wanting the same key at the same moment trigger one fetch whose result fans out to all of them. Every backend call carries fixed overhead regardless of size (serialization, shard lookup, authorization, syscalls), so folding operations together amortizes all of it — fewer, larger requests, lower CPU, steadier load — and since a use case is billed by the queries it sends, batching stretches its rate-limit budget too. Under a hot key, thousands of simultaneous callers collapse into a single read, so a hot key can never stampede one replica. This is also what let Meta retire its old client-side batching libraries, fragile and CPU-hungry and a steady source of incidents precisely because that complexity lived in a million binaries Meta didn’t control. Batches flush when a linger window elapses, a payload limit crosses, or a request cap hits, so added latency stays bounded; idle eviction and an in-flight execution cap keep the in-memory batches from becoming an out-of-memory risk when the backend slows.
Once traffic flows through one tier, capabilities accumulate there. Moving clients onto it was configuration only: flags per service and shard prefix, a percentage ramp, a region filter to limit blast radius, and a global kill switch. Tenant isolation comes from Discriminant Load Shedding — every request maps to a per-tenant bucket, buckets drain round-robin, and a flooding tenant fills only its own bucket while everyone else keeps draining. In a controlled overload above 90% CPU, across roughly 1,350 active buckets, only 6 — the actual noisy neighbors — were shedding, while the other ~1,344 executed 99.9% of their requests with zero rejections and goodput held near 97–98%, at about 8% CPU cost for the machinery. The caching tier serves hot reads in-process with a per-key fill lock (one fetch per herd) and stays fresh through a change-data-capture stream of write and checkpoint events, inside an explicit bounded-staleness contract. Load balancing copes with a tier mixing roughly 26-core and 126-core hosts: routing uses weighted consistent hashing, so a control-plane balancer nudges each host’s weight opposite to its recent CPU, with guardrails — damped, clamped, and throttled to move only the most-imbalanced hosts, since shifting weight on a cache tier means moving keys. Cross-region resilience adds global routing tables, “mega-regions” of nearby regions, and declared backup rings, each behind a percentage knob, with failover signals tuned to fire before a region tips into overload. Even transactions moved onto the gateway — the bookkeeping a thick client used to keep — consolidating two parallel implementations behind a flag across nine phases to 100% of transaction traffic with no reliability regression.
In production, ZGateway runs across dozens of regions in a handful of tiers — a general-purpose tier for the long tail of use cases, dedicated tiers for the largest customers, a separate high-throughput proxy tier — with per-use-case observability throughout. Next comes unifying all ZippyDB traffic, then making the tier programmable: agent-operated heuristics that expose the knobs and internal state as a control surface for AI agents, co-location that pushes parts of the gateway beside the database host for locality, and a multi-process split for hard fault isolation.
My take
The reveal worth keeping is that every capability in this post is downstream of position, not of cleverness. Batching across clients, discriminant shedding, hot-key collapse, weight-based balancing, cross-region failover — none are inventions unique to Meta; they become possible only at the point where many callers’ traffic already converges. The generalizable move is “put the control point where the fan-in already is,” and the deep version is arithmetic: the gateway converts an unbounded coupling — fan-in proportional to someone else’s client population — into a bounded one Meta owns. Capacity planning only became possible because the dominant variable became theirs.
Two honest caveats. The 97–98% figures are model-based estimates, as the post itself labels them, and the account comes from the team that built the tier — the ~6% overhead covers “an average use case,” and batching’s memory risk is visible in the two safety mechanisms it ships with. And the trade only pays at client populations you can’t coordinate; a small fleet of clients you own can keep pooling and retry logic in a library and skip the hop. It’s telling that the same shape appears in the reads beside this one — Figma’s DBProxy parsing SQL between application and pooler, OpenAI’s PgBouncer tiers in front of Postgres. Three very different companies converged on the same conclusion: when you can’t change the clients, change the thing they connect to.
Where this is useful
- Any shared backend whose client count grows faster than you can roll out client changes — storage clusters, internal APIs, message brokers.
- Deciding where cross-cutting concerns belong — retries, shaping, authorization, metrics — one managed tier versus every client binary.
- Multi-tenant platforms that need noisy-neighbor isolation; the per-tenant bucket design is concrete enough to study.
- Not worth it for a small, homogeneous client fleet you control — the extra hop and tier are pure cost there.
Apply this: Measure fan-in on your busiest storage host: count distinct client connections, and check whether that number grows with your client population. If it does, you’ve built the mesh — price a proxy tier before the next restart storm finds it for you.
What I want to remember
Remember: When your connection count scales with someone else’s fleet, you don’t own your own capacity. A proxy turns fan-in from everyone’s problem into one bounded number you control — that conversion, not any single feature, is why the tier earns its extra hop.