Scaling PostgreSQL to Power 800 Million ChatGPT Users
How OpenAI serves millions of Postgres queries per second for ChatGPT on one primary and nearly 50 read replicas, and the discipline that keeps it standing.
Every scaling story seems to begin with “so we sharded the database.” OpenAI serves millions of Postgres queries per second for 800 million ChatGPT users from a single primary — and the team that runs it treats that as the sane architecture, not a liability.
Source
Scaling PostgreSQL to power 800 million ChatGPT users
By: Bohan Zhang, OpenAI
Published: Jan 22 2026
Type: Engineering article
This is worth saving because it’s the most concrete public account of Postgres at ChatGPT scale: one primary on Azure’s managed PostgreSQL, nearly 50 read replicas across regions, and a specific overload pattern paired with a specific fix for each. It’s also an argument that the sharding decision is as much organizational as technical.
One primary, fifty replicas, and no sharding
Postgres has quietly run core parts of ChatGPT and OpenAI’s API for years, and the load grew more than 10x in the year before this post. The architecture that absorbed it is simple to state: a single primary — the one instance that accepts writes — on Azure PostgreSQL flexible server, with nearly 50 read replicas spread across regions. A read replica is a continuously updated copy of the database that serves reads, which is why this works at all: reads scale by adding replicas, writes don’t.
Simple to state, hard to keep alive. The article’s catalog of near-disasters follows one recurring shape: an upstream issue spikes load — a caching-layer failure causing widespread cache misses, a surge of expensive multi-table joins saturating CPU, a write storm from a feature launch — then utilization climbs, latency rises, requests time out, and client retries multiply the very load that caused the timeout. That feedback loop is what turns a spike into a site-wide incident.
Writes are the structurally weak side, and the post is direct about why. Postgres’s multiversion concurrency control (MVCC) keeps old row versions around so readers don’t block writers; the cost is that updating one field copies the entire row, and later queries wade through dead row versions to find the current one. Under heavy write load that means write amplification, read amplification, table and index bloat, and painful autovacuum tuning. So OpenAI’s strategy is to move shardable, write-heavy workloads to Azure Cosmos DB, a sharded store — and to stop adding new tables to Postgres entirely, with new workloads defaulting to the sharded systems. The existing Postgres stays unsharded because sharding it would mean touching hundreds of application endpoints, a migration measured in months or years, while its read-heavy profile plus optimization leaves comfortable headroom.
Within that frame, most of the post is a playbook of challenge-and-fix, and nearly every fix removes load rather than adding capacity.
Load on the primary is minimized from both directions. Reads go to replicas wherever possible; the reads that must stay on the primary, because they sit inside write transactions, are kept lean. On the write side, beyond the Cosmos DB migrations, the team fixed application bugs that caused redundant writes, introduced lazy writes to smooth spikes, and enforces strict rate limits on backfills.
Query optimization carries the sharpest anecdote: one query joining 12 tables was responsible for past high-severity incidents. The lessons — avoid complex multi-way joins, move join logic into the application when needed, and actually read the SQL your ORM generates — are unglamorous but load-bearing. So is configuring idle_in_transaction_session_timeout, because idle transactions can block autovacuum.
The single primary is still a single point of failure, and the mitigation is layered. Because most critical requests are reads, offloading them to replicas means a dead primary degrades the service rather than stopping it — no longer a SEV-0. For the primary itself, OpenAI runs high-availability mode with a hot standby: a continuously synchronized replica promoted within moments of a failure or planned maintenance. They credit the Azure PostgreSQL team with making those failovers safe under very high load. Replicas get multiple instances per region with capacity headroom, so losing one doesn’t take out a region.
Noisy neighbors get their own treatment. A shared instance can be dragged down by one feature’s inefficient queries, so traffic is split into low- and high-priority tiers routed to separate instances, and different products are isolated from each other the same way.
Connections are a named failure mode: Azure PostgreSQL caps an instance at 5,000 connections, and OpenAI has had incidents from connection storms exhausting them all. PgBouncer — a proxy that reuses a small pool of database connections across many clients, running in statement or transaction pooling mode — cut average connection time in their benchmarks from 50 milliseconds to 5. Proxies, clients, and replicas are co-located per region so connections stay cheap and short-lived, each replica runs multiple PgBouncer pods in Kubernetes behind one service, and idle timeouts are treated as critical configuration rather than an afterthought.
Caching gets a specific mechanism: cache locking and leasing. When many requests miss on the same cache key, only one acquires a lock and fetches from Postgres; the rest wait for the cache to refill. That one rule prevents a cache-miss storm from becoming a database stampede.
Even read replicas have a ceiling. The primary streams its write-ahead log (WAL — Postgres’s durable record of changes) to every replica, and at nearly 50 replicas that shipping consumes real network bandwidth and CPU while making replica lag harder to keep stable. OpenAI is working with Azure on cascading replication, where intermediate replicas relay the log to others, unlocking potentially more than a hundred replicas at the cost of trickier failover management. It’s still in testing.
Rate limiting exists at four layers — application, connection pooler, proxy, and query — because retry storms and sudden endpoint spikes exhaust CPU, I/O, and connections faster than capacity can respond. The ORM layer can rate-limit or outright block specific query digests (the normalized shape of a query), which is targeted load shedding: kill one bad query pattern without touching anything else.
Schema changes get a strict diet. Altering a column type can rewrite an entire table, so only lightweight changes are permitted, guarded by a strict 5-second timeout; indexes are created and dropped concurrently rather than blocking; new tables must live in the sharded systems; backfills are rate-limited even when that stretches them past a week.
The results: millions of queries per second, nearly 50 replicas with replication lag near zero, low double-digit millisecond p99 client-side latency, and five-nines availability. In twelve months there was exactly one SEV-0 — during the viral ChatGPT image-generation launch, when write traffic surged more than 10x as over 100 million new users signed up within a week. Ahead: more write migration to Cosmos DB, cascading replication, and eventually, maybe, sharded Postgres or another distributed system.
My take
Read the playbook as a whole and a pattern emerges: almost nothing in it adds database capacity. Every fix either removes load — reads to replicas, writes to Cosmos DB, joins into the application — or stops amplification: retry backoff, cache locking, pooled connections, timeout-guarded schema changes. The scaling insight isn’t that Postgres is secretly infinite. It’s that a single-primary system dies by amplification long before it dies by query count, and the amplifiers are fixable in software.
The quieter reframe is how the post prices sharding. The argument against it isn’t technical limits; it’s “hundreds of application endpoints, months or years.” That makes sharding an organizational bet about migration cost — and it only works because OpenAI already has somewhere else to put writes. Anyone reading this as “you don’t need to shard Postgres” is missing that the writes already left.
One caveat worth holding: the availability and latency numbers are self-reported by the team that runs the system, and the architecture leans on Azure-specific machinery — failover behavior under high load, the upcoming cascading replication — that a self-managed or smaller fleet can’t assume. The engineering is still the story, just not a benchmark to hold a smaller fleet to.
Where this is useful
- Postgres fleets seeing overload incidents with the spike, timeout, retry-amplification signature — the post maps each symptom to its fix.
- Capacity-planning arguments about when to shard — the honest cost model (endpoint count, migration duration) beats the usual “shard early” slogan.
- Teams with ORM-heavy codebases — the 12-table join and the query-digest blocking are the case for auditing generated SQL before an incident does it for you.
- Less applicable to write-heavy systems: MVCC makes writes the weak side, which is exactly the traffic OpenAI moved off Postgres.
Apply this: Before the next spike, turn on the boring knobs — set
idle_in_transaction_session_timeout, put a pooler with sane idle timeouts in front, and add real backoff to every retry path that can touch the database. Then find your widest ORM-generated join and decide whether it survives your next traffic surge.
What I want to remember
Remember: A single-writer Postgres fleet is usually killed by amplification, not query volume — retries, cache-miss herds, connection storms, and table-rewriting schema changes turn spikes into outages. Remove the amplifiers first; scaling reads is the easy half.