How OpenAI delivers low-latency voice AI at scale

A voice AI interaction is only as natural as the connection underneath it. The moment a user hears a pause where a response should be, the illusion of conversation breaks. At OpenAI’s scale, that meant rethinking what WebRTC actually looks like inside Kubernetes.

Source

How OpenAI delivers low-latency voice AI at scale

By: Yi Zhang, William McDonald
Published: May 04 2026
Type: Engineering article

Saved because the relay + transceiver split is a clean example of a broadly applicable pattern — separating routing from stateful protocol ownership — and because the ICE ufrag routing trick is the kind of elegant protocol reuse that rarely gets written up.

Routing WebRTC around Kubernetes

WebRTC is usually described as a technology for video calls. But OpenAI’s use of it points to a more fundamental property: it’s a standards-based protocol stack for sending a continuous audio stream from a client to a server in real time, with encryption, NAT traversal, codec negotiation, and jitter buffering already implemented across every browser and mobile platform. The alternative — building that stack from scratch per client — is an indefinite project. The more interesting question is what happens when you try to run WebRTC at scale inside Kubernetes, and the answer is that the standard deployment model breaks in two specific places.

The first breakage is port count. Classic WebRTC assigns one UDP port per session on the media server. That made sense for machines hosting hundreds of calls. At hundreds of thousands of concurrent sessions, it means exposing port ranges in the tens of thousands to the public internet. Cloud load balancers and Kubernetes aren’t designed for that. Large UDP port ranges expand the attack surface, complicate firewall policy, and conflict with elastic pod scheduling — you can’t autoscale if every pod needs to claim and hold a large stable port range.

The second breakage is state stickiness. ICE, the protocol WebRTC uses to establish and verify connectivity, maintains state across many packets. So does DTLS, which establishes the encrypted channel. The process that created a session must keep receiving that session’s packets, or the handshake fails. In Kubernetes, pods move, restart, and scale. A load balancer that distributes UDP packets round-robin will silently break sessions mid-handshake.

OpenAI’s solution separates these concerns into two services. The relay is a stateless UDP forwarder with a small, fixed public footprint — a VIP (virtual IP) and a small number of ports. The transceiver is the stateful WebRTC endpoint that owns all protocol state (ICE, DTLS, SRTP) but lives on an internal address unreachable from the public internet. The relay handles the port problem; the transceiver handles the state problem. From the client’s perspective nothing changes: it’s still standard WebRTC connecting to a single address.

The engineering challenge is first-packet routing. When the relay sees a UDP packet from a new client, it needs to know which transceiver to forward it to — without consulting an external lookup service that could be stale or unavailable. WebRTC hands you the solution for free. ICE connectivity checks carry an ufrag (username fragment), a short string that’s exchanged during session setup via SDP and echoed in every STUN binding request. OpenAI generates the server-side ufrag to encode routing metadata — enough for the relay to infer the destination cluster and owning transceiver. The relay parses just the STUN header, reads the ufrag, and forwards the packet. There is no hot-path external state lookup.

Once the flow is established, subsequent DTLS, RTP, and RTCP packets flow without re-parsing the ufrag — the relay maintains a minimal in-memory session keyed on the client’s source IP:port. If a relay restarts and loses that session, the next ICE STUN packet from the client rebuilds it automatically from the ufrag. A Redis cache holds the IP:port mapping after it’s first established, enabling faster recovery before the next STUN packet arrives.

Global Relay extends this pattern geographically. By deploying relay ingress points worldwide and using Cloudflare geo and proximity steering to select the closest transceiver cluster during signaling, both the initial HTTP setup and the first ICE packets enter OpenAI’s network at a point physically close to the user. Closer ingress means a shorter first-hop RTT, which directly shortens how long a user waits between starting a session and being able to speak. Geo-steered signaling puts the session in the right cluster; Global Relay puts the first media packet on a short path.

The implementation reinforces the design principle. The relay is written in Go and uses SO_REUSEPORT — a Linux socket option that lets multiple worker goroutines on the same machine share one UDP port, with the kernel distributing packets across them. Combined with runtime.LockOSThread pinning each reader goroutine to a specific OS thread, this tends to keep related packets on the same CPU core and reduces cache thrashing. Pre-allocated buffers minimize garbage collection pauses in the hot path. No kernel-bypass framework was needed because the relay’s job is narrow: parse one header field, look up a destination, forward the packet.

A common assumption about WebRTC at scale is that an SFU — selective forwarding unit — is the natural media plane foundation. SFUs make sense for multiparty sessions where you’re mixing and recording multiple streams. For 1:1 AI sessions where every millisecond of turn-taking delay matters, an SFU introduces protocol complexity and a WebRTC peer that the model side has to maintain. The transceiver model converts WebRTC to simpler internal protocols at the edge, letting every service behind it behave like a normal HTTP or gRPC endpoint rather than a real-time media node.

My take

The ufrag encoding trick is an instance of a general pattern worth internalizing: find a protocol-native field that’s already present in the first packet of every session and overload it with routing metadata. This sidesteps adding a hot-path lookup dependency, it’s self-healing on relay restart, and it works without any coordination between client and new infrastructure. The same logic shows up in consistent hashing with virtual nodes, in Thrift header routing, in gRPC trailers. The principle is the same: encode routing intent in data that travels with the payload, and let routing happen on the packet path rather than out-of-band.

Where this is useful

• Teams running WebRTC media at scale on Kubernetes where port exhaustion is a real constraint
• Any architecture where stateful protocol sessions need to survive pod rescheduling
• Designing session routing for low-latency, 1:1 client-to-server real-time protocols
• When NOT applicable: small-scale voice products where simple one-port-per-session works and SFU reuse is a genuine productivity win

Apply this: When designing UDP-based stateful sessions at scale, separate the thin routing layer (stateless, small public footprint) from the protocol terminator (stateful, internal) — and encode session ownership in a field that’s already in the first packet.

What I want to remember

Remember: Put hard session state in exactly one place — every additional service that owns protocol state is another reliability surface and another place where split-brain can occur.

Related reads

• Meta’s three-axis parallelism strategy for LLM inference
• Meta’s MLOW audio codec for bandwidth-constrained voice