MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet
Meta's MetaRoCE abandons the lossless-fabric assumption and rebuilds RDMA transport from scratch for Ethernet, pushing all intelligence to the NIC so the fabric can stay simple at million-GPU scale.
The standard assumption in AI networking is that lossless delivery is a precondition for RDMA performance: guarantee that no packet drops, and you get good throughput. MetaRoCE starts from the opposite premise — that the Ethernet fabric will lose packets — and builds a transport that performs better because of it.
Source
MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet
By: Meta
Published: Aug 24 2026
Type: Engineering article
Saved because it represents the clearest articulation of a transport philosophy shift: move intelligence from the fabric to the NIC, accept loss and disorder as normal, and build retransmission and multipathing into the endpoint. The open-source release through OCP makes it actionable beyond Meta’s own infrastructure.
What MetaRoCE gets right by assuming the fabric is broken
The 2026 paper introduces MetaRoCE as a clean-sheet redesign of RDMA transport for Ethernet, motivated by the limits of RoCEv2 at the scale Meta now operates — clusters of hundreds of thousands of GPUs spread across multiple data centers and regions. The central claim is that RoCEv2’s assumptions about the network — that the fabric will maintain packet order and avoid loss through Priority Flow Control (PFC) — become liabilities as scale increases, and that redesigning the transport to embrace loss and disorder produces better performance, not worse.
RoCEv2 delivers RDMA over UDP by relying on PFC at the link level to pause senders when receiver buffers fill, preventing loss before it can occur. The logic is sound for small clusters where pause propagation is fast and localized. At the scale of thousands of switches and millions of connections across multiple buildings and planes, PFC pause frames can ripple backward across the fabric and stall traffic that has nothing to do with the original congestion point — a phenomenon called PFC deadlock. Meta’s earlier RoCE work required significant DCQCN tuning and topology discipline to contain this. MetaRoCE abandons PFC entirely.
The core insight the paper makes explicit is worth dwelling on: “The fabric sees packets, but the NIC sees intent.” Traditional RDMA architectures embed intelligence in the fabric — switches enforce ordering and prevent loss — while the NIC executes instructions passed down from software. MetaRoCE inverts this. The NIC manages paths, detects loss, controls congestion, and repairs problems without coordination from the network. The fabric becomes deliberately simple: it only needs to support ECN marking and ECMP, capabilities every Ethernet switch already provides regardless of vendor. This topology independence is the practical payoff — the same MetaRoCE transport runs over fat-tree, multiplane, deep-buffer, and shallow-buffer fabrics, and over vendor clouds where you don’t control switch configuration. Nothing proprietary is required, so the fabric stays free to optimize for cost and cabling.
Out-of-order delivery is treated as a feature, not a problem to suppress. MetaRoCE sprays packets across many paths by design, so they arrive out of order routinely. The transport handles this through per-packet destination addressing: every Write packet carries the exact memory address it belongs to, so data is written directly to its final location on arrival with no intermediate reorder buffer and no head-of-line blocking. For Send operations, every packet carries a match token to a posted receive buffer, so even if earlier messages haven’t arrived, an incoming Send can land in the right buffer immediately, without a round trip to learn where the data goes. This changes the mental model: instead of thinking about connection state, you think about per-packet routing decisions that happen entirely at the NIC.
Multipathing is first-class rather than bolted on. Each connection carries multiple explicit paths, identified by distinct UDP source ports that the NIC can change at any time to steer traffic away from a degraded route. On multiplane fabrics — Meta runs both 4-plane and 8-plane configurations — plane selection is the NIC’s responsibility entirely, and throughput scales linearly with plane count because the NIC actively balances across planes. Each path tracks its own RTT and congestion window independently, which means the transport can distinguish a congested path from a failed path and react explicitly: a hot or broken link slows one path instead of stalling the entire connection. Traffic redistributes without application involvement or operator intervention.
Congestion control runs from both directions. The sender runs a conventional ECN-based AIMD algorithm per path, trimming the window on the specific path that received the ECN mark rather than globally. The receiver contributes in every acknowledgment — it returns a fair-share hint, the fraction of inbound bandwidth it has allocated to that sender, so the sender can target the right throughput directly instead of slowly discovering it through repeated mark-and-back-off cycles. The paper claims incast resolves in one or two round trips, compared to the prolonged recovery cycles RoCEv2 can exhibit under sudden many-to-one traffic. Loss detection uses a 256-bit selective acknowledgment bitvector per path. A gap in the bitvector is unambiguous evidence of loss — not reordering, because reordering is expected and handled separately — and it triggers retransmission of exactly the missing packet, on exactly the path that lost it, the moment the gap appears.
Unified connections reduce the state explosion that large-scale RDMA networks typically accumulate. Traditional RDMA gets more throughput or more parallelism by opening more Queue Pairs (QPs), each with its own congestion window blind to the rest, each consuming NIC state. MetaRoCE separates message ordering from bandwidth. A single connection carries many independent ordered streams — one per communicator or collective — and many paths below, under one shared congestion controller. The connection state stops growing with the parallelism of the workload. Existing RDMA Verbs APIs work without modification; enhanced multiplane features are exposed through extension APIs for applications that want them.
The validation results come from a 64-node AMD GPU cluster running RCCL collectives, comparing MetaRoCE against RoCEv2 across allreduce and all-to-all operations. At 1% packet loss — a rate that degrades RoCEv2 significantly — MetaRoCE maintained approximately 86% throughput. At 10% loss, it continued delivering useful bandwidth rather than collapsing. Multiplane validation across 4-plane and 8-plane topologies with up to 4,000 concurrent connections confirmed linear throughput scaling with plane count. During simulated plane failures, traffic redistributed autonomously. The implementation ran on AMD Pensando programmable NICs; additional NIC vendor implementations are underway.
The honest edge case is the NIC dependency. MetaRoCE’s intelligence lives at the endpoint, which means NIC hardware must implement the multipath management, per-path windowing, and reorder-tolerant reassembly logic. The paper validates on AMD Pensando programmable silicon, which carries a different cost and power profile from fixed-function ASICs. Whether the performance results hold on fixed-function NIC implementations — and at what area or latency cost — is the open question the paper leaves for follow-on work. The open-source release through OCP, including the full specification, a reference software implementation, and a compliance test suite, is the structure that makes that question answerable across vendors.
The misunderstanding to resist is treating MetaRoCE as a patch on RoCEv2. They share the Ethernet substrate and some protocol terminology, but the transport model is architecturally different: MetaRoCE assumes loss, mandates multipathing, eliminates PFC, and pushes all intelligence to the NIC. Deploying it means rethinking what role you expect the fabric to play — and that rethinking is the work, not the protocol spec.
My take
What the paper shows that is easy to miss: the lossless guarantee in traditional RDMA is a tax on the network, paid by every switch between the two endpoints. PFC propagates backward, DCQCN requires careful per-cluster tuning, and the topology must be designed around the failure modes of each. MetaRoCE offloads that tax to the endpoints, which already have more compute and more local information than any switch. The result is a transport that gets simpler, not harder, as the network grows — because additional switches and paths add bandwidth without adding coordination requirements. The OCP release also matters: a transport spec contributed openly means the vendor ecosystem has something to implement against, which is how InfiniBand built its performance advantage over time, but here with a fabric that any Ethernet switch can already support.
Where this is useful
- Designing a new AI cluster where you have NIC selection flexibility and want to avoid PFC-based lossless fabric complexity.
- Operating a multiplane or multi-data-center GPU cluster where RoCEv2’s ordering assumptions are already causing PFC events or deadlock.
- Evaluating transport protocols for inference serving at global scale, where tail latency across distributed model shards matters more than peak throughput.
- Retrofitting MetaRoCE onto an existing fleet of fixed-function NICs — the transport’s NIC-side complexity is non-trivial and the reference implementation was validated only on programmable silicon.
Apply this: If your RoCEv2 cluster shows PFC pause events that propagate beyond the immediate congestion point, map the pause propagation path before adjusting DCQCN parameters. That map is the argument for moving congestion intelligence to the NIC rather than tuning around it.
What I want to remember
Remember: “The fabric sees packets, but the NIC sees intent” — when you move intelligence from the switches to the endpoints, you get a transport that degrades gracefully under loss and disorder rather than amplifying them through the fabric with pause frames.