Inside the Hardware and Co-Design of MTIA
How Meta built its second-generation AI accelerator by treating PyTorch's operator set and memory access patterns as hardware requirements, not as post-silicon adaptation problems.
Custom silicon is assumed to be a faster GPU — a chip shaped by general-purpose accelerator thinking, then handed to software teams to port onto. MTIA inverts that: PyTorch’s operator set and Meta’s model memory access patterns arrived at the hardware design table first, and the chip’s ISA was written to match them.
Source
Inside the hardware and co-design of MTIA
By: Meta
Published: Aug 22 2024
Type: Engineering article
Saved because the co-design story here is more concrete than the initial MTIA chip announcements — specific silicon features are tied back to specific model behaviors, and the PyTorch compiler integration reveals where the real architectural constraints sat.
How the Software Stack Shapes the ISA
The standard framing of custom AI silicon goes like this: a chip team builds a processor, then a software team ports the ML framework to compile down to that chip’s instruction set. What Meta describes as co-design for MTIA is different in a way that sounds like a minor reordering but changes most of the outcomes. The software team — specifically the PyTorch compiler group and the model architecture teams for Meta’s recommendation and ranking workloads — produced detailed operator profiles and memory access patterns before the hardware team finalized the ISA. That sequence meant the chip’s on-chip SRAM layout, vector unit widths, and memory bandwidth ratios were sized to match what the models actually do, not what a general matrix engine would assume they do.
The practical impact is most visible at the memory hierarchy. Meta’s high-throughput inference workloads — ads ranking, content recommendation — are not dense matrix-multiply bottlenecks. They are embedding-lookup-heavy: the most frequent operation is a sparse table lookup followed by gather and pooling across variable-length feature sets. A chip designed from a peak FLOP/s perspective would allocate die area to tensor cores and constrain SRAM. MTIA’s co-design process surfaced that the actual bottleneck was memory bandwidth for embedding access, so the on-chip SRAM banks were organized around the specific feature-dimension shapes of Meta’s embedding tables. That chip would score poorly on standard dense-compute benchmarks while running well ahead of GPU-class alternatives on actual production workloads.
The PyTorch compiler was built concurrently with the hardware. This matters because compilers are the most direct diagnostic tool for ISA design. When the compiler team tried to lower a specific PyTorch operator and found it required ten MTIA instructions where a GPU’s implementation needed one, that gap had two possible resolutions: extend the ISA before tapeout, or accept the runtime cost. Discovering that gap before tapeout costs a design meeting. Discovering it after costs the next chip generation. The compiler team’s ability to produce lowered code against a software-simulated ISA, and then measure instruction count and latency, gave the hardware team concrete feedback on which operator patterns needed native support.
Developer experience deserves attention because it drives adoption speed. At Meta’s engineer count, the friction cost of requiring a custom SDK to target MTIA is real. The decision to expose MTIA through PyTorch proper meant that model authors write the same Python used for GPU training, and the compiler handles the target. The same model definition file runs training on GPU and inference on MTIA. This collapsed the surface area where inference-specific bugs could hide and reduced the number of people who needed to understand the underlying ISA to zero outside the compiler team.
The talk surfaces a specific co-design feedback loop that occurred multiple times through MTIA v2 development. The hardware team identified memory coalescing modes that were cheap to implement in silicon and would improve throughput for what looked like common access patterns in theory. When the compiler team profiled real production model graphs, those patterns appeared rarely enough that the die area wasn’t justified. The feature was cut. The inverse happened with operators that appeared frequently in ranking models but lacked native hardware support in v1 — software workarounds using sequences of simpler operations degraded production inference latency measurably. V2 added native support for those shapes. Both decisions required the compiler team to have profiling authority that could change hardware specs, not just adapt to them.
The model architecture teams were also active participants, not downstream consumers. When the hardware team proposed a constraint — say, a limit on tensor ranks that could be processed without DRAM spill — the model team evaluated whether existing architectures fit that constraint and whether known future model directions would. Some constraints were accepted because the model team could adapt; others were rejected because the constraint would have blocked planned model improvements. This three-way negotiation between chip architects, compiler engineers, and model authors is what “co-design” actually means when it works.
What this describes is a coordination structure before it is a technical solution. The technical work — ISA extension, operator lowering, SRAM bank sizing — is tractable once the right constraints are on the table. The failure mode in custom silicon programs is a chip team and a software team working toward a shared deadline with separate backlogs, discovering incompatibilities only after the die returns from the fab. MTIA v2 exists because the organization set up a structure where the software team’s workload profiles had veto power over hardware decisions while there was still time to change them.
My take
The underlying economic fact is that Meta cannot buy a v2 chip if v1 underperforms — it has to build the next one from scratch on a multi-year timeline. NVIDIA can iterate a CUDA kernel in weeks. Meta needs hardware-software fit close enough on the first or second silicon generation to justify continued investment. Every co-design example in this talk is evidence that the team understood that constraint and built feedback loops fast enough to act before tapeout.
Where this is useful
- Evaluating whether “co-design” in a vendor or internal chip pitch means joint ownership of trade-offs or just software adapting to finished hardware
- Analyzing inference latency bottlenecks when workloads are memory-bandwidth-bound rather than compute-bound — the architectural reasoning applies even without custom silicon
- When NOT applicable: GPU inference workloads already saturating standard tensor core operations don’t benefit from this kind of SRAM-first architectural thinking
Apply this: Profile your inference graph at the operator level before evaluating hardware — the actual bottleneck is rarely where peak FLOP/s numbers suggest it is.
What I want to remember
Remember: Co-design only works when the software team has authority to change hardware specifications before tapeout — after that, the only co-design remaining is compiler adaptation.