Python 3.14's Tail-Call Interpreter and the LLVM 19 Surprise

The 10-15% performance improvement from Python 3.14’s new tail-call interpreter was real, accurately measured, and honestly reported. The problem was that the baseline it compared against had been silently broken for five months by a compiler regression nobody had noticed.

Source

Performance of the Python 3.14 tail-call interpreter

By: Nelson Elhage
Published: Mar 09 2025
Type: Engineering article

I saved this because it is one of the clearest public examples of how a benchmark can be accurate and misleading simultaneously, and because the investigation uncovers something genuinely novel: the musttail attribute as a way to require that an optimization happen rather than hope it does.

When the baseline is the bug

CPython’s bytecode interpreter has used computed gotos for years. Instead of a single while (true) { switch(opcode) {} } loop where every opcode jumps back to one central dispatch point, each opcode ends with its own goto *opcode_table[next_opcode] — replicating the dispatch logic into each opcode’s body. The benefit is CPU branch-predictor locality: each opcode builds its own dispatch prediction history rather than sharing one indirect jump with every other opcode.

The complication is that Clang and LLVM don’t preserve computed gotos faithfully in their IR. For internal performance reasons, LLVM merges all those individual gotos into a single indirectbr instruction during compilation, then copies the dispatch logic back into each opcode body during code generation via a pass called tail duplication. This works correctly in Clang 18.

In Clang 19, the tail-duplication pass acquired a size limit designed to prevent pathologically-large code from blowing up compilation. On CPython’s interpreter — hundreds of opcodes — that limit caused the pass to bail out entirely. Where a Clang 18 build of _PyEval_EvalFrameDefault contained 332 distinct indirect jumps, the Clang 19 build contained 3. The optimization CPython wrote computed gotos to guarantee had been silently removed. The binary still ran correctly, so nothing caught it for five months.

Elhage quantified this with objdump | grep 'jmp\s+\*' — a concrete disassembly-level check that the expected code shape was actually present — and confirmed the regression accounted for roughly 9% slowdown on Intel and 12% on Apple M1 compared to a Clang 18 baseline.

The tail-call interpreter takes a different approach to the same dispatch problem. Each opcode is written as a C function that ends with a [[clang::musttail]] call to the next opcode’s function. The musttail attribute makes the optimization non-negotiable: either the compiler emits the call as a tail call, or compilation fails with a hard error. This sidesteps the Clang 19 regression not by fixing it but by using a mechanism with a compiler-enforced guarantee rather than one that depends on a specific optimizer pass making a specific choice. Benchmarked against a Clang 19 baseline, the tail-call interpreter appears 10-15% faster — but that gap is mostly measuring the regression it avoids. Against a Clang 18 baseline, the gain is 1-5%.

One sharp finding in the investigation: Clang 18, given the classic switch-based interpreter with no computed gotos at all, performs tail duplication anyway — achieving the exact code shape the computed gotos were written to guarantee, without the programmer asking. This means the elaborate computed-goto machinery existed to request an optimization the compiler was already inferring independently. GCC still needs the explicit form, but for modern Clang the computed gotos were unnecessary complexity — until a compiler version change made them actively harmful.

The deeper point is about the difference between hoping for an optimization and requiring it. The computed-goto interpreter asked the compiler to do the right thing and relied on that being stable across releases. The musttail attribute converts that assumption into a contract: deliver this optimization or refuse to compile. Elhage frames this as the more robust way to write performance-sensitive interpreter code going forward — not a faster algorithm, but a guarantee that performance intent survives the optimizer, even as compiler versions change.

What I want to remember

Remember: A benchmark comparison is accurate only within its own baseline — before generalizing “X is 10% faster than Y,” verify the baseline is actually healthy, because an optimization your code relied on may have been silently reverted by the compiler without changing any observable behavior.