Llama 4: 17 Billion Active Parameters, 10 Million Token Context, Two Competitive Models

Llama 4 Maverick has 400 billion total parameters and matches GPT-4o on multimodal and reasoning benchmarks — but its inference cost is closer to a 17B dense model than a 400B one. The question worth sitting with is why active parameter count, not total parameter count, turned out to be the number that matters most for deployment.

Source

The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation

By: Meta
Published: Apr 5, 2025
Type: Model announcement

The MoE efficiency story and Scout’s 10M context window each independently change the architectural decisions worth making for new applications. The post-training pipeline finding — that aggressive data pruning before online RL outperforms heavy supervised fine-tuning — connects to concurrent results from DeepSeek R1 and suggests a pattern generalizing across labs.

Why Active Parameters Are the Number That Matters

The conventional framing of model scale treats total parameter count as the primary quality indicator. Bigger models train on more data, learn richer representations, and perform better — and the compute cost of deploying them scales accordingly. The Llama 4 architecture makes a structural argument that this framing misses the number that actually determines deployment economics.

Llama 4 Maverick has 400 billion total parameters and 17 billion active parameters. In a dense model, every parameter processes every token. In a mixture-of-experts model, each token is routed to a shared expert and one of 128 specialized routed experts, activating roughly 17 billion parameters rather than 400 billion. All 400 billion parameters live in memory — VRAM requirements scale with total parameters — but the per-token compute cost scales with active parameters alone. Maverick runs on a single NVIDIA H100 DGX host, executes forward passes at roughly the inference cost of a 17B dense model, and delivers benchmark results that match or exceed GPT-4o and Gemini 2.0 Flash across multimodal and reasoning tasks. It is also competitive with DeepSeek V3 on coding and reasoning at less than half its active parameter count.

Llama 4 Scout makes the same argument more aggressively at the small end. Scout has 109 billion total parameters, 17 billion active parameters, and fits on a single H100 GPU with Int4 quantization. Its defining feature is a 10-million-token context window. The mechanism enabling that length is iRoPE — “interleaved Rotary Position Embeddings” — an attention architecture that alternates between standard RoPE layers and layers with no positional embedding at all. RoPE layers handle local syntactic and semantic structure; the NoPE layers allow long-range dependencies to form without positional encodings degrading at extreme sequence lengths. At inference time, Meta adds temperature scaling to the attention weights as an additional length generalization technique. Scout was pre-trained and post-trained at 256K context but generalizes to 10M at inference time through this combination of architectural choices.

A 10-million-token context window changes the engineering tradeoffs around retrieval-augmented generation. RAG pipelines exist largely because context windows were too small to hold relevant data directly — the standard workflow was to chunk documents, embed them, retrieve the most relevant pieces at query time, and hope the right context survived into the prompt. For datasets that fit under 10M tokens, Scout makes the retrieval step optional. Ten million tokens holds roughly 7,500 pages of text, a large production codebase, or years of structured user interaction records. Feeding the whole artifact directly eliminates the failure mode where relevant context simply doesn’t survive a retrieval cutoff — and reduces the architecture to a single-pass operation rather than a pipeline with multiple components that can each fail independently.

Native multimodality is the other structural shift in Llama 4. Previous Llama models treated vision as an add-on: a separate encoder processed images and injected representations into the text model at inference time. Llama 4 uses early fusion, where text tokens and image tokens enter the same model backbone from the start, pre-trained jointly on text, image, and video data. Meta’s vision encoder — based on MetaCLIP — was trained separately in conjunction with a frozen Llama model to calibrate its representations specifically to what the language model can use, rather than to general visual tasks. The result is multimodal understanding integrated into the model’s core processing rather than patched on after the fact. The pre-training data mixture covered over 30 trillion tokens — more than double Llama 3’s mixture — including 10 times more multilingual tokens across 200 languages.

The post-training pipeline warrants attention for a specific reason. Meta found that running heavy supervised fine-tuning followed by online RL followed by heavy DPO hit a ceiling — the SFT and DPO stages over-constrained the model’s behavior before RL had a chance to improve it, restricting the search space during the RL stage and producing suboptimal performance on reasoning, coding, and math. Their fix was aggressive data pruning: they removed more than 50% of SFT training examples flagged as easy by Llama-based quality judges, then ran online RL on the harder remaining set with a continuous curriculum that re-filtered prompts for medium-to-hard difficulty after each training cycle. A lightweight DPO pass handled only corner-case quality issues at the end. This pattern — prune easy data heavily, do lightweight SFT, run RL on hard data with dynamic filtering — matches the structure of DeepSeek R1’s post-training findings and suggests it may reflect something real about how online RL interacts with prior supervised training rather than being a coincidence across labs.

Llama 4 Behemoth — 288 billion active parameters, approximately two trillion total parameters — is not released for download but was used as the teacher model for Maverick during co-distillation. On MATH-500 and GPQA Diamond STEM benchmarks, Behemoth outperforms GPT-4.5, Claude Sonnet 3.7, and Gemini 2.0 Pro. The distillation process means Maverick’s weights absorbed knowledge from Behemoth’s forward passes during pre-training, with Meta developing a novel loss function that dynamically weights soft and hard targets through training. Maverick’s competitive benchmark position against models in its weight class reflects that investment — compute spent on the teacher during training rather than at inference time.

My take

The 10M context window in Scout is the result with the most immediate architectural consequences. Applications that currently require retrieval pipelines because the context window is too small to hold the full dataset may become single-context applications — simpler systems with fewer failure modes and no retrieval error surface. The MoE efficiency argument is well-established at this point, but Llama 4 is the clearest open-weight demonstration that 17B active parameters can match proprietary dense models at a fraction of the inference cost and hardware requirement.

Where this is useful

• Replacing RAG pipelines for datasets or codebases that fit within 10 million tokens, reducing architecture complexity and eliminating retrieval failure modes
• Running competitive multimodal inference on single-GPU hardware using Maverick’s MoE cost structure
• When NOT applicable: tasks where the full Behemoth-scale teacher reasoning is needed directly — Behemoth is not publicly released and its capabilities are only available through the distilled student models

Apply this: Before building a retrieval pipeline, calculate whether your full dataset fits under 10 million tokens — if it does, Scout’s context window makes chunking optional and simplifies the system architecture significantly.

What I want to remember

Remember: Active parameters determine inference cost, not total parameters — Maverick runs at roughly 17B inference cost despite 400B total parameters, and that gap is what makes frontier-competitive multimodal capability fit on a single H100 host.

Related reads

• Google Gemini 1.5 and long context
• DeepSeek R1 and open-weight reasoning