Baidu ERNIE 5.1 and efficient multimodal scaling
ERNIE 5.1 is worth tracking because Baidu is talking about model compression, elastic pre-training, agentic post-training, and cost-performance.
The instinct is to read “one-third the parameters” as one-third the capability. ERNIE 5.1 cuts total parameters to a third of its parent model and active parameters to a half — and Baidu claims flagship-level performance at about 6% of the pre-training cost of comparable models. If that holds, the expensive part of model progress just got a discount nobody predicted.
Source
By: Baidu ERNIE
Published: May 09 2026
Type: Release announcement
I follow the Chinese labs because they’re competing hardest on exactly this axis: capability per unit of compute. ERNIE 5.1 isn’t a “bigger model” story — it’s a “pay less, keep the capability” story, and the machinery behind that claim is more interesting than the benchmark table.
How you get flagship capability from a third of the parameters
The problem Baidu starts from is one every lab has: pre-training each model size separately is brutally expensive, and training all capabilities in one pot makes them fight — improve code and chat regresses, the “seesaw” effect their post names directly. ERNIE 5.1 attacks both with three ideas worth understanding.
First, pre-train a family, then extract a model. Traditional practice runs separate pre-training for each scale. Baidu’s Once-For-All framework instead runs one pre-training pass that jointly optimises a matrix of sub-models, varying three dials as it trains: depth (how many Transformer layers are active), expert capacity (how many experts in the MoE layers participate in routing), and sparsity (how many experts each token activates via variable Top-k routing — fewer is cheaper, more is smarter). ERNIE 5.1 is then the optimal sub-network extracted from that ERNIE 5.0 matrix: a third of the total parameters, half the active ones, inheriting the parent’s knowledge without paying for a fresh run. My mental model: you don’t plant a new tree for each size of furniture — you grow one tree and cut it to size.
Second, reinforcement learning rebuilt as an asynchronous pipeline. Agentic RL needs training, inference, reward scoring, and the agent loop to work together, and doing that synchronously wastes resources. Baidu’s disaggregated architecture puts an RL Controller over four decoupled subsystems, each deployed and scaled independently, with the stages overlapping like an assembly line. Two details stood out: FP8 operator unification plus a Rollout Router Replay technique that cuts training-inference divergence (they report the K3 KL measure halved at near-zero latency cost), and elastic CPU pooling that farms idle cluster CPUs for logic-heavy work like code sandboxes and verifiers.
Third, capabilities trained separately, then distilled together. Instead of one sequential SFT-then-RL pipeline, post-training runs four stages: unified SFT; parallel domain-expert models (code, reasoning, agentic) each with its own reward signals; on-policy distillation, where the student generates from its own distribution while learning from all the expert teachers via token-level reverse KL; and finally online RL reserved for high-entropy tasks — open-ended chat and creative writing — because distillation over-smooths exactly those outputs. That last honesty is the most technically credible line in the post: it admits a fashionable technique (distillation) has a failure mode and scopes it.
The claimed results: 1,223 on the Arena Search leaderboard (4th globally, 1st among Chinese models on May 9), agent scores on τ³-bench and SpreadsheetBench-Verified above DeepSeek-V4-Pro and approaching closed-source leaders, 99.6 on AIME26 with tool use (second only to Gemini 3.1 Pro), GPQA and MMLU-Pro near the frontier, and creative writing internally judged near Gemini 3.1 Pro. Standard caution applies twice over: these are vendor-run numbers, and “6% of comparable models” leaves the comparable set undefined.
My take
What this source reveals — without claiming it — is that pre-training is quietly becoming a capital asset instead of a recurring cost. Once-For-All means one big run amortises across a whole family of shipped models, and staged distillation means new capability gets assembled from parallel specialists rather than re-baked from scratch. The frontier labs’ moat is shifting from “we can afford the biggest run” to “we can extract and fuse the most value from runs we already paid for.” That’s a much more defensible moat, and a much worse omen for anyone hoping compute prices alone would level the field.
Where this is useful
- Budgeting model development: a sub-network extraction strategy can turn one pre-training run into several shippable sizes.
- Designing agentic RL infrastructure: the four-subsystem decoupling and CPU pooling are practical patterns for long-horizon training.
- Reading vendor efficiency claims: “X% of cost” means nothing until you know the comparable set and who ran the benchmark.
- Less relevant for inference-only teams — though the variable Top-k sparsity dial is a useful lens on MoE serving-cost trade-offs.
Apply this: next time a release claims an efficiency jump, look for the extraction mechanism — a parent model, a sub-network, a distillation stage. If there isn’t one, the claim is probably marketing.
What I want to remember
Remember: Model progress isn’t only scale. Elastic pre-training, sub-network extraction, and staged distillation can buy capability per unit of compute that raw size can’t — and the seesaw effect is the tell that a lab’s post-training pipeline is still single-stage.