Journey to 1000 Models: Scaling Instagram's Recommendation System
How Instagram built a model registry, launch tooling, and a stability metric to operationalize over 1000 ML models without losing track of what each one does or when one breaks.
The common assumption is that ML at scale is fundamentally a modeling problem — better architectures, more data, more compute. Instagram found that somewhere past a few hundred models, the real constraint became operational: no one knew what half the models did, launching a new one took days, and degraded predictions went unnoticed for long stretches. You can’t improve a system you can’t reason about.
Source
Journey to 1000 models: Scaling Instagram’s recommendation system
By: Meta
Published: May 21 2025
Type: Engineering article
This is one of the rare ML infrastructure posts that’s honest about the operational debt that accumulates when model count grows faster than tooling. The model registry, the launch flow, and the stability metric each solve a real failure mode that the article names concretely — and the naming convention for model types alone is worth reading carefully.
The Operational Collapse That Comes Before the Architecture Problem
Instagram’s recommendation stack covers more surface area than most people realize. Feed, Stories, and Reels are the visible surfaces, but the same ML ranking infrastructure drives which comments surface, which notifications get flagged as important, who you might tag in a post, and more. Within each surface, there are multiple layers: sourcing (retrieval), early-stage ranking, and late-stage ranking operate in sequence, progressively reducing candidate sets as operations grow more expensive. Each layer, on each surface, is a separate model. And within each layer, there is constant experimentation — weight adjustments, architecture changes, feature experiments — each producing its own model variant serving a fraction of traffic.
The result, once you account for all surfaces, all layers, and all experiments in flight, is over a thousand models. This is not waste; it reflects genuine product complexity. But it created three failure modes that compounded on each other.
Discovery failure — teams maintained separate sources of truth for their models, if they maintained one at all. An on-call engineer responding to an anomaly had to query multiple people before understanding whether the affected model was critical, experimental, or somewhere in between. This context gathering happened under time pressure, which meant errors and delays. The question “what does this model do and how important is it?” should have a one-sentence answer available instantly; instead it required an investigation.
Release failure — there was no consistent process for launching new models safely. Without a defined sequence of estimation, approval, preparation, scale-up, and finalization, teams improvised their own procedures. This slowed ML velocity — the distance between a successful experiment and a production model could be days — and it meant quality checks were inconsistently applied. Some models reached production without proper baselining; others were delayed by bottlenecks that a defined process would have eliminated.
Health failure — there was no consistent definition of what “model prediction quality” meant across different surfaces. Feed ranking and Stories ranking have different output distributions and different signal densities; a generic accuracy metric doesn’t capture meaningful degradation in either. Problems went unnoticed because the signal to notice them didn’t exist in a normalized, surface-agnostic form.
The solution the Instagram team built is three interlocking pieces, and the order matters: registry first, then launch tooling, then stability. Each piece depends on the one before.
The model registry is built on Configerator, Meta’s distributed configuration system. It stores model metadata in a schematized ledger with two primary record types: ModelMetadata (individual model instances) and ModelTypeMetadata (categories of models). The model type string is where the operational value lives — a type like ig_stories_tray_mtml encodes, in a single human-readable identifier, which app (ig), which surface (stories), which position in that surface (tray), and which model architecture (mtml — multi-task multi-label, common in late-stage ranking). These strings tag training flows, model checkpoints, and inference services, so every AI asset in the system carries its business context in its name rather than requiring a lookup.
Criticality is tracked via Meta’s company-wide GSI tier system — TIER0 through TIER4, corresponding to the maximum incident severity the model can cause. This matters because the incident response for an experiment serving 2% of Stories traffic is fundamentally different from the response for the baseline late-stage ranking model for Feed. Before the registry, teams would sometimes escalate their own models’ criticality to get better monitoring without sufficient justification; the registry required teams to prove business impact before claiming elevated tiers, which normalized the monitoring burden across Instagram.
Model launch tooling formalized the launch sequence into discrete phases with automation between them. The time to launch a new model went from days to hours. The more important outcome is consistency: every model now goes through estimation (what capacity does this need?), approval, preparation, graduated scale-up, and finalization in the same sequence, which means every model has been through the same quality gates. This uniformity is what makes automation possible — you can only automate a process that is actually a process.
Model stability is defined as a pioneering metric for prediction accuracy that works across surfaces with different output distributions. The article uses it to produce SLOs — service level objectives — for every model in the registry. An SLO here means: if this model’s prediction accuracy degrades below this threshold, it surfaces as a violation that the on-call team needs to respond to. Because the SLOs are derived from a normalized metric, an on-call engineer can look at the overall health of Instagram’s ML surface in a single view rather than interpreting surface-specific charts.
Together, these three pieces change the nature of the operational work. Before: an incident required gathering context from model owners, interpreting heterogeneous metrics, and improvising a response. After: the registry provides context automatically, stability SLOs surface the problem, and launch tooling provides the established playbook for remediation (roll back to baseline, end the experiment, or escalate). The on-call role shifts from detective to executor.
My take
The model registry is really incident response scaffolding wearing the costume of a metadata store. The metadata — model type, criticality, baseline and holdout IDs — is not interesting on its own. What it enables is that an on-call engineer at 2am can answer “what is this model, how important is it, and what’s the safe fallback” without waking anyone up. The decision to anchor criticality on the company-wide GSI tier system rather than an Instagram-internal scale is the specific design choice that made this stick: it gave social proof to the classification, made it legible to infra engineers outside Instagram, and prevented the team inflation that happens when each group gets to define their own importance scale. Operational tooling that creates new internal vocabulary almost always gets ignored; tooling that slots into existing vocabulary gets adopted.
Where this is useful
- Any ML platform team dealing with model sprawl — where the count of production models has outgrown the team’s ability to maintain shared context about them
- Designing on-call rotations for ML systems where prediction degradation is subtle and surface-specific
- Structuring model type naming conventions to encode operational context rather than just technical lineage
- Not applicable for teams with fewer than a few dozen production models — the registry overhead is not worth it until discovery and release failures are actually happening
Apply this: When designing a model registry, treat the model type string as the primary operational artifact — it should encode app, surface, funnel position, and architecture in a format an on-call engineer can read without documentation.
What I want to remember
Remember: Model stability as a surface-agnostic prediction accuracy metric is what enables uniform SLOs across heterogeneous ranking surfaces — without a normalized health signal, you can’t automate incident detection across a fleet of 1000 models with different output distributions.