Sequence Learning: A Paradigm Shift for Personalized Ads Recommendations
How Meta replaced hand-engineered sparse feature aggregations in its DLRM-based ads system with event-based sequence learning, recovering the temporal signal that aggregation was quietly destroying.
The standard story about recommendation systems is that more features make them better. Meta’s experience suggests the opposite problem: for years, the process of building features from user behavior was destroying the most informative part of that behavior — the order in which things happened.
Source
Sequence learning: A paradigm shift for personalized ads recommendations
By: Meta
Published: Nov 19 2024
Type: Engineering article
This is one of the cleaner explanations of why DLRMs with hand-engineered features hit a ceiling, and what it takes to actually move past that ceiling at production scale. The explanation of Event-Based Features as a standardized input contract is the part that most write-ups miss — the architecture shift and the infrastructure shift are inseparable, and this article shows why.
What Aggregation Takes Away
Meta’s ad recommendation models were built on deep learning recommendation models (DLRMs) fed by thousands of hand-engineered sparse features. A sparse feature typically looked like: “ads this person clicked in the last N days” → [ad-id-1, ad-id-2, …, ad-id-N], or “Facebook pages this person visited in the past M days with visit counts” → [(page-id-1, 45), (page-id-2, 30), …]. These aggregations were effective for learning general interest patterns, and the DLRM architecture was well-suited to working with them. The problem was structural, not accidental.
When you aggregate a person’s events into a bag-of-IDs with counts, you lose three things. First, sequence information: whether someone engaged with a fitness ad before a nutrition ad matters for predicting their next intent, but a bag-of-IDs treats both orderings the same. Second, event-level co-occurrence: if someone clicked an ad at a specific time from a specific placement, all those attributes get split across different feature slots and the connection between them disappears. Third, human intuition as a bottleneck: feature engineers have to decide which aggregations to build, which means patterns too complex or counterintuitive for a person to anticipate never get captured. You end up with a proliferating feature space — multiple overlapping aggregations with different time windows, each providing marginal incremental value while adding compute and maintenance overhead.
People’s purchase intent is dynamic and evolving. Someone researching a laptop two days ago and a monitor yesterday has a different next-step intent than someone who did those in the reverse order. A model that can’t see the order can’t make that distinction, so it falls back to weaker signals.
The redesign Meta implemented was not just a model architecture change — it required rebuilding the data pipeline, the feature input format, and the serving infrastructure. These three things had to change together, and that’s what makes this more interesting than a typical “we added attention” post.
Event-Based Features (EBFs) are the foundation of the new system. An EBF is a structured object that captures everything about a single event in a standardized way: which event stream it came from (e.g., “recent ad engagements”), how many recent events to include from that stream (sequence length), and the semantic and contextual information about each individual event — ad category, timestamp, placement, and so on. EBFs replace legacy sparse features as the primary inputs to the recommendation models. The key property is that they maintain the raw event sequence rather than aggregating it, which means the model sees the events in the order they happened, with their full attribute context intact.
The model architecture follows naturally from this input format. An event model first converts each EBF sequence into a sequence of event embeddings: it learns embeddings for each event attribute, compresses them with linear compression into a single per-event representation, then applies timestamp encoding to capture recency and order. The output is a sequence of dense vectors, one per event — analogous to how a language model produces a sequence of word embeddings. The vocabulary here is orders of magnitude larger than natural language, because events come from heterogeneous streams and involve millions of distinct entities, but the structure is the same.
A sequence model then consumes these event embeddings and produces a summary representation keyed by the ad being evaluated. This is where the architectural efficiency work matters most. A naive self-attention over N events has O(N²) complexity. Meta’s design uses multi-headed attention pooling to reduce this to O(M×N), where M is a tunable parameter much smaller than N. The model learns to compress the event sequence into M summary embeddings that are most predictive for the ad being ranked — rather than attending to every pair of events, it learns which aspects of the sequence matter for a given ad context.
Scaling this architecture required work in two directions simultaneously. Scaling the architecture itself meant developing a custom transformer with complex feature encoding schemes tailored to the recommendation domain. Scaling the event sequences meant making sequences longer and richer without exploding training and serving costs — longer sequences capture more behavioral history, but they also increase the M×N attention budget and storage requirements. The team built people-centric infrastructure to manage event storage and retrieval efficiently, since the data access patterns for sequential event data are fundamentally different from those for aggregated sparse features.
The reported outcome — 2-4% more conversions on select segments — understates what actually changed. The infrastructure refactoring required to reach this number was substantial: new data storage systems, new feature input formats, new serving optimization for sequence models. The conversion lift is the downstream signal; the more durable change is that the system can now consume sequential behavioral signal that was simply inaccessible before, which means future improvements don’t need to fight the feature aggregation bottleneck.
My take
EBFs as a standardized input contract are the non-obvious part of this story. When you read about the architecture shift — from DLRMs with aggregated sparse features to transformers with event sequences — it sounds like a model swap. But the contract that makes it work is the EBF specification: a defined schema for what an event stream is, how long each stream should be, and what per-event attributes to capture. That contract is what lets the infrastructure team build scalable event storage, the training team build the event model, and the serving team optimize latency — all against the same interface. Without the EBF contract, “shift to sequence learning” is just an architectural idea, not a production system. The paradigm shift in the title isn’t really about attention mechanisms; it’s about agreeing on what an event is.
Where this is useful
- Redesigning recommendation systems where you suspect user intent is temporal but your features don’t model it
- Evaluating whether your current feature engineering process is destroying signal at the aggregation step
- Understanding how to decompose a sequence learning system into testable components (event model + sequence model) with a stable contract between them
- Not applicable when user sessions are short enough that order genuinely doesn’t matter — e-commerce browse sessions have sequentiality worth modeling; single-query search results typically don’t
Apply this: Before adding more aggregated features to a recommendation model, check whether aggregation is the right operation — or whether the sequence itself is the feature you’re trying to build.
What I want to remember
Remember: Aggregating event sequences into bag-of-IDs features destroys sequence order, event co-occurrence, and the ability to learn non-intuitive patterns — the EBF approach preserves all three by treating each event as a structured object rather than a counted entity.