Meta's Full-stack HHVM Optimizations for GenAI

Meta’s web infrastructure was tuned for a specific traffic profile — fast requests, bounded I/O wait, hard 30-second timeouts — and it worked well for years. LLM inference violates all three constraints simultaneously, and the symptoms showed up immediately: timeout violations, thread starvation, and cold-start latency spikes that no amount of tuning the shared tier could fix without breaking other products.

Source

Meta’s Full-stack HHVM optimizations for GenAI

By: Phil Lopreiato, Zach Zundel
Published: May 20 2025
Type: Engineering article

Saved because the diagnosis — a new workload class that breaks multiple invariants of a shared infrastructure system — generalizes far beyond HHVM and GenAI, and the fix (a dedicated tenant rather than pushing shared configuration to its limits) is the answer teams reach after they’ve already tried the other approach and paid the cost.

Splitting a Web Tier to Serve LLM Traffic

Meta’s web tier runs on HHVM — the HipHop Virtual Machine — executing Hack code for every front-end request the company serves. The tier is operationally monolithic: all web traffic shares a common host pool, a common runtime configuration, and a set of constraints tuned over years for a specific traffic profile. That profile is front-end web requests: latency in the hundreds of milliseconds to low seconds, roughly 70% of execution time on I/O, and a hard 30-second request timeout that protects thread availability across the shared pool. The timeout exists because each in-flight request occupies a worker thread, of which there is a finite number. Requests that run too long starve new requests of available threads, producing user-visible unavailability.

GenAI changes two of those three properties at once. An LLM inference request spends as much as 90% of its time on I/O — waiting for the model to stream tokens back — and its total duration is measured in seconds, sometimes minutes. The timeout violation is immediate: a request that takes longer than 30 seconds is killed before the model finishes generating. The thread pressure is structural: more time on I/O with more threads would help, but longer requests mean each thread stays occupied longer, reducing the number of requests a host can serve simultaneously even before any timeout fires.

Attempting to solve this on the shared tier creates an obvious problem. Increasing the timeout for GenAI requests means thread starvation becomes a possibility for all other products on the same hosts. Increasing the thread pool to compensate increases per-host memory pressure across the board. The configuration space where GenAI traffic runs acceptably and standard front-end traffic runs safely does not overlap. The right answer is not a configuration patch; it is a separate operational envelope.

The team created a dedicated WWW tenant — a standalone deployment of the web tier that handles only GenAI traffic. This single decision is the prerequisite for everything else. On an isolated tier, changes that would be unsafe on shared infrastructure become safe, because their blast radius is limited to traffic that already violates the shared tier’s invariants.

The first change is the request timeout. Increasing it beyond 30 seconds is straightforward on the dedicated tenant — the containment problem that made it unsafe on the shared tier no longer applies.

The second change is thread-pool sizing. Normally, a web host runs a few hundred worker threads, the count set by dividing total available memory by the per-request memory limit. GenAI requests have different memory characteristics, and with adjusted per-request limits, the team landed on roughly 1,000 threads per GenAI host versus a couple of hundred on standard hosts. More threads in flight, each spending most of its time on I/O, keeps the CPU busier than fewer threads doing compute-heavy work.

The third change is JIT cache seeding. HHVM is a just-in-time compiled runtime. The first time any function executes on a fresh server, HHVM compiles it from bytecode to machine code. On a cold start, the first wave of requests absorbs this cost — compilation runs inline, and latency spikes until the cache is populated. HHVM’s Jump-Start feature lets a server load its JIT cache from compiled output produced by a previously warmed server. The team uses Jump-Start profiles from the main web tier on GenAI hosts, even though code overlap between standard web traffic and GenAI requests isn’t complete. Partial coverage is still a significant win: the shared framework code that runs on every request — routing, authentication, configuration loading — is fully warmed from the first request, and the GenAI-specific paths warm quickly on low incoming traffic.

The fourth change is request warm-up. Separate from JIT state, HHVM caches configuration values and service discovery information inline the first time they’re needed within a running server. The first real requests on a cold server pay the cost of those initial fetches. Warm-up requests — dummy requests executed at server startup whose results are discarded — populate these caches before user traffic arrives. For GenAI requests, this overhead is a larger fraction of total request time than it is for sub-second web requests, so the warm-up payoff is proportionally larger.

The fifth change addresses JIT coverage in steady state. Meta uses real-time configuration heavily for feature rollouts, which means the code paths active on a server change over time as flags flip. Jump-Start profiles from startup don’t cover code paths introduced after the last profile snapshot. Shadow traffic — replaying a fraction of real production requests against the host in a way that warms the JIT without affecting real responses — keeps coverage current as configuration changes.

Together these five changes produced a 30% latency improvement over the baseline of running GenAI traffic on the shared web tier. None of the individual techniques are novel — JIT warm-up, request warm-up, larger thread pools, shadow traffic all predate this deployment. The compound gain comes from applying them together in an isolated environment where each change is safe independently, rather than fighting the shared tier’s safety constraints from both sides at once.

My take

The 30% latency improvement is the headline, but the mechanism is what’s worth internalizing. None of the individual changes are novel — longer timeouts, more threads, JIT warming, warm-up requests, shadow traffic. All of these techniques exist and are standard. The compound gain comes from applying them together to a workload class that was simply misconfigured on shared infrastructure. The shared tier’s constraints exist to protect other traffic; applied to GenAI requests that violate every constraint, they produce measurable latency overhead that no individual config tweak could address safely. This is the pattern: before optimizing, check whether the workload belongs in the current operational envelope at all. If it breaks multiple invariants simultaneously, the answer is usually a new envelope, not a longer list of patches.

Where this is useful

• Deciding when a new traffic class should get its own deployment versus tuning shared infrastructure — the signal is “breaks multiple invariants simultaneously”
• Sizing thread pools for I/O-heavy long-running requests rather than the compute-heavy short-request profile most web infrastructure is tuned for
• Understanding HHVM cold start costs and the three-layer mitigation (Jump-Start, warm-up requests, shadow traffic) for production deployments
• When NOT applicable: workloads that fit the existing traffic distribution close enough that shared-tier tuning addresses the gap without creating risk for adjacent products

Apply this: Before optimizing a new workload class on shared infrastructure, check how many of the existing tier’s invariants it violates — if the answer is more than one, the cheapest path is usually a dedicated tenant, not a patch list.

What I want to remember

Remember: A workload that simultaneously breaks multiple invariants of a shared infrastructure system (timeout, thread count, I/O ratio, JIT coverage) is not a tuning problem — it needs its own operational envelope, and giving it one is almost always cheaper than the compound cost of safely accommodating it in the shared tier.

Related reads

• How Meta built Logarithm to index AI training logs at 100 GB/s
• Scaling LLM inference with tensor, context, and expert parallelism at Meta