Noisy Neighbor Detection with eBPF
How Netflix used eBPF hooks into the Linux scheduler to build continuous, sub-microsecond-overhead run queue latency monitoring that attributes CPU starvation to specific containers in their Titus multi-tenant platform.
You already have container CPU utilization metrics and p99 latency dashboards. But when a co-tenant burns every available CPU on the host, your container’s latency spikes before either of those numbers explains why — and by the time a traditional profiler arrives, the damage is done and the tooling adds overhead to a host that’s already struggling.
Source
Noisy Neighbor Detection with eBPF
By: Netflix Technology Blog (Jose Fernandez, Sebastien Dabdoub, Jason Koch, Artem Tkachuk)
Published: Sep 10 2024
Type: Engineering article
Saved because it is a complete, honest account of a production eBPF program — from kernel hook design through ring buffer to userspace metric emission — with real benchmark numbers and the trade-offs that shaped every design decision.
How run queue latency pins blame on the right container
The noisy neighbor problem in multi-tenant compute has a deceptive surface. When a container’s latency spikes, the symptom looks identical whether the container hit its own CPU quota or whether a neighbor is consuming the CPU cycles it needed. Both cases manifest as slow responses. The traditional response — deploying perf, attaching a profiler — either arrives too late or worsens the situation by adding its own overhead at the moment the host is already under pressure.
Netflix’s solution on Titus centers on a single kernel-level signal: run queue latency, the time a process spends waiting in the scheduler’s run queue after it becomes runnable but before it actually gets CPU time. A long wait means the CPU was busy with something else. Measured continuously and attributed to individual containers, this becomes an early indicator that fires in real time rather than in retrospect.
Three eBPF tracepoints form the measurement apparatus. The sched_wakeup and sched_wakeup_new hooks fire whenever a process transitions from sleeping to runnable. At that moment a nanosecond timestamp is stored in a BPF_MAP_TYPE_HASH keyed on process ID. When sched_switch fires — the point at which one process yields the CPU and another begins running — the code retrieves the incoming task’s stored timestamp and subtracts it from the current time. That delta is the run queue latency for that scheduling event.
Attributing a measurement to a container requires the task’s cgroup ID. That information is in the kernel’s task struct, but behind an RCU read-side lock. eBPF’s kfunc interface (bpf_rcu_read_lock / bpf_rcu_read_unlock) provides safe access from inside the eBPF program, letting the code traverse the task’s CSS set to the default cgroup’s knode ID without race conditions or memory safety violations.
Scheduling events happen at a rate of hundreds of thousands per second on a busy host. Shipping every one to user space would have the Go consumer burning measurable CPU just keeping up with the ring buffer. The fix is a per-cgroup-per-CPU rate limiter implemented entirely in eBPF: a BPF_MAP_TYPE_PERCPU_HASH stores the timestamp of the last emitted event for each cgroup ID on each CPU core. If the gap since the last emission is below a threshold, the event is dropped before it is even reserved in the ring buffer — never seen by user space, never causing a consumer wakeup. Rate limiting in kernel space rather than user space is the essential move; it eliminates the event before any downstream cost accrues.
Events that survive the rate limiter go into a BPF_MAP_TYPE_RINGBUF, chosen over perf buffers because it handles variable-length records without extra memory copies and avoids a syscall per batch. Each event carries the incoming task’s cgroup ID, the outgoing task’s cgroup ID (the preemption cause), the latency sample, and a timestamp. User space maps cgroup IDs to container names and emits two Atlas metrics: runq.latency as a percentile timer per container, and sched.switch.out as a preemption counter tagged with the cause.
The dual-metric design is the central contribution. runq.latency alone is not actionable because a container throttled by its own CPU quota looks identical to one being starved by a neighbor — both cases cause the run queue to back up. What distinguishes the noisy neighbor case is that sched.switch.out will simultaneously spike with a tag pointing to a different container or a system service. A spike in runq.latency paired with a spike in sched.switch.out attributed to an external cgroup is the definitive pattern. Without the second metric, every container that legitimately exhausts its CPU quota generates a false alarm.
The team measured overhead with their open-source bpftop tool. Each invocation of the three hooked tracepoints adds less than 600 nanoseconds, and a Java service under test showed no measurable application-level change with instrumentation on versus off. Several findings shaped the final choices: BPF_MAP_TYPE_HASH outperformed BPF_MAP_TYPE_TASK_STORAGE by nearly 2x and slightly outperformed BPF_MAP_TYPE_PERCPU_HASH, which was unexpected. BPF_MAP_TYPE_LRU_HASH added 40–50 nanoseconds per operation over a plain hash map. BPF_CORE_READ adds 20–30 nanoseconds per call — on tp_btf tracepoints, direct struct member access is safe and faster. Kernel tasks with PID 0 trigger these same hooks but need no monitoring, so early-exit conditions eliminate them before any map lookups run.
One edge case worth understanding: when the test container2 saturated all CPUs, the sched.switch.out metric attributed the spikes to system services rather than container2 directly. Those system services were triggered by container2’s demand and competed with container1 for CPU time. The signal is correct, but reading it requires one extra step: a high system-service preemption count alongside a CPU-maxed neighbor is the combined pattern that points to the root cause. Neither metric alone completes the picture.
My take
The dual-metric design is the real contribution here, not the eBPF mechanics. The observation that run queue latency is ambiguous without a preemption-cause counter is something most monitoring stacks never implement, so teams either miss noisy neighbors entirely or generate constant noise from quota-limited containers and treat it as signal.
Where this is useful
- Diagnosing unexplained latency spikes in multi-tenant container environments where container CPU utilization appears normal
- Building per-tenant scheduler fairness dashboards for internal compute platforms
- As a reference for the
sched_wakeup/sched_switchhook pair pattern — the timestamp-delta approach is reusable for any scheduler attribution problem - Not applicable when the latency root cause is network I/O, memory bandwidth contention, or kernel lock contention — run queue latency captures only CPU scheduling wait, not total execution delay
Apply this: Before building a noisy neighbor alert, emit both a run queue latency metric and a preemption-cause counter — one without the other fires on every container that legitimately saturates its own CPU quota.
What I want to remember
Remember: A run queue latency spike tells you something is ahead of you in the scheduler queue; only a simultaneous preemption-cause counter spike attributed to an external cgroup confirms it is someone else’s problem — and only that combination is worth alerting on.