We're Leaving Kubernetes
After six years running cloud development environments on Kubernetes at internet scale, Gitpod concluded the platform is fundamentally mismatched for system workloads like dev environments.
Kubernetes runs the production workloads of half the internet. Gitpod spent six years learning that this tells you nothing about whether it will work for your workload — because development environments aren’t production workloads.
Source
By: Christian Weichel, Alejandro de Brito Fontes
Published: Oct 31 2024
Type: Engineering article
The most thorough public account of Kubernetes’s limits as a platform for interactive, stateful compute. Six years and 1.5 million users gives Gitpod enough data to reach real conclusions — not guesses — about where the platform fights you.
When the platform fights the workload
The mismatch begins with state. A development environment accumulates gigabytes of source code, build caches, Docker layers, and test databases — all on the specific node it was scheduled to. Kubernetes’s core scheduling assumption is that containers are moveable: if a node degrades, reschedule its pods elsewhere. Development environments break this assumption entirely. There’s a 1-to-1 relationship between a developer and their environment, and that environment is bound to its node’s local storage. You cannot migrate it, which means the scheduler’s central capability is unavailable to you from the start.
The resource profile makes things worse. Production services have a relatively stable CPU footprint — web servers process requests, databases run queries at measurable rates. Development environments sit idle most of the time, then demand several cores within 100 milliseconds when a build fires or a language server initializes. Kubernetes’s CFS-based CPU throttling cannot handle this because it observes demand after the fact: you only know you needed more CPU bandwidth by inspecting nr_throttled in the cgroup’s cpu_stats. By the time any control loop can react to that signal, VS Code has already disconnected and the developer’s terminal is frozen. Gitpod implemented custom DaemonSet-based controllers, moved from cgroupv1 to cgroupv2, adjusted per-process priorities — none of it was a clean solution, and all of it required continuous second-by-second readjustment.
Security requirements created a third axis of conflict. Developers expect to install packages, run Docker, and sometimes spin up a Kubernetes cluster inside their environment. Giving containers actual root access grants root on the host node, which means access to the infrastructure and other users’ running environments. Gitpod’s answer was user namespaces: map the apparent UID 0 inside the container to an unprivileged UID on the host. Kubernetes didn’t support this until 1.25, so Gitpod built their own implementation starting at 1.22. That required shiftfs for filesystem UID shifting, seccomp notify to intercept /proc mount calls, eBPF device filter modifications for FUSE support, and a custom network namespace with slirp4netns to isolate CAP_NET_ADMIN inside the container from the host network. Every Kubernetes upgrade risked breaking one of these layers.
Storage was the fourth problem. Local SSDs gave the best IOPS but tied workspace data to a specific physical machine. PVCs are Kubernetes’s abstraction for portable storage, but Gitpod found them unreliable in practice: attachment and detachment timing was unpredictable, per-instance disk count limits constrained scheduling, and availability zone locality constraints made cross-AZ balancing nearly impossible. Their working solution was a DaemonSet handling uncompressed tar archives to and from S3, with a cgroup-based IO limiter to prevent one workspace’s backup from starving others on the same node. This is effectively a custom storage system built beside Kubernetes, not on top of it.
The micro-VM detour is worth understanding. Firecracker and Cloud Hypervisor promised cleaner isolation than containers, a real security boundary, and snapshot/resume with userfaultfd for near-instant workspace restoration. In 2023 evaluation, neither delivered: Firecracker lacked virtiofs and GPU support, Cloud Hypervisor lacked userfaultfd so restores were slow, and converting OCI images to VM-consumable filesystems added its own complexity and startup overhead. Most importantly, micro-VMs moved the state problem rather than solving it. Memory snapshots are large; large snapshots create harder scheduling constraints and slower startup — the same fundamental issue in a different container.
Gitpod Flex (now Ona) carries forward what worked in Kubernetes — declarative APIs, control loops based on control theory, reconciliation-based orchestration — while stripping what didn’t: generic container scheduling, CNI networking complexity, and security abstractions never designed for arbitrary user code. Self-hosting deploys in under three minutes because the full Kubernetes control plane doesn’t come with it.
My take
The distinction Gitpod implicitly surfaces is between application workloads and system workloads as infrastructure categories. Application workloads were designed from the start to run on someone else’s scheduler: mostly stateless, horizontally scalable, with no expectation of kernel-level capabilities. System workloads — development environments, code execution sandboxes, per-user compute kernels — require close control of OS interfaces, manage their own resource boundaries, and have security requirements that go through the kernel rather than around it. Kubernetes was built for the former. When you run the latter on it, you spend years engineering workarounds for the scheduler, the networking stack, and the security model. The lesson applies to anything in this category: if your workload routinely needs to punch through process isolation or manage cgroup hierarchies directly, the platform is working against you regardless of how well-supported it is.
Where this is useful
- When evaluating orchestration platforms for development environments, code sandbox services, or per-user interactive compute
- When diagnosing why a Kubernetes deployment requires an ever-growing stack of DaemonSets and kernel patches to function correctly
- Not applicable to standard web services and API backends that scale horizontally — Kubernetes remains the right call there, and this post says so explicitly
Apply this: Before picking an orchestration platform, classify the workload: application workload (stateless-ish, schedulable, horizontally scalable) or system workload (deeply stateful, kernel-touching, interactive)? The answer determines which platform abstractions help and which will fight you for years.
What I want to remember
Remember: Kubernetes is an application workload platform. When a workload requires deep OS permissions, non-migratable local state, and burst CPU within milliseconds, the platform’s scheduling assumptions work against you — no amount of tuning patches the fundamental shape mismatch.