Slog
↑↓Navigate↵Go to PageescClose
Soumitra Saha

Load Balancers, Caches, and CDNs Explained

I break down what load balancers, application caches, and CDNs each do, how L4 vs L7 routing differs, why sticky sessions are a trap, and when to use Redis vs a CDN.

Load Balancers, Caches, and CDNs Explained

Three things come up in almost every scaling conversation: load balancers, caches, and CDNs. They’re often described together, and developers sometimes reach for them interchangeably — adding a cache “to speed things up” and a CDN “for performance” without a clear picture of what each one actually does.

The confusion makes sense. All three can reduce how much work your origin servers do. But they operate at completely different layers, and using the wrong one for a given problem either doesn’t help or creates new issues.

I’m going to walk through each layer — what it does, where it sits in the request path, and what it can’t solve. By the end, the “which layer should I add?” question should feel answerable.

Series: Part 2 of 4 in the System Design series. Start with Part 1: Scaling from One User to One Million

On this page

What a load balancer actually does

A load balancer sits in front of your servers and decides which one handles each incoming request. That’s the whole job: distribute traffic across a pool of servers so no single one gets overwhelmed, and hide the fact that there are multiple servers from the outside world.

The immediate benefit is capacity. Four servers can handle roughly four times the request volume of one. The less obvious benefit is availability. A load balancer that performs health checks continuously polls each server — sending small probe requests and checking for a valid response. If a server stops responding, the load balancer stops sending traffic to it. New requests go to the healthy servers. Users don’t see the failure.

According to Cloudflare’s load balancing documentation, this process is called “failover” — automatic rerouting to backup servers when a primary server fails. Without it, a single server crash takes your entire service offline.

Load balancers also distinguish between static and dynamic routing strategies. A static algorithm like round-robin simply cycles through servers in order, regardless of their current load. A dynamic algorithm monitors server health and resource usage in real time, routing traffic toward the most available server. For most production workloads, dynamic routing is worth the extra configuration.

L4 vs L7: where the balancer reads the request

Load balancers operate at different layers of the network stack, and the layer determines what the balancer can see — and therefore what decisions it can make.

Layer 4 (transport layer) routes based on IP address and TCP port. It has no awareness of the protocol running on top — no HTTP headers, no cookie reading, no URL inspection. This makes it fast and simple. The tradeoff is inflexibility: it can’t route based on request content.

Layer 7 (application layer) understands HTTP. It can read request headers, inspect URL paths, and examine cookies. This opens up routing decisions that are genuinely useful:

  • Send /api/ requests to your API servers and /images/ requests to a media server
  • Route traffic based on the Host header to serve multiple domains from one pool
  • Direct health check endpoints to a lightweight service instead of your full application

Most web applications use L7 load balancers for this flexibility. If you’ve set up an Nginx reverse proxy, you’ve used a form of L7 load balancing already — Nginx reads the request and makes forwarding decisions based on path or host.

The practical difference: L4 is fine when all traffic goes to identical servers. Use L7 when you need to route different types of requests to different backend pools.

Sticky sessions: the fix that trades one problem for another

Here’s where things get interesting — and where a lot of developers first hit a problem they didn’t expect.

Traditional session storage keeps session data in memory on the server: your login state, your cart, your in-progress form. When you add a load balancer, requests from the same user can land on different servers. Server 1 has the session; Server 2 doesn’t. The user gets logged out, or their cart disappears.

The tempting fix is sticky sessions (also called session affinity): configure the load balancer to always route a given user’s requests to the same server. Use a cookie or the client’s IP address to keep them “pinned.” Problem solved, right?

Not quite. Sticky sessions work at the cost of even distribution. If users are pinned to specific servers, you can’t freely redistribute load by adding new servers — existing users stay stuck to the old ones. More critically, if a pinned server goes down, those users lose their sessions anyway. The sticky session didn’t actually solve the durability problem; it just delayed the failure mode.

The better fix — covered in Part 1 of this series — is to make your app servers stateless. Store session data in a shared store that any server can reach, so the load balancer can route any request to any server freely.

Check this before enabling sticky sessions

  • Is your app currently storing any per-user state in server memory?
  • Have you evaluated moving sessions to a shared external store instead?
  • Do you understand how your load balancer handles server failures when users are pinned?

Sticky sessions are sometimes the right short-term answer for a legacy application, but they’re a sign the underlying session architecture needs work.

Application caches sit between your app and the database

An application cache stores the result of an expensive operation — usually a database query — so subsequent requests can skip the database entirely. The cache lives between your application code and the database.

Two common approaches, with meaningfully different tradeoffs:

In-process cache stores data in the same memory heap as your application. Reads are extremely fast — no network round trip at all. The limitation is isolation: each server has its own independent cache. If you have four app servers, you have four separate caches. A cache hit on Server 1 is a cache miss on Server 2 for the same request. This makes sense for rarely-changing global data — configuration values, feature flags, compiled templates — but creates inconsistency for user-facing data under a load balancer.

Shared external cache runs as a separate service that every app server queries. Redis is the standard choice here. A cache hit on Server 1 is also a hit on Server 2, because they’re both reading from the same place. Redis responds in under a millisecond for most operations — orders of magnitude faster than a database query joining multiple tables.

The cost of shared caching is the network round trip to Redis on every cache read. For most web applications this is a worthwhile trade, especially compared to a database query. The API caching strategies post covers the application-level patterns in more depth if you want to go further.

The standard approach is to combine both: use an in-process cache for truly static data, and Redis for anything that varies by user or changes frequently.

CDN edge caching is not application caching

A CDN (Content Delivery Network) is a geographically distributed network of servers that cache content close to users. The key distinction from application caching: the CDN operates at the HTTP response layer, and it intercepts requests before they ever reach your origin infrastructure.

When a user in Berlin requests a page, a CDN edge server in Frankfurt can serve it — if the content is cached there — without the request crossing the Atlantic to your origin server. That eliminates both the latency and the origin load in one step.

CDNs cache based on the Cache-Control header in your HTTP responses. Set Cache-Control: public, max-age=3600 on a response, and CDN edge nodes worldwide can cache and serve that response for an hour without touching your origin. Cloudflare’s caching documentation describes this as a “cache hit” — the edge server delivers the content without a cache miss requiring a trip back to origin.

Here’s the part that surprises most developers: this doesn’t only apply to static files. A product catalog API endpoint, a public leaderboard, a blog post served from a CMS API — any response your server marks as public can be cached at the CDN edge. For read-heavy public data, this can eliminate the majority of origin requests without a line of application code changing.

But note the hard requirement: public, identical data only. A CDN serves the same cached response to everyone who requests the same URL. Personalized responses — a user’s dashboard, their order history, anything derived from a session cookie — cannot be cached at the CDN layer. Those requests always need to reach your origin. This is the architectural boundary between CDN caching and application caching: one serves identical responses to many users, the other serves tailored responses to specific users.

Cache invalidation: harder than it looks

Caching at any layer raises the same question: what happens when the underlying data changes?

TTL-based expiry is the simple approach. Set a time-to-live on each cached item. When the TTL expires, the next request fetches fresh data. Accept that cached data can be briefly stale. This works fine when a few seconds or minutes of delay doesn’t matter — a public product listing, a leaderboard score, a news headline.

Explicit invalidation deletes the cached item the moment the source data changes. User updates their profile → immediately delete the Redis cache key for that profile, and send a CDN purge request for the profile URL. More accurate, but it requires writing invalidation code in every path that modifies data.

The trouble with explicit invalidation is completeness. Real applications have many write paths, and it’s easy to miss one. A user updates their profile photo. You remember to clear the main profile cache key, but three months ago a different developer added a separate card view with its own cache key. The user sees the old photo in the card for an hour after the update went through.

Cache invalidation bugs are genuinely difficult to reproduce in development, because they only appear when the cache is warm and a specific write sequence happens. This is probably why the well-known joke about hard problems in computer science leads with cache invalidation — it’s hard in practice, not just in theory.

The practical recommendation: start with TTL-based expiry, keep TTLs short on data that changes often, and only add explicit invalidation where stale data causes a real user-facing problem. Premature explicit invalidation means maintaining more code than the problem warrants.

Three separate layers, three separate problems

The shifted mental model from this post: these aren’t three versions of the same tool. They’re separate layers that each solve a distinct part of the request chain.

  • Load balancer — distributes traffic across servers, hides server failures, enables horizontal scaling. It’s infrastructure plumbing, not performance optimisation.
  • Application cache — reduces database trips for frequently-read, expensive-to-compute results. It operates inside your infrastructure, between your app and the database.
  • CDN — reduces origin load and user-perceived latency for publicly cacheable content. It operates outside your infrastructure, between users and your origin servers.

You typically use all three together. A request comes in, hits the CDN edge (served from cache if available), passes through the load balancer (routed to a healthy app server), and the app server checks the application cache (skipping the database if the data is warm). Each layer does something the others can’t.

When something breaks or performs poorly, map the symptom to the right layer. Slow responses for users in a distant region? That’s a CDN problem, or the absence of one. A specific database query under load? Application cache coverage. Servers going down under traffic spikes? Load balancer health checks and server capacity.

Right now, if you’re building something that needs to scale: start with the load balancer and a stateless session store. Add application caching when specific queries become bottlenecks. Add a CDN when traffic is geographically distributed or when origin request volume is your constraint. Solve the layer that’s actually limiting you.

Sources