Soumitra SahaApril 9, 2026

System Design Basics: Scaling from One User to One Million

I walk through the real arc from a single-server app to a distributed system, showing where each stage breaks and what each layer of the architecture actually fixes.

Your first production app probably lives on one server: app process, database, and static files sharing the same CPU and RAM. That works for dozens of users without a problem.

Then traffic grows. Someone says “just add more servers.” So you add a second server, set up a load balancer, and users start getting randomly logged out.

That’s not a capacity problem. It’s a session state problem — and it’s one of several surprises hiding inside the phrase “scale horizontally.”

This post walks through the actual arc from a single server to a distributed system. At each step, I’ll show what breaks, why it breaks, and what the next layer of the architecture fixes. By the end, you’ll have a mental model for thinking about scale before you need it, not after you’re already under load.

Series: Part 1 of 4 in the System Design series.

Single server: one machine, everything on it

The starting point is familiar. One machine runs your web server, your app logic, and your database. Requests come in, the app queries the database, and the response goes back out.

This isn’t a bad architecture. It’s simple to deploy, simple to debug, and cheap to run. For most apps in their first year, it’s exactly the right choice.

What it can’t do is handle one process failing without taking everything else down. If the database process crashes, the app stops working. If a rogue query locks a table, every request waits. If the disk fills up, both the app and the database fail at the same time. Everything shares the same hardware: CPU, RAM, disk I/O, and network bandwidth.

That shared dependency is the fundamental problem you’re solving when you scale. Every architectural decision after this point is, in some way, about separating concerns so that one component failing doesn’t take down an unrelated one.

Vertical scaling: buying a bigger box

The most obvious response to running out of capacity is to upgrade the hardware. More RAM, more CPU cores, a faster disk — this is vertical scaling, and it often works well at first.

A machine with 32 GB RAM and 8 cores can handle significantly more load than one with 8 GB and 2 cores. On a cloud provider, upgrading to a larger instance takes minutes with no code changes and no architecture redesign.

The limits are real, though. Hardware has a ceiling: you can only add so much RAM to a single machine before the cost outpaces the capacity gain. A 16-core cloud instance might cost 8× the 2-core version but deliver only 4× the throughput, because most workloads don’t parallelize linearly. And you still have a single point of failure. One machine going down still takes your entire service offline.

Vertical scaling buys time. It doesn’t solve the underlying problem.

Adding servers — and the session problem

Horizontal scaling means running more than one instance of your application and distributing incoming traffic between them. A load balancer sits in front of your servers and decides which one handles each request.

Load balancers operate at different layers of the network stack:

Layer 4 (transport layer): Routes traffic based on IP address and TCP port. It has no knowledge of HTTP — it just distributes TCP connections. Fast and lightweight, but can’t inspect request content.
Layer 7 (application layer): Understands HTTP. It can read headers, inspect URL paths, and examine cookies. This lets you route /api/ traffic to one server pool and /static/ to a CDN origin. Most web applications use L7 load balancers for this flexibility.

Here’s where things break unexpectedly. Traditional web frameworks store session state in memory on the server: your login token, your shopping cart, your in-progress form data. When a user logs in and their request hits Server 1, Server 1 stores that session. The next request from the same user gets routed to Server 2 by the load balancer. Server 2 has no record of the session — because Server 1 owns it.

The result: the user gets logged out. Or their cart empties. Or a form submission they started on one page disappears on the next. Not because any server was overloaded, but because the architecture assumed one server would always handle requests from the same user.

Check this before adding a second server

Does your application store any per-user state in memory between requests?
Are sessions stored locally in the server process?
Does your app write temporary files that other servers will need to read?

If any of these are true, your app will behave incorrectly under a load balancer until you make the session layer stateless.

Making app servers stateless

The fix is to stop storing state on individual app servers. A stateless app server treats each incoming request as a self-contained operation: it reads from shared storage, computes a response, and sends it back. Nothing persists locally between requests.

Two common approaches work in practice:

Move sessions to a shared store. Store session data in a database or in-memory store that every server can reach — Redis is the standard choice here. When any request arrives at any server, it reads the session from Redis instead of local memory. The session belongs to the shared store, not to any individual server. This is the same approach I covered in the context of building a REST API in Node.js and Express.

Use signed tokens. Issue a signed JWT (JSON Web Token) on login. The token carries the user’s ID and claims, signed with a secret key. Any server can verify the signature and read the user’s identity without contacting a session store at all. Session state moves to the client side.

Each approach has trade-offs. JWTs can’t be revoked before expiry without maintaining a separate revocation list, which reintroduces state. Shared sessions are easy to revoke but add a Redis round trip to every authenticated request.

Once your app servers are stateless, the load balancer can send any request to any server. You can add servers to handle more traffic, remove them when load drops, and replace a crashed server without affecting other users.

The database becomes the next ceiling

Stateless app servers scale horizontally without much friction. Databases are a different story.

Most web applications are read-heavy. Searches, product listings, user profiles, news feeds — most traffic is reads. Writes (new posts, order creation, account updates) happen far less often. A relational database on a single primary instance can handle a significant volume of traffic, but eventually it becomes the constraint. Queries queue up. A slow analytical query holds up fast transactional queries. The database’s CPU stays pegged while app servers sit mostly idle.

The first tool for this is read replicas: additional database instances that receive a continuous stream of changes from the primary and serve all read traffic. Your application routes SELECT queries to replicas and INSERT, UPDATE, DELETE queries to the primary.

Two things to understand before relying on them:

Replication is asynchronous. A write reaches the primary first and propagates to replicas after a short delay — typically a few milliseconds under normal load. If your application reads immediately after a write, it might not see the updated value yet. This is called replication lag, and it matters in flows like “user updates their profile, then gets redirected to their profile page.” The profile page might briefly show the old data.

Replicas don’t help with write bottlenecks. If INSERT and UPDATE throughput is your constraint, adding replicas changes nothing. High write volume requires a different set of techniques: write queues, command/query separation, partitioning.

For most applications, read replicas combined with sensible connection pooling extend the useful life of a single primary substantially. You’ll likely need 3–5× your current read traffic before replicas stop being sufficient.

PostgreSQL’s approach to indexing and query performance is worth understanding before read replicas become your bottleneck — a missing index at this scale costs far more than it did at 100 users.

Two caches that reduce database pressure

Even with read replicas, some queries are expensive and some data gets requested so often that hitting the database on every request is wasteful. Caching keeps computed results closer to where they’re needed, skipping the database entirely for hot data.

CDN caching sits at the edge, geographically close to your users. Static assets — JavaScript bundles, CSS files, images — are the obvious candidates. A request for a cached asset never reaches your servers at all. But CDN caching extends to API responses too: a product catalog endpoint, a public leaderboard, a blog post — any response that includes Cache-Control: public, max-age=3600 can be stored at CDN edge nodes worldwide. For read-heavy public data, this can eliminate the majority of origin requests without a single line of application code changing.

In-memory caching operates inside your infrastructure, between app servers and the database. You query the database once, store the result in Redis with a TTL (time to live), and serve subsequent requests from memory. Redis responds in under a millisecond for most operations — orders of magnitude faster than a database query that joins three tables.

The hard part of caching is invalidation. If you cache a user’s profile, what happens when that user updates their settings? Two common patterns:

TTL-based expiry: Let the cache expire after N seconds. Accept that cached data can be slightly stale. Works well for data where a few seconds of delay doesn’t matter.
Explicit invalidation: When the source data changes, delete the corresponding cache key immediately. More consistent, but requires writing invalidation code in every path that modifies data.

Explicit invalidation sounds cleaner until you realize how many write paths there are in a real application, and how easy it is to miss one. A user sees their old profile picture for an hour after updating it. An item shows as in stock in the cache while the database records it as sold out. Cache invalidation bugs are some of the harder ones to reproduce in development, because they only appear when the cache is warm.

Scale is a moving target

The architecture that handles 1,000 users won’t handle 100,000 without changes. The one at 100,000 won’t reach a million in the same shape. Each order of magnitude in traffic reveals a different bottleneck.

What this means practically: you don’t need to build for a million users from day one. You need to understand where your current architecture will break next — and build the fix for that problem, not for imagined future ones.

The pattern repeats at every stage:

Identify the bottleneck: database CPU, network I/O, cache miss rate, a slow external API?
Understand why it’s the constraint
Apply the right technique: vertical scaling, adding servers, a caching layer, read replicas
Ship it, then look for the next bottleneck

Right now, if you’re on a single server, your next step is probably a stateless session layer and a load balancer — not a distributed database cluster. The goal is to know which layer breaks next, treat it, and repeat.