Background Jobs and Message Queues Explained

A user clicks “Export report.” Your REST API handler runs the export, builds the file, and emails a download link. It takes four seconds. The connection stays open for four seconds. On a busy afternoon, forty users do this at once — response times climb, the event loop is busy on every worker, and if a process restarts mid-export nobody gets their file.

This is the wall that background jobs are designed to solve. The API handler records a “generate this report” job, returns immediately, and a separate worker process picks up the job and does the heavy lifting in its own time.

The machinery behind that handoff — queues, producers, consumers, delivery semantics, and failure handling — is what this post covers.

Quick answer: Background jobs decouple slow or unreliable work from the HTTP request cycle. A queue acts as a persistent buffer between the component that creates work (the producer) and the component that executes it (the consumer). The queue is durable, the two sides never talk directly, and delivery guarantees determine whether a job can be processed more than once.

On this page

• Why synchronous work slows everything down
• The producer/consumer pattern
• Queue semantics: FIFO, priority, and delay
• Delivery guarantees: what “at-least-once” costs you
• Dead-letter queues: where failed jobs go
• Fan-out: one event, many workers
• When to reach for a queue

Why synchronous work slows everything down

An HTTP request is a synchronous handshake. The client sends a request, the server does work, the client waits. For reading a database record or computing a small value, this is fine.

It breaks down for three kinds of work:

• Slow operations — generating PDFs, resizing images, calling external APIs that can take a few seconds.
• Unreliable operations — sending email, triggering webhooks to third-party services that might time out or return a 503.
• Long-running work — processing that takes minutes, or tasks that need to run on a schedule.

Keeping any of these inside the request cycle means the user’s connection stays open until completion, and your server process is pinned for the entire duration. If the process restarts mid-operation, the work is simply lost — no record that it was requested, no way to retry.

Background jobs break the connection between “user made a request” and “work got done.” The request records the intent and returns immediately. The work runs separately, independently, with its own retry logic and failure handling.

The communication between intent and execution flows through a queue.

The producer/consumer pattern

A queue is a persistent store for messages — in this context, job descriptions. Two actors interact with it:

Producer — creates messages and puts them in the queue. In a typical web app, this is your API server. When a user triggers an export, the handler creates a job object with enough context to execute the work and pushes it onto the queue.

Consumer (also called a worker) — reads messages from the queue and does the actual work. Workers are often separate processes, sometimes running on separate machines. They pull a job from the queue, run the handler, and acknowledge that the job is done.

The producer and consumer never communicate directly. They share only the queue. This is loose coupling in practice: the API server doesn’t wait for the worker, doesn’t know which machine it’s on, and doesn’t know how fast it processes jobs.

One practical consequence: you can scale the two sides independently. If your report-generation jobs are backing up, you add more workers. If your API is under load, you add more API servers. Neither side needs to know about the other’s scale.

Queue semantics: FIFO, priority, and delay

Not all queues behave the same way. The three behaviours you’ll configure most often:

FIFO (first in, first out) is the default. Jobs are processed in the order they arrived. The first user to request an export gets their result first. This is the right model for most tasks.

Priority queues let you attach a numeric priority to each job. High-priority jobs get picked up before lower-priority ones, regardless of arrival order. A password-reset email should jump ahead of a weekly report digest. The trade-off is queue starvation — a constant flow of high-priority work can starve low-priority jobs indefinitely.

Delayed jobs are enqueued now but must not be processed until a specific time or after a minimum delay. They’re used for cases like “send a follow-up message 48 hours after signup” or “retry this webhook in five minutes.” Most implementations store delayed jobs in a separate sorted structure and move them into the main queue only when their scheduled time arrives. The Celery documentation on ETA and countdown describes this as a minimum time, not a guaranteed exact time — queue congestion or network latency can push execution later.

Think of the queue as a coffee shop line. FIFO is the normal queue. Priority is a VIP entrance. Delayed jobs are advance orders placed the night before — they sit aside until the right time slot.

Delivery guarantees: what “at-least-once” costs you

This is the piece that causes the most production surprises. There are three models:

At-most-once — the job is sent to a worker once. If the worker crashes before acknowledging it, the job is gone. Use this when a duplicate run causes more damage than a missed run (think irreversible financial operations). It’s rarely the right default.

At-least-once — the job is redelivered if the worker doesn’t acknowledge it within the visibility timeout. This guarantees the work gets done but allows duplicates. If a worker picks up a job, finishes the work, and crashes before sending the acknowledgement, the queue considers the job undelivered and hands it to another worker. That worker runs the handler a second time.

At-least-once is the default model in most systems. It means your job handlers must be idempotent: running the same handler twice with the same input should produce the same result and no side effects. Sending a password-reset email twice, charging a customer twice, or creating duplicate records are real at-least-once failure modes in non-idempotent handlers.

Exactly-once — each job is processed exactly one time, even across failures. This sounds like the obvious choice, but it’s hard to guarantee across distributed systems without costly coordination between the queue and your data store. Most practical systems settle for at-least-once delivery with idempotent handlers instead.

A common technique: before doing the work, write the job’s ID to your database inside a transaction. If the handler runs again, it checks for that ID, finds it already processed, and exits cleanly. This pattern — idempotency keys — is how payment systems avoid charging customers twice on retry.

Check this before moving on

• You can explain why at-least-once delivery requires idempotent job handlers
• You understand that a timeout acknowledgement is what triggers redelivery, not just a crash
• You know the trade-off: exactly-once semantics add coordination overhead that at-least-once avoids

Dead-letter queues: where failed jobs go

What happens when a job always fails? The handler throws an error because the input is malformed, the external API it calls is permanently down, or there’s a bug in the handler logic itself. Without a safety valve, that job retries indefinitely, consuming worker capacity and potentially blocking other jobs.

A dead-letter queue (DLQ) is the safety valve. It’s a separate queue that receives messages once they’ve exceeded a maximum number of retries. The AWS SQS documentation calls this the maxReceiveCount: after a message is received that many times without a successful acknowledgement, it’s moved to the DLQ instead of being redelivered again.

Once a job lands in the DLQ you can:

• Inspect the message payload and logs to understand why it failed
• Fix the underlying bug or external dependency
• Replay the messages back into the source queue to be reprocessed

A DLQ configuration should include a retention period longer than the source queue’s — so jobs don’t expire before you get around to investigating them — and an alert on DLQ arrivals so failures aren’t silent.

Without a DLQ, a persistently failing job either disappears silently (if you have a max-retry limit with no fallback) or loops forever. Neither outcome is acceptable in a system you actually operate.

Fan-out: one event, many workers

The patterns so far assume one queue, one type of worker. Fan-out breaks that assumption.

In fan-out, a single event is copied to multiple queues — and therefore to multiple different worker types — simultaneously. A concrete example: when a new user signs up, you might need to send a welcome email, create a CRM contact, log the event for analytics, and notify your team on a channel. Rather than one job that does all four, you publish a single “user signed up” event to a central exchange or topic, which copies it to four separate queues, each consumed by its own worker.

Fan-out is a publish-subscribe pattern, not point-to-point delivery. A fanout exchange routes a message to every queue bound to it, ignoring routing keys. The producer publishes once; the broker makes copies.

The benefit is independent failure isolation. If the CRM worker is slow, the email worker keeps running. If the analytics pipeline is down, the welcome email still goes out. Each consumer scales and fails on its own.

When to reach for a queue

Not everything needs a queue. A system with queues everywhere is just as painful as one that blocks on everything.

Work type	Queue useful?	Reason
Reading a database record to JSON	No	Fast, synchronous, low risk
Sending a welcome email on signup	Yes	Slow, external, should retry
Generating a large export file	Yes	Slow, ties up the request thread
Triggering a third-party webhook	Yes	Unreliable, needs retry logic
Invalidating a cache key	Depends	How stale is tolerable?
Processing a payment	Careful	Idempotency keys required

The model I use: if the operation takes more than a few hundred milliseconds, can fail from an external outage, or must retry on failure without user involvement, it belongs in a queue.

How you structure the worker code alongside your existing Express application is a separate question that depends on how your server is organised — the production Express project structure post covers where worker processes fit in a real deployment.

The shift that background jobs require

Background jobs change how you think about work: not as something that happens inside a request, but as something that gets recorded and executed asynchronously. Your API’s job is to accept and log intent. Your worker’s job is to act on it. The queue is the contract between them — durable, observable, and failure-tolerant.

The conceptual model here maps directly to what you’ll wire up in code. The follow-up post covers the practical implementation: queues, workers, delayed jobs, retry configuration, and dead-letter handling in a Node.js application using BullMQ.

Sources

• Part 1: RabbitMQ for beginners — What is RabbitMQ? — CloudAMQP
• Using dead-letter queues in Amazon SQS — AWS Documentation
• ETA and Countdown — Celery 5.6 Documentation
• Overview of Blocking vs Non-Blocking — nodejs.org