Tailscale — Tracking Down the 16-Year-Old WAL-Reset SQLite Bug
Tailscale's six-month forensic hunt for a rare SQLite checkpoint race that corrupted 19 production databases and had hidden in the engine for 16 years.
Choosing “boring” technology is supposed to rule out stories like this: Tailscale ran SQLite in a documented, supported configuration and still suffered 19 database corruptions in six months, caused by a race that had sat in the engine for an estimated 16 years.
Source
How we tracked down a 16-year-old SQLite bug
By: Tailscale (Alex Chan)
Published: Aug 12 2026
Type: Engineering article
This is worth saving because it’s a complete forensic arc — unexplained corruption, theories ruled out one by one, a clue from an unrelated recovery tool, a named root cause — and because the trigger wasn’t exotic code but an operational choice plenty of teams could make. It also doubles as a plain-English tour of SQLite’s write-ahead log.
How Tailscale cornered a 16-year-old data race
Start with the architecture that set the stage. To clients, Tailscale’s control plane is one endpoint; internally it’s split into coordination servers, or shards, with each customer’s private network (a tailnet) living on one shard at a time. Each shard keeps its state in a single SQLite database, accessed by one Go process — the single-writer setup SQLite is designed for. Tailscale ran SQLite this way since 2022, backed up since early 2023 by snapshotting the whole database file to S3 every few minutes.
In August 2025, a pipeline reading those backups flagged an error, and PRAGMA integrity_check confirmed corruption. SQLite corruption is possible but rare enough that you shouldn’t see it in normal operation, so they repaired the database and hunted for a cause. Then it happened again — 19 separate corruptions over six months.
The blast radius had limits: these databases hold tailnet metadata, never private keys or network traffic. But recovery stopped the shard’s control plane, so affected tailnets lost the admin console and API — for over an hour at first, shrinking with practice. Devices already online kept talking peer-to-peer, but a device that came online during the window couldn’t join. Most shards were never hit at all, yet each status-page incident eroded trust further.
The hunt resisted everything. No recent changes touched the SQLite-adjacent code, and a fine-tooth re-review found nothing. The incidents shared no common factor — not shard, customer, feature, time of day, or load — so the bug couldn’t be reproduced synthetically. The team deployed passive forensic telemetry to production and waited, on a schedule ranging from hours apart to a six-week silence broken by an unwelcome Christmas present. A professional support contract with SQLite’s developers added theories — broken POSIX locks on close(), mishandled memory ownership, thread-safety misuse — and each got ruled out.
While the search ran, recovery got faster: shards hard-stop on corruption, an automated monitor runs PRAGMA integrity_check over backups, and better runbooks cut response time under an hour. They also built a transaction-logging pipeline for recovery: every modifying SQL statement streamed to a separate log. Because SQLite is single-writer with serialisable transactions, that log is completely linear and deterministic — something a multi-writer database like Postgres or MySQL couldn’t guarantee. Replaying it against the last good backup should rebuild current state, bypassing the corruption. But in two incidents the replay failed for a stranger reason: data written and committed by one transaction was invisible to later transactions — a write vanished without an error. “That should be impossible!”
That impossibility pointed at checkpoints. SQLite stores a database as pages — small blocks of content. In write-ahead logging (WAL) mode, updated pages aren’t written straight into the main file; they’re appended to the WAL. Eventually those pages must be copied back into the database file, and that copy-back is a checkpoint. Most deployments let SQLite decide when to checkpoint; Tailscale takes manual control for fast, consistent backups — documented and supported, but non-standard, which made it a suspect as other theories fell. One metric fed the suspicion: during incidents, SQLite reported copying more pages out of the WAL than it contained. Ten in, twenty copied — something was deeply confused.
To see inside, SQLite’s developers wrapped the virtual filesystem layer — the bottom of the stack, under the code generator and the pager that splits work into pages — with a shim that traces what passes through. The shim, tmstmpvfs, was open-sourced; Tailscale deployed it and waited.
The logs nailed it: a rare data race between a checkpoint and a write transaction. If a write lands at exactly the wrong moment during a checkpoint, the checkpoint believes certain pages were already copied back when they weren’t. Those pages are never written; the data is permanently lost. Meanwhile pages that reference the missing ones — an index, say — do get written, so the file is internally inconsistent: corrupt. SQLite’s developers named it the WAL-Reset bug and estimated it had been present for at least 16 years. It survived because it was so rare they had to add code to deliberately trigger it in testing. The fix adds a check that detects when another thread has reset the WAL mid-checkpoint, and it shipped in SQLite 3.52.0.
Rollout then delivered a twist: after clean canary shards, the full deployment lit up the backup monitor with corruption in 13 databases — none of it real. The cause was a second, separate bug with stale expression indexes. Tailscale stored high-precision timestamps as text and converted them to floating point in a generated column; 3.52.0’s optimisation subtly changed that text-to-float rounding, so the indexes held mismatched values that read as corruption. SQLite withdrew 3.52.0 and re-released the race fix alone as 3.51.3, later adding self-healing indexes in 3.53.0; Tailscale switched to integer-second timestamps.
Even fixed, silence proves nothing — there had already been one deceptive six-week lull. So Tailscale patched their driver to log a warning whenever a write transaction and a WAL-reset overlapped: a warning plus an intact database would prove the fix was saving them. The warning took two months to fire once — its own measure of the race’s rarity, and of why 16 years of users never surfaced it. Four further months passed with zero incidents.
The post’s own conclusion is blunt: “running boring technology in a non-standard way is a risk.” The common paths are tested by sheer volume of users; step off them, even onto documented and supported ground, and you may be the one supplying the test data.
My take
The lesson everyone will quote is the boring-technology one, but the more transferable story is about evidence. Tailscale couldn’t reproduce the bug, so every advance came from instrumentation built for one purpose and read for another: the transaction log was built to speed recovery, and it produced the impossible vanishing-write clue; the checkpoint-page metric was routine observability, and it quantified the confusion; the post-fix warning turned an invisible race into a countable event, which converted “no incidents lately” into proof. For rare concurrency bugs, the bottleneck isn’t cleverness — it’s having a counter where the bug has to pass by. Notice also what didn’t work: careful code review of the suspicious paths found nothing, because the bug wasn’t in their code at all.
Where this is useful
- Running SQLite (or any embedded database) with manual checkpoint control, custom backup pipelines, or other off-default operational settings.
- Designing forensics for production bugs that won’t reproduce — passive telemetry plus a deterministic transaction log is the pattern.
- Planning recovery for single-writer databases: linear replay from a statement log beats rollback-and-lose-data once the log exists.
- Less relevant if you run SQLite’s default configuration — the post is explicit that most users never hit this.
Apply this: List where your stack leaves the vendor’s default operational path — manual checkpoints, custom timers, homegrown backup jobs — and for each, write down what signal would prove a rare race is happening there. If you can’t name the signal, that’s the gap to close first.
What I want to remember
Remember: A 16-year-old bug can sit dormant until your operational pattern is the one that trips it. When a bug won’t reproduce, build counters and logs that make it visible in production — Tailscale’s decisive clue came from a recovery tool, not from the investigation.