How WALs became a thing
Have you ever wondered why most modern databases are built on top of a WAL — a write-ahead log, which is an append-only log of changes? Why does nearly every storage engine — from PostgreSQL to SQLite to Kafka-derived systems — insist on writing every change to a sequential log before anything else?
This is because sequential writes (and reads) to a hard disk drive (HDD) are much faster than random ones. But to understand why that's the case — and why an append-only log became the foundation of modern storage engines — we first need to understand how storage devices like HDDs actually work.
How HDDs work
Disk Platters and Sector Layout
Hard disk drives store data on circular platters made of aluminum. Each platter's surface is divided into concentric circular paths called tracks, which are further subdivided into segments known as sectors.
Each sector typically holds 4 KB of data, and its internal layout has four parts arranged in sequence:
- Address Header: positioning metadata that lets the read/write head confirm which track and sector it is currently over.
- Data Block: the actual 4 KB of stored data.
- Error Correcting Code (ECC): a verification block that ensures the data was written and read back accurately.
- Gap: a small physical buffer between sectors, giving the head tolerance when writing.
The Read/Write Head and Electromagnetic Movement
The read/write head sits on an arm assembly and never physically touches the disk. As the platter spins at high speed, it generates a cushion of airflow that lets the head "fly" just 15 nanometers (about 100 atoms) above the surface.
To move the head across tracks, the drive uses a voice coil motor:
- A coil of wire attached to the rear of the arm sits between two strong neodymium magnets.
- Running current through the coil turns it into an electromagnet; its field pushes against the fixed magnets, sweeping the arm across the platter.
- Reversing the current reverses the force, driving the arm the opposite way — letting the drive position the head to within roughly 30 nanometers.
Sequential vs. Random Access
Question: Based on this mechanical design, which is faster — sequential reads/writes or random reads/writes, and why?
Answer: Sequential, by a wide margin.
- Random access pays a mechanical tax (seek + rotation): with data scattered across different tracks and sectors, the voice coil motor must physically sweep the arm to the right track (seek time), and then the head must wait for the platter to spin until the target sector passes underneath (rotational latency). You pay this tax on every single access.
- Sequential access streams: contiguous sectors along the same track are read or written in one continuous flow. The head is positioned once, and data simply streams as the platter spins beneath it — no arm repositioning, no waiting.
- The scale disparity: the electromagnetic processes that magnetize or sense bits operate at electronic speeds, while moving a physical arm and waiting for platter rotation happen on millisecond scales. Physical movement is orders of magnitude slower than the bit reading/writing itself — so the less you move the arm, the faster the drive.
The Natural Conclusion
So based on all of the above, it's quite natural that reading and writing sequentially is faster at the hardware level. This physical reality dovetails directly into database design: if the underlying storage is dramatically faster at sequential I/O, then the storage engine should be designed to only ever do sequential I/O.
That raises the design question the WAL answers: how do you take an arbitrary stream of updates — inserts, deletes, overwrites, arriving in any order — and persist them using nothing but appends? To leverage the hardware's sequential speed, databases needed a data structure that turns random logical updates into sequential physical writes, and one that a full database could then be built on top of. That structure is the append-only log — and that's how WALs became a thing.
This line of thinking eventually led Patrick O'Neil, Edward Cheng, Dieter Gawlick, and Esther O'Neil to co-author the seminal 1996 paper "The Log-Structured Merge Tree", which generalized the idea into a full storage engine architecture. In turn, most modern databases — from RocksDB to Cassandra to the SSTable-based engines we use today — adopted it. Read about it in The Log Structured Merge Tree.
Related reads
- The Log Structured Merge Tree — the storage engine architecture the append-only log enables
- SSTables: The Sorted String Table — what the log's data eventually becomes on disk