1. What It Is
Every durable database you name in an interview almost certainly uses a write-ahead log. We append changes to the WAL before applying them to data pages — that's how crash recovery works.
What:
Write-Ahead Logging (WAL) is an append-only sequential log stored on disk.
Primary purpose:
Providing ACID transaction durability and crash recovery at sub-millisecond latencies.
Usually used for:
Relational databases, key-value stores (LSM-Trees), message brokers, and consensus state machines.
2. Core Mental Model
WAL is the durability backbone inside PostgreSQL, RocksDB, Redis AOF, and Raft logs. If your design needs crash-safe commits or CDC, you need WAL semantics. If the store is purely ephemeral (session cache) or eventually consistent (analytics counter), skip it.
✍️ Log Intent Sequentially
Always record transaction intents to the append-only WAL first. Only update complex in-memory indexes and database pages after the log is written.
⚡ Sequential > Random I/O
Writing sequentially to disk is extremely fast, while updating random leaf pages inside B+Tree files requires slow disk heads search times.
🔄 Recoverable State
If power fails, memory is lost, but the WAL survives. Scan the log on boot to redo committed transactions and undo uncommitted updates.
In the room
WAL connects replication and durability: async replicas trail the primary's WAL; sync replication waits for WAL flush on replicas. If they ask "what happens on crash," say uncommitted WAL entries are replayed on restart.
3. Why It Matters in HLD
WAL is how databases survive crashes — write the log first, apply to pages later. Three lenses:
Needed When:
You require strict financial or transactional durability (ACID), fast write times, or real-time logical backup replication.
Avoids:
Silent database page corruptions, write bottleneck queues on live tables, and mismatched state machines after system crashes.
Optimizes For:
Transaction commit latency, hardware failure tolerance, point-in-time state recovery, and high write volume scaling.
4. Architecture & Data Flow
Walk the WAL path as interview steps. Step 1 — Commit: transaction appends record to WAL on disk. Step 2 — fsync: force WAL to persistent storage before ACK. Step 3 — Apply: background process updates in-memory/disk pages. Step 4 — Checkpoint: flush dirty pages, truncate old WAL segments. Step 5 — Replica: stream WAL to followers for replication.
The Crash Recovery Pipeline
When database processes crash, physical memory pages are wiped. On boot, the engine replays the log forward from the last known checkpoint:
In the room
When they ask "what happens on crash mid-transaction," walk WAL replay — uncommitted records roll back, committed records replay. That beats hand-waving about "ACID."
5. Key Characteristics
Write-ahead ordering, checkpoint frequency, and group commit — key characteristics we cite:
- Log Sequence Number (LSN): An ever-increasing unique integer assigned to every WAL transaction to verify precise sync order.
- Sequential Disk I/O: Eliminates mechanical head latency by only appending records to the end of the log file.
- Checkpointing: Background sweep flushes all dirty memory pages to disk, letting the engine safely truncate old WAL segments.
- Fsync policy trade-offs — durability vs latency vs throughput:
| Policy | Durability | Latency | Throughput |
|---|---|---|---|
fsync every commit | Max (Zero data loss) | High (~5-15ms per write) | Low |
fsync every N seconds (group commit) | Medium (Lose up to N sec) | Low | High (batched sequential writes) |
No fsync (OS buffered) | Low (All dirty buffer at risk) | Lowest (sub-millisecond) | Highest |
6. Strategic Tradeoffs
Durability guarantees cost write latency — we compare sync vs group commit:
| Benefit | Cost |
|---|---|
| Guaranteed Durability (ensures zero data loss under crash conditions by logging intents) | Hot Disk Hotspots (the WAL disk is a highly concurrent single point of write pressure) |
| Ultra-Fast Writes (replaces heavy, random B+Tree page writes with O(1) sequential appends) | Disk Space Consumption (un-checkpointed logs can quickly eat gigabytes of storage) |
| Replication Stream & Point-In-Time Recovery (simplifies log-shipping replication and logical rollback) | Recovery Startup Delay (crashed databases can take minutes to replay WAL at startup) |
7. Failure / Bottleneck Awareness
Disk full on WAL volume, replay time after crash — we name failure modes:
Problem: Running high-concurrency database queries with fsync every commit forces the physical storage controller to sync disk operations repeatedly, choking throughput and pushing latency to ~15ms.
Mitigation: Implement Group Commits (batching multiple concurrent transactions into a single physical WAL sync) or utilize high-speed NVMe storage with battery-backed write caches.
Problem: When database checkpoint sweeps trigger, the engine flushes massive amounts of dirty in-memory pages to database files on disk, saturating the disk controller and inducing user latency spikes.
Mitigation: Tune rate-limited checkpointing parameters (e.g. Postgres's checkpoint_completion_target) to spread page flushes continuously over time instead of in single burst storms.
Problem: Long-running transaction or broken replication listener prevents database engine from advancing checkpoints, letting WAL files accumulate indefinitely until disk space is fully exhausted.
Mitigation: Deploy disk alert monitors and configure strict max WAL segment storage limits (e.g., PostgreSQL's max_wal_size or active replica limits).
8. Common HLD Usage
PostgreSQL, MySQL, and Kafka all lean on append-only log durability:
| Problem | Usage |
|---|---|
| Payment Transaction Ledger | WAL with synchronous fsync on every commit to prevent double-spending |
| Redis Append-Only File (AOF) | WAL layer written asynchronously to disk to reconstruct Redis cache on reboot |
| LSM-Tree Storage Engine (RocksDB) | Sequential WAL protecting in-memory MemTable before data is flushed to SSTables |
| Distributed Consensus Logs (Raft/Paxos) | Raft log acts as the WAL replicated to a majority quorum before execution |
| Database Change Data Capture (CDC) | CDC pipelines stream updates by direct-parsing the database engine's binary WAL |
9. Decision Signals
Discuss WAL when explaining how databases achieve durability and replication:
- You are designing storage engines, transactional ledgers, or custom databases.
- You must support real-time point-in-time data recovery (PITR).
- You require absolute guarantees that "acknowledged" writes survive complete hardware power failures.
- You need to replicate transactions across network nodes reliably (log shipping / streaming replication).
- You face heavy write workloads and want to avoid random-access disk write overhead in the transaction path.
- Data is ephemeral (Redis cache without AOF, session tokens) — durability is not required.
- You use managed append-only logs (Kafka, concept #18) at the application layer — the broker's log replaces a custom WAL.
- Eventual consistency is acceptable and you already have replication via gossip or quorum (concept #08) — WAL is an implementation detail inside the DB, not your design focus.
11. Deep Dive (Optional)
Fsync Group Commit Internals
Under heavy write concurrency, engines batch commits into a single fsync via group commit instead of flushing after every transaction — trading a few milliseconds of latency for much higher throughput.
ARIES Recovery Algorithm Internals
Most recovery implementations follow the ARIES model in three phases on boot:
- Analysis Phase: Scan the WAL forward starting from the last checkpoint to identify active transactions (loser list) and dirty pages in memory at the moment of the crash.
- REDO Phase: Replay all logged operations forward (both committed and uncommitted) to return the database state to the exact point of crash.
- UNDO Phase: Scan the log backward to rollback (reverse) all transactions that were active but uncommitted (loser list) during the crash, ensuring database atomicity.
Streaming Replication & Log Shipping
In high-availability configurations, WAL is shipped across network nodes to keep replica databases synchronized:
- File-Based Log Shipping: The primary database writes complete WAL segment files (typically 16MB) and sends them to replicas on completion. This introduces data lag up to the segment write time.
- Streaming Replication: Replicas connect directly to the primary's WAL stream, receiving real-time byte updates down to the Log Sequence Number (LSN) level, reducing latency to near zero.
Redis Append-Only File (AOF) Compaction
In Redis, WAL is represented by the AOF file. Because Redis records every mutate command sequentially, AOF file sizes grow rapidly over time. To prevent disk overflow, Redis executes **AOF Rewrite** in the background: a child process forks, scans the current memory database state, and writes the minimum required commands to represent the final state (e.g., compaction of 100 increments into a single set command), cleanly replacing the historic WAL log.
Review
How helpful was this walkthrough?
Click a star to rate. We actively use this feedback to refine and update our system design content.
Discussion
Share your thoughts, ask questions, or help others.