study

Kleppmann & Riccomini, 2nd ed. (2025).

Chapter 2 · Defining Nonfunctional Requirements

• Tail-Latency Amplification — a backend call slow just 1% of the time, fanned out to 100 parallel calls, makes 63% of end-user requests slow. Predict "still ~99% fast"; measure ~37%.
• The Mean Hides the Tail — for right-skewed latency, 69% of requests are faster than "average," yet p99 is 6× the mean. The average overstates the typical request and understates the tail at once.
• Averaging Percentiles Is a Lie — average ten servers' p99s and you land 21% below the true pooled p99. Predict "close enough"; measure the gap.

Chapter 3 · Data Models and Query Languages

• ADD COLUMN Is Instant, Until It Isn't — adding a NOT NULL column with a default to a 10M-row Postgres table takes 0.8 ms, not the full rewrite everyone predicts — but a volatile default rewrites it in ~8 s.
• Reachability Needs Recursion — "is Boise in North America?" A two-level join silently answers no (it's 4 hops deep); WITH RECURSIVE gets it right at any depth. A fixed join count can't express transitive reachability.
• One Company, Two Names — a denormalized rename touches 666,666 rows vs 1; interrupt it and one company splits into two names — an inconsistency the normalized schema can't represent.
• The Document Wins the Whole Read and Loses the Field — a JSONB document reads ~20× fewer pages than the join for a whole profile, but ~12× more for a single field. Locality cuts both ways.
• You Can Index Inside a Document — a nested-field filter scans 2M rows (~98 ms); a GIN index makes it ~27× faster — with the catch that it serves @>, not ->> equality.
• The Ratings Matrix That's 99% Empty — a 1%-full ratings matrix is 200 MB dense vs 4 MB sparse (50×), following 1/(2×density). Netflix-scale dense is 34 GB.

Chapter 4 · Storage and Retrieval

• The Column Store Reads Only What It Needs — a 2-column query over a 30-column table makes a row store read 100% of the data and a column store 0.6% — 165× less — from layout alone.
• Sorting Rows Changes How Big They Are — a column store is 5.5× smaller than the row store; reordering the same rows shrinks it another 1.4×, one column collapsing 30×. Sort order is a compression lever.
• Ten Bits per Key Buys a 1% Miss Rate — a Bloom filter needs ~10 bits (not bytes) per key for a 0.8% false-positive rate, matching the formula — and its size is independent of key length.
• The Same Rows Cost 4× More in Random Order — load 10M rows into a B-tree in random key order and it's 4× slower with a 31% bigger index (69% vs 90% leaf fill). Why LSM writes sequentially.
• The Covering Index That Still Reads the Heap — a covering index keeps reading 1,637 heap blocks until one VACUUM sets the visibility map; then heap fetches drop to zero.
• LIKE Reads Every Row; an Inverted Index Reads Five Buffers — finding a word with ILIKE '%…%' scans 5M rows (~1.25 s); a tsvector+GIN index does it in ~19 ms, the probe touching 5 buffers — ~65× faster.
• Storing 230 MB Wrote 2 GB to Disk — an LSM (RocksDB) writes 2,091 MB to durably store 230 MB — a 9× write amplification, ~7× of it compaction — while the database ends up just 235 MB.

Chapter 5 · Encoding and Evolution

• The Same Record, 81 Bytes to 32 — one record encoded five ways: JSON 81 → MessagePack 66 → Protobuf 33 → Avro 32 bytes. Each rung drops a layer of self-description; 38% of the JSON is field names.
• Compression Closes the Gap — Avro is 2.7× smaller than JSON raw, but only 1.4× after gzip — the binary size win largely collapses once you compress a batch.
• The ID That Changed on the Way In — a 64-bit ID through a double-based JSON parser comes back off by 21, no error; a typed Protobuf int64 round-trips it exactly.
• Schema Evolution: Old Code Reads New Data, New Code Reads Old — add a field and old and new readers still interoperate (forward + backward compatible) — until you add one without a default, which breaks old data.
• The Field-Tag Landmine: Reuse a Tag, Silently Corrupt — reuse a Protobuf tag number and old bytes decode as the wrong field — a timestamp silently read as a user id, no error. A fresh tag is the only safe move.
• Varint & Zigzag — −1 Costs 10 Bytes, or 1 — a negative int64 sign-extends to 10 bytes; sint64 zigzag makes it 1. A batch of small negatives is 7.3× larger as int64.
• Avro Is the Smallest — and Unreadable Without Its Schema — Avro (21 bytes) beats Protobuf (24) by dropping tags, but decode with the wrong schema and it errors; Protobuf's tags survive a partial, renamed schema.

Chapter 6 · Replication

Reading companion — what to read closely, what to skim, where to stop and run one of these, and the passages that get misread.

• Your Own Write, Gone From the Replica — write to the leader, read an async replica, and your own comment comes back missing (count 0) for ~5 s; reading the leader is the fix.
• Refresh the Page, and the Comment Is Gone — two replicas at different lag make a user's reads run backward in time (1 then 0); sticky routing gives monotonic reads.
• The Same SQL, Two Different Rows — a follower replaying INSERT … now(), random() gets different values than the leader — why databases ship rows/WAL, not statements.
• Last-Write-Wins Discards the Write That Actually Happened Last — under a skewed clock, LWW keeps an older write and silently drops the one that truly happened last — no error, data gone.
• Quorums Overlap by w + r > n — And Still Read Stale — w + r > n forces the read and write sets to overlap (9/9 combos), yet a sloppy quorum still returns stale.
• Version Numbers Catch the Conflict LWW Hides — two concurrent writes: LWW keeps [eggs] and loses milk; version numbers keep both as siblings, losing nothing.
• A G-Counter Converges; a Naive Counter Loses 80 Increments — a naive merge keeps 100 of 180 increments (loses 44%); a G-Counter CRDT converges to the exact 180 under any merge order.

Chapter 7 · Sharding

• hash % N Is a Rebalancing Catastrophe — adding one node to ten moves 91% of all keys under hash % N; a fixed 1,000-shard layer moves just 9% — 10× less data over the wire, from indirection alone.
• Consistent Hashing Moves Only a Fair Share — rendezvous hashing moves 9% of keys and zero between nodes that didn't change; hash % N moves 91%, 163k of them pure churn between unchanged nodes.
• A Perfect Hash, and One Shard Still Melts — a good hash spreads keys evenly (1.06×), yet a Zipf celebrity key drives 19% of the load onto its one shard; splitting it helps but Zipf's head bites back (2.13×).
• Time-Ordered Keys Funnel Every Write Into One Shard — a timestamp partition key sends 100% of writes to the newest shard while 11 sit idle; a sensor-ID prefix spreads them to 1.02× the mean — at the cost of scattering the range query.
• Hashing the Key Destroys Range Queries — a 1,000-key range scan touches 1 shard under key-range sharding but all 16 under hash sharding; a composite key wins the range back.
• Local vs Global Secondary Indexes — color=red reads 8 shards locally vs 1 globally; writing one car touches 1 shard locally vs 3 globally. You choose which side pays.
• Why Cassandra Uses 16 Vnodes and ScyllaDB 256 — one random range per node leaves the busiest 2.8× over fair share; 16 vnodes cut it to 1.4×, 256 to 1.1× — the spread shrinking like 1/√V.

Chapter 8 · Transactions

Reading companion — what to read closely, what to skim, where to stop and run one of these, and the passages that get misread.

• Postgres Won't Dirty-Read Even When You Beg It To — set READ UNCOMMITTED (Postgres accepts it by name) and a concurrent session still reads the old committed value, never the uncommitted one — MVCC makes the dirty read impossible.
• Read Skew Makes $100 Vanish — Until REPEATABLE READ — under READ COMMITTED, two reads in one transaction sum to $900; REPEATABLE READ's single snapshot restores the true $1,000.
• The Lost Update: Two Increments, Final Value Off by One — two read-modify-write increments leave 43, not 44; an atomic value = value + 1 or FOR UPDATE fixes it, and REPEATABLE READ aborts one with 40001.
• Two On-Call Doctors Both Leave, and "Repeatable Read" Lets Them — both see 2 on call, both go off → zero on call; snapshot isolation permits it, only SERIALIZABLE aborts one with 40001.
• A Phantom Double-Books a Room, and FOR UPDATE Can't Stop It — both check the room is free (0) and both book it; FOR UPDATE can't lock a row that doesn't exist yet — SERIALIZABLE (40001) or an exclusion constraint (23P01) closes it.
• Under MVCC a Writer Doesn't Wait for a Reader — a writer commits in ~1 ms alongside an open reader that keeps seeing its snapshot; add one SELECT … FOR UPDATE and the writer blocks ~1 s.
• Two Transactions Deadlock — Postgres Shoots One, Doesn't Hang — opposite lock order deadlocks; after ~1 s the detector aborts a victim with 40P01 and the survivor commits. Consistent lock ordering prevents it.
• Compare-and-Set: the Loser Silently Gets UPDATE 0 — two guarded writes race; the loser's WHERE matches nothing → UPDATE 0 (a lost update prevented lock-free); at REPEATABLE READ it aborts with 40001 instead.
• One Row, Updated 100,000 Times — 8 kB or 2.75 MB? — an UPDATE is a delete + insert; 100k updates keep the HOT (unindexed) table flat at 8 kB but bloat the indexed one 344× to 2.75 MB, until VACUUM reclaims it.
• A Prepared Transaction Holds Its Lock Through a Full Server Restart — PREPARE TRANSACTION leaves an in-doubt txn that keeps its row lock and survives a restart in pg_prepared_xacts; only COMMIT/ROLLBACK PREPARED frees it.

Chapter 9 · The Trouble with Distributed Systems

• A Fencing Token Bounces a Zombie Leaseholder's Write — a paused leaseholder wakes, still thinks it's leader, and writes — corrupting storage. A monotonic fencing token the resource checks (33 < 34) makes the stale write bounce.
• There Is No Safe Failure-Detection Timeout — a "generous" 6×-median timeout still declares a healthy node dead 3.66% of the time; driving false positives to zero pushes detection delay toward infinity. A heavy tail has no safe cutoff.
• Commit-Wait: Fix an Uncertain Clock by Deliberately Waiting — a point timestamp from an uncertain clock orders causal writes backwards 38.9% of the time; Spanner's commit-wait (pause 2ε) drops it to 0 — the fix is going slower on purpose.
• Measuring a Duration With the Wall Clock Goes Negative — an NTP step-back mid-measurement makes wall-clock elapsed −3.000 s (a lease that never expires); the monotonic clock ignores the step and reads the true +2.000 s.

Chapter 10 · Consistency and Consensus

• A Linearizability Checker Rejects a History That "Looks Consistent" — every read returns a value that was really written, no op-by-op contradiction — yet a checker flags it not linearizable: once A read the new value, a later B may not read the old one.
• A Lamport Clock Inverts a Cross-Shard "Then", Leaking a Private Photo — set-private (ts 8) then upload (ts 2): with no message between shards there's no causal edge, so a snapshot read leaks the photo. A linearizable ID generator closes it.
• CAP Made Countable: a Partition Costs 12 Refusals or 3 Conflicts — the same partition makes the CP deployment refuse 12 of 20 requests and the AP one diverge on 3 keys. You don't pick 2 of 3; you pick which cost to pay.
• SERIALIZABLE Still Returns a Stale Read — It Is Not Linearizable — under SERIALIZABLE, a transaction's second read still returns 0 after another commits 1, and commits with no error — serializable order isn't real-time order.
• Only the Majority Side of a Partition Can Decide — split 5 nodes 3–2 and the minority just stops; and since every decision needs a majority round-trip, adding nodes raises fault tolerance but not throughput.
• A Backward NTP Step Reorders Timestamps; an HLC Stays Monotonic — a −100 ms step drops a later event's physical ts to 940 after 1030 (effects before causes); a hybrid logical clock absorbs it into a counter and stays ordered.
• A Lamport Timestamp Orders Two Concurrent Events; a Vector Clock Refuses — L(e9)=3 < 4=L(e4) tempts "e9 before e4", but the vector clocks are incomparable — genuinely concurrent. One integer can't tell order from coincidence.

Chapter 11 · Batch Processing

• An Aggregation's Memory Tracks Distinct Keys, Not Input Size — 2,000,000 identical lines cost a hash table one dict entry; hold the line count fixed and its memory swings 7.5× with distinct-key count while sort barely moves.
• A Broadcast Join Moves 124× Less Data Than a Reduce-Side Join — a reduce-side join shuffles both tables (60.5 MB); broadcasting the small side moves 500 KB for the identical result — 124× less network.
• One Hot Key Sends 85% of a Join to a Single Reducer — a celebrity key pins 86.9% of the job on one reducer; 8× the reducers buys ~1× the speedup. Salting the key cuts it to 17.9%.
• Materializing Between Stages Blocks the First Output — MapReduce writes a 3×-replicated intermediate and needs all 100k records before its first output; a pipelined dataflow emits after 101 — a 990× gap.
• Loading a Batch Job's Output Row-by-Row Is ~900× Slower — committing one row at a time takes ~28 s for 50k rows vs 0.03 s bulk (~900×), and a crashed retry duplicates rows that bulk-then-swap never leaks.

Chapter 12 · Stream Processing

• Windowing by Processing Time Invents a 61× Spike — a steady 1 event/second, bucketed by processing time after a 60 s consumer outage, shows a 61× spike that never happened; event-time buckets stay flat.
• A Straggler Silently Undercounts a Fired Window — a window fires at count 5; a late event for it is dropped with no error (published 5, true 6). A watermark + allowed-lateness emits the correction to 6.
• One Stream, One "5-Minute" Width, Four Different Groupings — two events 4m33s apart: a sliding window groups them, tumbling splits them, hopping shares none, session groups them. Alignment, not duration, decides.
• At-Least-Once Delivery Double-Counts a Sum — a crash between applying an event and checkpointing its offset replays 10 events → total 110, not 100; stamping the offset and skipping duplicates fixes it to 100.
• Dual Writes Diverge Two Stores Forever — interleaved writes leave the DB on B and the index on A with no error; routing both through one ordered CDC log converges them.
• A Compacted Changelog Rebuilds the Exact Database From 0.1% of the Records — 1,000,000 writes compact to 937 records (one per live key, 1,067×); replaying only those reconstructs the database exactly.
• A Naive Stream-Table Join Time-Travels — Replay Changes the Answer — a current-state tax lookup bills old sales at today's rate: $100 today but $80 if replayed last year; a point-in-time join gives a stable $94.
• Load Balancing Plus Redelivery Reorders an "In-Order" Stream — a stalled-then-redelivered message completes as m1, m2, m4, m3, m5 across two consumers; a single partitioned log preserves the order (but head-of-line blocks).

Chapter 13 · A Philosophy of Streaming Systems

• Retrying an "Atomic" Transfer Charges $22, Not $11 — atomicity isn't idempotence: a retried transfer moves $22, no error. A client-generated request ID with a UNIQUE constraint makes the retry a no-op — the end-to-end argument.
• A Multishard Transfer Stays Exactly-Once Through a Crash — Without 2PC — atomically append one request, derive both shard effects deterministically, dedup by ID: the money invariant (sum 200) survives a mid-transfer crash; drop the dedup and the payee is double-credited (211).
• Uniqueness Needs a Total Order; Async Multi-Leader Grants 'alice' Twice — two independent leaders each find alice free and both grant it (2 owners); routing claims through one totally-ordered partition grants exactly 1.
• Losing Causal Order Across Two Logs Notifies the Ex You Unfriended — unfriend-then-post across two independent logs arrives reordered, so the notification fires anyway; one ordered partition, or a causal-version reference, suppresses it.

Chapter 14 · Doing the Right Thing

• Dropping "Race" Doesn't Debias the Model — a Proxy Relearns It — "fairness through unawareness" fails: the disparate-impact ratio barely moves (0.29 → 0.44, still failing the 0.80 rule) when you drop the protected attribute, because a neighborhood proxy reconstructs it (AUC 0.96). Drop the proxy too and it reaches 1.03.

Provenance is the PostgreSQL 16 documentation rather than a book. Every transcript below was re-captured on PostgreSQL 16.14 (macOS 26.5.2 / arm64, Docker Engine 29.6.2, image digest pinned) — see the backlog, which records what re-verification changed.

Concurrency & locking

• Transaction Isolation Levels — walk READ COMMITTED → REPEATABLE READ → SERIALIZABLE in two psql sessions; each level produces the anomaly the next one prevents.
• Lock Contention — block a session on purpose, then walk from "something is slow" to the exact process holding the lock via pg_stat_activity, pg_blocking_pids() and pg_locks. A blocked query is not a black box.

Storage & reclamation

• MVCC and Vacuum — an UPDATE is an INSERT plus a tombstone, and the idle transaction everyone blames for blocking VACUUM turns out not to.

Query planning

• Index vs. Sequential Scan — four access paths priced by page count; the planner declines the index you just built at 99% selectivity, and is provably right.
• Query Planner Statistics — every plan is a bet on a guessed row count. When rows= and actual rows= disagree by a factor of a million, you get a Nested Loop where a Hash Join belonged — same SQL, same schema, same data.

Durability & replication

• WAL and Crash Recovery — when COMMIT returns your rows are not in the table file. SIGKILL the server and watch the table get reconstructed from the log.
• Replication Lag — write a million rows to the primary and watch the replica answer 1, then watch the byte distance between the two nodes climb and drain.

Provenance is the Docker docs and the Linux namespaces(7) / cgroup-v2 kernel documentation. Every transcript re-captured on Docker Engine 29.6.2 (Linux VM kernel 6.12.76-linuxkit), images pinned — see the backlog.

Isolation primitives

• Namespaces in Isolation — there is no container object in the kernel. One ordinary process tree, and per-process filters over which kernel-table entries you may see; docker run is a clone() with extra flags.
• cgroup Memory Limits — --memory=50m writes a number into a file. Exceed it and the kernel doesn't return NULL from malloc or raise anything — it sends SIGKILL, and your process gets no turn.

Filesystem & networking

• Union Filesystem Layers — an image is a stack of tarballs union-mounted topmost-wins, not a disk image; a container adds exactly one thin writable layer, which is why two containers from one image cost almost nothing.
• Container Networking — a healthy resolver at 127.0.0.11 answers NXDOMAIN for a container one bridge away, and DNS and reachability fail independently.