System design reference

03-system-design-curriculum.md tells you what to know and how to run the 45 minutes. This file is the depth behind it — the tradeoffs you need to be able to state out loud when an interviewer stops nodding and says “why?”.

How to use it: do not read this front-to-back. Work a problem, get stuck or hand-wave a component, then come back and read that one section. Week 3 and Week 4 ask you to write one page per building block in your own words — read the section here, close it, and write yours. The writing is the point; re-reading is nearly worthless.

Every section ends with “what scores” — the specific sentence that separates an L5 answer from an L4 one on that topic.

Contents

1. The framework, expanded
2. Estimation you can do out loud
3. Networking and entry
4. Storage
5. Distribution
6. Caching
7. Async and data flow
8. Reliability and operations
9. The 12 problems, one level deeper
10. The papers, and the tradeoff each one made
11. Language that scores, and language that sinks

1. The framework, expanded

The time-box lives in the curriculum. What follows is what “good” actually looks like inside each phase.

0–5 min · Scope

You are not gathering requirements. You are demonstrating that you narrow an unbounded problem before touching it — that is the GCA signal, and it is scored in every round.

Ask about, in roughly this order:

• Who the users are and what the one core flow is. “Is this consumer-scale public, or internal?”
• Read/write ratio. This single number drives caching, replication and storage choice more than any other. Feeds are ~100:1 read-heavy; metrics ingestion is write-heavy. Say which you’re assuming.
• Scale, but only as an order of magnitude. 10K users and 10M users are different systems; 10M and 12M are not.
• Latency target, and for which operation. “p99 under 200 ms on the read path; writes can be async” is a design constraint. “Fast” is not.
• Consistency requirement. Can a user read their own write immediately? Can two users briefly disagree? This is where most candidates under-ask.
• Explicitly out of scope. Say “I’m going to assume auth, abuse prevention and billing are handled elsewhere unless you want them” — then the interviewer can pull one back in if they care.

Write the requirements where you both can see them. Number them. You will refer back to them in the deep dive, and that is what makes the scoping look deliberate rather than ceremonial.

What scores: turning a vague ask into 3–5 functional and 3–4 non-functional requirements, and then actually using one of them later to reject a design option.

5–10 min · Estimate

See §2. The rule that matters: an estimate you never use is theatre. Compute the numbers that change a decision, and say which decision.

What scores: “That’s ~4 TB/year, which fits comfortably on one machine’s disk — so I’m not going to shard for capacity, only later for throughput if we need it.”

10–15 min · API and data model

Define 3–5 endpoints with real signatures — name, parameters, return shape, and pagination style. Then the entities, and the access patterns: which queries run, how often, by what key.

Say explicitly: access patterns drive the storage choice. Then choose storage from them, not from a vibe about scale.

What scores: cursor pagination over offset, with the reason — offsets drift when items are inserted, and get slower deeper into the result set.

15–30 min · High-level design

Boxes, in the order data flows: client → LB → service(s) → cache → storage, with async workers hanging off a queue. Draw it once, cleanly. Do not redraw.

Justify every box as you place it. A component you cannot justify is a component you should delete — and deleting it out loud is a positive signal, not a retreat.

Keep it a monolith-plus-storage until something forces a split. Premature microservices with no stated reason reads as cargo-culting. Good reasons to split: independent scaling profile, independent failure domain, a different team owning it.

What scores: “I’ll keep read and write paths in one service for now; if the read path needs to scale 100× harder, I’d split them so they can scale independently — but I don’t have evidence for that yet.”

30–40 min · Deep dive

This is the round. If the interviewer doesn’t pick, propose the most interesting component and say why it is interesting. Good default picks:

• the sharding key, and what happens on rebalance
• the hot key / celebrity problem
• the consistency guarantee on the critical path
• the fan-out strategy
• the index that makes the main query fast

Go deep enough that an explicit tradeoff appears. “I’d use consistent hashing” is not a deep dive. “Consistent hashing with ~200 virtual nodes per physical node, because plain modulo remaps nearly every key when the cluster resizes, and virtual nodes keep the load imbalance within a few percent” is a deep dive.

What scores: naming what breaks in your own design before being asked, then fixing it.

40–45 min · Failures, scale, wrap

Walk one failure at a time: a service instance dies, the cache dies, the primary dies, a region dies, a dependency gets slow (not down — slow is worse). For each: what the user sees, and what the system does.

Close by saying what you’d do with more time. That is not an apology; it shows you know what you left out.

What scores: treating degraded as a first-class state. “If the ranking service times out, I serve reverse-chronological and flag it in the response, rather than failing the request.”

2. Estimation you can do out loud

The only numbers to memorize

Seconds in a day        ≈ 86,400   →  round to 10^5
Seconds in a year       ≈ 3.15 × 10^7
1 million/day           ≈ 12 per second
1 billion/day           ≈ 12,000 per second
Peak                    = 2–5× average. Say which multiplier and why.

Latency, and what each number implies

That last row is the one that earns credit. Any design promising sub-100 ms globally from a single region is wrong on physics, and saying so is a strong signal.

The four estimates, in order

1. QPS. DAU × actions per user per day ÷ 10^5. Then peak = 2–5×.
2. Storage. writes/day × bytes/write × retention. Then ask: does this fit on one machine? A single machine today holds tens of TB. Most interview systems do not need sharding for capacity.
3. Bandwidth. QPS × payload. Matters most for media; this is what puts a CDN in the design.
4. Memory for cache. Apply the 80/20 rule: caching the hot 20% usually serves ~80% of reads. Working set = daily reads × 0.2 × object size. If that fits in RAM, say so — it’s often the single biggest latency win available.

Round aggressively. 86,400 → 10^5. Say “call it” out loud. Precision here is worthless and slow; the interviewer is watching whether the number changes what you build.

What scores: an estimate that kills an option. “At 50 GB total, a distributed cache is unnecessary complexity — this fits in memory on each replica.”

3. Networking and entry

DNS, anycast, CDN

DNS resolves a name to an IP and is cached at every layer, which makes DNS-based failover slow — TTLs are honoured unevenly and stale resolvers exist. Do not present “update DNS” as a fast failover mechanism.

Anycast advertises one IP from many locations; BGP routes the user to a topologically near site. This is how a global entry point gets low latency without the client choosing a region.

CDN caches static and cacheable dynamic content at the edge. It buys you three things: latency (content is near the user), origin offload (most requests never reach you), and absorption of traffic spikes. Key decisions: what is cacheable, TTL, and invalidation strategy. Versioned URLs (/asset.v7.js) sidestep invalidation entirely and should be your default answer.

Load balancing: L4 vs L7

L4 balances on IP/port. It’s fast, protocol-agnostic and cheap, but it cannot see the request, so it can’t route on path or do per-request retries.

L7 parses the request. You get path/header routing, TLS termination, per-request retries, sticky sessions and request-level metrics — at higher cost per request.

Real systems use both: L4 at the edge for raw throughput, L7 inside for routing. Say that.

Health checks must be deep enough to mean something but shallow enough not to cascade. A check that hits the database means one slow database marks every instance unhealthy and you take the whole service down yourself. Separate liveness (am I alive?) from readiness (should I get traffic?).

Balancing algorithms: round-robin is the default; least-connections is better when request costs vary; consistent hashing is used when you want the same key to land on the same instance for cache locality.

API gateway

One entry point doing auth, rate limiting, routing, request shaping and observability. The benefit is that cross-cutting concerns live in one place; the risk is that it becomes a single point of failure and a deployment bottleneck. Mention both.

Long polling vs SSE vs WebSockets

The real cost of WebSockets is not bandwidth — it’s state. A stateful connection means your load balancer can’t freely move the client, a deploy drops every connection, and you now need a registry mapping user → server so a message can find its recipient. Choose SSE when the traffic is one-directional; it is dramatically cheaper operationally.

What scores: choosing SSE over WebSockets for a notification feed and explaining that you avoid connection-state management for a flow that only ever goes one way.

4. Storage

SQL vs NoSQL — pick on access patterns, never on “scale”

“NoSQL because it scales” is the single most common L4 answer in this round. Modern SQL shards and replicates fine; the real question is what your data and queries look like.

Choose relational when: the data is relational and you need joins; you need multi-row transactional integrity; the query patterns will change (a flexible query language is the whole point); or you need strong secondary indexes.

Choose a document store when: entities are self-contained, are read whole, and the schema varies per record.

Choose a wide-column store (Bigtable, Cassandra) when: the workload is write-heavy, the access pattern is a known row-key lookup or range scan, and you want linear horizontal scaling. The schema is effectively a sorted map — (row, column, timestamp) → value — and row-key design is the whole design.

Choose a key-value store when: access is strictly by primary key and you want the lowest possible latency.

What scores: “Every query here is ‘get the last N events for this user’, which is a row-key prefix scan. That’s a wide-column store, and I’d make the row key userId#reverseTimestamp so the newest rows sort first.”

B-tree vs LSM-tree

This is the tradeoff behind almost every storage choice, and knowing it by name is a differentiator.

B-tree (Postgres, MySQL/InnoDB): updates in place. Reads are predictable — a handful of seeks. Writes cost random I/O and must be journaled for crash safety. Good for read-heavy and mixed workloads.

LSM-tree (Bigtable, Cassandra, RocksDB, LevelDB): writes go to an in-memory table plus a sequential write-ahead log, later flushed to immutable sorted files (SSTables) and merged by compaction. Writes are sequential and therefore very fast. Reads may have to check several files — mitigated by bloom filters and block indexes.

The cost people forget: compaction. It consumes disk I/O and CPU in the background, and it causes latency spikes on the read path at the worst moments. Naming compaction as the price of write throughput is exactly the depth this round is looking for.

Indexing

An index trades write cost and storage for read speed. Every additional index makes every write slower — say this when you add one.

• Primary/clustered: the table’s physical order. One per table.
• Secondary: a separate structure pointing back to rows.
• Composite: order matters. An index on (a, b) serves queries on a and on a, b — not on b alone. This is a common interview probe.
• Covering: contains every column the query needs, so the query never touches the base table.

Why secondary indexes are hard once sharded: the index entry and the row can live on different shards. Two options, and you should name both. Local (per-shard) index: writes stay local and cheap, but a query without the shard key must scatter to every shard and gather. Global index: queries are cheap, but each write now touches two shards and needs a distributed transaction or an async pipeline to stay consistent. There is no free option — pick one and defend it.

Object vs block vs file

Object storage (GCS, S3): immutable blobs by key, effectively unlimited, cheap, high latency, HTTP access, no partial writes. This is where images, video, backups and logs go. Never store blobs in your database — store them in object storage and keep the URL in the row. Uploads should go direct from client to object storage via a pre-signed URL, so bytes never pass through your service. That one detail signals real experience.

Block storage: a raw virtual disk attached to one machine. This is what your database runs on.

File storage: a shared POSIX filesystem (NFS-style). Convenient, hard to scale, usually the wrong answer at interview scale.

Time-series and columnar

Time-series data (metrics, events) is append-only, queried by time range, and aggregated rather than read row-by-row. That justifies a dedicated store: timestamp-ordered layout, aggressive compression of similar adjacent values, and downsampling — keep full resolution for hours, then roll up to minute and hour granularity for older data. Retention is a first-class design decision, not an afterthought.

Columnar stores (BigQuery-style) read only the columns a query touches and compress each column independently, which is why analytics belongs on a separate path from your serving database. Running analytics on the primary is how you take down the serving path.

What scores: separating the serving path from the analytics path, and naming the pipeline between them, rather than adding “and also we run reports on it” to the primary database.

5. Distribution

Partitioning (sharding)

Range partitioning: keys split into contiguous ranges. Range scans are efficient; the danger is hotspots. Partitioning by timestamp means every write goes to the newest shard — a classic self-inflicted wound worth naming.

Hash partitioning: hash the key, take the bucket. Distribution is even; range scans are gone.

Consistent hashing: keys and nodes both map onto a ring; a key belongs to the next node clockwise. When a node joins or leaves, only the keys between it and its neighbour move — roughly 1/N of the data, instead of nearly all of it under plain modulo. Plain hashing is only acceptable if the node count never changes, which it always does.

Virtual nodes fix the remaining problem: with one point per node the ring is lumpy and a departing node dumps its entire load onto one neighbour. Give each physical node ~100–200 ring positions and both the imbalance and the failover spike smooth out.

Choosing the shard key is the highest-leverage decision in the design. It must spread load evenly, and it must be present on your most common query, or every read becomes a scatter-gather. When one key is inherently hot, split it: append a bucket suffix (celebrity#0..N) and fan reads across the buckets.

Rebalancing: never rebalance by recomputing hash % N. Use consistent hashing, or allocate a fixed large number of logical partitions (say 1024) up front and move whole partitions between physical nodes. Moving a partition means copying data while it is being written to — so you double-write, backfill, verify, then cut over. Describing that sequence is a strong senior signal.

Replication

Leader–follower: one node takes writes and streams a log to followers that serve reads. Simple, and the default for good reason. Two questions decide the behaviour:

• Synchronous or asynchronous? Sync means a write isn’t acknowledged until a follower has it — durable, slower, and unavailable if a follower is down. Async is fast but loses recently acknowledged writes on failover. Semi-sync (wait for exactly one follower) is the usual compromise, and naming it is a good sign.
• How is failover handled? Automatic failover risks split brain — two nodes both believing they lead. That’s what fencing tokens and a consensus-backed lock service exist to prevent.

The everyday trap: reading from an async follower right after a write returns the old value. Fix it by routing reads-after-write to the leader for a short window, or by tracking the write position per session.

Multi-leader: writes accepted in several regions. Low local write latency, but conflicts are now guaranteed and you must resolve them — last-write-wins (simple, silently loses data), per-field merge, or CRDTs (correct, complex).

Leaderless (quorum): write to W nodes, read from R of N. If R + W > N, a read is guaranteed to see at least one node with the latest write. W=N, R=1 optimizes reads; W=1, R=N optimizes writes. Sloppy quorums keep you available during partitions at the cost of temporary inconsistency, repaired later by read repair and anti-entropy.

Consistency models

Ordered from strongest and most expensive:

• Linearizable / strong: behaves as a single copy; every read sees the latest write. Costs a consensus round trip, so cross-region strong consistency costs you the RTT — unavoidably.
• Sequential: all nodes see operations in the same order, not necessarily real-time order.
• Causal: operations that are causally related appear in order everywhere; concurrent ones may not. This is usually the right answer for social and messaging systems — a reply must never appear before the message it answers, but unrelated posts can float.
• Read-your-writes: you always see your own updates. The minimum most products actually need.
• Monotonic reads: you never see time go backwards. Pin a session to a replica and you get this.
• Eventual: replicas converge, given quiet. Fine for counters, view counts and feeds.

Pick per-operation, not per-system. “Money movement is strongly consistent; the balance display can be eventually consistent” is a much better answer than picking one for the whole design.

CAP, and why PACELC is the better tool

CAP says: during a network partition, choose availability or consistency. Its limitation is that partitions are rare, so CAP says nothing about normal operation.

PACELC completes it: if Partitioned, choose A or C; Else, choose Latency or Consistency. The “else” branch is where your system lives 99.9% of the time, and it’s the interesting tradeoff: every synchronous replication or consensus round trip you add is latency you’re paying, all day, to make the rare partition safer.

What scores: “During a partition I’d stay available and reconcile after, because a stale count is acceptable here. But even without a partition, I’m choosing latency over consistency on this path — and that means clients must tolerate a stale read.”

Consensus

Raft and Paxos solve one problem: get a majority to agree on an ordered log of operations. At interview depth, know this much — leader election plus a replicated log, requiring a majority quorum, which is why clusters are odd-sized (3, 5) and tolerate (n−1)/2 failures.

Know the cost: every committed write needs a majority round trip. Across regions that’s tens of milliseconds. This is why consensus is used for metadata and coordination — cluster membership, shard assignment, leader locks, configuration — and rarely on the per-request data path.

Do not hand-roll it. The correct answer is “a lock service” or “an existing consensus-backed store”, which is precisely the lesson of Chubby.

Distributed transactions

Two-phase commit: a coordinator asks all participants to prepare, then tells them to commit. Correct, and it blocks — participants hold locks while awaiting the decision, and a coordinator crash between phases leaves them stuck. Avoid it on hot paths.

Saga: a chain of local transactions, each with a compensating action to undo it. No global locks, and the system is temporarily inconsistent by design. You need compensations to be idempotent, and you must accept that some states are visible mid-saga.

Outbox pattern: the reliable way to “update the database and publish an event”. Write the row and the event into the same database transaction, then a relay tails the outbox table and publishes. This removes the dual-write failure where one succeeds and the other doesn’t. Knowing this by name is a strong signal.

Idempotency keys: the client sends a unique key with the request; the server stores the result against that key and replays it on retry. This is what makes at-least-once delivery safe, and it belongs in almost every design that involves retries or payments.

6. Caching

Placement

Client → CDN → API gateway → in-process → distributed cache → database’s own buffer pool. Each layer is faster and less consistent than the one below it. Name the layer you mean.

In-process (a map in the service) is ~100 ns and free, but each instance has its own copy, so invalidation is now a distributed problem and the hit rate falls as you add instances. Distributed (Redis/Memcached) is one shared truth at ~1 ms, with a network hop and a new dependency to operate.

Write strategies

• Cache-aside (lazy loading): application checks the cache, on a miss reads the DB and populates it. The default. Only cached data is ever requested data. Downside: every miss pays full latency, and the cache can go stale on writes unless you invalidate.
• Write-through: write to cache and DB together. Cache is never stale; writes are slower; you cache data nobody reads.
• Write-back: write to cache, flush to DB asynchronously. Fastest writes, and you lose data if the cache dies before the flush. Only acceptable where loss is tolerable.

Default to cache-aside with explicit invalidation on write, and say why.

Eviction and TTL

LRU is the sane default. LFU resists a scan wiping your hot set but adapts slowly to shifting popularity. TTL is your safety net against stale data and the bug you haven’t found yet — put a TTL on everything, even data you invalidate explicitly.

The three failure modes worth naming

Cache stampede / thundering herd. A hot key expires; a thousand concurrent requests all miss and hit the database simultaneously. Fixes: request coalescing (one request recomputes, the rest wait on it), jittered TTLs so keys don’t expire in lockstep, and early recomputation before expiry.

Cache penetration. Requests for keys that don’t exist bypass the cache every time — trivially weaponizable. Fix: cache the negative result with a short TTL, or front it with a bloom filter.

Hot key. One key exceeds a single cache node’s capacity. Fixes: replicate that key across nodes; split it into key#0..N and read a random bucket; or add a small in-process cache in front, which is often enough on its own.

What scores: volunteering the stampede before being asked, and fixing it with coalescing plus jitter rather than just “increase the TTL”.

7. Async and data flow

Queue vs pub/sub vs log

Message queue: work distributed to competing consumers; each message handled once. Use for tasks — send the email, transcode the video.

Pub/sub: each subscriber gets every message. Use for fan-out notification.

Log-based streaming (Kafka): an ordered, durable, replayable partitioned log. Consumers track their own offset. The killer property is replay — a new consumer, or one recovering from a bug, can re-read history. Queues delete on acknowledge; logs retain. That difference decides which one you want.

Kafka mechanics worth stating: ordering is guaranteed within a partition only, so the partition key determines what is ordered (key by userId and that user’s events stay ordered). Parallelism is capped by partition count. A consumer group splits partitions among its members.

Delivery semantics

• At-most-once: fire and forget; may lose messages.
• At-least-once: retry until acknowledged; may duplicate. This is the realistic default.
• Exactly-once: does not exist end-to-end across systems. What exists is at-least-once delivery plus idempotent processing, which produces an exactly-once effect. Say it that way — it is a precise, well-known distinction and it lands well.

Idempotency in practice: a natural unique key (order ID) plus a dedup store, or a conditional write that fails if the record already exists.

Batch vs stream

Batch (MapReduce-style): high latency, high throughput, easy to reason about, trivially re-runnable over a fixed input. Still correct for daily aggregation and backfills.

Stream: continuous, low latency, and it must handle late and out-of-order events.

Streaming vocabulary you should use by name:

• Event time vs processing time — when it happened vs when you saw it. Always aggregate on event time.
• Windows — tumbling (fixed, non-overlapping), sliding (overlapping), session (gap-delimited).
• Watermark — the system’s assertion that events older than T have probably all arrived. It decides when a window can close.
• Allowed lateness — what happens to an event that arrives after the watermark: drop it, or re-open the window and emit a correction.

The pragmatic architecture: stream for fresh approximate numbers, batch nightly for the correct ones, and reconcile. Say you’d reconcile — that’s the senior half of the answer.

Backpressure and retries

Backpressure is what a system does when it cannot keep up. The choices are: block the producer, buffer (until you run out of memory), or shed load — reject explicitly. Shedding fast is usually correct; an unbounded buffer just moves the outage and makes it harder to diagnose.

Retries must be exponential and jittered. Synchronized retries from thousands of clients are a self-inflicted DDoS, and the jitter is what prevents it. Always cap attempts, and always pair retries with idempotency or you will duplicate work.

Dead-letter queue: after N failures, move the message aside rather than blocking the partition behind it, and alert. Without a DLQ one poison message stalls a whole consumer group.

8. Reliability and operations

SLI, SLO, error budget

An SLI is a measured number (p99 latency, success rate). An SLO is the target (99.9% of requests succeed). The error budget is what’s left — 99.9% over 30 days permits about 43 minutes of failure.

Why it matters in an interview: the error budget converts reliability from an argument into a number. “We’re inside budget, so we ship the feature; we’ve burnt it, so we freeze and fix.” That framing is a Leadership signal as much as a technical one.

Always specify percentiles. Averages hide the users who are suffering. p99 is the number that describes your worst customers, and in a fan-out system it’s the number that describes nearly everyone — if one request touches 100 services, almost every request hits someone’s p99.

Failure containment

• Timeouts. Every remote call needs one. A call with no timeout is a resource leak waiting for a bad day. Your timeout must be shorter than your caller’s, or you’re both holding connections for a result nobody will use.
• Circuit breaker. After N failures, stop calling and fail immediately; probe occasionally to see if it recovered. This stops a slow dependency from consuming all your threads.
• Bulkhead. Separate resource pools per dependency, so one slow downstream can’t exhaust the pool the rest of the system needs.
• Graceful degradation. Decide in advance which features are droppable. Search without personalization beats no search.
• Load shedding. Under overload, reject cheaply and early — ideally lowest-value traffic first. A system that degrades predictably at its limit is worth more than one that’s slightly faster and collapses.

Rate limiting

• Token bucket: tokens refill at a fixed rate; a request takes one. Allows bursts up to bucket size. The usual best answer.
• Leaky bucket: processes at a constant rate; smooths bursts entirely.
• Fixed window: simple counter per interval, but permits 2× the limit across a window boundary — a known flaw you should be able to name.
• Sliding window log: exact, stores a timestamp per request, expensive at scale.
• Sliding window counter: weighted blend of current and previous window. The usual production compromise — nearly exact, cheap.

Distributed enforcement is the real question: a central store (Redis) is accurate but adds a hop and a dependency; per-node limits with the quota divided by node count are cheap but wrong when traffic is uneven. Say which you’d pick and what it costs.

Observability

Metrics for what (cheap, aggregate, alertable). Logs for detail (expensive, structured, sampled). Traces for where the time went in a distributed request (sampled, and the only tool that answers “which of these 30 services is slow?”).

Sampling is mandatory at scale — tracing everything costs more than serving. This is Dapper’s contribution: low-overhead tracing via a propagated trace ID and aggressive sampling.

Alert on symptoms (users are getting errors), not causes (CPU is high). Cause-based alerts page you for things that don’t matter and miss the things that do.

Deploys and multi-region

Blue/green: two environments, switch traffic, roll back by switching back. Fast rollback, double the infrastructure.

Canary: route a small percentage to the new version and watch the metrics. Slower, but limits blast radius — the better default.

Both require backward-compatible schema changes, because two versions run simultaneously. Expand/contract: add the new column, write both, migrate, read new, drop old. Say this whenever a schema change appears.

Active–passive: one region serves, another stands by. Cheaper, and failover is a real event with real RTO. Active–active: all regions serve. Better latency and utilization, but now you have multi-leader writes and conflict resolution.

RTO is how long until you’re back; RPO is how much data you can lose. Async cross-region replication means RPO > 0 — quantify it rather than implying zero.

9. The 12 problems, one level deeper

For each: the crux, the design decision that carries the round, and the failure mode to raise yourself. These are skeletons, not scripts — an answer recited from memory is audible, and the round is scored on how you reason, not what you recall.

Tier 1 — foundational

1. URL shortener. Crux: ID generation and read latency. Counter with base62 encoding (short, sequential, therefore enumerable — say that), or a hash of the URL truncated with collision handling, or pre-generated keys handed out in blocks. Extremely read-heavy, so cache aggressively and serve redirects from the edge; 301 is cacheable forever but kills your analytics, 302 keeps analytics and costs a round trip — state the tradeoff. Raise it yourself: custom alias collisions, and expiry/cleanup of dead links.

2. Rate limiter. Crux: distributed counters under contention. Token bucket in Redis; the counter update must be atomic (single command or a Lua script) or concurrent requests lose updates. The interesting tradeoff is accuracy vs cost: central Redis is exact and adds a hop to every request; local per-node limits are free and wrong under uneven traffic. Decide what happens when Redis is down — fail open or fail closed, and justify it. Raise it yourself: the fixed-window boundary burst, and hot keys for your largest customers.

3. Web crawler. Crux: politeness and dedup at scale. A frontier of URLs, partitioned by domain so one worker owns a domain and can respect its crawl delay and robots.txt. URL dedup at billions of scale is where a bloom filter genuinely belongs — tiny memory, no false negatives, and false positives merely mean you skip a page. Content dedup needs a different tool: checksums for exact copies, SimHash-style fingerprints for near-duplicates. Raise it yourself: crawler traps (infinite calendars, session IDs in URLs), and re-crawl prioritization for pages that change often.

4. Search autocomplete / typeahead. Crux: latency budget, and the offline/online split. The end-to-end budget is a few tens of milliseconds, which rules out computing rankings at request time. Build the index offline from query logs, serve from an in-memory trie with the top-K completions precomputed at each node. Shard the trie by prefix, and note that prefix sharding is inherently skewed — "a" gets vastly more traffic than "z". Raise it yourself: how new/trending queries enter the index, and why you’d accept a delay there.

Tier 2 — the classics

5. News feed. Crux: fan-out on write vs read, and the celebrity problem. Fan-out on write pushes each post into every follower’s feed: reads are trivially fast, writes are expensive, and a user with 50M followers triggers 50M writes. Fan-out on read merges at query time: writes are cheap, reads are expensive and slow. The real answer is hybrid — push for normal accounts, pull for celebrities, merge at read time — and being able to say why the hybrid exists is the whole point of the question. Raise it yourself: cursor-based pagination on a feed that mutates under the reader, and ranking as a separately-scaled, failure-tolerant component.

6. Chat / messaging. Crux: connection state and ordering. WebSocket connections are stateful, so you need a registry mapping user → connected server, and a way to route a message to that server. Ordering should come from a per-conversation sequence number, not wall-clock timestamps — client clocks lie and are trivially spoofed. Undelivered messages queue per recipient until their device acknowledges. Raise it yourself: what a deploy does to every open connection, and how presence avoids O(n²) notification storms in large groups.

7. Distributed cache. Crux: consistent hashing and membership. Consistent hashing with virtual nodes for placement; replication so a node loss doesn’t lose the whole shard; gossip or a coordination service for membership. Decide what a cache miss caused by a node failure does to the database behind it — that stampede is the actual risk. Raise it yourself: rebalancing while serving traffic, and hot-key replication.

8. Video streaming. Crux: the transcoding pipeline and delivery. Upload goes directly to object storage (pre-signed URL), which enqueues a transcoding job. Split the video into segments and transcode them in parallel into several bitrate ladders — this is a DAG of tasks, and it should be idempotent so a failed segment can be retried alone. Delivery is adaptive bitrate over HLS/DASH from a CDN: the client measures its own bandwidth and picks the next segment’s quality. Metadata lives in a normal database; bytes never do. Raise it yourself: the cost asymmetry of transcoding everything vs transcoding on first view for the long tail.

Tier 3 — Google-flavored

9. Google Drive / Dropbox. Crux: chunking, dedup and sync. Split files into content-addressed chunks (hash the chunk, name it by its hash). This gives you deduplication for free, and delta sync — changing one byte re-uploads one chunk, not the file. Sync is a metadata problem: a per-file version vector or sequence number tells the client what it’s missing. Conflicts happen when two devices edit while offline; the honest answer is usually to keep both versions rather than silently merge. Raise it yourself: many small files making metadata, not bytes, the bottleneck.

10. Distributed job scheduler. Crux: at-least-once execution without duplicate side effects. A leader (elected via a consensus-backed lock) assigns work; workers lease tasks and heartbeat. If a worker dies, its lease expires and the task is reassigned — which means the task may run twice, so execution must be idempotent. Cron semantics need care: what happens to a missed run during an outage — skip it, or catch up and risk a thundering herd of backlog? Raise it yourself: stragglers (run a backup copy of a slow task and take the first result — this is straight out of MapReduce), and preventing one tenant’s jobs from starving everyone else’s.

11. Metrics and monitoring. Crux: ingestion volume and retention cost. Enormous write rate, almost no updates, queries by time range and label. That’s an LSM-backed time-series store with heavy compression. Control cost with downsampling and tiered retention — full resolution for hours, minute rollups for weeks, hourly for a year. Cardinality is the failure mode nobody expects: a label containing user IDs or request IDs multiplies your series count until the system falls over. Alerting evaluates rules on a schedule and must itself be more reliable than the systems it watches. Raise it yourself: cardinality explosion, and why the alerting path must not depend on the pipeline it monitors.

12. Ad click aggregator. Crux: counting correctly with duplicate and late events. Stream events through a partitioned log keyed by ad ID, aggregate in event-time windows, dedup on a client-generated event ID. Accept that the streaming number is approximate and run a batch reconciliation that recomputes the authoritative number from raw events — this is the lambda shape, and money-adjacent counting is exactly where it’s justified. Handle late events with a watermark plus a defined allowed-lateness policy, and say what happens to events later than that. Raise it yourself: click fraud inflating counts, and the fact that “exactly-once” here means idempotent aggregation, not magic.

10. The papers, and the tradeoff each one made

Referencing these by name is a genuine differentiator — but only if you can state what was given up. A name with no tradeoff attached sounds like reading a list.

Optional, for bonus range: Borg (cluster scheduling, bin-packing, priority preemption) and Monarch (in-memory time-series at planetary scale) — the latter is literally problem 11.

11. Language that scores, and language that sinks

Say these

• “Let me restate the problem and confirm what’s in scope.”
• “I’m assuming X — tell me if that’s wrong.” (Assumptions are fine. Unstated ones are not.)
• “Both work. I’d pick A because of requirement 3, and here’s what A costs me.”
• “That estimate changes my decision: I no longer need to shard.”
• “The weakest point in what I’ve drawn is X. Here’s how I’d address it.”
• “This is eventually consistent, which means a user could briefly see a stale count. For this feature I think that’s acceptable — for balances it wouldn’t be.”
• “With more time I’d go deeper on X.”

Avoid these

• “We’ll use NoSQL because it scales.” Name the access pattern instead.
• “We’ll add a cache.” Which layer, which keys, what TTL, what invalidation, what on miss?
• Microservices with no forcing reason. Complexity you can’t justify reads as cargo-culting.
• “Exactly-once delivery.” At-least-once plus idempotency. Be precise.
• Silence. An unspoken thought scores zero. If you’re stuck, narrate the options you’re weighing and why each doesn’t fit yet — struggling out loud is scored, silence is scored against you.
• Arguing with a hint. Hints are scored. Take them quickly and gracefully; explore rather than defend.
• Design before scope. The most common failure in this round, and the most avoidable.

The two-sentence self-check, after every practice run

Did every major choice come with a stated alternative I rejected and a reason? Did I name a failure mode before the interviewer had to ask for one?

If either answer is no, the design was a description, not a defence — and this round scores the defence. Log it in 05-practice-log.md while it’s fresh.