Module Quiz

Complete this quiz after finishing all concept and practice pages.

Current Module Questions

Question 1: The Fallacies in Action

You are told the network between two services "has 5ms latency and is basically reliable." Which two fallacies of distributed computing does this statement commit, and what concrete design review question would you ask to check whether the dependent code respects reality?

Answer: Fallacies 1 ("the network is reliable") and 2 ("latency is zero"). "Basically reliable" hides tail partitions and packet loss; "5ms" hides 99th-percentile spikes and tail-at-scale. Ask: "What timeout does the caller use, and what happens to in-flight requests when a partition lasts 30 seconds?" If the code does not have a timeout, a retry policy, and an idempotency story, it has the bug.

Question 2: Partial Failure Reasoning

A client calls POST /charge on a payments service. TCP reports the connection reset after 4 seconds. Write down, as an enumeration, every distinct post-condition the system could be in, and what the client must do in each.

Answer:

1. Charge was not received by the server (network dropped it before). Client can safely retry.
2. Charge was received, server crashed before processing. Client must retry; server must be idempotent on the idempotency key.
3. Charge was processed, server crashed before reply. Client must retry; the idempotency store returns the stored result.
4. Charge was processed, reply was sent but lost on the wire. Same as case 3 from the client's view.

The client cannot distinguish these. The correct behavior is: retry with the same idempotency key. Without an idempotency key, retry is a user-visible bug (double charge).

Question 3: Clocks

Why can a millisecond of NTP-synchronized wall-clock time not be trusted to order two events on different hosts? Give two distinct physical reasons and one algorithmic reason.

Answer:

• Physical: NTP's best-case accuracy is tens of milliseconds on a LAN, higher over WAN.
• Physical: clocks can jump forward or backward at NTP sync or leap-second adjustments; a wall-clock read can return a value smaller than the previous read.
• Algorithmic: clock skew between hosts is uncontrolled by your code; an event stamped at t on host A and t+5ms on host B may actually have happened in the opposite real-time order.

Question 4: Lamport and Concurrency

If L(a) = 5 and L(b) = 7, can you conclude a -> b? Why or why not, and what can you conclude?

Answer: You cannot conclude a -> b. The implication goes one way: if a -> b, then L(a) < L(b), but the converse does not hold. a and b may be concurrent. What you can conclude: b did not happen before a (because that would require L(b) < L(a), which is false).

Question 5: Vector Clock Comparison

Vector clock V1 = [3, 1, 4] on process P1, V2 = [2, 2, 4] on process P2. Is V1 < V2, V2 < V1, equal, or concurrent?

Answer: Concurrent. V1[0] > V2[0] but V1[1] < V2[1], so neither dominates. This means the two events are neither causally ordered; a system doing LWW by timestamp alone here would silently discard one.

Question 6: Heartbeat Tuning - What Goes Wrong

A team sets heartbeat interval = 100ms, timeout = 150ms for a Raft cluster. GC pauses on the JVM can reach 500ms. What goes wrong, and how does phi-accrual improve the situation?

Answer: Every GC pause longer than 150ms triggers a false "leader dead" suspicion. Followers start elections, cause term churn, throughput collapses during GC-heavy load. Phi-accrual keeps a rolling distribution of heartbeat inter-arrivals; when GC pauses widen the tail, the effective threshold widens with them, so momentary pauses do not cross the phi threshold. The detector trades quicker detection for tunable false-positive rate, which is the right trade-off under bursty latency.

Question 7: Byzantine vs Crash

Your system runs on AWS with managed hardware and your own code running on it. Do you need Byzantine fault tolerance? What specific assumption must hold to safely assume only crash-stop?

Answer: No, BFT is not needed in most corporate deployments. The assumption: you trust the hypervisor, the hardware, and all node operators. If a node can fail silently but not actively lie, crash-stop with a failure detector is enough. BFT is justified only when nodes can be compromised (multi-tenant chains, blockchains, multi-organization consensus) -- at the cost of requiring 3f+1 replicas to tolerate f failures, vs 2f+1 for crash-stop.

Question 8: Why Consensus is Hard

Explain in one paragraph why leader election reduces to consensus, and why consensus reduces to atomic commit.

Answer: Leader election requires all non-faulty nodes to agree on exactly one leader despite concurrent candidacies; that is literally agreement on a single value, which is consensus. Atomic commit requires all participants to decide the same outcome (commit or abort) despite failures; consensus over {commit, abort} is exactly this. Any algorithm that solves consensus can solve both; any impossibility result on consensus (FLP) applies to both.

Question 9: Paxos Safety

In Paxos, why must a proposer, during Phase 2, propose the value with the highest-numbered accepted proposal it saw in Phase 1 (rather than its own preferred value)?

Answer: Because an earlier round may have already chosen a value (a majority had accepted it). The Phase 1 quorum intersects every Phase 2 quorum, so if any value was chosen, at least one responder in Phase 1 has seen it. The proposer must respect it to preserve the "only one value is ever chosen" safety invariant. If the proposer ignored it and proposed a different value, two values could be chosen in different rounds -- violating agreement.

Question 10: A Simple Raft Question

Raft's leader crashes after replicating entry e at term T to a minority of followers (e is not committed). A new leader is elected at term T+1. Is the new leader allowed to overwrite e? Cite the specific Raft rule.

Answer: Yes, the new leader can overwrite e. Because e was not committed, no application has observed it. The new leader's log may or may not contain e; if it does not, then when the new leader sends AppendEntries, followers that have e will truncate it. This is consistent with Raft's safety rules: the new leader was elected because its log was at least as up-to-date as a majority's; any entry that was committed would have been on that majority and therefore on the new leader.

Question 11: What Goes Wrong - Split Brain

Two datacenters run a 4-node Raft cluster (2 nodes each). The link between them fails for 30 seconds. What happens, and which specific design choice would have prevented the outage?

Answer: Neither partition has a majority (3 of 4), so neither can elect a leader, neither can commit writes, and the cluster is unavailable for 30 seconds. The fundamental design error was running 4 nodes (even number) split evenly across two datacenters. Fixes: (a) run 5 nodes split 3/2 or 2/2 + a witness in a third zone, so one datacenter holds the majority; (b) use a three-zone design. Note: no change to Raft itself fixes this - the problem is the quorum topology.

Question 12: What Goes Wrong - Bad Retry

A client wrote "if timeout, retry up to 3 times with exponential backoff" on top of a non-idempotent POST /transfer endpoint. A network blip causes the first two attempts to time out although the server processed them. The third succeeds and returns 200. Describe the user-visible bug and the minimal fix.

Answer: The user was charged three times. Each timeout did not mean "request not received" - it meant "unknown." The retry logic amplified the unknown into duplicate processing. Minimal fix: the client sends an Idempotency-Key header on every attempt and the server stores (key -> result) with a TTL, returning the stored result on duplicates. This converts at-least-once delivery into effectively-once processing without changing the underlying network guarantees.

Question 13: Lease and Fencing

A lock service grants a 30-second lease to client A. A GC pause of 45 seconds on A makes the lease expire. B acquires the lease, reads and writes. Then A wakes up, still believing it holds the lease, and issues a write to the storage backend. What mechanism must the storage backend have, and what does it do with A's write?

Answer: The storage backend must check a monotonic fencing token issued by the lock service with each grant. B's token is higher than A's. The backend remembers the highest token it has seen; when A's write arrives with an older token, the backend rejects it. Without fencing, A would corrupt B's data. The lease alone is insufficient because the client and backend disagree on whether the lease has expired.

Question 14: Coordination Service Choice

You need service discovery and leader election for a 50-node internal service. You are deciding between ZooKeeper, etcd, and Consul. What is the deciding factor, and which would you pick?

Answer: The deciding factor is which ecosystem already runs in your environment. If you already run Kubernetes, etcd is essentially free (Kubernetes ships with it). If you run HashiCorp tooling (Vault, Nomad), Consul composes better. If you run JVM infrastructure (Kafka pre-KRaft, HBase, HDFS), ZooKeeper is the default. All three solve the same problem at this scale; operational fit beats theoretical differences.

Question 15: End-to-End Argument

Why is transport-level reliability (TCP, at-least-once message queues) insufficient to guarantee "the user's intent was executed exactly once"?

Answer: Because transport-level reliability delivers a byte stream; it does not promise that the application on the other side processed the request, nor that a duplicate was detected. The real correctness property (exactly-once processing) lives at the application layer, where the idempotency key and the side-effect table live. TCP or Kafka's "exactly-once" semantics protect the transport; the application must still deduplicate. This is the end-to-end argument in one sentence: reliability must be implemented where the correctness requirement actually lives.

Interleaved Review Questions

Prior Module Question 1 (S6 Module 4: Linearizability)

What is the difference between linearizability and serializability, and why does consensus buy you linearizability but not serializability?

Answer: Serializability is about multi-key transactions: there exists a serial order of transactions consistent with the observed effects. Linearizability is about single-object registers: every read returns the most recent write in real time. Consensus (Raft/Paxos) orders log entries into a single totally-ordered replicated state machine; applied to a single-key register that makes the register linearizable. Serializability over multi-key transactions requires additional concurrency control (2PL or SSI) on top of the consensus-ordered log.

Prior Module Question 2 (S6 Module 3: Replication)

Why does a synchronous replication design not automatically give you consensus?

Answer: Synchronous replication guarantees that a committed write is on a quorum of replicas before acknowledgement. It does not guarantee that all nodes agree on the order of writes during failures, nor that exactly one leader exists after a partition heals. Consensus adds the ordering and agreement layer; synchronous replication without consensus can still exhibit split-brain if both sides of a partition accept writes independently.

Prior Module Question 3 (S6 Module 2: LSM Trees)

How does a coordination service like etcd use an LSM-tree or B-tree internally, and why does it matter for its operational characteristics?

Answer: etcd uses a bbolt (B+tree) key-value store underneath the Raft log. This gives it point-query performance suitable for service discovery and key lookup, but limits its write throughput compared to a pure LSM. Operationally this means etcd is excellent for low-volume metadata but not a general-purpose database; you should not use etcd as a high-throughput append log -- that is what Kafka is for.

Prior Module Question 4 (S5 Module 4: Networking)

TCP's retransmit timer and Raft's election timer are both "wait then react" mechanisms. Why is it safe to tune TCP's RTO aggressively but dangerous to tune Raft's election timeout below a few hundred milliseconds on the JVM?

Answer: TCP retransmits data, which is idempotent at the byte level; spurious retransmits waste bandwidth but do not corrupt state. Raft's election timeout triggers a term change that can invalidate the current leader and force log reconciliation. Spurious elections under GC pauses cost throughput and burn disk for persistent term/voted-for state. The leader must serve heartbeats faster than the smallest election timeout, which on a GC-paused JVM can be hundreds of milliseconds. Aggressive tuning on TCP is a latency tradeoff; aggressive tuning on Raft's timer is a correctness-adjacent stability problem.

Prior Module Question 5 (S5 Module 3: OS Memory)

GC pauses affect distributed-systems correctness in a way memory management on a single machine does not. Explain the gap.

Answer: On a single machine, a GC pause stalls the process; from outside nothing observable happens until the pause ends. In a distributed system, other nodes continue to run. They observe the paused node as non-responsive. If their heartbeat timeout is shorter than the pause, they remove it from membership, elect a new leader, or issue a fencing token. When the paused node resumes, it holds state (a lease, a leader role) that the system has already revoked, and its subsequent writes violate safety unless something (fencing tokens, lease checking) rejects them. Memory management on one box is local; GC pauses on one box become a global event.

Self-Assessment and Remediation

Mastery Level (90-100% correct):

• Ready for Module 6 (scalable data systems) and the capstone with confidence.
• Your instinct should now be: when you see a distributed bug, mentally map it onto fallacies, failure model, clocks, or consensus invariants.

Proficient Level (75-89% correct):

• Review missed concepts. Most gaps are in the causal-reasoning questions (2, 11, 12) -- those reward fluency, not memorization.
• Redo Practice 2 (Failure Model Workshop) and one Kata from Practice 4.

Developing Level (60-74% correct):

• Rework Cluster 1 (Inescapable Reality) end to end. Most failures here are "still thinking single-machine."
• Redo Practice 1 (Time and Ordering Lab) by hand.
• Re-read DDIA Chapter 8 start to finish; it will re-anchor your intuition.

Insufficient Level (<60% correct):

• The module has not landed. Restart from the module index.md diagnostic questions. Identify which three concepts you cannot state in one sentence, and read the concept page plus its first "Read This Only If Stuck" link for each. Do not proceed to Module 6 until you can pass this quiz above 75%.