Skip to main content

Log-Based Brokers: Kafka's Design and Retention

What This Concept Is​

A log-based broker stores messages as an append-only, immutable, partitioned log. Kafka is the canonical example; Apache Pulsar, AWS Kinesis, and Redpanda share the same core shape.

The key departures from a classical queue (Concept 07):

  1. Messages are not removed on consumption. Reading does not delete. The broker retains messages for a configured retention period (hours, days, weeks, forever).
  2. Each consumer tracks its own position in the log via an offset. Two independent consumers can read the same partition at different offsets.
  3. Partitions are the unit of parallelism, not consumers. A topic is split into N partitions, each an ordered log; consumers within a group share partitions (Concept 09).
  4. Order is per-partition, not per-topic. Messages inside one partition are strictly ordered; between partitions there is no ordering guarantee.
topic "order.placed"   (3 partitions, retention = 7 days)

partition 0:  | m0 | m1 | m4 | m7 | m9 | ...  (offsets 0..)
partition 1:  | m2 | m5 | m8 | ...
partition 2:  | m3 | m6 | ...

Consumer A (offset 5 on p0):  reads m7, m9, ...
Consumer B (offset 0 on p0):  reads from m0  (replaying 7 days)

Why It Matters Here​

Log-based brokers solve problems queues cannot:

  • Replay. A new consumer can start from the beginning of retention and catch up. Queues cannot do this; the messages are gone.
  • Multiple independent consumers at different speeds. Each consumer group has its own offsets; a slow batch job at 10 MB/s does not affect the online consumer at 1 GB/s.
  • Rebuild derived state. Drop a read model, reset the consumer to offset 0, re-consume. The log is the history.
  • Long-term event history as a first-class artifact. Retention can be set to years (at cost). Kafka with infinite retention + compaction is effectively an event store (see Concept 13).

The cost: the broker is more expensive and complicated than a queue. If you only ever need work distribution with DLQs, use a queue. If you need replay, multiple subscribers, or the log as history, use a log-based broker.

Retention as a Design Tool​

Retention is a first-class knob, not an afterthought:

Retention settingConsequence
Short (hours)Consumers must be caught up; good for high-throughput transient streams; cheap storage
Medium (days)Default; covers consumer outages, redeploys, bug-reprocess windows
Long (months-years)Log acts as history; expensive storage; enables rebuild of read models from scratch
Log compaction (by key)Retain latest value per key indefinitely, tombstone on delete. Turns the topic into a keyed current-state table
Time-based + compactedHybrid: recent history + compact latest-per-key for older data

Compaction is the feature that lets a Kafka topic serve as the source of truth for current state (latest price per SKU, latest profile per user) while still carrying the stream of changes. This is the foundation of Kafka as a "database turned inside out" pattern.

Kafka Concept Map in One Sketch​

  Topic "orders"                      (retention: 7 days, compaction: none)
  +----------+    +----------+    +----------+
  | p0       |    | p1       |    | p2       |
  | [m0..m9] |    | [m2..m8] |    | [m3..m6] |
  +----------+    +----------+    +----------+
      ^               ^               ^
      | replicate     | replicate     | replicate
  +---+---------------+---------------+---+
  | Broker 1   Broker 2   Broker 3       |   (leader per partition, followers replicate)
  +--------------------------------------+

  Producer writes --(partitioner, keyed by order_id)--> broker leader of that partition
  Producer receives ack after min in-sync replicas persist (acks=all)

  Consumer group "billing":
      c1 -> [p0]    c2 -> [p1, p2]    (one consumer per partition inside a group)
  Consumer group "analytics":
      a1 -> [p0, p1, p2]               (reads independently of "billing")

Key behaviors you must understand:

  • Producer chooses the partition by hashing a key (typically the aggregate ID). Same key -> same partition -> ordered.
  • Replication (replication.factor) guarantees durability: acks=all writes wait for all in-sync replicas.
  • Leader election happens on broker failure; the new leader resumes from the replicated log.
  • Consumer offsets are stored (in Kafka itself, the __consumer_offsets internal topic) and committed independently per consumer group.

Common Confusion / Misconception​

"Kafka is just a queue." Kafka is a log. You can make it behave like a queue (1 partition + 1 consumer group = FIFO queue with redelivery), but that is a degenerate use. The power is replay and multi-subscriber fan-out.

"Kafka is always ordered." Per-partition yes, per-topic no. If you produce to a topic with 12 partitions without choosing a key, messages are spread across partitions and there is no global order. When order matters, you must key-partition (see Concept 09).

"We can delete individual messages." Generally no. You can compact by key (which drops older values for the same key), or use tombstones, or shrink retention. Individual message deletion is not an operation.

"At-least-once is a Kafka feature." At-least-once is a consumer-behavior choice based on when you commit offsets. Commit after processing -> at-least-once. Commit before processing -> at-most-once. "Exactly once" is a much stronger claim with many qualifiers (see Concept 12).

"Kafka replaces my database." Kafka + compaction can serve as a source of truth for some workloads, but it is not a relational DB. It lacks ad-hoc queries, indexes, transactions across keys, and many other features. It complements your DB; it rarely replaces it.

How To Use It​

When picking log-based over queue-based:

Operational checklist:

  1. Partition count: must be ≥ your peak consumer concurrency per group, and a number you can live with (increasing is disruptive for keyed topics).
  2. Keying strategy: choose the key that gives per-entity ordering and even distribution.
  3. Replication factor: ≥ 3 for production, with min.insync.replicas=2.
  4. Retention: pick by use case (transient stream vs history vs compacted state).
  5. Compaction: on if you want "latest per key"; off for pure event streams.
  6. Schema management: Avro/Protobuf with a schema registry; JSON is acceptable if you are disciplined.

Check Yourself​

  1. Why does "reading does not remove" enable replay? How is that different from a queue?
  2. What exactly does acks=all guarantee and not guarantee about a producer's message?
  3. When is log compaction appropriate, and how does it differ from time-based retention?
  4. Why do partitions -- not consumers -- define parallelism in Kafka?

Mini Drill or Application​

In 20 minutes:

  1. Design a Kafka topic for order events in a retailer. Choose partitions, keying, retention, compaction. Justify each.
  2. Write the producer code (pseudocode) for OrderPlaced with the right key.
  3. Describe how a new analytics team consumes the topic independently of the existing billing consumer.
  4. Describe what happens when a broker dies.

Transfer to Adjacent Domains​

  • Projections (Concept 14) and event sourcing (Concept 13). Long-retention + compacted topics are the substrate that makes "drop a projection, rebuild from scratch" routine. If your retention is shorter than your rebuild horizon, CQRS on Kafka is structurally unsafe.
  • Stream processing (Kafka Streams, Flink, ksqlDB). Stream processors are Kafka consumers with local state + automatic rebalancing -- the whole paradigm assumes the log-based model. A classical queue cannot host the same patterns.
  • Data lakehouse ingest (S6). Log-based brokers are the dominant ingest path into Iceberg/Delta/Hudi tables. "Kafka -> Flink -> Iceberg" is the modern counterpart to "Kafka -> HDFS -> Hive."
  • Database replication. Kafka's partitioned log is deliberately modeled on Postgres/MySQL replication logs. Understanding Kafka replication ≈ understanding distributed-log replication in general -- useful for S6 DB deep-dives.
  • Cost engineering. Retention and replication factor are the two knobs that dominate Kafka cost. Treat "retention = 30 days, RF=3, compacted" as a financial decision, not just a technical one.

Read This Only If Stuck​