Kafka & Event Streaming Basics

Kafka is a distributed, append-only log. Not a queue that hands out messages and deletes them — a durable log that just keeps appending records to the end, like a never-ending append-only file that consumers read through at their own pace.

In simple language — Kafka is a giant shared logbook. Producers write new lines at the bottom, and each reader keeps their own bookmark for how far they’ve read.

Why a log and not a queue

This is the mental model everything else hangs off. A traditional queue deletes a message once it’s consumed — the broker tracks what’s been delivered. Kafka does the opposite: it keeps every record for a retention period, and each consumer tracks its own position. The broker doesn’t remember who read what; the consumer does.

That flip is the whole reason Kafka exists. Because records stick around and readers own their position, we get replay — a new consumer can start from the beginning and read all of history, or a broken consumer can rewind and re-process. A queue can’t do that; once a message is gone, it’s gone.

Core concepts

A handful of terms, and they all fit together:

• Broker — a single Kafka server. A cluster is several brokers sharing the load and replicating data for durability.
• Topic — a named stream of records, like payments or clicks. This is what producers write to and consumers read from.
• Partition — a topic is split into partitions, and each partition is its own ordered, append-only log. This is the unit of parallelism and ordering.
• Offset — a record’s sequential position within a partition (0, 1, 2, …). A consumer’s progress is just “I’m at offset N in partition P”. The bookmark.
• Producer — writes records to a topic. It picks which partition (often by hashing a key, so all records for one key land in the same partition).
• Consumer group — a set of consumers that split the partitions of a topic between them. Kafka guarantees each partition is read by exactly one consumer within a group, which is how work gets divided.

Why partitions matter

Partitions buy us two things that are usually in tension: ordering and parallelism.

• Ordering is guaranteed per partition, not per topic. Kafka promises records in one partition are read in the exact order they were written. Across the whole topic? No global order. So if we need related events ordered (all events for order-42), we give them the same key — Kafka hashes the key to a partition, so they all land in the same partition and stay ordered.
• Parallelism comes from having many partitions. Each partition can be processed by a different consumer at the same time. Want more throughput? Add partitions and consumers. The ceiling on parallelism is the partition count — more consumers than partitions and the extras just sit idle.

Think of it like — one partition is a single-file queue (perfectly ordered but slow), and splitting into many partitions is opening more checkout lanes (fast, but order is only kept within a lane).

Retention and replay

Kafka keeps records for a configured retention window — by time (say 7 days) or by size — regardless of whether anyone consumed them. Nothing is deleted just because it was read.

Because of that, and because consumers own their offsets, replay is trivial: reset the consumer group’s offset back to 0 (or to a timestamp) and re-read everything. This is huge for real work — reprocessing after a bug fix, backfilling a brand-new service with all historical events, or spinning up an analytics job that reads the full stream from the start.

# rewind a consumer group to the beginning and re-process the whole topic
kafka-consumer-groups --bootstrap-server localhost:9092 \
  --group analytics --topic orders --reset-offsets --to-earliest --execute

Queue vs log, one more time

The distinction worth nailing: in a queue, the broker owns delivery state and deletes consumed messages — one shot, then gone. In a log like Kafka, records are retained and each consumer group tracks its own offset independently — so five different services can read the same topic at their own speeds, and any of them can rewind. Same stream, many independent readers, full history.

Interview soundbite

Kafka is a distributed, append-only log rather than a queue. A topic is split into partitions, each an ordered log; a record’s position is its offset. Ordering is guaranteed per partition (use a key to keep related events together), and partitions give parallelism since each is read by exactly one consumer in a consumer group. The key difference from a queue: Kafka retains records for a retention window and consumers track their own offsets — so multiple consumer groups read the same stream independently, and any of them can rewind to replay history.