Streams

Streams are Redis’s answer to “I want Kafka but I already have Redis.” They’re an append-only log of entries, each with a monotonically increasing ID, supporting consumer groups for distributed processing, message acknowledgment, and replay. Added in Redis 5.0.

In simple language: think of it like a list that remembers everyone who’s read what, lets multiple workers split the work, and lets a new consumer catch up from the beginning whenever it wants.

The basics: XADD and XREAD

Each entry is a set of field-value pairs and gets an ID like 1716720000123-0 (milliseconds-sequence).

XADD events * type "login" userId "42"
# "1716720000123-0"

XADD events * type "purchase" userId "42" amount "99.99"
# "1716720000456-0"

XLEN events
# (integer) 2

XRANGE events - +
# 1) 1) "1716720000123-0"
#    2) 1) "type"
#       2) "login"
#       3) "userId"
#       4) "42"
# 2) ...

The * tells Redis to auto-generate the ID. The - and + in XRANGE mean “from beginning” and “to end.”

Reading new entries

XREAD blocks until new entries arrive, like BLPOP for streams:

XREAD BLOCK 0 STREAMS events $
# $ means "only entries added AFTER this call"

Consumer groups — the killer feature

This is what makes streams Kafka-lite. A consumer group lets multiple workers split the stream, each entry delivered to exactly one consumer in the group, with explicit acknowledgment.

Setting up

# Create the group, starting from the very beginning
XGROUP CREATE events workers $ MKSTREAM
# $ = "start from new entries only", 0 = "start from beginning"

# Worker A reads
XREADGROUP GROUP workers worker-A COUNT 10 BLOCK 0 STREAMS events >

# > = "give me entries never delivered to anyone in this group"

# After processing
XACK events workers 1716720000123-0

What if a worker crashes?

Redis remembers every entry delivered but not yet acked — the Pending Entries List (PEL). We can inspect it and reassign stuck entries:

XPENDING events workers
# Summary: how many pending per consumer

XPENDING events workers - + 10 worker-A
# Detail: specific entry IDs that worker-A hasn't acked

XCLAIM events workers worker-B 60000 1716720000123-0
# Reassign that entry from whoever to worker-B if it's been pending > 60s

Newer Redis has XAUTOCLAIM which automates this — sweep up entries pending longer than X and reassign them.

Streams vs Lists vs Kafka

• Lists (LPUSH/BRPOP): one consumer wins each item, no replay, no ack. Simple work queue.
• Streams: many consumers per group, explicit ack, replay from any point, multi-group fanout. Real event log.
• Kafka: same shape as streams, plus partitioning across brokers, infinite retention, exactly-once semantics, mature ecosystem. Use Kafka if you’re already operating it or if you need its scale.

For most apps under ~100k events/sec, Redis streams are plenty.

Real-world: order processing

// Producer (web server)
await redis.xadd("orders", "*",
  "orderId", id, "userId", userId, "amount", amount
);

// Consumer (worker)
while (true) {
  const res = await redis.xreadgroup(
    "GROUP", "fulfillment", "worker-1",
    "COUNT", 10, "BLOCK", 0,
    "STREAMS", "orders", ">"
  );
  for (const [stream, entries] of res) {
    for (const [id, fields] of entries) {
      await processOrder(fields);
      await redis.xack("orders", "fulfillment", id);
    }
  }
}

Trimming

Streams grow forever unless we trim. Use MAXLEN to keep only the most recent N:

XADD events MAXLEN ~ 1000000 * type "login" userId "42"
# ~ means "approximate" — way cheaper than exact trimming

XTRIM events MAXLEN 500000

The interview soundbite

“Streams give Redis a real event log with consumer groups and ack semantics, so we can build fan-out, retry, and replay without dragging in Kafka. The Pending Entries List + XCLAIM is the recovery story — that’s what makes them ‘reliable’ rather than fire-and-forget like lists.”