Redis Streams in Distributed Systems

In event-driven architecture and modern distributed systems, asynchronous communication between services (microservices) or task orchestration for background workers is a core problem.

For a long time, Redis Pub/Sub and the Redis List (LPUSH / RPOP) data structure have been familiar tools thanks to extremely fast in-memory performance and low deployment cost. However, as system scale grows, the inherent limits of these solutions around reliability and load balancing begin to show.

The arrival of Redis Streams brought the distributed log model into the Redis ecosystem. This article analyzes the architectural idea behind Redis Streams, compares its strengths and weaknesses in detail with Redis Pub/Sub, and clarifies when each solution should be chosen in practice.

---

1. Core Idea: An Immutable Append-Only Log

At its core, Redis Streams is an append-only log data structure stored in memory:

• Each message pushed into the stream is assigned a unique Message ID that increases over time (in timestamp-sequence form).
• Once written into the stream, data is stored persistently in RAM and can be configured to sync to disk (AOF/RDB); it does not disappear automatically after being read.
• Consuming services (consumers) can read data sequentially in real time, or replay history from any point in the past.

                    REDIS STREAM (Append-Only Log)
 ┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
 │ Msg: 1001-0 │──►│ Msg: 1002-0 │──►│ Msg: 1003-0 │──►│ Msg: 1004-0 │──► [Continue appending...]
 └─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘
  (Processed)       (In progress)     (Waiting to be read) (Newly arrived)

---

2. Architectural Comparison: Redis Streams vs. Redis Pub/Sub

To understand the difference clearly, place the two mechanisms side by side in terms of how they publish data and how they guarantee delivery:

[REDIS PUB/SUB: Fire-and-Forget]
Producer ────► [Channel] ────► Consumer 1 (Online)   ── Receives the message
                         ────► Consumer 2 (Offline)  ── Lost permanently!

[REDIS STREAMS: Log Trail & Consumer Groups]
Producer ────► [Stream Log] ── (Persisted in memory)
                     │
                     ├─── Worker 1 (Consumer Group) ── Processes Msg 1 (ACK completed)
                     └─── Worker 2 (Consumer Group) ── Processes Msg 2 (Waiting for ACK)
                     └─── Worker 3 (Back online)     ── Continues reading from Msg 3

Criterion	Redis Pub/Sub	Redis Streams
Delivery model	Broadcast / Fan-out<br>Sends the message to every currently connected subscriber.	Distributed log (Log + Consumer Groups)<br>Supports both broadcast and work-sharing.
Persistence	No storage (In-memory transient)<br>If no one is listening, the message is discarded immediately.	Persistent storage<br>Messages remain in the Stream until they are actively trimmed.
Delivery guarantee	At-most-once<br>Messages are easily lost if the network lags or the consumer crashes.	At-least-once<br>Includes acknowledgment (ACK) and retry when failures occur.
Load balancing	Not supported<br>Every subscriber receives an identical copy.	Strongly supported via Consumer Groups<br>Tasks are distributed evenly across a group of workers.
Replayability	Not possible<br>Consumers only receive messages sent after they subscribe.	Fully possible<br>History can be read from any position or timestamp.
Memory pressure	No RAM cost for storing messages.	Consumes RAM according to stream length (an upper bound is needed).

---

3. Three Pillars That Make Redis Streams Fit Distributed Systems

1. Consumer Groups

In heavy-load workloads (for example: document OCR, image compression, large-scale data processing), a single worker cannot keep up. We need a group of many workers sharing the workload.

Redis Streams provides the Consumer Groups concept, similar to Apache Kafka:

• Workers in the same group share the reading of different messages, ensuring no two workers process the same message.
• When a new worker is added to scale out, the system redistributes load automatically without complex configuration changes.

flowchart LR
    Stream[(Redis Stream)]
    
    subgraph Group["Consumer Group: OCR_Workers"]
        W1[Worker 1]
        W2[Worker 2]
        W3[Worker 3]
    end
    
    Stream -->|Assign Task A| W1
    Stream -->|Assign Task B| W2
    Stream -->|Assign Task C| W3

---

2. Guaranteeing No Data Loss (ACK & Pending Entries List)

With Pub/Sub, if a worker receives a message but then loses power or crashes midway, that work disappears permanently without anyone knowing.

Redis Streams fully addresses this risk with the Pending Entries List (PEL):

• When a worker receives a message, that message moves into a “Pending” state.
• Only when the worker finishes processing and sends a successful acknowledgment (ACK) is the message marked complete.
• If a worker dies midway, the message remains in the PEL. Other workers can inspect tasks that have been stuck beyond the allowed time and claim the right to process them (Claiming/Auto-claim), ensuring the task is never dropped.

---

3. Intelligent Memory Control (Stream Trimming)

Because a Stream stores data persistently, without a control strategy Redis RAM will fill up quickly.

Redis Streams allows configuration of a maximum length limit (Capped Streams):

• When the number of messages exceeds the allowed threshold (for example: keep at most the 10,000 most recent records), the system automatically removes the oldest messages.
• It supports approximate trimming, which frees memory in the background without degrading the CPU performance of the Redis Server.

---

4. Practical Selection Criteria

A clear understanding of the architectural nature helps us make accurate technology decisions without wasting resources:

[WHAT IS YOUR PROBLEM?]
      │
      ├── Need ultra-fast messaging, and losing a few messages is acceptable?
      │   (Live chat, Notifications, GPS coordinate updates)
      │   └──► CHOOSE: Redis Pub/Sub
      │
      ├── Need a reliable queue, worker load balancing, and a retry mechanism?
      │   (Document OCR queues, report export, internal transaction processing)
      │   └──► CHOOSE: Redis Streams
      │
      └── Need to store billions of events over many months, with enterprise-grade complex routing?
          (Enterprise-wide Event Sourcing, Big Data analytics)
          └──► CHOOSE: Apache Kafka / RabbitMQ

When should you choose Redis Pub/Sub?

• Real-time push: Deliver push notifications and UI status updates over WebSocket.
• Instant measurement data (Metrics / Heartbeat): Service health heartbeats and realtime chart data — where the latest information matters most, and dropped older information does not need to be replayed.

When should you choose Redis Streams?

• Distributed background task processing (Task Queue): Orchestrate heavy work such as document digitization, OCR, and search-index generation across many workers running in parallel.
• Asynchronous data synchronization between services: Preserve data integrity when one service needs to notify others of state without being interrupted if the receiving service is temporarily offline.
• A lean alternative to Kafka/RabbitMQ: When Redis is already in the stack and messaging demand is medium to large, using Redis Streams saves operational cost and significantly reduces infrastructure complexity compared with standing up a heavy Kafka cluster.

---

Conclusion

Redis Streams was not created to fully replace Redis Pub/Sub, but to fill the missing link around durability, fault tolerance, and load balancing in the Redis ecosystem.

By combining an immutable log architecture with a flexible consumer-group mechanism, Redis Streams offers a solution that is “just strong enough, and extremely lightweight” for distributed messaging problems, helping organizations build reliable task-processing systems at the most optimized operational cost.

---

Shared by the engineering team at BK Hightech.