RabbitMQ Streams
Stream queues, offsets and replay, super streams, and when to choose a stream over a classic or quorum queue. An optional advanced deep-dive.
Prerequisite:Queues and Messages Deep DiveYou’ll need: RabbitMQ 3.9+ with the stream plugin enabled.
Optional advanced module. Streams are a specialized tool. You can build every pattern in this course without them. Read this when you need message replay or very high fan-out throughput, and skip it otherwise without breaking the main path.
What you’ll be able to do after this module
- Explain how a stream differs from a classic or quorum queue.
- Understand offsets and how consumers replay history.
- Describe super streams for partitioned scale.
- Decide when a stream is the right choice.
1. What a stream is
A stream is an append-only, replayable log stored on disk, similar in spirit to a Kafka topic but native to RabbitMQ. Unlike a classic or quorum queue, reading a message does not remove it. Messages stay in the log until a retention policy (by age or size) removes them, and any number of consumers can read the same log independently.
flowchart LR
P["Producer"]
subgraph stream ["orders.stream (append-only log)"]
m0["offset 0"] --> m1["offset 1"] --> m2["offset 2"] --> m3["offset 3"]
end
P --> stream
stream -->|"reads from offset 0"| C1["Analytics (replay)"]
stream -->|"reads from last"| C2["Live dashboard"]
2. Offsets and replay
Each message has an offset, its position in the log. A consumer chooses where to start:
| Start point | Meaning |
|---|---|
first | Read the whole retained history from the beginning |
last | Start at the last chunk |
next | Only messages published from now on |
| a specific offset or timestamp | Resume from an exact point |
This is the defining feature: a new consumer can replay past events (rebuild a read model, backfill analytics) without the producer republishing anything. Classic and quorum queues cannot do this because consumption is destructive.
Streams can be consumed over AMQP, but the dedicated stream protocol is far faster for high-throughput cases. With Spring, a stream is still reachable through familiar listener abstractions for AMQP access.
3. Super streams
A single stream is one log on one set of nodes. A super stream partitions a logical stream into several sub-streams so throughput scales horizontally, much like partitions in a distributed log. Producers route messages to partitions by a key, and a group of consumers divides the partitions among themselves.
Reach for super streams only when a single stream’s throughput is a proven bottleneck.
4. Streams vs classic vs quorum
| Classic / Quorum queue | Stream | |
|---|---|---|
| Consumption | Destructive (message removed on ack) | Non-destructive (log retained) |
| Replay | No | Yes, from any offset |
| Multiple independent readers | Each needs its own queue and a fan-out | All read the same log |
| Best for | Task queues, work distribution | Event replay, high fan-out, audit logs |
Rule of thumb: use quorum queues for reliable work distribution (the default for this course), and reach for a stream only when you specifically need replay or many readers of the same high-volume feed.
5. Further learning: Spring Cloud Stream
For a higher-level, broker-agnostic programming model, Spring Cloud Stream lets you write producers and consumers as simple functions and bind them to RabbitMQ (or other brokers) through configuration. It is worth exploring once you are comfortable with raw Spring AMQP, but it is a separate framework beyond this course’s scope, so treat this as a pointer for later study rather than a required step.
Checkpoint
You should now be able to:
- Explain why stream consumption is non-destructive and replayable.
- Choose a starting offset for a consumer.
- Describe when super streams add value.
- Decide between a stream and a classic or quorum queue.