Path 1 · lesson 2 of 6
Messaging vs event streaming: queues and logs
A queue hands each message to one worker and removes it once acknowledged. A log keeps records and lets each reader track its own position. Compare Kafka and RabbitMQ on that basis.
What you will learn
- Explain what happens to a message after it is consumed in a queue and in a log, and who keeps track of progress in each
- Predict how adding consumers changes throughput and ordering in a RabbitMQ queue and in a Kafka consumer group
- Compare Kafka and RabbitMQ on routing, acknowledgement, ordering, replay and failure handling using each project's own documentation
- Recognise where the two overlap today: RabbitMQ streams, and share groups in Kafka
- Choose queue or log semantics for a workload from its requirements
Before you start
The concept
Message brokers come in two families that look alike from a distance, since both sit between a sender and a receiver, and behave very differently up close. The difference is one question: what happens to a message after someone has read it?
In a queue, a message is a piece of work waiting to be done. The broker gives it to one consumer, waits for that consumer to acknowledge it, and then removes it. The queue is meant to drain: an empty queue is the healthy state. In a log, a record is an entry in a history. Reading it changes nothing in the log. It stays until a retention rule removes it, and each reader keeps track of how far it has read.
Queue semantics (traditional messaging)
- A message is delivered to one consumer and removed once that consumer acknowledges it
- The broker tracks the state of every message: ready, delivered but unacknowledged, done
- Several consumers on one queue compete: each message goes to one of them, so adding consumers adds throughput directly
- Each message is acknowledged or rejected on its own; a rejected one can be requeued or dead-lettered
- The broker routes: rules decide which queues get a copy of each message
- Order holds for a single consumer; with several consumers and redeliveries it does not
- Once consumed, a message is gone. A new consumer sees only what arrives after its queue exists
Log semantics (event streaming)
- A record is appended to a partition and stays there until retention removes it, read or not
- The broker stores the records; each consumer group stores one position (offset) per partition
- Within a group, a partition belongs to one consumer at a time, so parallelism is capped by the partition count
- Progress is a position: committing offset N means everything before N is done. There is no per-record acknowledgement
- The broker does not route. Producers choose a topic and partition; consumers choose which topics to read
- Order is guaranteed inside a partition, for every reader, every time it is read
- Any group can go back to an earlier offset and read again, and a new group can start from the oldest record
Neither model is better. They answer different needs. A queue is a natural fit for work to be done once: resize this image, send this email. A log is a natural fit for facts that several parties need, in order, possibly more than once: an order was placed, a price changed.
Why it exists
Queues came first. They solved a real problem: let a web request hand slow work to a background worker and return, and let workers be added when the backlog grows. Per-message acknowledgement is exactly what that job needs. If a worker dies holding a message, the broker gives the message to another worker.
The model strains when the data is not work for one party but information for many. Consider an OrderPlaced message that payments, inventory, notifications and analytics all need. In a queue-based broker each of them needs its own queue, and the broker copies every message into each one. That works for four readers. It becomes expensive with many, and it has two harder limits: a consumer that connects next month gets nothing from before its queue was created, and a consumer that discovers a bug cannot ask for last week's messages again, because they were deleted when it acknowledged them.
The Kafka design documentation describes the other cost. When the broker must know the state of every message for every consumer, it has to lock each message when it hands it out and mark it when it is acknowledged, and it must deal with messages that were handed out and never acknowledged. Kafka avoids that bookkeeping by making a consumer's state a single integer per partition. The side effect, which the designers call essential, is that a consumer can deliberately rewind and re-read.
RabbitMQ's own documentation is candid about the same trade-off from the other side. Its streams guide says queue types have destructive consume behaviour, so consumed messages cannot be re-read; that delivering one message to many subscribers means binding a dedicated queue for each; and that most queues are designed to converge towards empty and can perform worse with millions of messages waiting. Streams were added to RabbitMQ to cover those cases.
How it works inside
How a RabbitMQ queue moves a message
In RabbitMQ's AMQP 0-9-1 model a publisher never writes to a queue directly. It publishes to an exchange with a routing key, and the exchange places a copy in every queue whose binding matches. A direct exchange matches the routing key exactly, a fanout exchange copies to every bound queue, a topic exchange matches patterns, and a headers exchange matches on message headers. Routing logic therefore lives in the broker and can be changed without touching the publisher.
The broker then pushes messages to subscribed consumers. With several consumers on a queue it normally dispatches round-robin. Each consumer acknowledges each delivery (basic.ack), or rejects it (basic.nack or basic.reject) with a flag saying whether to requeue it. A prefetch limit caps how many unacknowledged deliveries a consumer may hold, which is how a slow consumer avoids being flooded. If a consumer's connection drops with deliveries outstanding, the broker requeues them automatically. Quorum queues count deliveries and drop or dead-letter a message that exceeds the delivery limit, which defaults to 20 since RabbitMQ 4.0.
How a Kafka topic moves a record
A producer appends a record to one partition of a topic, chosen from the record's key. The broker does nothing more with it than store and replicate it. Consumers pull: each fetch asks for records from a given offset. Inside a consumer group every partition is assigned to exactly one member, which reads it in order and periodically commits the offset of the next record it will read. A second group has its own assignments and its own offsets, and reads the same records without any copying on the broker.
Failure handling follows from that. If a consumer dies, its partitions are given to another member, which resumes from the last committed offset; whatever was processed after that commit is processed again. If a record cannot be processed, the consumer must itself choose to retry, skip, or publish it to a dead letter topic and move on. Kafka's consumer group protocol has no reject operation.
Reading does not remove anything
Simulation · not a real clusterWatch records stay in the partition after they are read, and leave only when retention deletes the oldest segment.
Kafka and RabbitMQ side by side
| Question | RabbitMQ queue (classic or quorum) | Kafka topic with a consumer group |
|---|---|---|
| What is stored? | Messages waiting for a consumer | An ordered history of records, per partition |
| After a consumer handles it? | Removed once acknowledged | Nothing changes; retention deletes it later (7 days by default) |
| Who tracks progress? | The broker, per message | The consumer group: one committed offset per partition |
| How does data reach the consumer? | Broker pushes, limited by prefetch (a pull call also exists) | Consumer pulls with fetch requests |
| Who decides where a message goes? | Exchanges and bindings in the broker | The producer picks topic and partition; consumers pick topics |
| Several independent readers? | One queue per reader, each with its own copy | One group id per reader, all reading the same log |
| Scaling one reader | Add consumers to the queue; each takes messages as it is free | Add consumers up to the partition count; extra ones sit idle |
| Ordering | FIFO for one consumer; priorities, several consumers and requeues can reorder | Guaranteed within a partition; none across partitions |
| A message fails | Reject it; requeue or dead-letter through a dead letter exchange | Application decides: retry, skip, or publish to a dead letter topic |
| Replay old data | Not possible: consumed messages are deleted | Reset the group's offsets or start a new group |
The two have grown towards each other
RabbitMQ has streams. A stream is declared like a queue with x-queue-type set to stream, and the documentation describes it as an append-only log with non-destructive consumer semantics: consumers can attach at the first message, the last, a numeric offset or a timestamp, and read the same messages repeatedly until retention (x-max-age, x-max-length-bytes) expires them. Super streams, available since RabbitMQ 3.11, partition a stream much as Kafka partitions a topic. So "RabbitMQ cannot replay" is true of its queues and false of the product.
Kafka has share groups. KIP-932, "Queues for Kafka", adds a second kind of group next to consumer groups. Per the Kafka documentation, consumers in a share group cooperatively consume from the same partitions, can outnumber the partitions, acknowledge records individually, and can release or reject a record. The broker counts delivery attempts (limit 5 by default) and holds each delivered record under a time-limited lock (30 seconds by default). The upgrade notes list it as a preview in Kafka 4.1 and production-ready in 4.2. The documentation recommends share groups where records are processed one at a time and not as part of an ordered stream.
Architecture
Drawing diagram…
Top: a publisher sends to an exchange that routes copies into a payments queue served by two workers and an emails queue served by one worker. Bottom: a producer appends to the orders topic and three consumer groups read it, each with its own offsets.
flowchart TB
subgraph Q["Queue model - RabbitMQ"]
PUB["Publisher"] --> EX["Exchange"]
EX -->|"binding payments"| Q1["Queue payments"]
EX -->|"binding emails"| Q2["Queue emails"]
Q1 --> W1["Payment worker 1"]
Q1 --> W2["Payment worker 2"]
Q2 --> W3["Email worker"]
end
subgraph L["Log model - Kafka"]
PR["Producer"] --> TP["Topic orders - partitions 0 to 2 - retained 7 days"]
TP --> GA["Group payment-service - own offsets"]
TP --> GB["Group notification-service - own offsets"]
TP --> GC["Group analytics - started last week from offset 0"]
end- In the queue model the exchange decides where copies go. The two payment workers compete for messages in one queue; whichever is free takes the next one, and the message disappears when it is acknowledged.
- Adding a third kind of reader in the queue model means declaring a new queue and binding it. It receives messages from that moment on.
- In the log model there is one stored copy of each record.
payment-service,notification-serviceandanalyticsdiffer only in the offsets they have committed. - The
analyticsgroup was added later and still read from offset 0, because the records were retained. Nothing was configured on the broker or the producer to make that possible.
Message flow
Drawing diagram…
In the queue half, the producer publishes m1, the queue pushes it to a worker, the worker acknowledges and the queue removes m1. In the log half, the producer appends r1 at offset 57, the payment-service group fetches it and commits offset 58, and a new group a week later fetches from offset 0 and still receives r1.
sequenceDiagram participant P as Producer participant Q as RabbitMQ queue participant W as Worker participant K as Kafka partition participant G as payment-service group participant N as New group next week Note over P,W: Queue - deliver then delete P->>Q: Publish m1 via exchange Q->>W: Push m1 W->>Q: basic.ack m1 Q->>Q: Remove m1 Note over P,N: Log - append then keep P->>K: Produce r1 K->>K: Append r1 at offset 57 G->>K: Fetch from offset 57 K-->>G: r1 G->>K: Commit offset 58 N->>K: Fetch from offset 0 K-->>N: Records 0 to 57 including r1
- In the queue, the acknowledgement is about one message and causes its removal. If the worker had crashed before
basic.ack, the broker would requeue m1 and push it to a worker again. - In the log, the commit of 58 is about a position in the partition. It causes no change to the stored records.
- The last two arrows are what a queue cannot do: a reader that did not exist when r1 was written still receives it, in its original place in the order.
In a real system
A typical online shop has both kinds of workload, and it is common to see both models in one company.
Order events: a log
The orders topic has 6 partitions and is keyed by order id. OrderPlaced, OrderPaid and OrderCancelled for one order share a key, so they sit in one partition in the order they happened. Four consumer groups read it: payment-service, inventory-service, notification-service and analytics. Order matters (do not ship a cancelled order), several teams need every event, and when the analytics job is rewritten it re-reads the last seven days. Every one of those needs is a log property.
Invoice PDF rendering: a queue
After an order is paid, a PDF invoice must be rendered. Each job takes between two and forty seconds, jobs are unrelated to each other, nobody needs them in order, and nobody wants last week's jobs again. The team wants thirty workers at month end and three on a quiet day, and a job that fails five times should be parked for a human. This is queue-shaped: competing consumers, per-message acknowledgement, a delivery limit and a dead letter destination. A RabbitMQ quorum queue does this directly. On a Kafka 4.2 or later cluster a share group is the equivalent.
Forcing either workload into the other model is possible and awkward. PDF rendering on a classic consumer group needs thirty partitions to run thirty workers, and one slow job delays every job behind it in its partition. Order events on plain queues need a queue per consuming team and a separate archive to answer "what happened last Tuesday".
Java and Spring Boot
Log semantics in code: two listeners with different group ids each receive every record, and an operator can rewind one group without touching the other. This builds on the application from the previous lesson.
package com.example.orders;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderListeners { private static final Logger log = LoggerFactory.getLogger(OrderListeners.class); /** Group 1: takes payment for each order. */ @KafkaListener(id = "payments", topics = "orders", groupId = "payment-service") public void charge(ConsumerRecord<String, OrderPlaced> record) { log.info("[payment-service] {}-{}@{} charge {}", record.topic(), record.partition(), record.offset(), record.value().orderId()); } /** Group 2: counts revenue. Same topic, same records, its own offsets. */ @KafkaListener(id = "analytics", topics = "orders", groupId = "analytics") public void count(ConsumerRecord<String, OrderPlaced> record) { log.info("[analytics] {}-{}@{} add {} {}", record.topic(), record.partition(), record.offset(), record.value().amountMinor(), record.value().currency()); }}Why it is written this way
- The two methods differ only in
groupId. That alone makes Kafka deliver every record to both: each group has its own committed offsets. - With a queue broker the same result needs two queues bound to an exchange. Here nothing is declared on the broker for the second reader.
idnames the listener container andgroupIdsets the consumer group explicitly. WithoutgroupId, Spring uses theidas the group id.- Run two copies of the application and each group's partitions are split between the copies. Each record is still handled once per group.
# 1. Where is each group? One row per partition: committed offset, end of log, lag.bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --describe --group analytics# 2. Stop the analytics consumers first: the group must be inactive when its offsets are reset.# 3. Preview: which offsets would the group move to? (--dry-run is the default)bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --reset-offsets --group analytics --topic orders \ --to-datetime 2026-11-09T09:00:00.000 --dry-run# 4. Apply it, then start the consumers again. payment-service is not affected.bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --reset-offsets --group analytics --topic orders \ --to-datetime 2026-11-09T09:00:00.000 --executeWhy it is written this way
- Replay is an operation on one group's offsets. The records are not re-sent and no other group notices.
- The Kafka documentation says to make sure the group's consumer instances are inactive before resetting, so stop the consumers first.
- Other targets include
--to-earliest,--to-latest,--shift-byand--to-offset. Always look at the--dry-runoutput before using--execute. - Replay means the consumer processes records it has processed before. That is only safe if processing a record twice gives the same result as once.
Configuration
The settings that give a Kafka topic its log behaviour, plus the two that shape share groups. Defaults are from the Apache Kafka 4.4 documentation.
retention.ms (topic)Default 604800000 (7 days)
- What it does
- How long records are kept before old segments are deleted. This, not consumption, is what removes data from a topic.
- When to change it
- Set it to the longest time a consumer may be offline or may need to look back. The docs call it an SLA on how soon consumers must read their data.
cleanup.policy (topic)Default delete
- What it does
deletediscards old segments by time or size.compactkeeps at least the latest record for each key instead.- When to change it
- Keep
deletefor event streams.compactsuits topics that hold the current value per key; see Log compaction.
auto.offset.reset (consumer)Default latest
- What it does
- Where a group with no committed offset starts: the oldest retained record (
earliest) or only new records (latest). - When to change it
- A new group that should process history needs
earliest. With the default, a new group behaves like a freshly declared queue and sees only what arrives afterwards.
offsets.retention.minutes (broker)Default 10080
- What it does
- How long the committed offsets of a group are kept after the group has lost all its consumers.
- When to change it
- A group that stays stopped for longer than this (7 days) loses its position and falls back to
auto.offset.resetwhen it returns. Raise it if consumers can be off for longer.
group.share.delivery.count.limit (broker)Default 5
- What it does
- For share groups only: the maximum number of delivery attempts for a record before it is no longer redelivered.
- When to change it
- This is the queue-style poison message protection that consumer groups do not have. Tune it to how many retries a job deserves.
share.record.lock.duration.ms (group)Default 30000 (30 seconds)
- What it does
- For share groups only: how long a delivered record stays locked to one consumer before it becomes available for another attempt.
- When to change it
- Set it above the normal processing time of one record, or records will be redelivered while the first consumer is still working on them.
Hands-on exercise
More consumers than partitions
See the one place where queue intuition most often misleads people who are new to Kafka: adding consumers to a group does not always add throughput.
Steps · 0 of 6 done
A group of consumers on a 3-partition topic
Simulation · not a real clusterAdd consumers one at a time and watch the partition assignment. Then go past the partition count.
What you should see
The fourth consumer gets no partitions and does no work: with 3 partitions, at most 3 consumers of one group can be active. When a working consumer leaves, the idle one takes over its partition. Only raising the partition count lets the fourth consumer contribute. On a queue, the fourth worker would have started taking messages immediately.
What can go wrong
Scaling a consumer group like a worker pool
- Symptom
- Lag is growing, so the team raises the consumer count from 4 to 12 on a 4-partition topic. Lag keeps growing at the same rate.
- Cause
- A partition is read by one member of a group at a time. Eight of the twelve consumers have no partition and sit idle.
- Fix
- Make each consumer faster, or add partitions (knowing that this changes which partition a key maps to). If the work needs no ordering, a queue or a share group is the better fit.
Assuming a read record is gone
- Symptom
- Sensitive data that was consumed and handled weeks ago is found still readable in the topic, or a new consumer group unexpectedly processes thousands of old records on its first start.
- Cause
- Consumption does not delete. Records live until retention removes them, and a new group configured with
earlieststarts at the oldest one. - Fix
- Set retention deliberately per topic, and decide for every new group whether it should start from
earliestorlatest.
One bad record stops a partition
- Symptom
- Lag on one partition climbs while the others stay at zero. The consumer log shows the same offset failing again and again.
- Cause
- Progress is a position, so the consumer cannot move past a record it keeps failing on, and there is no broker-side reject as there is for a queue.
- Fix
- Decide in the consumer what happens after a bounded number of attempts: skip with an alert, or publish the record to a dead letter topic and continue. See Poison messages and error handling.
Expecting the broker to route or filter
- Symptom
- A team coming from RabbitMQ looks for the Kafka equivalent of a topic exchange so that a consumer receives only
order.cancelledevents. - Cause
- Kafka brokers do not inspect records or route them. A consumer receives everything in the partitions it reads.
- Fix
- Either filter in the consumer and ignore what it does not need, or put event types that have different audiences on different topics.
Replaying into a consumer that is not idempotent
- Symptom
- After an offset reset to fix a reporting bug, customers receive a second copy of last week's emails.
- Cause
- Replay delivers old records again to everything in that group. The handler had a side effect that is not safe to repeat.
- Fix
- Make handlers idempotent before relying on replay, and keep side effects that must not repeat in a separate group from the one you intend to rewind.
How to troubleshoot
Not sure whether a Kafka group is keeping up
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
LAGisLOG-END-OFFSETminusCURRENT-OFFSETfor each partition. This is the Kafka counterpart of queue depth, with one difference: it is per group. Two groups on the same topic have different lag.
Extra consumers were added and throughput did not change
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
- How to read it
- Lists each member with its
#PARTITIONScount andASSIGNMENT. Members showing 0 are idle: the group has more consumers than the topic has partitions.
Not sure what kinds of groups exist on a 4.x cluster
Check
bin/kafka-groups.sh --bootstrap-server localhost:9092 --list
- How to read it
- Prints each group with its
TYPE(Consumer, Share and so on) andPROTOCOL. A group you expected to behave like a queue must show type Share; share groups are then inspected withbin/kafka-share-groups.sh.
A RabbitMQ queue is growing
Check
rabbitmqctl list_queues name messages_ready messages_unacknowledged
- How to read it
messages_readyare waiting for a consumer: there are too few consumers or they are too slow. A highmessages_unacknowledgedmeans consumers hold deliveries without acknowledging them, so look at prefetch and at stuck handlers.
Key takeaways
- A queue deletes a message when one consumer acknowledges it. A log keeps records until retention removes them, and readers only move a position.
- In a queue the broker tracks every message. In Kafka a consumer group tracks one offset per partition, which is cheap and is what makes replay and independent readers possible.
- Queue consumers compete, so more consumers means more throughput. A Kafka consumer group is capped at one active consumer per partition.
- Kafka guarantees order within a partition for every reader. A queue with several consumers or requeues does not preserve order.
- RabbitMQ routes in the broker with exchanges and bindings. Kafka does not route: producers pick the partition and consumers pick topics.
- Both products now offer both models: RabbitMQ streams are logs, and Kafka share groups (production-ready since 4.2) give queue-style consumption.
- Choose by workload: independent jobs done once suit a queue; ordered facts that several readers need, or need again, suit a log.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A video platform must transcode uploads. Each job takes 20 seconds to 15 minutes, jobs are independent, order does not matter, and nobody needs old jobs again. Load is bursty: the team wants to run 5 workers normally and 60 during peaks, and a file that fails 3 times should be set aside.
Which design fits these requirements with the least extra machinery?
Scenario 2 of 3
A RabbitMQ fanout exchange copies order events into three queues for payments, inventory and email. A new recommendations team needs every order from the past 5 days to build its first model, then live events from now on. The existing queues are drained continuously.
What does this requirement tell you about the current design?
Scenario 3 of 3
A shipments topic has 8 partitions keyed by shipment id. A group of 8 consumers processes 400 records per second in total, but 900 arrive, and lag grows by 500 per second. Handling a record is a 20 ms database write. Events for one shipment must stay in order.
Which change addresses the lag without breaking the ordering requirement?
References
References: Apache Kafka 4.4 documentation: Design (consumer position, push vs pull, share groups) ↗ · Apache Kafka 4.4 documentation: Upgrading (status of Queues for Kafka in 4.1 and 4.2) ↗ · KIP-932: Queues for Kafka ↗ · Apache Kafka 4.4 documentation: Basic Kafka operations (managing consumer and share groups) ↗ · Apache Kafka 4.4 documentation: Topic configs ↗ · Apache Kafka 4.4 documentation: Broker configs (share group settings) ↗ · Apache Kafka 4.4 documentation: Group configs ↗ · RabbitMQ documentation: Queues ↗ · RabbitMQ documentation: AMQP 0-9-1 model (exchanges and bindings) ↗ · RabbitMQ documentation: Consumer acknowledgements and publisher confirms ↗ · RabbitMQ documentation: Streams and super streams ↗ · RabbitMQ documentation: Quorum queues (delivery limit) ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener annotation ↗ · Report a problem