Skip to content

Path 1 · lesson 2 of 6

Foundation22 minApache Kafka 4.4 (KRaft) · RabbitMQ 4.4 documentation · Java 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 (kafka-clients 3.9)

Messaging vs event streaming: queues and logs

A queue hands each message to one worker and removes it once acknowledged. A log keeps records and lets each reader track its own position. Compare Kafka and RabbitMQ on that basis.

What you will learn

  • Explain what happens to a message after it is consumed in a queue and in a log, and who keeps track of progress in each
  • Predict how adding consumers changes throughput and ordering in a RabbitMQ queue and in a Kafka consumer group
  • Compare Kafka and RabbitMQ on routing, acknowledgement, ordering, replay and failure handling using each project's own documentation
  • Recognise where the two overlap today: RabbitMQ streams, and share groups in Kafka
  • Choose queue or log semantics for a workload from its requirements

Before you start

The concept

Message brokers come in two families that look alike from a distance, since both sit between a sender and a receiver, and behave very differently up close. The difference is one question: what happens to a message after someone has read it?

In a queue, a message is a piece of work waiting to be done. The broker gives it to one consumer, waits for that consumer to acknowledge it, and then removes it. The queue is meant to drain: an empty queue is the healthy state. In a log, a record is an entry in a history. Reading it changes nothing in the log. It stays until a retention rule removes it, and each reader keeps track of how far it has read.

Queue semantics (traditional messaging)

  • A message is delivered to one consumer and removed once that consumer acknowledges it
  • The broker tracks the state of every message: ready, delivered but unacknowledged, done
  • Several consumers on one queue compete: each message goes to one of them, so adding consumers adds throughput directly
  • Each message is acknowledged or rejected on its own; a rejected one can be requeued or dead-lettered
  • The broker routes: rules decide which queues get a copy of each message
  • Order holds for a single consumer; with several consumers and redeliveries it does not
  • Once consumed, a message is gone. A new consumer sees only what arrives after its queue exists

Log semantics (event streaming)

  • A record is appended to a partition and stays there until retention removes it, read or not
  • The broker stores the records; each consumer group stores one position (offset) per partition
  • Within a group, a partition belongs to one consumer at a time, so parallelism is capped by the partition count
  • Progress is a position: committing offset N means everything before N is done. There is no per-record acknowledgement
  • The broker does not route. Producers choose a topic and partition; consumers choose which topics to read
  • Order is guaranteed inside a partition, for every reader, every time it is read
  • Any group can go back to an earlier offset and read again, and a new group can start from the oldest record

Neither model is better. They answer different needs. A queue is a natural fit for work to be done once: resize this image, send this email. A log is a natural fit for facts that several parties need, in order, possibly more than once: an order was placed, a price changed.

Why it exists

Queues came first. They solved a real problem: let a web request hand slow work to a background worker and return, and let workers be added when the backlog grows. Per-message acknowledgement is exactly what that job needs. If a worker dies holding a message, the broker gives the message to another worker.

The model strains when the data is not work for one party but information for many. Consider an OrderPlaced message that payments, inventory, notifications and analytics all need. In a queue-based broker each of them needs its own queue, and the broker copies every message into each one. That works for four readers. It becomes expensive with many, and it has two harder limits: a consumer that connects next month gets nothing from before its queue was created, and a consumer that discovers a bug cannot ask for last week's messages again, because they were deleted when it acknowledged them.

The Kafka design documentation describes the other cost. When the broker must know the state of every message for every consumer, it has to lock each message when it hands it out and mark it when it is acknowledged, and it must deal with messages that were handed out and never acknowledged. Kafka avoids that bookkeeping by making a consumer's state a single integer per partition. The side effect, which the designers call essential, is that a consumer can deliberately rewind and re-read.

RabbitMQ's own documentation is candid about the same trade-off from the other side. Its streams guide says queue types have destructive consume behaviour, so consumed messages cannot be re-read; that delivering one message to many subscribers means binding a dedicated queue for each; and that most queues are designed to converge towards empty and can perform worse with millions of messages waiting. Streams were added to RabbitMQ to cover those cases.

How it works inside

How a RabbitMQ queue moves a message

In RabbitMQ's AMQP 0-9-1 model a publisher never writes to a queue directly. It publishes to an exchange with a routing key, and the exchange places a copy in every queue whose binding matches. A direct exchange matches the routing key exactly, a fanout exchange copies to every bound queue, a topic exchange matches patterns, and a headers exchange matches on message headers. Routing logic therefore lives in the broker and can be changed without touching the publisher.

The broker then pushes messages to subscribed consumers. With several consumers on a queue it normally dispatches round-robin. Each consumer acknowledges each delivery (basic.ack), or rejects it (basic.nack or basic.reject) with a flag saying whether to requeue it. A prefetch limit caps how many unacknowledged deliveries a consumer may hold, which is how a slow consumer avoids being flooded. If a consumer's connection drops with deliveries outstanding, the broker requeues them automatically. Quorum queues count deliveries and drop or dead-letter a message that exceeds the delivery limit, which defaults to 20 since RabbitMQ 4.0.

How a Kafka topic moves a record

A producer appends a record to one partition of a topic, chosen from the record's key. The broker does nothing more with it than store and replicate it. Consumers pull: each fetch asks for records from a given offset. Inside a consumer group every partition is assigned to exactly one member, which reads it in order and periodically commits the offset of the next record it will read. A second group has its own assignments and its own offsets, and reads the same records without any copying on the broker.

Failure handling follows from that. If a consumer dies, its partitions are given to another member, which resumes from the last committed offset; whatever was processed after that commit is processed again. If a record cannot be processed, the consumer must itself choose to retry, skip, or publish it to a dead letter topic and move on. Kafka's consumer group protocol has no reject operation.

Reading does not remove anything

Simulation · not a real cluster

Watch records stay in the partition after they are read, and leave only when retention deletes the oldest segment.

Kafka and RabbitMQ side by side

QuestionRabbitMQ queue (classic or quorum)Kafka topic with a consumer group
What is stored?Messages waiting for a consumerAn ordered history of records, per partition
After a consumer handles it?Removed once acknowledgedNothing changes; retention deletes it later (7 days by default)
Who tracks progress?The broker, per messageThe consumer group: one committed offset per partition
How does data reach the consumer?Broker pushes, limited by prefetch (a pull call also exists)Consumer pulls with fetch requests
Who decides where a message goes?Exchanges and bindings in the brokerThe producer picks topic and partition; consumers pick topics
Several independent readers?One queue per reader, each with its own copyOne group id per reader, all reading the same log
Scaling one readerAdd consumers to the queue; each takes messages as it is freeAdd consumers up to the partition count; extra ones sit idle
OrderingFIFO for one consumer; priorities, several consumers and requeues can reorderGuaranteed within a partition; none across partitions
A message failsReject it; requeue or dead-letter through a dead letter exchangeApplication decides: retry, skip, or publish to a dead letter topic
Replay old dataNot possible: consumed messages are deletedReset the group's offsets or start a new group
The queue column describes RabbitMQ queues, not RabbitMQ streams. The Kafka column describes consumer groups, not share groups.

The two have grown towards each other

RabbitMQ has streams. A stream is declared like a queue with x-queue-type set to stream, and the documentation describes it as an append-only log with non-destructive consumer semantics: consumers can attach at the first message, the last, a numeric offset or a timestamp, and read the same messages repeatedly until retention (x-max-age, x-max-length-bytes) expires them. Super streams, available since RabbitMQ 3.11, partition a stream much as Kafka partitions a topic. So "RabbitMQ cannot replay" is true of its queues and false of the product.

Kafka has share groups. KIP-932, "Queues for Kafka", adds a second kind of group next to consumer groups. Per the Kafka documentation, consumers in a share group cooperatively consume from the same partitions, can outnumber the partitions, acknowledge records individually, and can release or reject a record. The broker counts delivery attempts (limit 5 by default) and holds each delivered record under a time-limited lock (30 seconds by default). The upgrade notes list it as a preview in Kafka 4.1 and production-ready in 4.2. The documentation recommends share groups where records are processed one at a time and not as part of an ordered stream.

Architecture

Drawing diagram…

Top: a publisher sends to an exchange that routes copies into a payments queue served by two workers and an emails queue served by one worker. Bottom: a producer appends to the orders topic and three consumer groups read it, each with its own offsets.

flowchart TB
  subgraph Q["Queue model - RabbitMQ"]
    PUB["Publisher"] --> EX["Exchange"]
    EX -->|"binding payments"| Q1["Queue payments"]
    EX -->|"binding emails"| Q2["Queue emails"]
    Q1 --> W1["Payment worker 1"]
    Q1 --> W2["Payment worker 2"]
    Q2 --> W3["Email worker"]
  end
  subgraph L["Log model - Kafka"]
    PR["Producer"] --> TP["Topic orders - partitions 0 to 2 - retained 7 days"]
    TP --> GA["Group payment-service - own offsets"]
    TP --> GB["Group notification-service - own offsets"]
    TP --> GC["Group analytics - started last week from offset 0"]
  end
A queue broker copies a message into one queue per reader and deletes each copy when acknowledged. A log keeps one copy and each group keeps a position.
  • In the queue model the exchange decides where copies go. The two payment workers compete for messages in one queue; whichever is free takes the next one, and the message disappears when it is acknowledged.
  • Adding a third kind of reader in the queue model means declaring a new queue and binding it. It receives messages from that moment on.
  • In the log model there is one stored copy of each record. payment-service, notification-service and analytics differ only in the offsets they have committed.
  • The analytics group was added later and still read from offset 0, because the records were retained. Nothing was configured on the broker or the producer to make that possible.

Message flow

Drawing diagram…

In the queue half, the producer publishes m1, the queue pushes it to a worker, the worker acknowledges and the queue removes m1. In the log half, the producer appends r1 at offset 57, the payment-service group fetches it and commits offset 58, and a new group a week later fetches from offset 0 and still receives r1.

sequenceDiagram
  participant P as Producer
  participant Q as RabbitMQ queue
  participant W as Worker
  participant K as Kafka partition
  participant G as payment-service group
  participant N as New group next week
  Note over P,W: Queue - deliver then delete
  P->>Q: Publish m1 via exchange
  Q->>W: Push m1
  W->>Q: basic.ack m1
  Q->>Q: Remove m1
  Note over P,N: Log - append then keep
  P->>K: Produce r1
  K->>K: Append r1 at offset 57
  G->>K: Fetch from offset 57
  K-->>G: r1
  G->>K: Commit offset 58
  N->>K: Fetch from offset 0
  K-->>N: Records 0 to 57 including r1
The same publish, followed through a queue and through a log.
  1. In the queue, the acknowledgement is about one message and causes its removal. If the worker had crashed before basic.ack, the broker would requeue m1 and push it to a worker again.
  2. In the log, the commit of 58 is about a position in the partition. It causes no change to the stored records.
  3. The last two arrows are what a queue cannot do: a reader that did not exist when r1 was written still receives it, in its original place in the order.

In a real system

A typical online shop has both kinds of workload, and it is common to see both models in one company.

Order events: a log

The orders topic has 6 partitions and is keyed by order id. OrderPlaced, OrderPaid and OrderCancelled for one order share a key, so they sit in one partition in the order they happened. Four consumer groups read it: payment-service, inventory-service, notification-service and analytics. Order matters (do not ship a cancelled order), several teams need every event, and when the analytics job is rewritten it re-reads the last seven days. Every one of those needs is a log property.

Invoice PDF rendering: a queue

After an order is paid, a PDF invoice must be rendered. Each job takes between two and forty seconds, jobs are unrelated to each other, nobody needs them in order, and nobody wants last week's jobs again. The team wants thirty workers at month end and three on a quiet day, and a job that fails five times should be parked for a human. This is queue-shaped: competing consumers, per-message acknowledgement, a delivery limit and a dead letter destination. A RabbitMQ quorum queue does this directly. On a Kafka 4.2 or later cluster a share group is the equivalent.

Forcing either workload into the other model is possible and awkward. PDF rendering on a classic consumer group needs thirty partitions to run thirty workers, and one slow job delays every job behind it in its partition. Order events on plain queues need a queue per consuming team and a separate archive to answer "what happened last Tuesday".

Java and Spring Boot

Log semantics in code: two listeners with different group ids each receive every record, and an operator can rewind one group without touching the other. This builds on the application from the previous lesson.

src/main/java/com/example/orders/OrderListeners.javaJava
package com.example.orders;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderListeners {    private static final Logger log = LoggerFactory.getLogger(OrderListeners.class);    /** Group 1: takes payment for each order. */    @KafkaListener(id = "payments", topics = "orders", groupId = "payment-service")    public void charge(ConsumerRecord<String, OrderPlaced> record) {        log.info("[payment-service] {}-{}@{} charge {}",                record.topic(), record.partition(), record.offset(), record.value().orderId());    }    /** Group 2: counts revenue. Same topic, same records, its own offsets. */    @KafkaListener(id = "analytics", topics = "orders", groupId = "analytics")    public void count(ConsumerRecord<String, OrderPlaced> record) {        log.info("[analytics] {}-{}@{} add {} {}",                record.topic(), record.partition(), record.offset(),                record.value().amountMinor(), record.value().currency());    }}

Why it is written this way

  • The two methods differ only in groupId. That alone makes Kafka deliver every record to both: each group has its own committed offsets.
  • With a queue broker the same result needs two queues bound to an exchange. Here nothing is declared on the broker for the second reader.
  • id names the listener container and groupId sets the consumer group explicitly. Without groupId, Spring uses the id as the group id.
  • Run two copies of the application and each group's partitions are split between the copies. Each record is still handled once per group.

Configuration

The settings that give a Kafka topic its log behaviour, plus the two that shape share groups. Defaults are from the Apache Kafka 4.4 documentation.

retention.ms (topic)Default 604800000 (7 days)
What it does
How long records are kept before old segments are deleted. This, not consumption, is what removes data from a topic.
When to change it
Set it to the longest time a consumer may be offline or may need to look back. The docs call it an SLA on how soon consumers must read their data.
cleanup.policy (topic)Default delete
What it does
delete discards old segments by time or size. compact keeps at least the latest record for each key instead.
When to change it
Keep delete for event streams. compact suits topics that hold the current value per key; see Log compaction.
auto.offset.reset (consumer)Default latest
What it does
Where a group with no committed offset starts: the oldest retained record (earliest) or only new records (latest).
When to change it
A new group that should process history needs earliest. With the default, a new group behaves like a freshly declared queue and sees only what arrives afterwards.
offsets.retention.minutes (broker)Default 10080
What it does
How long the committed offsets of a group are kept after the group has lost all its consumers.
When to change it
A group that stays stopped for longer than this (7 days) loses its position and falls back to auto.offset.reset when it returns. Raise it if consumers can be off for longer.
group.share.delivery.count.limit (broker)Default 5
What it does
For share groups only: the maximum number of delivery attempts for a record before it is no longer redelivered.
When to change it
This is the queue-style poison message protection that consumer groups do not have. Tune it to how many retries a job deserves.
share.record.lock.duration.ms (group)Default 30000 (30 seconds)
What it does
For share groups only: how long a delivered record stays locked to one consumer before it becomes available for another attempt.
When to change it
Set it above the normal processing time of one record, or records will be redelivered while the first consumer is still working on them.

Hands-on exercise

More consumers than partitions

See the one place where queue intuition most often misleads people who are new to Kafka: adding consumers to a group does not always add throughput.

Steps · 0 of 6 done

A group of consumers on a 3-partition topic

Simulation · not a real cluster

Add consumers one at a time and watch the partition assignment. Then go past the partition count.

Take it further in the lab: Consumer group assignment

What can go wrong

Scaling a consumer group like a worker pool
Symptom
Lag is growing, so the team raises the consumer count from 4 to 12 on a 4-partition topic. Lag keeps growing at the same rate.
Cause
A partition is read by one member of a group at a time. Eight of the twelve consumers have no partition and sit idle.
Fix
Make each consumer faster, or add partitions (knowing that this changes which partition a key maps to). If the work needs no ordering, a queue or a share group is the better fit.
Assuming a read record is gone
Symptom
Sensitive data that was consumed and handled weeks ago is found still readable in the topic, or a new consumer group unexpectedly processes thousands of old records on its first start.
Cause
Consumption does not delete. Records live until retention removes them, and a new group configured with earliest starts at the oldest one.
Fix
Set retention deliberately per topic, and decide for every new group whether it should start from earliest or latest.
One bad record stops a partition
Symptom
Lag on one partition climbs while the others stay at zero. The consumer log shows the same offset failing again and again.
Cause
Progress is a position, so the consumer cannot move past a record it keeps failing on, and there is no broker-side reject as there is for a queue.
Fix
Decide in the consumer what happens after a bounded number of attempts: skip with an alert, or publish the record to a dead letter topic and continue. See Poison messages and error handling.
Expecting the broker to route or filter
Symptom
A team coming from RabbitMQ looks for the Kafka equivalent of a topic exchange so that a consumer receives only order.cancelled events.
Cause
Kafka brokers do not inspect records or route them. A consumer receives everything in the partitions it reads.
Fix
Either filter in the consumer and ignore what it does not need, or put event types that have different audiences on different topics.
Replaying into a consumer that is not idempotent
Symptom
After an offset reset to fix a reporting bug, customers receive a second copy of last week's emails.
Cause
Replay delivers old records again to everything in that group. The handler had a side effect that is not safe to repeat.
Fix
Make handlers idempotent before relying on replay, and keep side effects that must not repeat in a separate group from the one you intend to rewind.

How to troubleshoot

Not sure whether a Kafka group is keeping up

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
LAG is LOG-END-OFFSET minus CURRENT-OFFSET for each partition. This is the Kafka counterpart of queue depth, with one difference: it is per group. Two groups on the same topic have different lag.
Extra consumers were added and throughput did not change

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
How to read it
Lists each member with its #PARTITIONS count and ASSIGNMENT. Members showing 0 are idle: the group has more consumers than the topic has partitions.
Not sure what kinds of groups exist on a 4.x cluster

Check

bin/kafka-groups.sh --bootstrap-server localhost:9092 --list
How to read it
Prints each group with its TYPE (Consumer, Share and so on) and PROTOCOL. A group you expected to behave like a queue must show type Share; share groups are then inspected with bin/kafka-share-groups.sh.
A RabbitMQ queue is growing

Check

rabbitmqctl list_queues name messages_ready messages_unacknowledged
How to read it
messages_ready are waiting for a consumer: there are too few consumers or they are too slow. A high messages_unacknowledged means consumers hold deliveries without acknowledging them, so look at prefetch and at stuck handlers.

Key takeaways

  • A queue deletes a message when one consumer acknowledges it. A log keeps records until retention removes them, and readers only move a position.
  • In a queue the broker tracks every message. In Kafka a consumer group tracks one offset per partition, which is cheap and is what makes replay and independent readers possible.
  • Queue consumers compete, so more consumers means more throughput. A Kafka consumer group is capped at one active consumer per partition.
  • Kafka guarantees order within a partition for every reader. A queue with several consumers or requeues does not preserve order.
  • RabbitMQ routes in the broker with exchanges and bindings. Kafka does not route: producers pick the partition and consumers pick topics.
  • Both products now offer both models: RabbitMQ streams are logs, and Kafka share groups (production-ready since 4.2) give queue-style consumption.
  • Choose by workload: independent jobs done once suit a queue; ordered facts that several readers need, or need again, suit a log.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A video platform must transcode uploads. Each job takes 20 seconds to 15 minutes, jobs are independent, order does not matter, and nobody needs old jobs again. Load is bursty: the team wants to run 5 workers normally and 60 during peaks, and a file that fails 3 times should be set aside.

Which design fits these requirements with the least extra machinery?

Scenario 2 of 3

A RabbitMQ fanout exchange copies order events into three queues for payments, inventory and email. A new recommendations team needs every order from the past 5 days to build its first model, then live events from now on. The existing queues are drained continuously.

What does this requirement tell you about the current design?

Scenario 3 of 3

A shipments topic has 8 partitions keyed by shipment id. A group of 8 consumers processes 400 records per second in total, but 900 arrive, and lag grows by 500 per second. Handling a record is a 20 ms database write. Events for one shipment must stay in order.

Which change addresses the lag without breaking the ordering requirement?

References

References: Apache Kafka 4.4 documentation: Design (consumer position, push vs pull, share groups) ↗ · Apache Kafka 4.4 documentation: Upgrading (status of Queues for Kafka in 4.1 and 4.2) ↗ · KIP-932: Queues for Kafka ↗ · Apache Kafka 4.4 documentation: Basic Kafka operations (managing consumer and share groups) ↗ · Apache Kafka 4.4 documentation: Topic configs ↗ · Apache Kafka 4.4 documentation: Broker configs (share group settings) ↗ · Apache Kafka 4.4 documentation: Group configs ↗ · RabbitMQ documentation: Queues ↗ · RabbitMQ documentation: AMQP 0-9-1 model (exchanges and bindings) ↗ · RabbitMQ documentation: Consumer acknowledgements and publisher confirms ↗ · RabbitMQ documentation: Streams and super streams ↗ · RabbitMQ documentation: Quorum queues (delivery limit) ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener annotation ↗ · Report a problem