Skip to content

Path 7 · lesson 8 of 8

Intermediate30 minApache Kafka 4.x (configuration reference read for 4.1) · Spring for Apache Kafka 3.3 on Spring Boot 3.x · Java 21 · PostgreSQL

Request-reply and event fan-out

Two messaging shapes on Kafka: asking a question and waiting for the answer on a reply topic, and publishing one event to many independent consumer groups. One of them fits Kafka far better.

What you will learn

  • Implement request-reply over Kafka with a reply topic and a correlation id, using ReplyingKafkaTemplate and @SendTo
  • Explain why a reply can reach the wrong application instance and configure one of the three ways around it
  • Say exactly what a reply timeout does and does not mean, and design requests that are safe under it
  • Decide when a plain HTTP or gRPC call is the better tool, and when an asynchronous command with a durable result is
  • Fan one event out to several consumer groups, and add a new consumer without changing the producer
  • Treat the event schema as a public API and change it without breaking consumers you do not know

Before you start

The concept

This lesson covers two shapes of communication between services. They are often presented together because both are "messaging patterns", but they sit at opposite ends of how well they suit Kafka.

Request-reply is a conversation: service A sends a request and needs an answer to continue. Over Kafka, A publishes the request to a request topic with two headers: a correlation id that identifies this request, and the name of a reply topic. Service B consumes the request, does the work and publishes the answer to the reply topic with the same correlation id. A consumes the reply topic and matches the answer to the waiting caller.

Fan-out is an announcement: the Order Service publishes OrderCreated once, and every interested service reads it in its own consumer group. Payment charges, Inventory reserves, Notification sends an email, analytics counts. The producer does not know who reads the event and does not wait for anyone.

Request-reply over Kafka: trade-offs

Advantages

  • The caller needs no network route to the responder, only to the brokers
  • Requests are buffered: a short responder restart delays answers without failing them, if the caller's timeout allows
  • Responders scale as a consumer group, with no load balancer or service discovery
  • Requests for one key are handled in order by one responder instance
  • The same request stream can be read by other groups, for audit or analytics

Disadvantages

  • Higher and less predictable latency: two hops, batching, fetch waits, and any consumer lag on the request topic
  • The reply must find the one instance that is waiting, which Kafka's group assignment does not arrange
  • A timeout does not cancel anything: the request stays in the log and is processed later
  • Waiting callers live in memory. If that instance restarts, the reply has nobody to go to
  • A rebalance on either side stalls every in-flight request
  • Two topics, a reply consumer per instance and several timeouts to operate, for what an HTTP call does directly

Fan-out: trade-offs

Advantages

  • A new consumer is a new group id: no change, deployment or permission from the producing team's code
  • Consumers fail, lag, retry and replay independently. A slow one does not slow the others or the producer
  • The event is stored once and read many times. An extra group adds fetch traffic, not storage
  • Each group reads a partition in the same order, so every consumer sees one order's events in sequence
  • A new consumer can start from the beginning and build its state from history

Disadvantages

  • The event schema becomes a public API with consumers the producer cannot list
  • No one sees the whole flow: what happens after OrderCreated is spread over many services
  • The producer gets no result. Anything it needs back must come as another event
  • Every consumer needs its own idempotency, retry and dead letter handling
  • Consumers are eventually consistent with each other: Notification may email before Inventory has reserved

Why it exists

The problem request-reply tries to solve

A team has moved its services onto Kafka and then meets an interaction that is a question: at checkout the Order Service wants to know whether a SKU is available before it accepts the order. Opening an HTTP path to Inventory feels like a step back: another protocol, another set of credentials, service discovery, a circuit breaker. So the question is sent through Kafka as well. That works, and it is supported by Spring for Apache Kafka. It also brings every difficulty of this lesson, because a log is being used as a telephone.

The better first question is whether the caller must wait. In the order system it does not: the Order Service accepts the order as PENDING, publishes OrderCreated, and learns the stock result later from an inventory.events record that it handles like any other event. That is an asynchronous command and result. The state of the conversation is a row in a database, not a thread waiting on a future, and it survives restarts. The saga lesson builds exactly this.

The problem fan-out solves

Without fan-out, the Order Service calls Payment, then Inventory, then Notification. Each new consumer of "an order was placed" means a change and a deployment of the Order Service, one more dependency that can fail checkout, and one more team to coordinate with. Checkout becomes as slow as the sum of the calls and as available as their product.

With traditional queues the same goal needs one queue per consumer and a producer or exchange that copies each message into all of them. Kafka's design removes the copying: a topic is a log, and a consumer group is only a set of committed positions in that log. Any number of groups can read the same records, each at its own pace. The Kafka introduction puts it this way: a topic can have zero, one or many consumers, and events can be read as often as needed because, unlike in traditional messaging systems, they are not deleted after consumption.

How it works inside

How ReplyingKafkaTemplate works

  1. sendAndReceive(record) generates a correlation id, adds the headers kafka_correlationId and kafka_replyTopic (and optionally kafka_replyPartition), stores a future in an in-memory map under the correlation id, and sends the record.
  2. It returns a RequestReplyFuture. Its getSendFuture() tells you whether the send succeeded. The future itself completes when the reply arrives.
  3. On the responder, a @KafkaListener method annotated with @SendTo returns the answer. The listener infrastructure sends it to the topic named in the reply-topic header and copies the correlation id.
  4. The template owns a listener container on the reply topic. When a reply arrives it looks up the correlation id in its map and completes the future.
  5. If no reply arrives within the reply timeout (the template's defaultReplyTimeout, 5 seconds by default), the future completes exceptionally with a KafkaReplyTimeoutException and the entry is removed from the map.

The reply has to reach the instance that asked

The map of waiting futures lives in one JVM. Run three instances of the Order Service with one consumer group on the reply topic, and Kafka assigns each reply partition to one of them. A reply lands on a partition, and the instance that owns that partition is usually not the one that sent the request. It finds no matching correlation id and discards the reply, and the real requester times out. The Spring reference gives three ways to arrange it:

ApproachHowCost
A reply topic per instanceEach instance listens on its own topic and names it in the reply-topic header.Topics must be created and removed as instances come and go. Awkward with autoscaling.
A shared reply topic, a different group.id per instanceEvery instance is its own consumer group, so every instance receives every reply and ignores those whose correlation id it does not know. Set sharedReplyTopic to true so unknown replies are logged at DEBUG, not ERROR.Reply traffic is multiplied by the number of instances. Each restart creates a new group if the id is random.
A shared reply topic, a fixed partition per instanceEach instance is manually assigned one partition and sends it in the kafka_replyPartition header. The responder writes the reply to that partition.No group management on the reply container, and something must hand out partition numbers to instances.

With the shared-topic approach, the reply consumer should start at the end of the topic (auto.offset.reset=latest), because a new group has no interest in old replies. Then a request sent before the reply container has its partitions assigned can have its reply written before the consumer's start position. waitForAssignment on the template exists for this: call it before the first request.

What a timeout means

A timeout in request-reply over Kafka means only one thing: the caller stopped waiting. It does not mean the request failed, and nothing is cancelled. The request record is in the log. If the responder was restarting or lagging, it processes the request later and publishes a reply that nobody is waiting for. Three rules follow:

  • Requests should be queries or idempotent commands. If a timed-out "reserve stock" is retried by the caller, both copies are eventually processed. A query has no side effect to repeat.
  • Give requests a deadline. Put the time after which the answer is useless in the request. A responder working through a backlog can then skip expired requests cheaply.
  • Latency includes consumer lag. The reply time is produce, wait in the request topic behind every earlier request, process, produce, fetch. A responder that is 10 seconds behind makes every caller time out, and each retry adds to the backlog.

There are two more timeouts to keep in order. The producer's delivery.timeout.ms (default 2 minutes) bounds how long the send may keep trying. A request whose reply timeout has passed can still be delivered a minute later. And the HTTP request that triggered all this has its own timeout upstream. Keep them nested: reply timeout shorter than the upstream timeout, and the producer's delivery timeout not far above the reply timeout for this template.

Several replies to one request

AggregatingReplyingKafkaTemplate is the scatter-gather variant: one request is consumed by several responders (each in its own group) and the template collects their replies. A release strategy that you supply is consulted on each reply and decides when the set is complete. With returnPartialOnTimeout (false by default) a timeout returns what has arrived so far, as long as there is at least one reply. Its reply container must use manual acknowledgement. The same instance-routing and timeout rules apply.

Fan-out: one log, many positions

Each consumer group commits its own offsets per partition. payment-service at offset 9,412 and analytics at offset 2,007 on the same partition do not affect each other. Within a group a partition is read by one member, so the group's parallelism is capped by the partition count. Across groups there is no limit, and every group receives every record.

Drawing diagram…

Sequence diagram: the Order Service writes OrderCreated at offset 9412. The payment and inventory groups each read it and commit their own offset. The email notification group reads it but is slow and falls behind. A new analytics group joins later with no committed offset, and auto.offset.reset decides where it starts. The email group commits later without having affected the others.

sequenceDiagram
  participant O as Order Service
  participant K as orders.events
  participant P as Group payment-service
  participant I as Group inventory-service
  participant E as Group notification-email
  participant A as Group order-analytics, added later
  O->>K: OrderCreated o-1001 at offset 9412
  K->>P: offset 9412
  P->>K: commit 9413 for payment-service
  K->>I: offset 9412
  I->>K: commit 9413 for inventory-service
  K->>E: offset 9412
  Note over E: email provider slow, group falls behind
  Note over A: new group id, no committed offset
  A->>K: join, auto.offset.reset decides the start
  K->>A: records from the chosen start position
  E->>K: commit 9413 for notification-email
One record, four groups, four independent positions. The producer is not involved after the first arrow.

Adding a consumer without touching the producer

  1. Choose a new group.id. Reusing an existing one does not add a consumer. It joins that group and takes a share of its partitions, so each service then sees only part of the events.
  2. Decide where the group starts. With no committed offsets, auto.offset.reset applies: the default latest means "events from now on", and earliest means "everything the topic still holds".
  3. Grant the new service read access to the topic and to its group, if the cluster uses ACLs.
  4. Deploy. The producer's code, configuration and deployment are unchanged.

The event is a public API

With direct calls, a producer knows its callers and can change them together with the interface. With fan-out it cannot even list its consumers. Any field in OrderCreated may be relied on by a team the Order Service's developers have never met. That changes how the schema must be treated:

  • Change additively. Add optional fields. Do not remove or rename a field, change its type, or change what it means.
  • Make consumers tolerant. A consumer ignores fields it does not know and does not fail on a new enum value.
  • Publish facts, not internals. Include what happened and the data consumers need to act on it. Leave out the producer's table layout and internal status codes, because whatever is published will be depended on.
  • A breaking change is a new contract. Publish a new event type or a new topic alongside the old one and retire the old one when its consumers have moved.
  • Enforce it. A schema registry with a compatibility rule rejects a breaking change at build or publish time. See schema evolution.

Fan-out to notification channels

DesignHowGoodBad
A group per channelnotification-email, notification-sms, notification-push each consume orders.events.Channels fail and retry independently. An SMS provider outage does not delay email. A new channel is a new group.Each channel reads and filters the whole stream, and repeats the lookup of customer preferences.
One group that dispatchesOne consumer reads the event and calls every channel in turn.One place for preferences and templates.A failing channel blocks or forces a retry of the others, so channels that succeeded are sent again unless each send is deduplicated.
One group that fans out againOne consumer decides which channels apply and publishes one command per channel to notifications.email, notifications.sms. Channel workers consume those.Preferences decided once, channels still independent.More topics, and the dispatcher's publish must be made reliable (an outbox, or a Kafka transaction since it is topic to topic).

Failure scenarios

Crash pointResulting stateRecovery
Request-reply: responder down when the request is sentRequest waits in the topic. Caller times out.The responder processes the request after restart and replies to nobody. Safe only if the request is a query or idempotent. A deadline lets it skip the work.
Request-reply: requester instance crashes while waitingThe in-memory future is gone. The reply arrives and matches nothing.The upstream caller sees an error and retries, creating a new request. Nothing connects it to the old one.
Request-reply: reply lands on another instanceThe owner of that partition discards it. The requester times out.No recovery at runtime. Fix the reply routing: a group or partition per instance.
Request-reply: responder crashes after the work, before the reply is sentOffset not committed, no replyThe request is redelivered and processed again, and a reply is sent, possibly after the caller has timed out.
Request-reply: rebalance of the reply containerReplies are not consumed until assignment completesIn-flight requests time out. With a new group and latest, replies written during the gap are skipped.
Fan-out: one consumer group is downIts lag grows. Other groups and the producer are unaffected.It resumes from its committed offset, provided it returns within the topic's retention.
Fan-out: a consumer crashes after its effect, before the offset commitEffect applied, offset not advancedThe record is redelivered to that group only. Its idempotency check absorbs it.
Fan-out: the producer ships an incompatible schemaSome consumers fail to deserialize every new recordTheir error handling dead-letters the records or stalls. Fix forward with a compatible schema, then replay each affected group's dead letters.
Crash or fault at each point, what is left behind, and how the system recovers.

Architecture

Drawing diagram…

Flowchart with two parts. In request-reply, two Order Service instances send requests carrying a correlation id and reply topic to a request topic. The inventory group consumes them and writes replies with the same correlation id to a reply topic, which both instances read in their own consumer groups, each keeping only replies it is waiting for. In fan-out, the Order Service writes OrderCreated to orders.events and five separate consumer groups read it: payment, inventory, email notification, SMS notification and a later analytics group.

flowchart TB
  subgraph RR["Request-reply, a conversation"]
    A1["Order Service instance 1"] -->|"request with correlationId and replyTopic"| RQ["inventory.stock-check.requests"]
    A2["Order Service instance 2"] -->|"request"| RQ
    RQ --> INV["Group inventory-stock-check"]
    INV -->|"reply with the same correlationId"| RP["inventory.stock-check.replies"]
    RP -->|"own group id, matches its correlationId"| A1
    RP -->|"own group id, ignores unknown ids"| A2
  end
  subgraph FO["Fan-out, an announcement"]
    OS["Order Service"] -->|"OrderCreated, key orderId"| OE["orders.events"]
    OE --> G1["Group payment-service"]
    OE --> G2["Group inventory-service"]
    OE --> G3["Group notification-email"]
    OE --> G4["Group notification-sms"]
    OE --> G5["Group order-analytics, added later"]
  end
Top: every reply must find the one instance that asked. Bottom: the producer writes once and does not know its readers.
  • In the top half arrows form a loop. The requester depends on the responder being up and fast, exactly as with an HTTP call, with two topics in between.
  • Both Order Service instances read the reply topic in their own group, so both receive every reply. That duplication is the price of making sure the right one gets it.
  • In the bottom half all arrows point away from the producer. Nothing flows back, so no consumer can slow down or fail checkout.
  • The analytics group was added without any change above the topic. That one-way dependency is what fan-out buys.

Message flow

Drawing diagram…

Sequence diagram: Order instance 1 sends request c-71 naming the reply topic and stores a future with a two second timeout. The inventory responder reads it, queries stock and writes the reply. Both order instances receive the reply. Instance 2 ignores it and instance 1 completes its future. Instance 1 then sends request c-72 while the responder is restarting. After two seconds the future fails with a reply timeout. The responder later processes c-72 and replies, and instance 1 drops the reply because no request is pending.

sequenceDiagram
  participant A1 as Order instance 1
  participant A2 as Order instance 2
  participant RQ as stock-check.requests
  participant IV as Inventory responder
  participant RP as stock-check.replies
  A1->>RQ: request c-71, replyTopic stock-check.replies
  Note over A1: future stored under c-71, timeout 2 s
  RQ->>IV: request c-71
  IV->>IV: SELECT available for the sku
  IV->>RP: reply c-71, available true
  RP->>A2: reply c-71
  Note over A2: unknown correlation id, ignored
  RP->>A1: reply c-71
  Note over A1: future c-71 completes
  A1->>RQ: request c-72
  Note over IV: responder restarting for 5 s
  Note over A1: 2 s pass, KafkaReplyTimeoutException
  RQ->>IV: request c-72, delivered after restart
  IV->>RP: reply c-72
  RP->>A1: reply c-72
  Note over A1: no pending request c-72, reply dropped
The first request succeeds through a shared reply topic. The second times out, yet is still processed and answered later.
  1. Both instances receive reply c-71. Instance 2's discard is normal with a shared reply topic, which is why sharedReplyTopic lowers the log level for it.
  2. Request c-72 was never lost and never failed. It was answered late. The caller cannot distinguish "late" from "never", so it must not assume the work did not happen.
  3. The responder's query for c-72 was wasted work. Under load this waste grows: late requests are processed while new ones queue behind them.

In a real system

In a typical order platform, fan-out is the backbone and request-reply over Kafka is rare. orders.events has 12 partitions, replication factor 3, the order id as key, and these readers:

Consumer groupDoesStarts atIf it is down
payment-serviceCharges the customerearliest (never skip an order)Orders stay PENDING until it catches up
inventory-serviceReserves stockearliestReservations are delayed
notification-emailOrder confirmation emaillatest when first deployedEmails are late. Nothing else is affected.
notification-smsText message, if the customer opted inlatest when first deployedTexts are late
order-analyticsLoads the warehouseearliest, to backfillDashboards are stale

Each of the five has its own lag metric, its own retry and dead letter topics (see retry and dead letter patterns) and its own on-call owner. The Order Service's team owns the topic and the event contract, and reviews schema changes as it would review a change to a public REST API.

The stock question at checkout is answered three different ways in such a system, depending on what the caller needs. The product page asks Inventory over HTTP, because a person is waiting and a stale answer is acceptable. Checkout does not ask at all: it accepts the order and reacts to StockReserved or OutOfStock events. Request-reply over Kafka is kept for one integration where the responder sits in a network zone that only the Kafka cluster can reach.

When to use request-reply over Kafka

  • The responder can only be reached through Kafka: a network boundary, or a partner integration that is topic-based.
  • The request is a query or an idempotent command, and the caller can tolerate a slow or missing answer.
  • You want the request stream to be recorded and readable by others, and you accept the latency.
  • Volumes are modest and you have decided how each instance receives its own replies.

When HTTP or gRPC is the better choice

  • A user or an upstream request is waiting, and latency of a few milliseconds matters.
  • The caller needs a definite failure: connection refused, a 503, a deadline exceeded that also cancels the work on the server.
  • You need standard tooling for synchronous calls: load balancing, retries with budgets, circuit breakers, tracing.
  • The interaction is a read of current state with no value in being stored in a log.

When neither: use an asynchronous command and result

  • The work is slow or may be retried for minutes: a payment, a reservation, a document render.
  • The caller can record "in progress" and continue. The result arrives as an event and updates a row.
  • The conversation must survive a restart of the caller. A waiting future does not. A database row does.

When to use fan-out

  • Several services react to the same business fact, and the producer should not depend on any of them.
  • New reactions are expected over time: fraud checks, loyalty points, analytics.
  • Consumers can be idempotent and can work with eventual consistency between each other.

When not to use fan-out

  • The producer needs the outcome before it can answer its own caller. That is a call, or a saga with an explicit result.
  • Steps must happen in a strict sequence across services, with compensation on failure. Independent consumers of one event give no such sequence: use an orchestrated saga.
  • The message is meant for exactly one known recipient and is an instruction, not a fact. Send a command to that service's own topic.
  • The payload contains data that not every reader may see. Topic ACLs are per topic, not per field.

Java and Spring Boot

Both shapes on Spring Boot 3. First the stock check as request-reply: the requester's configuration with a shared reply topic and a group per instance, the client, and the responder with @SendTo. Then fan-out: two notification channels as separate consumer groups on orders.events, and the order service handling a stock result as an event. Keys and values are JSON strings.

db/schema.sqlSQL
-- Inventory Service database. The stock check is a query: no transaction, no state change.--   SELECT available FROM stock WHERE sku = ?;CREATE TABLE stock (    sku        varchar(64) PRIMARY KEY,    available  integer     NOT NULL CHECK (available >= 0));-- Notification Service database. One row per event and channel, so each channel-- sends at most once however often its consumer group sees the event.CREATE TABLE notification_delivery (    event_id   varchar(128) NOT NULL,    channel    varchar(16)  NOT NULL,                -- EMAIL, SMS    status     varchar(16)  NOT NULL,                -- PENDING, SENT    created_at timestamptz  NOT NULL DEFAULT now(),    sent_at    timestamptz,    PRIMARY KEY (event_id, channel));--   Claim:    INSERT INTO notification_delivery (event_id, channel, status)--             VALUES (?, ?, 'PENDING') ON CONFLICT DO NOTHING;--   Check:    SELECT status FROM notification_delivery WHERE event_id = ? AND channel = ?;--   Complete: UPDATE notification_delivery SET status = 'SENT', sent_at = now()--             WHERE event_id = ? AND channel = ? AND status = 'PENDING';-- Order Service database. The asynchronous alternative to request-reply: the pending-- conversation is the order row, and the result event completes it.CREATE TABLE orders (    id          varchar(36) PRIMARY KEY,    customer_id varchar(36) NOT NULL,    status      varchar(16) NOT NULL,                -- PENDING, CONFIRMED, REJECTED    updated_at  timestamptz NOT NULL DEFAULT now());--   On StockReserved:  UPDATE orders SET status = 'CONFIRMED', updated_at = now()--                      WHERE id = ? AND status = 'PENDING';--   On OutOfStock:     UPDATE orders SET status = 'REJECTED', updated_at = now()--                      WHERE id = ? AND status = 'PENDING';--   0 rows updated means the result was already applied: a duplicate, safely ignored.

Why it is written this way

  • Request-reply has no table of its own. The pending requests are held in the requester's memory, which is the weakness the lesson describes.
  • The asynchronous alternative keeps the pending state in orders.status. Any instance can apply the result, and a restart loses nothing.
  • The conditional WHERE status = 'PENDING' makes the result handler idempotent without a separate dedup table.
  • notification_delivery is keyed by event and channel because fan-out delivers the same event to each channel's group once, and redelivers within a group after a crash.

Configuration

Settings that decide how request-reply behaves under delay, and where a new fan-out consumer starts. Spring values are from the 3.3 reference, Kafka defaults from the 4.1 configuration reference.

ReplyingKafkaTemplate.defaultReplyTimeoutDefault 5 seconds
What it does
How long the template keeps a request's future before failing it with KafkaReplyTimeoutException, when no timeout is passed to sendAndReceive.
When to change it
Set it below the timeout of whatever called you. Remember that it stops the waiting only, not the processing of the request.
ReplyingKafkaTemplate.sharedReplyTopicDefault false
What it does
When true, a reply with an unknown correlation id is logged at DEBUG. When false it is logged at ERROR.
When to change it
Set it to true when instances share a reply topic with a group id each. Leave it false otherwise, so that a misrouted reply is visible.
group.id (reply container)Default none
What it does
The consumer group of the requester's reply consumer.
When to change it
Must differ per instance on a shared reply topic. With one group for all instances, replies go to whichever instance owns the partition.
auto.offset.resetDefault latest
What it does
Where a group with no committed offset starts reading: at the end (latest), at the oldest available record (earliest), or fail (none).
When to change it
Reply consumers: latest. New fan-out consumers: decide per consumer. earliest backfills state and also repeats side effects for every retained event.
AggregatingReplyingKafkaTemplate.returnPartialOnTimeoutDefault false
What it does
When true, a timeout completes the future with the replies received so far, if there is at least one, and does not fail it.
When to change it
Turn it on when a partial answer is useful, such as quotes from several carriers, and check how many replies you actually got.
delivery.timeout.ms (requester's producer)Default 120000 (2 minutes)
What it does
Upper bound on the time the producer keeps trying to deliver a record after send() returns.
When to change it
For a request-reply producer, lower it towards the reply timeout. Otherwise a request can be delivered long after its caller gave up.
retention.ms (reply topic)Default 604800000 (7 days)
What it does
How long the broker keeps records in the topic.
When to change it
A reply is worthless after the reply timeout. A short retention, such as an hour, keeps the reply topic small. Do not apply the same reasoning to request topics you want to audit.
offsets.retention.minutes (broker)Default 10080
What it does
How long committed offsets of a group with no members are kept (7 days).
When to change it
Bounds how long abandoned reply groups with random ids linger, and how long a stopped fan-out consumer keeps its position.

Hands-on exercise

Where does the reply go?

Use the consumer group simulator to see why a reply topic read by one consumer group cannot route a reply to the instance that asked, then reason about fan-out with separate groups on the same topic.

Steps · 0 of 5 done

One group on a reply topic

Simulation · not a real cluster

Three instances share one consumer group on a three-partition reply topic. Crash and add consumers and watch which instance owns each partition.

Take it further in the lab: Consumer group assignment

What can go wrong

One consumer group on the reply topic
Symptom
With one instance everything works. With three, about two thirds of requests time out, and the log shows replies that match no pending request.
Cause
All instances share a group on the reply topic, so each reply is delivered to whichever instance owns its partition, which is usually not the requester.
Fix
Give each instance its own group id on the shared reply topic and set sharedReplyTopic, or use a reply topic or a fixed reply partition per instance.
Treating a reply timeout as a failed request
Symptom
Stock is reserved twice for some orders, always after a slow period in the Inventory Service.
Cause
A command was sent as request-reply. The caller timed out and retried, and the responder later processed both copies.
Fix
Send only queries or idempotent commands this way, with an idempotency key and a deadline. Better, model the command as an asynchronous event flow with a stored result.
Request-reply on a user-facing path
Symptom
Checkout latency is fine on average and several seconds at the 99th percentile, with spikes at every deployment of either service.
Cause
Each request waits for two produce-consume hops, and a rebalance of the responder group or of a reply consumer stalls all in-flight requests.
Fix
Use HTTP or gRPC for the synchronous question, or remove the question by accepting the order and reacting to the result event.
Adding a consumer by reusing a group id
Symptom
After the new analytics service is deployed, the Payment Service processes only part of the orders, and so does analytics.
Cause
The new service was configured with group.id=payment-service. It joined that group and took a share of the partitions.
Fix
Every independent consumer of a topic gets its own group id. Name groups after the consuming service and check with kafka-consumer-groups.sh --describe --members.
A new consumer that replays history into the real world
Symptom
Customers receive text messages about orders from last week on the day the SMS channel is launched.
Cause
The new group was started with auto.offset.reset=earliest and processed every retained OrderCreated.
Fix
Start side-effecting consumers at latest, or filter by event time. Use earliest for consumers that only build state.
A breaking change to a fanned-out event
Symptom
After an Order Service release, Notification and analytics fail on every new record while Payment keeps working.
Cause
A field was renamed. The producing team checked the consumers it knew about, and others depended on the old name.
Fix
Only additive changes. Enforce compatibility with a schema registry check in the producer's build, and publish breaking changes as a new event or topic.
One dispatcher for all channels
Symptom
During an SMS provider outage, customers receive the same confirmation email several times.
Cause
One consumer sends email and then SMS for each event. The SMS failure makes the record retry, and the email is sent again on each attempt.
Fix
Use a consumer group per channel, and deduplicate each channel by event id so that a retry cannot repeat a send that succeeded.

How to troubleshoot

Requests time out although the responder is healthy

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group inventory-stock-check
How to read it
LAG on the request topic is waiting time added to every request. Lag larger than the request rate times the reply timeout means every caller times out before its request is even read.
Replies are produced but callers still time out

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --list, then --describe --members on the groups that read the reply topic
How to read it
With a shared reply topic there should be one group per running instance, each owning all partitions. A single group with the partitions split among instances is the misrouting case.
Need to see whether a request was answered at all

Check

kafka-console-consumer.sh --bootstrap-server broker:9092 --topic inventory.stock-check.replies --from-beginning --property print.headers=true --property print.timestamp=true
How to read it
Look for the kafka_correlationId header and compare the reply's timestamp with the request's. A reply that exists but is later than the timeout is a latency problem, not a lost message. The correlation id is binary and may not print readably. Recent 4.x releases deprecate --property here in favour of --formatter-property.
Not sure who consumes an event topic

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --list, then --describe --group <name> for each group
How to read it
Every group with rows for orders.events is a consumer of your contract. Groups with no active members and old offsets are consumers that are stopped, or gone but not yet expired.
One fan-out consumer is behind and the others are not

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group notification-sms
How to read it
Lag in one group only is that consumer's own problem: a slow provider, a failing record, too few instances. It does not affect other groups. It becomes data loss only if the group falls further behind than the topic's retention.

Key takeaways

  • Request-reply over Kafka needs a reply topic, a correlation id and a way for the reply to reach the instance that is waiting.
  • A reply timeout means the caller stopped waiting. The request is still in the log and will be processed, so send only queries or idempotent commands.
  • For a synchronous question, HTTP or gRPC is usually simpler, faster and clearer in failure. Use Kafka request-reply only when there is a specific reason.
  • Often the better design removes the waiting: record a pending state, publish, and complete it when the result event arrives.
  • Fan-out is what Kafka is built for: one topic, one consumer group per interested service, each with its own offsets, lag and failures.
  • Adding a consumer is a new group id and a deliberate start position. Reusing a group id splits the stream instead.
  • A fanned-out event is a public API. Change it additively and enforce compatibility.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

An Order Service uses ReplyingKafkaTemplate with reply topic stock.replies (6 partitions) and reply group order-service-replies. In testing with one instance, every request succeeds in about 40 ms. In production with 3 instances, roughly two out of three requests fail after the 5 second timeout, and the log has ERROR lines about replies with no pending request.

What is the most direct fix?

Scenario 2 of 3

A team sends ReserveStock commands with request-reply over Kafka and a 3 second reply timeout. On timeout the Order Service retries once. After an Inventory deployment that took 20 seconds, 310 orders have two reservations each. Inventory's logs show no errors.

What best explains the double reservations?

Scenario 3 of 3

The marketing team builds a loyalty service that awards points for each order. They want to consume orders.events (7 days retention, about 900,000 orders in it). A developer suggests configuring it with group.id=notification-email "since that group already has access", and auto.offset.reset=earliest to credit recent orders.

How should the loyalty consumer be set up?

References

References: Spring for Apache Kafka 3.3 reference: Sending messages (ReplyingKafkaTemplate, reply topic and partition headers, AggregatingReplyingKafkaTemplate) ↗ · Spring for Apache Kafka 3.3 reference: Forwarding listener results using @SendTo ↗ · Spring for Apache Kafka 3.3 Javadoc: ReplyingKafkaTemplate ↗ · Apache Kafka 4.1: introduction (topics have many subscribers, events are not deleted after consumption) ↗ · Apache Kafka 4.1: consumer configuration reference (group.id, auto.offset.reset) ↗ · Apache Kafka 4.1: producer configuration reference (delivery.timeout.ms) ↗ · Apache Kafka 4.1: basic operations, managing consumer groups ↗ · Enterprise Integration Patterns: Request-Reply ↗ · Enterprise Integration Patterns: Publish-Subscribe Channel ↗ · Report a problem