Path 7 · lesson 8 of 8
Request-reply and event fan-out
Two messaging shapes on Kafka: asking a question and waiting for the answer on a reply topic, and publishing one event to many independent consumer groups. One of them fits Kafka far better.
What you will learn
- Implement request-reply over Kafka with a reply topic and a correlation id, using
ReplyingKafkaTemplateand@SendTo - Explain why a reply can reach the wrong application instance and configure one of the three ways around it
- Say exactly what a reply timeout does and does not mean, and design requests that are safe under it
- Decide when a plain HTTP or gRPC call is the better tool, and when an asynchronous command with a durable result is
- Fan one event out to several consumer groups, and add a new consumer without changing the producer
- Treat the event schema as a public API and change it without breaking consumers you do not know
Before you start
The concept
This lesson covers two shapes of communication between services. They are often presented together because both are "messaging patterns", but they sit at opposite ends of how well they suit Kafka.
Request-reply is a conversation: service A sends a request and needs an answer to continue. Over Kafka, A publishes the request to a request topic with two headers: a correlation id that identifies this request, and the name of a reply topic. Service B consumes the request, does the work and publishes the answer to the reply topic with the same correlation id. A consumes the reply topic and matches the answer to the waiting caller.
Fan-out is an announcement: the Order Service publishes OrderCreated once, and every interested service reads it in its own consumer group. Payment charges, Inventory reserves, Notification sends an email, analytics counts. The producer does not know who reads the event and does not wait for anyone.
Request-reply over Kafka: trade-offs
Advantages
- The caller needs no network route to the responder, only to the brokers
- Requests are buffered: a short responder restart delays answers without failing them, if the caller's timeout allows
- Responders scale as a consumer group, with no load balancer or service discovery
- Requests for one key are handled in order by one responder instance
- The same request stream can be read by other groups, for audit or analytics
Disadvantages
- Higher and less predictable latency: two hops, batching, fetch waits, and any consumer lag on the request topic
- The reply must find the one instance that is waiting, which Kafka's group assignment does not arrange
- A timeout does not cancel anything: the request stays in the log and is processed later
- Waiting callers live in memory. If that instance restarts, the reply has nobody to go to
- A rebalance on either side stalls every in-flight request
- Two topics, a reply consumer per instance and several timeouts to operate, for what an HTTP call does directly
Fan-out: trade-offs
Advantages
- A new consumer is a new group id: no change, deployment or permission from the producing team's code
- Consumers fail, lag, retry and replay independently. A slow one does not slow the others or the producer
- The event is stored once and read many times. An extra group adds fetch traffic, not storage
- Each group reads a partition in the same order, so every consumer sees one order's events in sequence
- A new consumer can start from the beginning and build its state from history
Disadvantages
- The event schema becomes a public API with consumers the producer cannot list
- No one sees the whole flow: what happens after
OrderCreatedis spread over many services - The producer gets no result. Anything it needs back must come as another event
- Every consumer needs its own idempotency, retry and dead letter handling
- Consumers are eventually consistent with each other: Notification may email before Inventory has reserved
Why it exists
The problem request-reply tries to solve
A team has moved its services onto Kafka and then meets an interaction that is a question: at checkout the Order Service wants to know whether a SKU is available before it accepts the order. Opening an HTTP path to Inventory feels like a step back: another protocol, another set of credentials, service discovery, a circuit breaker. So the question is sent through Kafka as well. That works, and it is supported by Spring for Apache Kafka. It also brings every difficulty of this lesson, because a log is being used as a telephone.
The better first question is whether the caller must wait. In the order system it does not: the Order Service accepts the order as PENDING, publishes OrderCreated, and learns the stock result later from an inventory.events record that it handles like any other event. That is an asynchronous command and result. The state of the conversation is a row in a database, not a thread waiting on a future, and it survives restarts. The saga lesson builds exactly this.
The problem fan-out solves
Without fan-out, the Order Service calls Payment, then Inventory, then Notification. Each new consumer of "an order was placed" means a change and a deployment of the Order Service, one more dependency that can fail checkout, and one more team to coordinate with. Checkout becomes as slow as the sum of the calls and as available as their product.
With traditional queues the same goal needs one queue per consumer and a producer or exchange that copies each message into all of them. Kafka's design removes the copying: a topic is a log, and a consumer group is only a set of committed positions in that log. Any number of groups can read the same records, each at its own pace. The Kafka introduction puts it this way: a topic can have zero, one or many consumers, and events can be read as often as needed because, unlike in traditional messaging systems, they are not deleted after consumption.
How it works inside
How ReplyingKafkaTemplate works
sendAndReceive(record)generates a correlation id, adds the headerskafka_correlationIdandkafka_replyTopic(and optionallykafka_replyPartition), stores a future in an in-memory map under the correlation id, and sends the record.- It returns a
RequestReplyFuture. ItsgetSendFuture()tells you whether the send succeeded. The future itself completes when the reply arrives. - On the responder, a
@KafkaListenermethod annotated with@SendToreturns the answer. The listener infrastructure sends it to the topic named in the reply-topic header and copies the correlation id. - The template owns a listener container on the reply topic. When a reply arrives it looks up the correlation id in its map and completes the future.
- If no reply arrives within the reply timeout (the template's
defaultReplyTimeout, 5 seconds by default), the future completes exceptionally with aKafkaReplyTimeoutExceptionand the entry is removed from the map.
The reply has to reach the instance that asked
The map of waiting futures lives in one JVM. Run three instances of the Order Service with one consumer group on the reply topic, and Kafka assigns each reply partition to one of them. A reply lands on a partition, and the instance that owns that partition is usually not the one that sent the request. It finds no matching correlation id and discards the reply, and the real requester times out. The Spring reference gives three ways to arrange it:
| Approach | How | Cost |
|---|---|---|
| A reply topic per instance | Each instance listens on its own topic and names it in the reply-topic header. | Topics must be created and removed as instances come and go. Awkward with autoscaling. |
A shared reply topic, a different group.id per instance | Every instance is its own consumer group, so every instance receives every reply and ignores those whose correlation id it does not know. Set sharedReplyTopic to true so unknown replies are logged at DEBUG, not ERROR. | Reply traffic is multiplied by the number of instances. Each restart creates a new group if the id is random. |
| A shared reply topic, a fixed partition per instance | Each instance is manually assigned one partition and sends it in the kafka_replyPartition header. The responder writes the reply to that partition. | No group management on the reply container, and something must hand out partition numbers to instances. |
With the shared-topic approach, the reply consumer should start at the end of the topic (auto.offset.reset=latest), because a new group has no interest in old replies. Then a request sent before the reply container has its partitions assigned can have its reply written before the consumer's start position. waitForAssignment on the template exists for this: call it before the first request.
What a timeout means
A timeout in request-reply over Kafka means only one thing: the caller stopped waiting. It does not mean the request failed, and nothing is cancelled. The request record is in the log. If the responder was restarting or lagging, it processes the request later and publishes a reply that nobody is waiting for. Three rules follow:
- Requests should be queries or idempotent commands. If a timed-out "reserve stock" is retried by the caller, both copies are eventually processed. A query has no side effect to repeat.
- Give requests a deadline. Put the time after which the answer is useless in the request. A responder working through a backlog can then skip expired requests cheaply.
- Latency includes consumer lag. The reply time is produce, wait in the request topic behind every earlier request, process, produce, fetch. A responder that is 10 seconds behind makes every caller time out, and each retry adds to the backlog.
There are two more timeouts to keep in order. The producer's delivery.timeout.ms (default 2 minutes) bounds how long the send may keep trying. A request whose reply timeout has passed can still be delivered a minute later. And the HTTP request that triggered all this has its own timeout upstream. Keep them nested: reply timeout shorter than the upstream timeout, and the producer's delivery timeout not far above the reply timeout for this template.
Several replies to one request
AggregatingReplyingKafkaTemplate is the scatter-gather variant: one request is consumed by several responders (each in its own group) and the template collects their replies. A release strategy that you supply is consulted on each reply and decides when the set is complete. With returnPartialOnTimeout (false by default) a timeout returns what has arrived so far, as long as there is at least one reply. Its reply container must use manual acknowledgement. The same instance-routing and timeout rules apply.
Fan-out: one log, many positions
Each consumer group commits its own offsets per partition. payment-service at offset 9,412 and analytics at offset 2,007 on the same partition do not affect each other. Within a group a partition is read by one member, so the group's parallelism is capped by the partition count. Across groups there is no limit, and every group receives every record.
Drawing diagram…
Sequence diagram: the Order Service writes OrderCreated at offset 9412. The payment and inventory groups each read it and commit their own offset. The email notification group reads it but is slow and falls behind. A new analytics group joins later with no committed offset, and auto.offset.reset decides where it starts. The email group commits later without having affected the others.
sequenceDiagram participant O as Order Service participant K as orders.events participant P as Group payment-service participant I as Group inventory-service participant E as Group notification-email participant A as Group order-analytics, added later O->>K: OrderCreated o-1001 at offset 9412 K->>P: offset 9412 P->>K: commit 9413 for payment-service K->>I: offset 9412 I->>K: commit 9413 for inventory-service K->>E: offset 9412 Note over E: email provider slow, group falls behind Note over A: new group id, no committed offset A->>K: join, auto.offset.reset decides the start K->>A: records from the chosen start position E->>K: commit 9413 for notification-email
Adding a consumer without touching the producer
- Choose a new
group.id. Reusing an existing one does not add a consumer. It joins that group and takes a share of its partitions, so each service then sees only part of the events. - Decide where the group starts. With no committed offsets,
auto.offset.resetapplies: the defaultlatestmeans "events from now on", andearliestmeans "everything the topic still holds". - Grant the new service read access to the topic and to its group, if the cluster uses ACLs.
- Deploy. The producer's code, configuration and deployment are unchanged.
The event is a public API
With direct calls, a producer knows its callers and can change them together with the interface. With fan-out it cannot even list its consumers. Any field in OrderCreated may be relied on by a team the Order Service's developers have never met. That changes how the schema must be treated:
- Change additively. Add optional fields. Do not remove or rename a field, change its type, or change what it means.
- Make consumers tolerant. A consumer ignores fields it does not know and does not fail on a new enum value.
- Publish facts, not internals. Include what happened and the data consumers need to act on it. Leave out the producer's table layout and internal status codes, because whatever is published will be depended on.
- A breaking change is a new contract. Publish a new event type or a new topic alongside the old one and retire the old one when its consumers have moved.
- Enforce it. A schema registry with a compatibility rule rejects a breaking change at build or publish time. See schema evolution.
Fan-out to notification channels
| Design | How | Good | Bad |
|---|---|---|---|
| A group per channel | notification-email, notification-sms, notification-push each consume orders.events. | Channels fail and retry independently. An SMS provider outage does not delay email. A new channel is a new group. | Each channel reads and filters the whole stream, and repeats the lookup of customer preferences. |
| One group that dispatches | One consumer reads the event and calls every channel in turn. | One place for preferences and templates. | A failing channel blocks or forces a retry of the others, so channels that succeeded are sent again unless each send is deduplicated. |
| One group that fans out again | One consumer decides which channels apply and publishes one command per channel to notifications.email, notifications.sms. Channel workers consume those. | Preferences decided once, channels still independent. | More topics, and the dispatcher's publish must be made reliable (an outbox, or a Kafka transaction since it is topic to topic). |
Failure scenarios
| Crash point | Resulting state | Recovery |
|---|---|---|
| Request-reply: responder down when the request is sent | Request waits in the topic. Caller times out. | The responder processes the request after restart and replies to nobody. Safe only if the request is a query or idempotent. A deadline lets it skip the work. |
| Request-reply: requester instance crashes while waiting | The in-memory future is gone. The reply arrives and matches nothing. | The upstream caller sees an error and retries, creating a new request. Nothing connects it to the old one. |
| Request-reply: reply lands on another instance | The owner of that partition discards it. The requester times out. | No recovery at runtime. Fix the reply routing: a group or partition per instance. |
| Request-reply: responder crashes after the work, before the reply is sent | Offset not committed, no reply | The request is redelivered and processed again, and a reply is sent, possibly after the caller has timed out. |
| Request-reply: rebalance of the reply container | Replies are not consumed until assignment completes | In-flight requests time out. With a new group and latest, replies written during the gap are skipped. |
| Fan-out: one consumer group is down | Its lag grows. Other groups and the producer are unaffected. | It resumes from its committed offset, provided it returns within the topic's retention. |
| Fan-out: a consumer crashes after its effect, before the offset commit | Effect applied, offset not advanced | The record is redelivered to that group only. Its idempotency check absorbs it. |
| Fan-out: the producer ships an incompatible schema | Some consumers fail to deserialize every new record | Their error handling dead-letters the records or stalls. Fix forward with a compatible schema, then replay each affected group's dead letters. |
Architecture
Drawing diagram…
Flowchart with two parts. In request-reply, two Order Service instances send requests carrying a correlation id and reply topic to a request topic. The inventory group consumes them and writes replies with the same correlation id to a reply topic, which both instances read in their own consumer groups, each keeping only replies it is waiting for. In fan-out, the Order Service writes OrderCreated to orders.events and five separate consumer groups read it: payment, inventory, email notification, SMS notification and a later analytics group.
flowchart TB
subgraph RR["Request-reply, a conversation"]
A1["Order Service instance 1"] -->|"request with correlationId and replyTopic"| RQ["inventory.stock-check.requests"]
A2["Order Service instance 2"] -->|"request"| RQ
RQ --> INV["Group inventory-stock-check"]
INV -->|"reply with the same correlationId"| RP["inventory.stock-check.replies"]
RP -->|"own group id, matches its correlationId"| A1
RP -->|"own group id, ignores unknown ids"| A2
end
subgraph FO["Fan-out, an announcement"]
OS["Order Service"] -->|"OrderCreated, key orderId"| OE["orders.events"]
OE --> G1["Group payment-service"]
OE --> G2["Group inventory-service"]
OE --> G3["Group notification-email"]
OE --> G4["Group notification-sms"]
OE --> G5["Group order-analytics, added later"]
end- In the top half arrows form a loop. The requester depends on the responder being up and fast, exactly as with an HTTP call, with two topics in between.
- Both Order Service instances read the reply topic in their own group, so both receive every reply. That duplication is the price of making sure the right one gets it.
- In the bottom half all arrows point away from the producer. Nothing flows back, so no consumer can slow down or fail checkout.
- The analytics group was added without any change above the topic. That one-way dependency is what fan-out buys.
Message flow
Drawing diagram…
Sequence diagram: Order instance 1 sends request c-71 naming the reply topic and stores a future with a two second timeout. The inventory responder reads it, queries stock and writes the reply. Both order instances receive the reply. Instance 2 ignores it and instance 1 completes its future. Instance 1 then sends request c-72 while the responder is restarting. After two seconds the future fails with a reply timeout. The responder later processes c-72 and replies, and instance 1 drops the reply because no request is pending.
sequenceDiagram participant A1 as Order instance 1 participant A2 as Order instance 2 participant RQ as stock-check.requests participant IV as Inventory responder participant RP as stock-check.replies A1->>RQ: request c-71, replyTopic stock-check.replies Note over A1: future stored under c-71, timeout 2 s RQ->>IV: request c-71 IV->>IV: SELECT available for the sku IV->>RP: reply c-71, available true RP->>A2: reply c-71 Note over A2: unknown correlation id, ignored RP->>A1: reply c-71 Note over A1: future c-71 completes A1->>RQ: request c-72 Note over IV: responder restarting for 5 s Note over A1: 2 s pass, KafkaReplyTimeoutException RQ->>IV: request c-72, delivered after restart IV->>RP: reply c-72 RP->>A1: reply c-72 Note over A1: no pending request c-72, reply dropped
- Both instances receive reply c-71. Instance 2's discard is normal with a shared reply topic, which is why
sharedReplyTopiclowers the log level for it. - Request c-72 was never lost and never failed. It was answered late. The caller cannot distinguish "late" from "never", so it must not assume the work did not happen.
- The responder's query for c-72 was wasted work. Under load this waste grows: late requests are processed while new ones queue behind them.
In a real system
In a typical order platform, fan-out is the backbone and request-reply over Kafka is rare. orders.events has 12 partitions, replication factor 3, the order id as key, and these readers:
| Consumer group | Does | Starts at | If it is down |
|---|---|---|---|
payment-service | Charges the customer | earliest (never skip an order) | Orders stay PENDING until it catches up |
inventory-service | Reserves stock | earliest | Reservations are delayed |
notification-email | Order confirmation email | latest when first deployed | Emails are late. Nothing else is affected. |
notification-sms | Text message, if the customer opted in | latest when first deployed | Texts are late |
order-analytics | Loads the warehouse | earliest, to backfill | Dashboards are stale |
Each of the five has its own lag metric, its own retry and dead letter topics (see retry and dead letter patterns) and its own on-call owner. The Order Service's team owns the topic and the event contract, and reviews schema changes as it would review a change to a public REST API.
The stock question at checkout is answered three different ways in such a system, depending on what the caller needs. The product page asks Inventory over HTTP, because a person is waiting and a stale answer is acceptable. Checkout does not ask at all: it accepts the order and reacts to StockReserved or OutOfStock events. Request-reply over Kafka is kept for one integration where the responder sits in a network zone that only the Kafka cluster can reach.
When to use request-reply over Kafka
- The responder can only be reached through Kafka: a network boundary, or a partner integration that is topic-based.
- The request is a query or an idempotent command, and the caller can tolerate a slow or missing answer.
- You want the request stream to be recorded and readable by others, and you accept the latency.
- Volumes are modest and you have decided how each instance receives its own replies.
When HTTP or gRPC is the better choice
- A user or an upstream request is waiting, and latency of a few milliseconds matters.
- The caller needs a definite failure: connection refused, a 503, a deadline exceeded that also cancels the work on the server.
- You need standard tooling for synchronous calls: load balancing, retries with budgets, circuit breakers, tracing.
- The interaction is a read of current state with no value in being stored in a log.
When neither: use an asynchronous command and result
- The work is slow or may be retried for minutes: a payment, a reservation, a document render.
- The caller can record "in progress" and continue. The result arrives as an event and updates a row.
- The conversation must survive a restart of the caller. A waiting future does not. A database row does.
When to use fan-out
- Several services react to the same business fact, and the producer should not depend on any of them.
- New reactions are expected over time: fraud checks, loyalty points, analytics.
- Consumers can be idempotent and can work with eventual consistency between each other.
When not to use fan-out
- The producer needs the outcome before it can answer its own caller. That is a call, or a saga with an explicit result.
- Steps must happen in a strict sequence across services, with compensation on failure. Independent consumers of one event give no such sequence: use an orchestrated saga.
- The message is meant for exactly one known recipient and is an instruction, not a fact. Send a command to that service's own topic.
- The payload contains data that not every reader may see. Topic ACLs are per topic, not per field.
Java and Spring Boot
Both shapes on Spring Boot 3. First the stock check as request-reply: the requester's configuration with a shared reply topic and a group per instance, the client, and the responder with @SendTo. Then fan-out: two notification channels as separate consumer groups on orders.events, and the order service handling a stock result as an event. Keys and values are JSON strings.
-- Inventory Service database. The stock check is a query: no transaction, no state change.-- SELECT available FROM stock WHERE sku = ?;CREATE TABLE stock ( sku varchar(64) PRIMARY KEY, available integer NOT NULL CHECK (available >= 0));-- Notification Service database. One row per event and channel, so each channel-- sends at most once however often its consumer group sees the event.CREATE TABLE notification_delivery ( event_id varchar(128) NOT NULL, channel varchar(16) NOT NULL, -- EMAIL, SMS status varchar(16) NOT NULL, -- PENDING, SENT created_at timestamptz NOT NULL DEFAULT now(), sent_at timestamptz, PRIMARY KEY (event_id, channel));-- Claim: INSERT INTO notification_delivery (event_id, channel, status)-- VALUES (?, ?, 'PENDING') ON CONFLICT DO NOTHING;-- Check: SELECT status FROM notification_delivery WHERE event_id = ? AND channel = ?;-- Complete: UPDATE notification_delivery SET status = 'SENT', sent_at = now()-- WHERE event_id = ? AND channel = ? AND status = 'PENDING';-- Order Service database. The asynchronous alternative to request-reply: the pending-- conversation is the order row, and the result event completes it.CREATE TABLE orders ( id varchar(36) PRIMARY KEY, customer_id varchar(36) NOT NULL, status varchar(16) NOT NULL, -- PENDING, CONFIRMED, REJECTED updated_at timestamptz NOT NULL DEFAULT now());-- On StockReserved: UPDATE orders SET status = 'CONFIRMED', updated_at = now()-- WHERE id = ? AND status = 'PENDING';-- On OutOfStock: UPDATE orders SET status = 'REJECTED', updated_at = now()-- WHERE id = ? AND status = 'PENDING';-- 0 rows updated means the result was already applied: a duplicate, safely ignored.Why it is written this way
- Request-reply has no table of its own. The pending requests are held in the requester's memory, which is the weakness the lesson describes.
- The asynchronous alternative keeps the pending state in
orders.status. Any instance can apply the result, and a restart loses nothing. - The conditional
WHERE status = 'PENDING'makes the result handler idempotent without a separate dedup table. notification_deliveryis keyed by event and channel because fan-out delivers the same event to each channel's group once, and redelivers within a group after a crash.
package com.example.orders.stock;import java.time.Duration;import java.util.Properties;import java.util.UUID;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;import org.springframework.kafka.core.ProducerFactory;import org.springframework.kafka.listener.ConcurrentMessageListenerContainer;import org.springframework.kafka.requestreply.ReplyingKafkaTemplate;@Configurationpublic class StockCheckConfig { public static final String REQUEST_TOPIC = "inventory.stock-check.requests"; public static final String REPLY_TOPIC = "inventory.stock-check.replies"; /** Every instance reads the shared reply topic in its own consumer group. */ @Bean public ConcurrentMessageListenerContainer<String, String> stockReplyContainer( ConcurrentKafkaListenerContainerFactory<String, String> factory) { ConcurrentMessageListenerContainer<String, String> container = factory.createContainer(REPLY_TOPIC); container.getContainerProperties().setGroupId("order-service-stock-replies-" + UUID.randomUUID()); Properties consumerProps = new Properties(); // A new group has no use for replies to requests it never sent. consumerProps.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest"); container.getContainerProperties().setKafkaConsumerProperties(consumerProps); container.setAutoStartup(false); // the template starts and stops it return container; } @Bean public ReplyingKafkaTemplate<String, String, String> stockCheckTemplate( ProducerFactory<String, String> producerFactory, ConcurrentMessageListenerContainer<String, String> stockReplyContainer) { ReplyingKafkaTemplate<String, String, String> template = new ReplyingKafkaTemplate<>(producerFactory, stockReplyContainer); template.setSharedReplyTopic(true); // replies for other instances are expected template.setDefaultReplyTimeout(Duration.ofSeconds(2)); return template; }}Why it is written this way
- The reply container listens to a single topic, so the template detects it and sets the reply-topic header on every request by itself.
- A unique group id per instance means each instance receives all replies. This is the option the reference recommends for autoscaling, at the cost of extra traffic.
setSharedReplyTopic(true)only changes logging: a reply with an unknown correlation id is logged at DEBUG, not ERROR.- Random group ids leave an empty group behind at every restart until its offsets expire (
offsets.retention.minutes, 7 days by default). A stable per-instance id, such as a pod ordinal, avoids the clutter. ReplyingKafkaTemplateis itself aKafkaTemplate. Once you declare it, check which template the rest of the application injects, since Spring Boot's auto-configured one may no longer be created.
package com.example.orders.stock;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.time.Duration;import java.util.concurrent.ExecutionException;import java.util.concurrent.TimeUnit;import java.util.concurrent.TimeoutException;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.producer.ProducerRecord;import org.springframework.kafka.requestreply.ReplyingKafkaTemplate;import org.springframework.kafka.requestreply.RequestReplyFuture;import org.springframework.stereotype.Component;@Componentpublic class StockCheckClient { public enum Availability { AVAILABLE, UNAVAILABLE, UNKNOWN } public record StockCheckRequest(String sku, int quantity, long deadlineEpochMs) {} public record StockCheckReply(String sku, boolean available, boolean expired) {} private static final Duration REPLY_TIMEOUT = Duration.ofSeconds(2); private final ReplyingKafkaTemplate<String, String, String> template; private final ObjectMapper json; public StockCheckClient(ReplyingKafkaTemplate<String, String, String> template, ObjectMapper json) { this.template = template; this.json = json; } public Availability check(String sku, int quantity) { try { long deadline = System.currentTimeMillis() + REPLY_TIMEOUT.toMillis(); String payload = json.writeValueAsString(new StockCheckRequest(sku, quantity, deadline)); var request = new ProducerRecord<>(StockCheckConfig.REQUEST_TOPIC, sku, payload); RequestReplyFuture<String, String, String> future = template.sendAndReceive(request, REPLY_TIMEOUT); future.getSendFuture().get(1, TimeUnit.SECONDS); // did the request reach the broker? ConsumerRecord<String, String> reply = future.get(REPLY_TIMEOUT.toMillis() + 500, TimeUnit.MILLISECONDS); StockCheckReply answer = json.readValue(reply.value(), StockCheckReply.class); if (answer.expired()) { return Availability.UNKNOWN; } return answer.available() ? Availability.AVAILABLE : Availability.UNAVAILABLE; } catch (ExecutionException | TimeoutException e) { // Includes KafkaReplyTimeoutException as the cause. The request may still be processed later. return Availability.UNKNOWN; } catch (InterruptedException e) { Thread.currentThread().interrupt(); return Availability.UNKNOWN; } catch (JsonProcessingException e) { throw new IllegalStateException("Stock check payload could not be mapped", e); } }}Why it is written this way
- There are three outcomes, not two.
UNKNOWNis a normal result of this pattern and the caller must have a rule for it, such as accepting the order and letting the reservation step decide. getSendFuture()and the reply future fail for different reasons. The first says the request never reached Kafka. The second says no answer came back in time.- The deadline travels in the request, so the responder can refuse work whose answer nobody is waiting for any more.
- The request is a query. Retrying it after a timeout is harmless. The same client around a command such as "reserve stock" would need an idempotency key.
- Call
template.waitForAssignment(Duration.ofSeconds(10))once at startup before serving traffic, so the first replies are not written before the reply consumer's start position.
package com.example.inventory;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.util.List;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.messaging.handler.annotation.SendTo;import org.springframework.stereotype.Component;@Componentpublic class StockCheckResponder { public record StockCheckRequest(String sku, int quantity, long deadlineEpochMs) {} public record StockCheckReply(String sku, boolean available, boolean expired) {} private final JdbcTemplate jdbc; private final ObjectMapper json; public StockCheckResponder(JdbcTemplate jdbc, ObjectMapper json) { this.jdbc = jdbc; this.json = json; } @KafkaListener(id = "stock-check-responder", topics = "inventory.stock-check.requests", groupId = "inventory-stock-check") @SendTo // no value: reply to the topic named in the request's reply-topic header public String onStockCheck(String payload) throws JsonProcessingException { StockCheckRequest request = json.readValue(payload, StockCheckRequest.class); if (System.currentTimeMillis() > request.deadlineEpochMs()) { // The caller has stopped waiting. Answer cheaply, without touching the database. return json.writeValueAsString(new StockCheckReply(request.sku(), false, true)); } List<Integer> available = jdbc.queryForList( "SELECT available FROM stock WHERE sku = ?", Integer.class, request.sku()); boolean enough = !available.isEmpty() && available.get(0) >= request.quantity(); return json.writeValueAsString(new StockCheckReply(request.sku(), enough, false)); }}Why it is written this way
@SendTowith no value uses the reply-topic header of the incoming request. The listener infrastructure copies the correlation id into the reply, so this method never touches it.- The reply is sent with the listener container factory's reply template. Spring Boot wires its auto-configured
KafkaTemplateinto the factory for this. - The handler only reads. It does not reserve anything, because a request that timed out at the caller is still processed here, and a retried one is processed twice.
- Responders are an ordinary consumer group. Adding instances up to the partition count of the request topic adds capacity.
- The answer is true at the moment of the query only. Stock can be gone a millisecond later, which is one more reason checkout should rely on the reservation event and not on this check.
package com.example.notifications;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.nio.charset.StandardCharsets;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;/** Two channels, two consumer groups, one topic. Each group receives every order event. */@Componentpublic class OrderCreatedFanOut { public record OrderCreated(String orderId, String customerId, long totalCents) {} /** One implementation per channel. The key lets the provider recognise a repeated send. */ public interface ChannelSender { void send(String customerId, String orderId, String idempotencyKey); } private final JdbcTemplate jdbc; private final ObjectMapper json; private final ChannelSender emailSender; private final ChannelSender smsSender; public OrderCreatedFanOut(JdbcTemplate jdbc, ObjectMapper json, ChannelSender emailSender, ChannelSender smsSender) { this.jdbc = jdbc; this.json = json; this.emailSender = emailSender; this.smsSender = smsSender; } @KafkaListener(id = "notify-email", topics = "orders.events", groupId = "notification-email") public void email(ConsumerRecord<String, String> record) throws JsonProcessingException { deliver(record, "EMAIL", emailSender); } @KafkaListener(id = "notify-sms", topics = "orders.events", groupId = "notification-sms") public void sms(ConsumerRecord<String, String> record) throws JsonProcessingException { deliver(record, "SMS", smsSender); } private void deliver(ConsumerRecord<String, String> record, String channel, ChannelSender sender) throws JsonProcessingException { if (!"OrderCreated".equals(header(record, "eventType"))) { return; // other event types on this topic are not ours } String eventId = header(record, "id"); OrderCreated event = json.readValue(record.value(), OrderCreated.class); jdbc.update(""" INSERT INTO notification_delivery (event_id, channel, status) VALUES (?, ?, 'PENDING') ON CONFLICT DO NOTHING""", eventId, channel); String status = jdbc.queryForObject(""" SELECT status FROM notification_delivery WHERE event_id = ? AND channel = ?""", String.class, eventId, channel); if ("SENT".equals(status)) { return; // already delivered on this channel } sender.send(event.customerId(), event.orderId(), eventId + ":" + channel); jdbc.update(""" UPDATE notification_delivery SET status = 'SENT', sent_at = now() WHERE event_id = ? AND channel = ? AND status = 'PENDING'""", eventId, channel); } private static String header(ConsumerRecord<String, String> record, String name) { Header h = record.headers().lastHeader(name); if (h == null) { throw new IllegalArgumentException("Missing header " + name); } return new String(h.value(), StandardCharsets.UTF_8); }}Why it is written this way
- The two listeners differ only in
groupId, and that is the fan-out. Each group has its own offsets, so an SMS provider outage stopsnotification-smswhilenotification-emailcontinues. - The Order Service is not mentioned anywhere and was not changed to add the SMS channel.
- The listener reads only the fields it needs and ignores the rest of the payload, so the producer can add fields without breaking it.
- Claim, send with a stable key, complete: the sequence from the idempotent consumer pattern for an effect outside the database.
- Both groups start at
latestby default when first deployed. For a notification channel that is the right choice:earliestwould message customers about old orders. emailSenderandsmsSenderare two beans of the same type, matched by constructor parameter name.
package com.example.orders;import java.nio.charset.StandardCharsets;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;/** The asynchronous alternative to request-reply: the answer is an event, the pending state is a row. */@Componentpublic class StockResultListener { private final JdbcTemplate jdbc; public StockResultListener(JdbcTemplate jdbc) { this.jdbc = jdbc; } @KafkaListener(id = "order-stock-results", topics = "inventory.events", groupId = "order-service") public void onInventoryEvent(ConsumerRecord<String, String> record) { Header type = record.headers().lastHeader("eventType"); if (type == null) { return; } String newStatus = switch (new String(type.value(), StandardCharsets.UTF_8)) { case "StockReserved" -> "CONFIRMED"; case "OutOfStock" -> "REJECTED"; default -> null; }; if (newStatus == null) { return; } // The record key is the order id. That is the correlation, and it is stored in the database. jdbc.update(""" UPDATE orders SET status = ?, updated_at = now() WHERE id = ? AND status = 'PENDING'""", newStatus, record.key()); }}Why it is written this way
- Any instance of the Order Service can handle the result, because the correlation is the order id in the database and not a future in one JVM.
- There is no timeout that silently abandons the conversation. A result that arrives minutes later still completes the order.
- The conditional update applies the result once. A redelivered or duplicate event updates zero rows.
- The cost is in the API: the client receives
PENDINGand learns the outcome by polling the order or from a notification.
Configuration
Settings that decide how request-reply behaves under delay, and where a new fan-out consumer starts. Spring values are from the 3.3 reference, Kafka defaults from the 4.1 configuration reference.
ReplyingKafkaTemplate.defaultReplyTimeoutDefault 5 seconds
- What it does
- How long the template keeps a request's future before failing it with
KafkaReplyTimeoutException, when no timeout is passed tosendAndReceive. - When to change it
- Set it below the timeout of whatever called you. Remember that it stops the waiting only, not the processing of the request.
ReplyingKafkaTemplate.sharedReplyTopicDefault false
- What it does
- When true, a reply with an unknown correlation id is logged at DEBUG. When false it is logged at ERROR.
- When to change it
- Set it to true when instances share a reply topic with a group id each. Leave it false otherwise, so that a misrouted reply is visible.
group.id (reply container)Default none
- What it does
- The consumer group of the requester's reply consumer.
- When to change it
- Must differ per instance on a shared reply topic. With one group for all instances, replies go to whichever instance owns the partition.
auto.offset.resetDefault latest
- What it does
- Where a group with no committed offset starts reading: at the end (
latest), at the oldest available record (earliest), or fail (none). - When to change it
- Reply consumers:
latest. New fan-out consumers: decide per consumer.earliestbackfills state and also repeats side effects for every retained event.
AggregatingReplyingKafkaTemplate.returnPartialOnTimeoutDefault false
- What it does
- When true, a timeout completes the future with the replies received so far, if there is at least one, and does not fail it.
- When to change it
- Turn it on when a partial answer is useful, such as quotes from several carriers, and check how many replies you actually got.
delivery.timeout.ms (requester's producer)Default 120000 (2 minutes)
- What it does
- Upper bound on the time the producer keeps trying to deliver a record after
send()returns. - When to change it
- For a request-reply producer, lower it towards the reply timeout. Otherwise a request can be delivered long after its caller gave up.
retention.ms (reply topic)Default 604800000 (7 days)
- What it does
- How long the broker keeps records in the topic.
- When to change it
- A reply is worthless after the reply timeout. A short retention, such as an hour, keeps the reply topic small. Do not apply the same reasoning to request topics you want to audit.
offsets.retention.minutes (broker)Default 10080
- What it does
- How long committed offsets of a group with no members are kept (7 days).
- When to change it
- Bounds how long abandoned reply groups with random ids linger, and how long a stopped fan-out consumer keeps its position.
Hands-on exercise
Where does the reply go?
Use the consumer group simulator to see why a reply topic read by one consumer group cannot route a reply to the instance that asked, then reason about fan-out with separate groups on the same topic.
Steps · 0 of 5 done
One group on a reply topic
Simulation · not a real clusterThree instances share one consumer group on a three-partition reply topic. Crash and add consumers and watch which instance owns each partition.
What you should see
Each partition belongs to exactly one instance, so a reply reaches the requester only when it happens to land on a partition that the requester owns. After a crash, partitions move and waiting requests time out. An instance with no partition never receives any reply. A second consumer group is unaffected by all of this: it has its own assignment and offsets and receives every record, which is why fan-out uses a group per consumer and a shared reply topic needs a group per instance.
What can go wrong
One consumer group on the reply topic
- Symptom
- With one instance everything works. With three, about two thirds of requests time out, and the log shows replies that match no pending request.
- Cause
- All instances share a group on the reply topic, so each reply is delivered to whichever instance owns its partition, which is usually not the requester.
- Fix
- Give each instance its own group id on the shared reply topic and set
sharedReplyTopic, or use a reply topic or a fixed reply partition per instance.
Treating a reply timeout as a failed request
- Symptom
- Stock is reserved twice for some orders, always after a slow period in the Inventory Service.
- Cause
- A command was sent as request-reply. The caller timed out and retried, and the responder later processed both copies.
- Fix
- Send only queries or idempotent commands this way, with an idempotency key and a deadline. Better, model the command as an asynchronous event flow with a stored result.
Request-reply on a user-facing path
- Symptom
- Checkout latency is fine on average and several seconds at the 99th percentile, with spikes at every deployment of either service.
- Cause
- Each request waits for two produce-consume hops, and a rebalance of the responder group or of a reply consumer stalls all in-flight requests.
- Fix
- Use HTTP or gRPC for the synchronous question, or remove the question by accepting the order and reacting to the result event.
Adding a consumer by reusing a group id
- Symptom
- After the new analytics service is deployed, the Payment Service processes only part of the orders, and so does analytics.
- Cause
- The new service was configured with
group.id=payment-service. It joined that group and took a share of the partitions. - Fix
- Every independent consumer of a topic gets its own group id. Name groups after the consuming service and check with
kafka-consumer-groups.sh --describe --members.
A new consumer that replays history into the real world
- Symptom
- Customers receive text messages about orders from last week on the day the SMS channel is launched.
- Cause
- The new group was started with
auto.offset.reset=earliestand processed every retainedOrderCreated. - Fix
- Start side-effecting consumers at
latest, or filter by event time. Useearliestfor consumers that only build state.
A breaking change to a fanned-out event
- Symptom
- After an Order Service release, Notification and analytics fail on every new record while Payment keeps working.
- Cause
- A field was renamed. The producing team checked the consumers it knew about, and others depended on the old name.
- Fix
- Only additive changes. Enforce compatibility with a schema registry check in the producer's build, and publish breaking changes as a new event or topic.
One dispatcher for all channels
- Symptom
- During an SMS provider outage, customers receive the same confirmation email several times.
- Cause
- One consumer sends email and then SMS for each event. The SMS failure makes the record retry, and the email is sent again on each attempt.
- Fix
- Use a consumer group per channel, and deduplicate each channel by event id so that a retry cannot repeat a send that succeeded.
How to troubleshoot
Requests time out although the responder is healthy
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group inventory-stock-check
- How to read it
- LAG on the request topic is waiting time added to every request. Lag larger than the request rate times the reply timeout means every caller times out before its request is even read.
Replies are produced but callers still time out
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --list, then --describe --members on the groups that read the reply topic
- How to read it
- With a shared reply topic there should be one group per running instance, each owning all partitions. A single group with the partitions split among instances is the misrouting case.
Need to see whether a request was answered at all
Check
kafka-console-consumer.sh --bootstrap-server broker:9092 --topic inventory.stock-check.replies --from-beginning --property print.headers=true --property print.timestamp=true
- How to read it
- Look for the
kafka_correlationIdheader and compare the reply's timestamp with the request's. A reply that exists but is later than the timeout is a latency problem, not a lost message. The correlation id is binary and may not print readably. Recent 4.x releases deprecate--propertyhere in favour of--formatter-property.
Not sure who consumes an event topic
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --list, then --describe --group <name> for each group
- How to read it
- Every group with rows for
orders.eventsis a consumer of your contract. Groups with no active members and old offsets are consumers that are stopped, or gone but not yet expired.
One fan-out consumer is behind and the others are not
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group notification-sms
- How to read it
- Lag in one group only is that consumer's own problem: a slow provider, a failing record, too few instances. It does not affect other groups. It becomes data loss only if the group falls further behind than the topic's retention.
Key takeaways
- Request-reply over Kafka needs a reply topic, a correlation id and a way for the reply to reach the instance that is waiting.
- A reply timeout means the caller stopped waiting. The request is still in the log and will be processed, so send only queries or idempotent commands.
- For a synchronous question, HTTP or gRPC is usually simpler, faster and clearer in failure. Use Kafka request-reply only when there is a specific reason.
- Often the better design removes the waiting: record a pending state, publish, and complete it when the result event arrives.
- Fan-out is what Kafka is built for: one topic, one consumer group per interested service, each with its own offsets, lag and failures.
- Adding a consumer is a new group id and a deliberate start position. Reusing a group id splits the stream instead.
- A fanned-out event is a public API. Change it additively and enforce compatibility.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
An Order Service uses ReplyingKafkaTemplate with reply topic stock.replies (6 partitions) and reply group order-service-replies. In testing with one instance, every request succeeds in about 40 ms. In production with 3 instances, roughly two out of three requests fail after the 5 second timeout, and the log has ERROR lines about replies with no pending request.
What is the most direct fix?
Scenario 2 of 3
A team sends ReserveStock commands with request-reply over Kafka and a 3 second reply timeout. On timeout the Order Service retries once. After an Inventory deployment that took 20 seconds, 310 orders have two reservations each. Inventory's logs show no errors.
What best explains the double reservations?
Scenario 3 of 3
The marketing team builds a loyalty service that awards points for each order. They want to consume orders.events (7 days retention, about 900,000 orders in it). A developer suggests configuring it with group.id=notification-email "since that group already has access", and auto.offset.reset=earliest to credit recent orders.
How should the loyalty consumer be set up?
References
References: Spring for Apache Kafka 3.3 reference: Sending messages (ReplyingKafkaTemplate, reply topic and partition headers, AggregatingReplyingKafkaTemplate) ↗ · Spring for Apache Kafka 3.3 reference: Forwarding listener results using @SendTo ↗ · Spring for Apache Kafka 3.3 Javadoc: ReplyingKafkaTemplate ↗ · Apache Kafka 4.1: introduction (topics have many subscribers, events are not deleted after consumption) ↗ · Apache Kafka 4.1: consumer configuration reference (group.id, auto.offset.reset) ↗ · Apache Kafka 4.1: producer configuration reference (delivery.timeout.ms) ↗ · Apache Kafka 4.1: basic operations, managing consumer groups ↗ · Enterprise Integration Patterns: Request-Reply ↗ · Enterprise Integration Patterns: Publish-Subscribe Channel ↗ · Report a problem