Path 6 · lesson 3 of 5
Retries, back-off and retry topics
Blocking retries keep order and stall the partition; retry topics keep the partition moving and give up order. How each works, how they meet max.poll.interval.ms, and how to retry safely.
What you will learn
- Explain what a blocking retry does to the partition, to the consumer thread and to the
max.poll.interval.msclock - Design an exponential back-off with a cap and jitter, and work out its longest single delay and its total duration
- Describe how
@RetryableTopicmoves a failed record through a chain of retry topics and what that does to ordering - Choose between blocking retries, retry topics and pausing the listener for a given topic and failure
- Make the retried work idempotent so that a second attempt after a half-finished first one does no harm
Before you start
The concept
A retry is the right response to a transient failure: the record is fine, and the thing it needs (a database, a payment provider, a lock) is not available right now. The question is where the record waits while it is not being processed. There are two places.
Blocking retry: wait in the partition
- The consumer keeps the failed record, waits, and delivers it to the listener again
- Nothing behind it in the partition is processed meanwhile
- Order within the partition is fully preserved
- No extra topics. The retry state lives in the consumer's memory
- The waiting happens on the consumer thread, under the
max.poll.interval.msclock
Non-blocking retry: wait in another topic
- The consumer publishes the failed record to a retry topic and commits past it
- The partition continues with the next record at once
- Order is lost: later records for the same key can be processed first
- One extra topic and consumer per delay step, plus a dead letter topic
- The waiting happens in the retry topic's consumer, which pauses until the record is due
Whichever place you choose, the delay between attempts should grow. A dependency that is down for thirty seconds is not helped by ten attempts in the first second, and a dependency that is struggling is harmed by them. Exponential back-off multiplies the delay after each failure, a cap stops it growing beyond a useful maximum, and jitter spreads the attempts of many consumers so they do not all return at the same instant.
Why it exists
The simplest consumer has one behaviour for a failure: the exception propagates, the offset is not committed, and the record is fetched again immediately. For a brief failure this works. For a dependency that is down for a minute it is a hot loop that fails thousands of times, fills the log, and adds load to the dependency just as it tries to come back.
Back-off was added to fix the hot loop: wait between attempts, and wait longer each time. That solves the load problem and exposes the next one. While a record waits in place, its whole partition waits with it. If one order in ten thousand needs four retries of several seconds each, every order behind it on that partition is late by that amount, and the lag alarm fires for a problem that affects one record.
Retry topics were invented to remove that stall: put the failed record somewhere else and let the partition continue. The price is the one property the partition was giving you for free, which is order. So neither approach replaces the other. Blocking retries are what you use when the order of records matters more than the delay; retry topics are what you use when records are independent and the delay matters more.
There is a third situation that neither handles well: the dependency is down for every record. Blocking retries stall every partition, which is at least honest. Retry topics move the entire stream through the retry chain into the dead letter topic, record by record, and the main topic looks healthy while nothing succeeds. For that case the right response is to stop consuming until the dependency returns.
How it works inside
Blocking retries in the listener container
In Spring for Apache Kafka, blocking retries are done by DefaultErrorHandler. When the listener throws a retryable exception, the handler asks its BackOff for the next delay, suspends the consumer thread for that long, and then seeks the partition back to the failed offset so that the next poll() returns the record again. When the BackOff says stop, the recoverer is called (see Poison messages and error handling). The default is 9 retries with no delay.
Two details matter. First, the thread that sleeps is the thread that polls, so every partition assigned to that consumer waits, not only the one with the failing record. Second, the retry count is kept in the memory of that consumer. If the partition moves to another instance, the count starts again at one.
The interaction with max.poll.interval.ms
A consumer that does not call poll() within max.poll.interval.ms (default 5 minutes) is considered stuck. The client leaves the group, and its partitions are assigned to another member. A blocking back-off is time spent not polling.
The container polls again between attempts, so the clock restarts after each delay. What must fit inside the interval is therefore one delay plus the processing time around it, not the sum of all delays. A back-off of 1, 2, 4, 8 and 10 seconds is no risk with the default interval. A single 6-minute wait is.
- The listener fails and the handler sleeps for 6 minutes on the consumer thread.
- After 5 minutes without
poll(), the client leaves the group. The partition goes to instance B. - Instance B reads the same record from the last committed offset, as attempt 1 of its own count. It fails and sleeps for 6 minutes.
- Instance A wakes up, finds it no longer owns the partition, and rejoins. Another rebalance.
- Instance B is removed in turn. The record moves round the group, the attempt count never reaches its limit, and no partition of the group makes steady progress.
If you need blocking delays longer than the poll interval, Spring provides ContainerPausingBackOffHandler. Instead of sleeping, it pauses the listener container for the delay: the consumer keeps calling poll(), receives nothing, stays in the group, and resumes afterwards. The partition is still blocked, which is the point, but the group is stable.
Exponential back-off, cap and jitter
The delay before retry number n is min(cap, initial × multiplier^(n-1)). With an initial delay of 1 second, a multiplier of 2 and a cap of 10 seconds, six retries wait 1, 2, 4, 8, 10 and 10 seconds: 35 seconds in total before the recoverer is called. Spring's ExponentialBackOffWithMaxRetries implements exactly this.
Without the cap, the delay doubles past any useful value: the twelfth retry would wait over half an hour. Without jitter, every consumer that failed at the same moment, because the database went down at that moment, retries at the same moments too, and the recovering database receives its load in synchronised waves. Jitter replaces each delay with a random value in a range, for example between half the computed delay and the full delay. The animated lesson Back-off and jitter shows the waves and how jitter flattens them.
ExponentialBackOff in Spring Framework 6.2, the version used by Spring Boot 3.5, has no jitter setting. BackOff is a small interface, so a jittered version is a few lines, shown in the implementation section.
Non-blocking retries with @RetryableTopic
Adding @RetryableTopic to a @KafkaListener method makes the framework create a chain of topics and a listener container for each. With 4 attempts and an exponential back-off of 1 second and multiplier 2 on payments.requested, the chain is payments.requested-retry-1000, payments.requested-retry-2000, payments.requested-retry-4000 and payments.requested-dlt. The first attempt on the main topic counts, so 4 attempts means 3 retries.
- The listener fails on the main topic. The framework publishes the record to the first retry topic, with a header that holds the time at which it is due, and the main container commits past the record.
- The retry topic's consumer reads the record and compares the due time with the clock. If the record is not due, it pauses that partition and resumes it when the time comes. It keeps polling meanwhile, so
max.poll.interval.msis not at risk. - When the record is due, the same listener method is called again. If it succeeds, the record is done.
- If it fails, the record is published to the next retry topic with a new due time, and so on.
- When the attempts are used up, or the exception is one of the fatal types such as
DeserializationException, the record goes to the dead letter topic. A method annotated with@DltHandlerin the same class receives it; without one, a default handler only logs.
| Aspect | Default | Consequence |
|---|---|---|
| Attempts | 3, including the first | Two retries, then the dead letter topic |
| Back-off | Fixed, 1000 ms | With a multiplier it becomes exponential; the maximum delay then defaults to 30 seconds |
| Topic names | Suffix -retry plus the delay in ms, and -dlt | With exponential back-off, one topic per distinct delay; attempts at the maximum delay share the last topic (since 3.2) |
| Topic creation | Automatic, 1 partition, broker default replication | Set numPartitions to match the main topic: records are sent to the same partition number they came from |
| Consumer groups | The listener's group id plus the topic suffix | Each retry topic has its own group, so its rebalances do not disturb the main listener |
| Exceptions | Retry on all, except the global fatal list | Use exclude to send your permanent failures straight to the dead letter topic |
Some limits are stated in the reference and are worth knowing before you design around this feature: non-blocking retries are not supported with batch listeners, they cannot be combined with container transactions, and the delay is a lower bound. A record is never processed before its due time, but it can be processed later if the retry consumer is busy with an earlier record.
The delay of each retry topic is fixed when the application starts and belongs to the topic. Every record in -retry-2000 waits about two seconds. So the chain gives you exponential growth and a cap, but not per-record jitter.
What retry topics do to ordering
Kafka guarantees order within a partition, for the records as the broker appended them. A retry topic takes a record out of its partition and appends a copy somewhere else, later. Suppose OrderUpdated version 1 for order 42 fails and goes to the retry topic. Version 2 for order 42 arrives on the main topic a moment later and succeeds. Two seconds afterwards version 1 is retried, succeeds, and overwrites version 2. The reference says it in one sentence: by using this strategy you lose Kafka's ordering guarantees for that topic.
This is harmless when records are independent, such as one payment request per payment id or one email per notification. It is a correctness bug when later records for a key build on earlier ones. If you still want retry topics there, the handler must detect stale records itself, for example by comparing a version number in the event with the version already stored, and ignore the older one.
Choosing
| Situation | Use | Why |
|---|---|---|
| Records for a key must be applied in order | Blocking retries, bounded, then dead-letter or stop | The stall is the mechanism that keeps the order |
| Records are independent and a few fail now and then | Retry topics | One slow record should not delay thousands of healthy ones |
| Failures usually clear within a second or two | A few short blocking retries | Cheaper than a round trip through another topic |
| The dependency is down for every record | Long blocking back-off, or pause or stop the container | Retry topics would carry the whole stream to the dead letter topic |
| Permanent failure | No retries: dead-letter at once | Waiting cannot change the outcome |
The two can be combined. A configuration class that extends RetryTopicConfigurationSupport can name exceptions that get a few blocking retries first and move to the retry topics only if those fail. That suits failures that often clear quickly and would also hit the next records, such as a database connection error.
Retry topics & DLQ
Simulation, not a real clusterThe animated lesson shows a failing record stalling its partition with in-place retries, then leaving through a retry topic while the records behind it continue.
Step 1 · The pattern
One at a time, in order
A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.
- message
- processed
Architecture
Drawing diagram…
A flowchart in which a record from payments.requested is tried by the listener, and on failure moves through three retry topics with delays of 1, 2 and 4 seconds, each followed by another attempt, ending in the dead letter topic and its handler when all four attempts fail.
flowchart LR MAIN["payments.requested"] --> L0["Listener, attempt 1"] L0 -- "Success" --> DONE["Payment charged"] L0 -- "Retryable failure" --> R1["payments.requested-retry-1000"] L0 -- "Fatal or excluded exception" --> DLT["payments.requested-dlt"] R1 --> L1["Listener, attempt 2 after 1 s"] L1 -- "Success" --> DONE L1 -- "Failure" --> R2["payments.requested-retry-2000"] R2 --> L2["Listener, attempt 3 after 2 s"] L2 -- "Success" --> DONE L2 -- "Failure" --> R3["payments.requested-retry-4000"] R3 --> L3["Listener, attempt 4 after 4 s"] L3 -- "Success" --> DONE L3 -- "Failure" --> DLT DLT --> H["DltHandler method"]
- Every box labelled Listener is the same method in your code, run by a different listener container: one for the main topic and one for each retry topic. The record you receive on attempt 3 is a copy that was published twice on its way there.
- A record leaves the main topic after its first failure and never returns to it. From that moment the main partition's offset has moved on and the order relative to its neighbours is gone.
- There are two ways into the dead letter topic: exhausting the attempts, and throwing an exception that is classified as fatal or excluded, which skips the chain entirely.
- Each retry topic holds records with one delay. That is why the delay can be enforced by pausing a partition: the record at the front is always the one that is due first.
Message flow
Drawing diagram…
A sequence diagram in which the main listener fails to charge payment p-17, publishes it to the first retry topic and commits, then continues with p-18, while the retry listeners wait for the due time, fail once more, forward the record to the second retry topic and finally succeed.
sequenceDiagram participant M as payments.requested participant L as Main listener participant P as Payment provider participant R1 as retry-1000 topic participant L1 as Retry listener 1000 participant R2 as retry-2000 topic participant L2 as Retry listener 2000 M->>L: Record offset 88, payment p-17 L->>P: Charge p-17 P--xL: Timeout L->>R1: Publish p-17 with due time now plus 1 s L->>M: Commit offset 89 M->>L: Record offset 89, payment p-18 Note over L: Main partition continues R1->>L1: Record p-17, not due yet Note over L1: Pause partition until due time L1->>P: Charge p-17 P--xL1: Timeout L1->>R2: Publish p-17 with due time now plus 2 s R2->>L2: Record p-17 Note over L2: Pause partition until due time L2->>P: Charge p-17 P-->>L2: Charged L2->>R2: Commit offset
- The main listener's failure costs the partition one publish and one commit. Payment p-18 is processed while p-17 is still waiting, which is both the benefit and the loss of order.
- The retry listener does not sleep. It pauses the partition and keeps polling, so it stays in its group however long the delay is.
- The payment provider is called three times for p-17. The first two calls timed out, which does not prove they did nothing. The charge is only safe because the provider is given the payment id as an idempotency key.
- If the publish to the retry topic succeeds and the commit of offset 89 then fails, the main listener receives offset 88 again and publishes a second copy. Retry topics can therefore duplicate a record as well as delay it.
In a real system
A typical setup with two topics in the same system that need opposite answers. Both are consumed by Spring Boot services with three instances each.
| payments.requested | orders.status | |
|---|---|---|
| Key and partitions | Payment id, 6 partitions | Order id, 12 partitions |
| Meaning of a record | Charge this payment once | The order moved to this state: PLACED, PAID, SHIPPED, CANCELLED |
| Records for one key | One | Several, and each depends on the ones before it |
| Typical transient failure | The payment provider times out for a few requests per minute | A row lock in the orders database is held for a moment |
| Retry choice | @RetryableTopic: 4 attempts, 1 s doubling, then dead letter topic | Blocking: 5 retries, 200 ms doubling to a 5 s cap with jitter, then stop the container |
| Why | Payments are independent. One slow provider call should not delay the other payments on the partition | Processing SHIPPED before PAID would corrupt the order. A short stall is the correct behaviour |
For the payments topic, the worst case for one record is 1 + 2 + 4 = 7 seconds of waiting plus four provider timeouts, and the main partition never waits at all. For the status topic, the worst case is about 200 + 400 + 800 + 1600 + 3200 ms, roughly 6 seconds, during which that consumer's partitions are stalled. Both numbers are far below the 5-minute poll interval.
When the payment provider has a full outage, the payments listener is not left to drain into the dead letter topic. An operator, or a circuit breaker in the service, stops the listener container, the records wait in payments.requested where they are safe and in order, and the lag alarm states the real situation.
Java and Spring Boot
Three parts: a listener with non-blocking retries and a dead letter handler, a jittered back-off for the blocking case, and the idempotent processing that makes either kind of retry safe.
package com.example.payments;import java.nio.charset.StandardCharsets;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.DltHandler;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.annotation.RetryableTopic;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.retry.annotation.Backoff;import org.springframework.stereotype.Component;@Componentpublic class PaymentListener { private static final Logger log = LoggerFactory.getLogger(PaymentListener.class); private final PaymentProcessor paymentProcessor; private final FailedPaymentStore failedPayments; public PaymentListener(PaymentProcessor paymentProcessor, FailedPaymentStore failedPayments) { this.paymentProcessor = paymentProcessor; this.failedPayments = failedPayments; } @RetryableTopic( attempts = "4", backoff = @Backoff(delay = 1000, multiplier = 2, maxDelay = 10000), numPartitions = "6", exclude = InvalidPaymentException.class) @KafkaListener(topics = "payments.requested", groupId = "payment-service") public void onPaymentRequested(PaymentRequested request) { paymentProcessor.process(request); } @DltHandler public void onDeadLetter(ConsumerRecord<String, PaymentRequested> record) { Header reason = record.headers().lastHeader(KafkaHeaders.DLT_EXCEPTION_MESSAGE); String message = reason == null ? "unknown" : new String(reason.value(), StandardCharsets.UTF_8); log.error("Payment {} gave up after all attempts: {}", record.key(), message); failedPayments.save(record.key(), record.value(), message); }}Why it is written this way
attempts = "4"with a 1 second delay and multiplier 2 createspayments.requested-retry-1000,-retry-2000,-retry-4000andpayments.requested-dlt. The same listener method serves all of them.numPartitions = "6"matches the main topic. The default is 1 partition for the automatically created topics, while records are forwarded to the partition number they came from.excludesends a permanent failure straight to the dead letter topic. Retrying an invalid payment three times only delays the same result.- The annotation looks for a bean named
defaultRetryTopicKafkaTemplate, thenkafkaTemplate, to forward records. It must be able to serialisePaymentRequested, and rawbyte[]too if the consumer usesErrorHandlingDeserializer(the template from the previous lesson does both). - The
@DltHandlerstores the failure where people will look at it. The default, when no such method exists, only writes a log line. - This listener must not be a batch listener and must not use container transactions: non-blocking retries support neither.
package com.example.orders;import java.util.concurrent.ThreadLocalRandom;import org.springframework.util.backoff.BackOff;import org.springframework.util.backoff.BackOffExecution;/** * Exponential back-off with a cap and jitter. Retry n waits a random time between half of * min(maxMs, initialMs * multiplier^(n-1)) and the whole of it. */public record JitteredExponentialBackOff(long initialMs, double multiplier, long maxMs, int maxRetries) implements BackOff { @Override public BackOffExecution start() { return new BackOffExecution() { private int retry; @Override public long nextBackOff() { if (retry >= maxRetries) { return BackOffExecution.STOP; } double uncapped = initialMs * Math.pow(multiplier, retry); long ceiling = (long) Math.min((double) maxMs, uncapped); retry++; long half = ceiling / 2; return half + ThreadLocalRandom.current().nextLong(ceiling - half + 1); } }; }}Why it is written this way
start()is called once per failing record, so each record gets its own retry counter.- The cap is applied before the jitter, so the longest possible single delay is exactly
maxMs. That is the number to compare withmax.poll.interval.ms. - Keeping half of the delay fixed guarantees a real pause; randomising the other half spreads consumers that failed at the same moment.
BackOffExecution.STOPtellsDefaultErrorHandlerthat retries are exhausted and the recoverer should be called.
package com.example.orders;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.listener.DefaultErrorHandler;@Configurationpublic class OrderStatusErrorHandlingConfig { private static final Logger log = LoggerFactory.getLogger(OrderStatusErrorHandlingConfig.class); @Bean public DefaultErrorHandler orderStatusErrorHandler() { // 5 retries: about 0.2, 0.4, 0.8, 1.6 and 3.2 seconds, each with jitter, capped at 5 s. DefaultErrorHandler handler = new DefaultErrorHandler( (record, exception) -> { // Order matters on this topic: do not move on silently. log.error("Order status {} at {}-{} offset {} failed after retries", record.key(), record.topic(), record.partition(), record.offset(), exception); throw new IllegalStateException("Refusing to skip an order status event", exception); }, new JitteredExponentialBackOff(200, 2.0, 5000, 5)); handler.addNotRetryableExceptions(InvalidOrderStatusException.class); return handler; }}Why it is written this way
- Blocking retries for a topic where order per key matters. The longest single delay is 5 seconds, far below the 5-minute
max.poll.interval.ms, so the consumer stays in the group. - A recoverer that throws makes the handler redeliver the record and start the back-off again. The partition stays on this record, and the lag alarm reports it, instead of later status events being applied on top of a missing one.
- For a hard stop that needs a manual restart, use
CommonContainerStoppingErrorHandler. For blocking delays longer than the poll interval, give the handler aContainerPausingBackOffHandler. InvalidOrderStatusExceptionis this application's own type for a permanent failure. It skips the retries and goes to the recoverer on the first delivery.
package com.example.payments;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.stereotype.Service;import org.springframework.transaction.annotation.Transactional;@Servicepublic class PaymentProcessor { private final JdbcTemplate jdbc; private final PaymentGateway gateway; public PaymentProcessor(JdbcTemplate jdbc, PaymentGateway gateway) { this.jdbc = jdbc; this.gateway = gateway; } @Transactional public void process(PaymentRequested request) { // payment_id is the primary key. A second attempt for a finished payment inserts nothing. int claimed = jdbc.update( "INSERT INTO processed_payments (payment_id, order_id, amount_minor, currency) " + "VALUES (?, ?, ?, ?) ON CONFLICT (payment_id) DO NOTHING", request.paymentId(), request.orderId(), request.amountMinor(), request.currency()); if (claimed == 0) { return; } // The provider deduplicates on the idempotency key, so a retry after a timeout // cannot charge the customer twice. gateway.charge(request.paymentId(), request.amountMinor(), request.currency()); }}Why it is written this way
- The key comes from the event (
paymentId), never from the attempt. Every retry, from either mechanism, and every duplicate copy carries the same key. - If
gateway.chargethrows, the transaction rolls back and the row disappears, so the next attempt tries again. If it succeeded and the commit then fails, the retry calls the provider again with the same key and the provider returns the earlier result. - Two layers are needed because there are two systems: the unique row protects this service's database, the idempotency key protects the provider. A Kafka transaction would cover neither.
ON CONFLICT ... DO NOTHINGis PostgreSQL syntax.PaymentGatewaystands for the provider's client.
spring: kafka: bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092 consumer: group-id: payment-service enable-auto-commit: false max-poll-records: 100 properties: max.poll.interval.ms: 300000 producer: acks: all value-serializer: org.springframework.kafka.support.serializer.JsonSerializer listener: ack-mode: recordWhy it is written this way
max-poll-records: 100with a provider timeout of 3 seconds gives a worst case of 300 seconds for one poll, which is the whole interval. In practice set the timeout or the batch size lower so there is a margin.- The reference suggests
RECORDacknowledgement with retry topics: the main offset is committed as soon as the failed record has been forwarded. - The producer section configures the template that forwards records to the retry topics.
acks: allmatters here as much as for any other write: a lost retry record is a lost payment.
Configuration
The settings that bound a retry. Kafka defaults are from the 4.4 configuration reference; Spring defaults are from the Spring for Apache Kafka 3.3 reference and Javadoc.
max.poll.interval.msDefault 300000 (5 minutes)
- What it does
- Consumer. Longest allowed time between two
poll()calls. Beyond it the client leaves the group and its partitions are reassigned. - When to change it
- Every blocking delay plus the processing of one poll's records must fit inside it. Raise it only after shortening the delays and the batch.
max.poll.recordsDefault 500
- What it does
- Consumer. Maximum number of records returned by one
poll(). - When to change it
- Lower it when single records are slow. It sets how much work stands between two polls.
DefaultErrorHandler back-offDefault FixedBackOff(0, 9): 9 retries, no delay
- What it does
- Blocking retries: the delays between redeliveries of a failed record on the consumer thread.
- When to change it
- Replace it with a capped exponential back-off. Keep the longest single delay well under
max.poll.interval.ms, or useContainerPausingBackOffHandler.
@RetryableTopic attemptsDefault 3
- What it does
- Total deliveries before the record goes to the dead letter topic. The first delivery on the main topic counts.
- When to change it
- With exponential back-off each distinct delay is a topic. Choose a number whose total waiting time matches how long the dependency is typically unavailable.
@RetryableTopic backoff (@Backoff delay, multiplier, maxDelay)Default Fixed delay of 1000 ms. With a multiplier, maxDelay defaults to 30000 ms
- What it does
- The delay of each step in the retry chain. A multiplier above 1 makes it exponential, up to
maxDelay. - When to change it
- Set
maxDelayexplicitly. Changing these values changes the retry topic names, so plan the change like a topic migration.
@RetryableTopic numPartitions / replicationFactorDefault 1 / -1 (the broker default)
- What it does
- Partition count and replication factor of the automatically created retry and dead letter topics.
- When to change it
- Match the main topic's partition count. In production, give these topics the same replication settings as the main topic, or create them yourself and set
autoCreateTopicsto false.
@RetryableTopic include / excludeDefault Retry on every exception, except the global fatal list
- What it does
- Which exception types go through the retry chain. Excluded types and the fatal list go directly to the dead letter topic.
- When to change it
- Exclude your own permanent failures. The fatal list is the same as
DefaultErrorHandler's:DeserializationException,ClassCastExceptionand the conversion exceptions.
@RetryableTopic dltStrategyDefault ALWAYS_RETRY_ON_ERROR
- What it does
- What happens when the
@DltHandleritself fails: the record is sent to the dead letter topic again, the consumer fails without forwarding (FAIL_ON_ERROR), or there is no dead letter topic at all (NO_DLT). - When to change it
- Keep the dead letter handler simple enough that it rarely fails: store the record and return.
NO_DLTdrops records after the last attempt.
retry.backoff.ms / retry.backoff.max.msDefault 100 / 1000 (1 second)
- What it does
- Kafka client. Back-off between the client's own retries of a failed request to a broker, exponential with 20% jitter.
- When to change it
- Leave them. They do not control how your listener is retried.
Hands-on exercise
Stall a partition with retries, then move the retries out of the way
Make a dependency fail for some records, watch blocking retries hold up everything behind them, push the delay past the poll interval, then switch to retry topics and see what happens to lag and to order.
Steps · 0 of 6 done
What you should see
Without back-off the failing records are retried in a tight loop. With blocking back-off the dependency is spared, but lag rises on the affected partitions for as long as a record waits. A delay beyond the poll interval causes repeated rebalances and the attempt count never reaches its limit. With retry topics the main lag stays near zero and failing records move along the chain, and the two updates for one key finish in the wrong order until the handler ignores the stale one.
What can go wrong
A blocking delay longer than max.poll.interval.ms
- Symptom
- Rebalances every few minutes,
CommitFailedExceptionin the log, a record that is retried for hours and never reaches the dead letter topic. - Cause
- The error handler sleeps on the consumer thread longer than the poll interval. The client leaves the group, the partition moves, and the new owner starts counting attempts from one.
- Fix
- Cap each delay well below the interval, or use
ContainerPausingBackOffHandler, or move long waits into retry topics.
Retry topics on a topic where order matters
- Symptom
- Now and then an entity ends up in an older state: an order that shows PAID after it was SHIPPED. It correlates with brief errors in the consumer.
- Cause
- The earlier event failed and was retried from a retry topic after the later event for the same key had been processed on the main topic.
- Fix
- Use blocking retries for that topic, or make the handler compare a version in the event with the stored one and ignore older events.
Retries that repeat a side effect
- Symptom
- Customers are charged twice, or receive the same email three times, during an incident with many timeouts.
- Cause
- A timeout was treated as nothing happened. The first attempt had completed at the provider, and the retry did it again without an idempotency key.
- Fix
- Derive an idempotency key from the event, pass it to the provider, and keep a unique record of processed keys in your own database.
An outage drained into the dead letter topic
- Symptom
- After a 10-minute outage of the payment provider, the main topic has no lag and the dead letter topic holds every payment from that period.
- Cause
- With retry topics, each record takes a few seconds to exhaust its attempts and is then dead-lettered. The chain is far shorter than the outage.
- Fix
- Stop or pause the main listener while the dependency is down, for example from a circuit breaker, and keep a tested way to replay the dead letter topic.
Retry topics created with one partition
- Symptom
- Retries are slow under load, and all retried records are handled by one instance.
- Cause
- The automatically created retry topics have 1 partition by default, so one consumer thread handles every retry, and each waiting record delays the ones behind it.
- Fix
- Set
numPartitionsto the main topic's partition count, or create the topics yourself with the right settings.
No jitter on a shared dependency
- Symptom
- After a database failover, load arrives in spikes at 1, 2, 4 and 8 seconds, and the database falls over again on a spike.
- Cause
- Every consumer failed at the same moment and uses the same deterministic back-off, so all of them retry at the same moments.
- Fix
- Add jitter to blocking back-off, and limit how many records are in flight against the dependency at once.
How to troubleshoot
Rebalances started after retries with back-off were added
Check
Consumer metric time-between-poll-max (kafka.consumer:type=consumer-metrics) compared with max.poll.interval.ms, and CommitFailedException in the log
- How to read it
- A maximum close to or above the interval means the thread is blocked too long between polls. Compare it with the largest delay of the back-off plus the time one poll's records take.
You want to know whether records are stuck in the retry chain
Check
kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group payment-service-retry-1000
- How to read it
- Each retry topic has its own group, named from the listener's group id and the topic suffix. A small lag that comes and goes is normal: records wait there until due. Lag that keeps growing means retries arrive faster than one consumer can work through them.
You are not sure which retry topics exist
Check
kafka-topics.sh --bootstrap-server kafka-1:9092 --list
- How to read it
- Look for the main topic's name with
-retry-<delay>and-dltsuffixes. Topics from an older back-off configuration stay behind after the delays were changed and may still hold records.
The main topic is healthy but work is not getting done
Check
kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group payment-service-dlt
- How to read it
- A rising LOG-END-OFFSET on the dead letter topic while the main topic has no lag means records are failing all their attempts. The stream is being diverted, not processed.
A record was retried and you need to know from where and why
Check
kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic payments.requested-dlt --from-beginning --formatter-property print.headers=true
- How to read it
- The headers include the original topic, partition and offset and the last exception. Tools older than Kafka 4.2 take
--property print.headers=true.
Key takeaways
- A retry is for transient failures only, and it runs your listener again. Make the work idempotent with a key taken from the event before choosing any retry mechanism.
- Blocking retries keep order and stall the partition. Each single delay, plus processing time, must fit inside
max.poll.interval.ms. - Exponential back-off needs a cap, so delays stay useful, and jitter, so consumers that failed together do not retry together.
@RetryableTopicforwards a failed record to a chain of delay topics and commits past it. The main partition keeps moving and order for that key is lost.- Use blocking retries where records for a key depend on each other, and retry topics where records are independent.
- When a dependency is down for every record, stop or pause consumption. Retry topics would carry the whole stream to the dead letter topic.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
An order-status consumer uses DefaultErrorHandler with a fixed back-off of 10 minutes and 3 retries, added to ride out maintenance windows of the orders database. max.poll.interval.ms is the default. During last night's 20-minute window the group rebalanced continuously, and afterwards the first failing record showed dozens of delivery attempts across all three instances.
What explains both symptoms?
Scenario 2 of 3
A team adds @RetryableTopic (4 attempts, 2 s doubling) to the consumer of inventory.levels, keyed by product id. Each event carries the absolute stock level of a product and the handler writes it to a table. Soon after, support reports products whose shown stock is briefly correct and then jumps back to an older value. The errors correlate with short database lock timeouts.
What should the team change?
Scenario 3 of 3
A notification consumer sends an SMS per OrderShipped event through a provider API, with 3 blocking retries at 1, 2 and 4 seconds. During a provider slowdown, calls time out after 3 seconds, and many customers receive the same SMS two or three times. The provider's dashboard shows that the timed-out requests were accepted and sent.
Which change stops the repeated messages while keeping the retries?
References
References: Spring for Apache Kafka 3.3 reference: Non-blocking retries (overview and limits) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, how the pattern works (ordering) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, features (back-off, attempts, exception classifier, topic auto-creation) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, topic naming ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, DLT strategies (@DltHandler) ↗ · Spring for Apache Kafka 3.3 reference: Combining blocking and non-blocking retries ↗ · Spring for Apache Kafka 3.3 reference: Back-off delay precision ↗ · Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler, back-off handlers, ExponentialBackOffWithMaxRetries) ↗ · Spring for Apache Kafka 3.3 Javadoc: @RetryableTopic ↗ · Apache Kafka documentation: Consumer configs (max.poll.interval.ms, max.poll.records, retry.backoff.ms) ↗ · Spring Framework 6.2 Javadoc: ExponentialBackOff ↗ · Report a problem