Skip to content

Path 6 · lesson 3 of 5

Advanced30 minApache Kafka 4.x (defaults checked against the 4.4 documentation) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 and Spring Framework 6.2

Retries, back-off and retry topics

Blocking retries keep order and stall the partition; retry topics keep the partition moving and give up order. How each works, how they meet max.poll.interval.ms, and how to retry safely.

What you will learn

  • Explain what a blocking retry does to the partition, to the consumer thread and to the max.poll.interval.ms clock
  • Design an exponential back-off with a cap and jitter, and work out its longest single delay and its total duration
  • Describe how @RetryableTopic moves a failed record through a chain of retry topics and what that does to ordering
  • Choose between blocking retries, retry topics and pausing the listener for a given topic and failure
  • Make the retried work idempotent so that a second attempt after a half-finished first one does no harm

Before you start

The concept

A retry is the right response to a transient failure: the record is fine, and the thing it needs (a database, a payment provider, a lock) is not available right now. The question is where the record waits while it is not being processed. There are two places.

Blocking retry: wait in the partition

  • The consumer keeps the failed record, waits, and delivers it to the listener again
  • Nothing behind it in the partition is processed meanwhile
  • Order within the partition is fully preserved
  • No extra topics. The retry state lives in the consumer's memory
  • The waiting happens on the consumer thread, under the max.poll.interval.ms clock

Non-blocking retry: wait in another topic

  • The consumer publishes the failed record to a retry topic and commits past it
  • The partition continues with the next record at once
  • Order is lost: later records for the same key can be processed first
  • One extra topic and consumer per delay step, plus a dead letter topic
  • The waiting happens in the retry topic's consumer, which pauses until the record is due

Whichever place you choose, the delay between attempts should grow. A dependency that is down for thirty seconds is not helped by ten attempts in the first second, and a dependency that is struggling is harmed by them. Exponential back-off multiplies the delay after each failure, a cap stops it growing beyond a useful maximum, and jitter spreads the attempts of many consumers so they do not all return at the same instant.

Why it exists

The simplest consumer has one behaviour for a failure: the exception propagates, the offset is not committed, and the record is fetched again immediately. For a brief failure this works. For a dependency that is down for a minute it is a hot loop that fails thousands of times, fills the log, and adds load to the dependency just as it tries to come back.

Back-off was added to fix the hot loop: wait between attempts, and wait longer each time. That solves the load problem and exposes the next one. While a record waits in place, its whole partition waits with it. If one order in ten thousand needs four retries of several seconds each, every order behind it on that partition is late by that amount, and the lag alarm fires for a problem that affects one record.

Retry topics were invented to remove that stall: put the failed record somewhere else and let the partition continue. The price is the one property the partition was giving you for free, which is order. So neither approach replaces the other. Blocking retries are what you use when the order of records matters more than the delay; retry topics are what you use when records are independent and the delay matters more.

There is a third situation that neither handles well: the dependency is down for every record. Blocking retries stall every partition, which is at least honest. Retry topics move the entire stream through the retry chain into the dead letter topic, record by record, and the main topic looks healthy while nothing succeeds. For that case the right response is to stop consuming until the dependency returns.

How it works inside

Blocking retries in the listener container

In Spring for Apache Kafka, blocking retries are done by DefaultErrorHandler. When the listener throws a retryable exception, the handler asks its BackOff for the next delay, suspends the consumer thread for that long, and then seeks the partition back to the failed offset so that the next poll() returns the record again. When the BackOff says stop, the recoverer is called (see Poison messages and error handling). The default is 9 retries with no delay.

Two details matter. First, the thread that sleeps is the thread that polls, so every partition assigned to that consumer waits, not only the one with the failing record. Second, the retry count is kept in the memory of that consumer. If the partition moves to another instance, the count starts again at one.

The interaction with max.poll.interval.ms

A consumer that does not call poll() within max.poll.interval.ms (default 5 minutes) is considered stuck. The client leaves the group, and its partitions are assigned to another member. A blocking back-off is time spent not polling.

The container polls again between attempts, so the clock restarts after each delay. What must fit inside the interval is therefore one delay plus the processing time around it, not the sum of all delays. A back-off of 1, 2, 4, 8 and 10 seconds is no risk with the default interval. A single 6-minute wait is.

  1. The listener fails and the handler sleeps for 6 minutes on the consumer thread.
  2. After 5 minutes without poll(), the client leaves the group. The partition goes to instance B.
  3. Instance B reads the same record from the last committed offset, as attempt 1 of its own count. It fails and sleeps for 6 minutes.
  4. Instance A wakes up, finds it no longer owns the partition, and rejoins. Another rebalance.
  5. Instance B is removed in turn. The record moves round the group, the attempt count never reaches its limit, and no partition of the group makes steady progress.

If you need blocking delays longer than the poll interval, Spring provides ContainerPausingBackOffHandler. Instead of sleeping, it pauses the listener container for the delay: the consumer keeps calling poll(), receives nothing, stays in the group, and resumes afterwards. The partition is still blocked, which is the point, but the group is stable.

Exponential back-off, cap and jitter

The delay before retry number n is min(cap, initial × multiplier^(n-1)). With an initial delay of 1 second, a multiplier of 2 and a cap of 10 seconds, six retries wait 1, 2, 4, 8, 10 and 10 seconds: 35 seconds in total before the recoverer is called. Spring's ExponentialBackOffWithMaxRetries implements exactly this.

Without the cap, the delay doubles past any useful value: the twelfth retry would wait over half an hour. Without jitter, every consumer that failed at the same moment, because the database went down at that moment, retries at the same moments too, and the recovering database receives its load in synchronised waves. Jitter replaces each delay with a random value in a range, for example between half the computed delay and the full delay. The animated lesson Back-off and jitter shows the waves and how jitter flattens them.

ExponentialBackOff in Spring Framework 6.2, the version used by Spring Boot 3.5, has no jitter setting. BackOff is a small interface, so a jittered version is a few lines, shown in the implementation section.

Non-blocking retries with @RetryableTopic

Adding @RetryableTopic to a @KafkaListener method makes the framework create a chain of topics and a listener container for each. With 4 attempts and an exponential back-off of 1 second and multiplier 2 on payments.requested, the chain is payments.requested-retry-1000, payments.requested-retry-2000, payments.requested-retry-4000 and payments.requested-dlt. The first attempt on the main topic counts, so 4 attempts means 3 retries.

  1. The listener fails on the main topic. The framework publishes the record to the first retry topic, with a header that holds the time at which it is due, and the main container commits past the record.
  2. The retry topic's consumer reads the record and compares the due time with the clock. If the record is not due, it pauses that partition and resumes it when the time comes. It keeps polling meanwhile, so max.poll.interval.ms is not at risk.
  3. When the record is due, the same listener method is called again. If it succeeds, the record is done.
  4. If it fails, the record is published to the next retry topic with a new due time, and so on.
  5. When the attempts are used up, or the exception is one of the fatal types such as DeserializationException, the record goes to the dead letter topic. A method annotated with @DltHandler in the same class receives it; without one, a default handler only logs.
AspectDefaultConsequence
Attempts3, including the firstTwo retries, then the dead letter topic
Back-offFixed, 1000 msWith a multiplier it becomes exponential; the maximum delay then defaults to 30 seconds
Topic namesSuffix -retry plus the delay in ms, and -dltWith exponential back-off, one topic per distinct delay; attempts at the maximum delay share the last topic (since 3.2)
Topic creationAutomatic, 1 partition, broker default replicationSet numPartitions to match the main topic: records are sent to the same partition number they came from
Consumer groupsThe listener's group id plus the topic suffixEach retry topic has its own group, so its rebalances do not disturb the main listener
ExceptionsRetry on all, except the global fatal listUse exclude to send your permanent failures straight to the dead letter topic
Defaults of @RetryableTopic in Spring for Apache Kafka 3.3

Some limits are stated in the reference and are worth knowing before you design around this feature: non-blocking retries are not supported with batch listeners, they cannot be combined with container transactions, and the delay is a lower bound. A record is never processed before its due time, but it can be processed later if the retry consumer is busy with an earlier record.

The delay of each retry topic is fixed when the application starts and belongs to the topic. Every record in -retry-2000 waits about two seconds. So the chain gives you exponential growth and a cap, but not per-record jitter.

What retry topics do to ordering

Kafka guarantees order within a partition, for the records as the broker appended them. A retry topic takes a record out of its partition and appends a copy somewhere else, later. Suppose OrderUpdated version 1 for order 42 fails and goes to the retry topic. Version 2 for order 42 arrives on the main topic a moment later and succeeds. Two seconds afterwards version 1 is retried, succeeds, and overwrites version 2. The reference says it in one sentence: by using this strategy you lose Kafka's ordering guarantees for that topic.

This is harmless when records are independent, such as one payment request per payment id or one email per notification. It is a correctness bug when later records for a key build on earlier ones. If you still want retry topics there, the handler must detect stale records itself, for example by comparing a version number in the event with the version already stored, and ignore the older one.

Choosing

SituationUseWhy
Records for a key must be applied in orderBlocking retries, bounded, then dead-letter or stopThe stall is the mechanism that keeps the order
Records are independent and a few fail now and thenRetry topicsOne slow record should not delay thousands of healthy ones
Failures usually clear within a second or twoA few short blocking retriesCheaper than a round trip through another topic
The dependency is down for every recordLong blocking back-off, or pause or stop the containerRetry topics would carry the whole stream to the dead letter topic
Permanent failureNo retries: dead-letter at onceWaiting cannot change the outcome

The two can be combined. A configuration class that extends RetryTopicConfigurationSupport can name exceptions that get a few blocking retries first and move to the retry topics only if those fail. That suits failures that often clear quickly and would also hit the next records, such as a database connection error.

Retry topics & DLQ

Simulation, not a real cluster

The animated lesson shows a failing record stalling its partition with in-place retries, then leaving through a retry topic while the records behind it continue.

Step 1 · The pattern

One at a time, in order

A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.

  • message
  • processed
Open the full animated lesson

Architecture

Drawing diagram…

A flowchart in which a record from payments.requested is tried by the listener, and on failure moves through three retry topics with delays of 1, 2 and 4 seconds, each followed by another attempt, ending in the dead letter topic and its handler when all four attempts fail.

flowchart LR
  MAIN["payments.requested"] --> L0["Listener, attempt 1"]
  L0 -- "Success" --> DONE["Payment charged"]
  L0 -- "Retryable failure" --> R1["payments.requested-retry-1000"]
  L0 -- "Fatal or excluded exception" --> DLT["payments.requested-dlt"]
  R1 --> L1["Listener, attempt 2 after 1 s"]
  L1 -- "Success" --> DONE
  L1 -- "Failure" --> R2["payments.requested-retry-2000"]
  R2 --> L2["Listener, attempt 3 after 2 s"]
  L2 -- "Success" --> DONE
  L2 -- "Failure" --> R3["payments.requested-retry-4000"]
  R3 --> L3["Listener, attempt 4 after 4 s"]
  L3 -- "Success" --> DONE
  L3 -- "Failure" --> DLT
  DLT --> H["DltHandler method"]
The chain created by @RetryableTopic with 4 attempts, a 1 second delay and a multiplier of 2.
  • Every box labelled Listener is the same method in your code, run by a different listener container: one for the main topic and one for each retry topic. The record you receive on attempt 3 is a copy that was published twice on its way there.
  • A record leaves the main topic after its first failure and never returns to it. From that moment the main partition's offset has moved on and the order relative to its neighbours is gone.
  • There are two ways into the dead letter topic: exhausting the attempts, and throwing an exception that is classified as fatal or excluded, which skips the chain entirely.
  • Each retry topic holds records with one delay. That is why the delay can be enforced by pausing a partition: the record at the front is always the one that is due first.

Message flow

Drawing diagram…

A sequence diagram in which the main listener fails to charge payment p-17, publishes it to the first retry topic and commits, then continues with p-18, while the retry listeners wait for the due time, fail once more, forward the record to the second retry topic and finally succeed.

sequenceDiagram
  participant M as payments.requested
  participant L as Main listener
  participant P as Payment provider
  participant R1 as retry-1000 topic
  participant L1 as Retry listener 1000
  participant R2 as retry-2000 topic
  participant L2 as Retry listener 2000
  M->>L: Record offset 88, payment p-17
  L->>P: Charge p-17
  P--xL: Timeout
  L->>R1: Publish p-17 with due time now plus 1 s
  L->>M: Commit offset 89
  M->>L: Record offset 89, payment p-18
  Note over L: Main partition continues
  R1->>L1: Record p-17, not due yet
  Note over L1: Pause partition until due time
  L1->>P: Charge p-17
  P--xL1: Timeout
  L1->>R2: Publish p-17 with due time now plus 2 s
  R2->>L2: Record p-17
  Note over L2: Pause partition until due time
  L2->>P: Charge p-17
  P-->>L2: Charged
  L2->>R2: Commit offset
Payment p-17 fails twice and succeeds on its third attempt, while the main partition has long moved on to p-18.
  1. The main listener's failure costs the partition one publish and one commit. Payment p-18 is processed while p-17 is still waiting, which is both the benefit and the loss of order.
  2. The retry listener does not sleep. It pauses the partition and keeps polling, so it stays in its group however long the delay is.
  3. The payment provider is called three times for p-17. The first two calls timed out, which does not prove they did nothing. The charge is only safe because the provider is given the payment id as an idempotency key.
  4. If the publish to the retry topic succeeds and the commit of offset 89 then fails, the main listener receives offset 88 again and publishes a second copy. Retry topics can therefore duplicate a record as well as delay it.

In a real system

A typical setup with two topics in the same system that need opposite answers. Both are consumed by Spring Boot services with three instances each.

payments.requestedorders.status
Key and partitionsPayment id, 6 partitionsOrder id, 12 partitions
Meaning of a recordCharge this payment onceThe order moved to this state: PLACED, PAID, SHIPPED, CANCELLED
Records for one keyOneSeveral, and each depends on the ones before it
Typical transient failureThe payment provider times out for a few requests per minuteA row lock in the orders database is held for a moment
Retry choice@RetryableTopic: 4 attempts, 1 s doubling, then dead letter topicBlocking: 5 retries, 200 ms doubling to a 5 s cap with jitter, then stop the container
WhyPayments are independent. One slow provider call should not delay the other payments on the partitionProcessing SHIPPED before PAID would corrupt the order. A short stall is the correct behaviour

For the payments topic, the worst case for one record is 1 + 2 + 4 = 7 seconds of waiting plus four provider timeouts, and the main partition never waits at all. For the status topic, the worst case is about 200 + 400 + 800 + 1600 + 3200 ms, roughly 6 seconds, during which that consumer's partitions are stalled. Both numbers are far below the 5-minute poll interval.

When the payment provider has a full outage, the payments listener is not left to drain into the dead letter topic. An operator, or a circuit breaker in the service, stops the listener container, the records wait in payments.requested where they are safe and in order, and the lag alarm states the real situation.

Java and Spring Boot

Three parts: a listener with non-blocking retries and a dead letter handler, a jittered back-off for the blocking case, and the idempotent processing that makes either kind of retry safe.

payment-service/src/main/java/com/example/payments/PaymentListener.javaJava
package com.example.payments;import java.nio.charset.StandardCharsets;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.DltHandler;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.annotation.RetryableTopic;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.retry.annotation.Backoff;import org.springframework.stereotype.Component;@Componentpublic class PaymentListener {    private static final Logger log = LoggerFactory.getLogger(PaymentListener.class);    private final PaymentProcessor paymentProcessor;    private final FailedPaymentStore failedPayments;    public PaymentListener(PaymentProcessor paymentProcessor, FailedPaymentStore failedPayments) {        this.paymentProcessor = paymentProcessor;        this.failedPayments = failedPayments;    }    @RetryableTopic(            attempts = "4",            backoff = @Backoff(delay = 1000, multiplier = 2, maxDelay = 10000),            numPartitions = "6",            exclude = InvalidPaymentException.class)    @KafkaListener(topics = "payments.requested", groupId = "payment-service")    public void onPaymentRequested(PaymentRequested request) {        paymentProcessor.process(request);    }    @DltHandler    public void onDeadLetter(ConsumerRecord<String, PaymentRequested> record) {        Header reason = record.headers().lastHeader(KafkaHeaders.DLT_EXCEPTION_MESSAGE);        String message = reason == null ? "unknown" : new String(reason.value(), StandardCharsets.UTF_8);        log.error("Payment {} gave up after all attempts: {}", record.key(), message);        failedPayments.save(record.key(), record.value(), message);    }}

Why it is written this way

  • attempts = "4" with a 1 second delay and multiplier 2 creates payments.requested-retry-1000, -retry-2000, -retry-4000 and payments.requested-dlt. The same listener method serves all of them.
  • numPartitions = "6" matches the main topic. The default is 1 partition for the automatically created topics, while records are forwarded to the partition number they came from.
  • exclude sends a permanent failure straight to the dead letter topic. Retrying an invalid payment three times only delays the same result.
  • The annotation looks for a bean named defaultRetryTopicKafkaTemplate, then kafkaTemplate, to forward records. It must be able to serialise PaymentRequested, and raw byte[] too if the consumer uses ErrorHandlingDeserializer (the template from the previous lesson does both).
  • The @DltHandler stores the failure where people will look at it. The default, when no such method exists, only writes a log line.
  • This listener must not be a batch listener and must not use container transactions: non-blocking retries support neither.

Configuration

The settings that bound a retry. Kafka defaults are from the 4.4 configuration reference; Spring defaults are from the Spring for Apache Kafka 3.3 reference and Javadoc.

max.poll.interval.msDefault 300000 (5 minutes)
What it does
Consumer. Longest allowed time between two poll() calls. Beyond it the client leaves the group and its partitions are reassigned.
When to change it
Every blocking delay plus the processing of one poll's records must fit inside it. Raise it only after shortening the delays and the batch.
max.poll.recordsDefault 500
What it does
Consumer. Maximum number of records returned by one poll().
When to change it
Lower it when single records are slow. It sets how much work stands between two polls.
DefaultErrorHandler back-offDefault FixedBackOff(0, 9): 9 retries, no delay
What it does
Blocking retries: the delays between redeliveries of a failed record on the consumer thread.
When to change it
Replace it with a capped exponential back-off. Keep the longest single delay well under max.poll.interval.ms, or use ContainerPausingBackOffHandler.
@RetryableTopic attemptsDefault 3
What it does
Total deliveries before the record goes to the dead letter topic. The first delivery on the main topic counts.
When to change it
With exponential back-off each distinct delay is a topic. Choose a number whose total waiting time matches how long the dependency is typically unavailable.
@RetryableTopic backoff (@Backoff delay, multiplier, maxDelay)Default Fixed delay of 1000 ms. With a multiplier, maxDelay defaults to 30000 ms
What it does
The delay of each step in the retry chain. A multiplier above 1 makes it exponential, up to maxDelay.
When to change it
Set maxDelay explicitly. Changing these values changes the retry topic names, so plan the change like a topic migration.
@RetryableTopic numPartitions / replicationFactorDefault 1 / -1 (the broker default)
What it does
Partition count and replication factor of the automatically created retry and dead letter topics.
When to change it
Match the main topic's partition count. In production, give these topics the same replication settings as the main topic, or create them yourself and set autoCreateTopics to false.
@RetryableTopic include / excludeDefault Retry on every exception, except the global fatal list
What it does
Which exception types go through the retry chain. Excluded types and the fatal list go directly to the dead letter topic.
When to change it
Exclude your own permanent failures. The fatal list is the same as DefaultErrorHandler's: DeserializationException, ClassCastException and the conversion exceptions.
@RetryableTopic dltStrategyDefault ALWAYS_RETRY_ON_ERROR
What it does
What happens when the @DltHandler itself fails: the record is sent to the dead letter topic again, the consumer fails without forwarding (FAIL_ON_ERROR), or there is no dead letter topic at all (NO_DLT).
When to change it
Keep the dead letter handler simple enough that it rarely fails: store the record and return. NO_DLT drops records after the last attempt.
retry.backoff.ms / retry.backoff.max.msDefault 100 / 1000 (1 second)
What it does
Kafka client. Back-off between the client's own retries of a failed request to a broker, exponential with 20% jitter.
When to change it
Leave them. They do not control how your listener is retried.

Hands-on exercise

Stall a partition with retries, then move the retries out of the way

Make a dependency fail for some records, watch blocking retries hold up everything behind them, push the delay past the poll interval, then switch to retry topics and see what happens to lag and to order.

Steps · 0 of 6 done

Take it further in the lab: Retry and backoff strategy

What can go wrong

A blocking delay longer than max.poll.interval.ms
Symptom
Rebalances every few minutes, CommitFailedException in the log, a record that is retried for hours and never reaches the dead letter topic.
Cause
The error handler sleeps on the consumer thread longer than the poll interval. The client leaves the group, the partition moves, and the new owner starts counting attempts from one.
Fix
Cap each delay well below the interval, or use ContainerPausingBackOffHandler, or move long waits into retry topics.
Retry topics on a topic where order matters
Symptom
Now and then an entity ends up in an older state: an order that shows PAID after it was SHIPPED. It correlates with brief errors in the consumer.
Cause
The earlier event failed and was retried from a retry topic after the later event for the same key had been processed on the main topic.
Fix
Use blocking retries for that topic, or make the handler compare a version in the event with the stored one and ignore older events.
Retries that repeat a side effect
Symptom
Customers are charged twice, or receive the same email three times, during an incident with many timeouts.
Cause
A timeout was treated as nothing happened. The first attempt had completed at the provider, and the retry did it again without an idempotency key.
Fix
Derive an idempotency key from the event, pass it to the provider, and keep a unique record of processed keys in your own database.
An outage drained into the dead letter topic
Symptom
After a 10-minute outage of the payment provider, the main topic has no lag and the dead letter topic holds every payment from that period.
Cause
With retry topics, each record takes a few seconds to exhaust its attempts and is then dead-lettered. The chain is far shorter than the outage.
Fix
Stop or pause the main listener while the dependency is down, for example from a circuit breaker, and keep a tested way to replay the dead letter topic.
Retry topics created with one partition
Symptom
Retries are slow under load, and all retried records are handled by one instance.
Cause
The automatically created retry topics have 1 partition by default, so one consumer thread handles every retry, and each waiting record delays the ones behind it.
Fix
Set numPartitions to the main topic's partition count, or create the topics yourself with the right settings.
No jitter on a shared dependency
Symptom
After a database failover, load arrives in spikes at 1, 2, 4 and 8 seconds, and the database falls over again on a spike.
Cause
Every consumer failed at the same moment and uses the same deterministic back-off, so all of them retry at the same moments.
Fix
Add jitter to blocking back-off, and limit how many records are in flight against the dependency at once.

How to troubleshoot

Rebalances started after retries with back-off were added

Check

Consumer metric time-between-poll-max (kafka.consumer:type=consumer-metrics) compared with max.poll.interval.ms, and CommitFailedException in the log
How to read it
A maximum close to or above the interval means the thread is blocked too long between polls. Compare it with the largest delay of the back-off plus the time one poll's records take.
You want to know whether records are stuck in the retry chain

Check

kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group payment-service-retry-1000
How to read it
Each retry topic has its own group, named from the listener's group id and the topic suffix. A small lag that comes and goes is normal: records wait there until due. Lag that keeps growing means retries arrive faster than one consumer can work through them.
You are not sure which retry topics exist

Check

kafka-topics.sh --bootstrap-server kafka-1:9092 --list
How to read it
Look for the main topic's name with -retry-<delay> and -dlt suffixes. Topics from an older back-off configuration stay behind after the delays were changed and may still hold records.
The main topic is healthy but work is not getting done

Check

kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group payment-service-dlt
How to read it
A rising LOG-END-OFFSET on the dead letter topic while the main topic has no lag means records are failing all their attempts. The stream is being diverted, not processed.
A record was retried and you need to know from where and why

Check

kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic payments.requested-dlt --from-beginning --formatter-property print.headers=true
How to read it
The headers include the original topic, partition and offset and the last exception. Tools older than Kafka 4.2 take --property print.headers=true.

Key takeaways

  • A retry is for transient failures only, and it runs your listener again. Make the work idempotent with a key taken from the event before choosing any retry mechanism.
  • Blocking retries keep order and stall the partition. Each single delay, plus processing time, must fit inside max.poll.interval.ms.
  • Exponential back-off needs a cap, so delays stay useful, and jitter, so consumers that failed together do not retry together.
  • @RetryableTopic forwards a failed record to a chain of delay topics and commits past it. The main partition keeps moving and order for that key is lost.
  • Use blocking retries where records for a key depend on each other, and retry topics where records are independent.
  • When a dependency is down for every record, stop or pause consumption. Retry topics would carry the whole stream to the dead letter topic.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

An order-status consumer uses DefaultErrorHandler with a fixed back-off of 10 minutes and 3 retries, added to ride out maintenance windows of the orders database. max.poll.interval.ms is the default. During last night's 20-minute window the group rebalanced continuously, and afterwards the first failing record showed dozens of delivery attempts across all three instances.

What explains both symptoms?

Scenario 2 of 3

A team adds @RetryableTopic (4 attempts, 2 s doubling) to the consumer of inventory.levels, keyed by product id. Each event carries the absolute stock level of a product and the handler writes it to a table. Soon after, support reports products whose shown stock is briefly correct and then jumps back to an older value. The errors correlate with short database lock timeouts.

What should the team change?

Scenario 3 of 3

A notification consumer sends an SMS per OrderShipped event through a provider API, with 3 blocking retries at 1, 2 and 4 seconds. During a provider slowdown, calls time out after 3 seconds, and many customers receive the same SMS two or three times. The provider's dashboard shows that the timed-out requests were accepted and sent.

Which change stops the repeated messages while keeping the retries?

References

References: Spring for Apache Kafka 3.3 reference: Non-blocking retries (overview and limits) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, how the pattern works (ordering) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, features (back-off, attempts, exception classifier, topic auto-creation) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, topic naming ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, DLT strategies (@DltHandler) ↗ · Spring for Apache Kafka 3.3 reference: Combining blocking and non-blocking retries ↗ · Spring for Apache Kafka 3.3 reference: Back-off delay precision ↗ · Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler, back-off handlers, ExponentialBackOffWithMaxRetries) ↗ · Spring for Apache Kafka 3.3 Javadoc: @RetryableTopic ↗ · Apache Kafka documentation: Consumer configs (max.poll.interval.ms, max.poll.records, retry.backoff.ms) ↗ · Spring Framework 6.2 Javadoc: ExponentialBackOff ↗ · Report a problem