Path 9 · lesson 4 of 6
Error handling, retries and dead letters in Spring
One production-shaped Spring configuration: DefaultErrorHandler with a back-off, a dead-letter recoverer, ErrorHandlingDeserializer, and @RetryableTopic when retries must not block.
What you will learn
- State what a Spring listener does with a failing record when you configure nothing, and why that is rarely what a payment flow wants
- Configure a DefaultErrorHandler with a bounded back-off, not-retryable exceptions and a DeadLetterPublishingRecoverer with an explicit destination
- Wrap deserializers in ErrorHandlingDeserializer so a malformed payload reaches the error handler and is published as its original bytes
- Choose between blocking retries and @RetryableTopic, and say what each costs in ordering and throughput
- Handle a failure inside a batch listener without sending the whole batch to the dead-letter topic
- Write an integration test that proves a bad record ends in the dead-letter topic and the listener keeps going
Before you start
The concept
The reliability path explained the ideas: what a poison message is, why retries need a back-off and what a dead-letter topic is for. This lesson is the Spring implementation guide. It names the classes, shows where each one sits, and ends with one configuration you can adapt.
In Spring for Apache Kafka a listener method never talks to the consumer. A listener container owns the consumer, runs the poll loop and calls your method once per record. When the method throws, the exception goes back to the container, and the container hands it to a CommonErrorHandler. That one object decides everything that follows: redeliver now, wait and redeliver, or give up and call a recoverer. After a recovery the container commits the record's offset and the partition moves on.
| Piece | Class | What it decides |
|---|---|---|
| Error handler | DefaultErrorHandler | Whether a failed record is delivered again, and when it stops trying |
| Back-off | FixedBackOff, ExponentialBackOffWithMaxRetries | How many redeliveries and how long to wait before each |
| Classification | addNotRetryableExceptions(...) | Which exceptions skip the retries and go straight to recovery |
| Recoverer | DeadLetterPublishingRecoverer | What happens to a record that is given up on: publish it elsewhere, with failure headers |
| Destination resolver | a BiFunction of record and exception | Which topic and partition the dead-letter record goes to |
| Deserializer wrapper | ErrorHandlingDeserializer | That a payload which cannot be decoded becomes a record the error handler can see |
| Non-blocking retries | @RetryableTopic, @DltHandler | That a failed record waits in a retry topic while the partition keeps flowing |
| Transactional listeners | DefaultAfterRollbackProcessor | The same decisions when the container runs a Kafka transaction (see the next lesson) |
SeekToCurrentErrorHandler and the other legacy handlers were replaced by DefaultErrorHandler in 2.8.Why it exists
A raw KafkaConsumer loop gives you no error policy at all. If your code throws and you poll again, the consumer's position has already moved past the record, so it is skipped. If you seek back, you loop on it for ever. Every team that wrote its own loop had to write the same things by hand: seek back to the failed offset, count attempts per record, sleep without missing max.poll.interval.ms, publish the record somewhere when giving up, and commit past it only when that publish succeeded.
Spring moved all of that into the container so that the policy is configuration and the listener stays business code. The price is that the policy has defaults, and the defaults were chosen so that a demo never hangs. Two of them matter in production:
- Log and skip after ten immediate attempts. A database that is down for thirty seconds turns every record polled in that time into ten stack traces and a skipped offset.
- No protection against undecodable bytes. A deserializer runs inside
poll(), before the container has a record to give to anyone. The Spring reference says it plainly: when a deserializer fails, Spring has no way to handle the problem, because it occurs beforepoll()returns. Without the wrapper shown below, the consumer fails on the same offset again and again and the partition stops.
Retry topics & DLQ
Simulation, not a real clusterThe animated Build lesson shows a poison record blocking a partition, then retry topics and a dead-letter topic releasing it. It is a simulation, not a real broker.
Step 1 · The pattern
One at a time, in order
A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.
- message
- processed
A third reason to be explicit is naming. Up to Spring for Apache Kafka 3.2 the default dead-letter topic was the source topic plus .DLT. From 3.3 it is the source topic plus -dlt. A project that relied on the default started writing to a different topic after an upgrade. A destination resolver in your own code removes that surprise.
How it works inside
What DefaultErrorHandler does with one failed record
- The listener throws. The container wraps the exception in a
ListenerExecutionFailedExceptionand, before calling the error handler, commits the offsets of the records that already succeeded. - The handler classifies the exception, looking through the causes. Six types are fatal by default and are never retried:
DeserializationException,MessageConversionException,ConversionException,MethodArgumentResolutionException,NoSuchMethodExceptionandClassCastException.addNotRetryableExceptionsadds your own. - For a retryable exception it asks the
BackOfffor the next wait. It sleeps on the consumer thread, then seeks the consumer back so that the failed record and everything after it from that poll are fetched again. The nextpoll()returns the failed record first. - When the back-off is exhausted, or the exception is not retryable, it calls the recoverer. If the recoverer returns normally the record counts as handled, its offset is committed and the listener receives the next record.
- If the recoverer throws (the dead-letter publish failed, for example), the record is not skipped. It is sought again and, by default, goes through the whole back-off sequence once more.
How a bad payload gets to the handler
ErrorHandlingDeserializer is a deserializer that delegates to the real one. When the delegate throws, the wrapper returns null as the value and puts a DeserializationException, containing the cause and the raw bytes, in a record header. poll() therefore succeeds. The container sees the header, does not call your listener, and passes the record to the error handler. DeserializationException is in the fatal list, so there are no retries: the recoverer runs on the first failure.
DeadLetterPublishingRecoverer then restores the record's value to the original bytes that failed to decode. That has a consequence on the producer side: the template it publishes with must be able to send both your normal objects and a raw byte[]. A producer configured only with a JSON serializer would write the bytes as a JSON-encoded string, which is no longer the message that arrived. The configuration below uses DelegatingByTypeSerializer for that.
What the dead-letter record contains
| Aspect | Behaviour in 3.3 | Why it matters |
|---|---|---|
| Default destination | <source topic>-dlt, same partition number as the failed record | The dead-letter topic needs at least as many partitions as the source. If the resolver returns a negative partition, the producer chooses one. |
| Partition check | If the chosen partition does not exist, the partition is left unset | Records still arrive, but no longer line up with the source partition. |
| Key and value | Copied from the failed record; the original bytes for a deserialization failure | The record can be replayed to the source topic unchanged. |
| Origin headers | KafkaHeaders.DLT_ORIGINAL_TOPIC, DLT_ORIGINAL_PARTITION, DLT_ORIGINAL_OFFSET, DLT_ORIGINAL_TIMESTAMP, DLT_ORIGINAL_CONSUMER_GROUP | You can find the exact source record and the group that gave up on it. |
| Exception headers | DLT_EXCEPTION_FQCN, DLT_EXCEPTION_CAUSE_FQCN, DLT_EXCEPTION_MESSAGE, DLT_EXCEPTION_STACKTRACE | Enough to triage without the service logs. The stack trace can be large; excludeHeader removes it. |
| Publish failure | With failIfSendResultIsError the recoverer waits for the broker's answer and throws if the send failed | A record must never be skipped on the strength of a publish that did not happen. |
Blocking retries or retry topics
DefaultErrorHandler (blocking)
- The failed record is retried in place; later records of the partition wait.
- Order within the partition is kept.
- Total wait is bounded by
max.poll.interval.ms, so think seconds. - No extra topics. Works with batch listeners and with transactions (through the after-rollback processor).
- Fits failures where the next record would fail too: database down, dependency unreachable.
@RetryableTopic (non-blocking)
- The failed record is published to a retry topic with a due time; the main partition moves on.
- Order is lost for that record. The reference says so directly: you lose Kafka's ordering guarantees for that topic.
- Delays of minutes or hours are fine; the retry consumer pauses its partition until the record is due.
- Creates retry topics and a DLT, with extra listener containers. Not supported for batch listeners, and cannot be combined with container transactions.
- Fits failures tied to one record or one customer: a rate-limited account, a recipient's mail server.
With @RetryableTopic the defaults are 3 attempts in total (the first delivery counts) with a fixed 1 second delay, retry topics suffixed -retry and a dead-letter topic suffixed -dlt, each auto-created with one partition and the broker's default replication factor. If the class has no @DltHandler method, a default handler only logs what arrives in the DLT. The same six fatal exception types skip the retry topics and go straight to the DLT.
Batch listeners, briefly
A batch listener receives a list, so the container cannot know which element failed unless you say so. Throw BatchListenerFailedException with the index of the failing record: the handler commits the offsets before that index, and retries or recovers only that record, redelivering the rest. Any other exception makes the handler retry the complete batch from memory, and when the back-off is exhausted the recoverer is called for every record in the batch. One bad record then sends a whole poll of good records to the dead-letter topic.
Architecture
Drawing diagram…
Flowchart. A polled record either fails deserialization and goes to the error handler with an exception header, or reaches the listener. A listener exception also goes to the error handler, which either waits and seeks back for another delivery, or calls the dead-letter recoverer. A successful dead-letter publish lets the offset be committed; a failed publish causes another seek.
flowchart TD
POLL["Container polls orders.order-placed"] --> DES{"Value deserializes"}
DES -->|no| HDR["ErrorHandlingDeserializer returns null value plus exception header"]
DES -->|yes| LIS["OrderPlacedListener"]
LIS -->|returns| ACK["Offset committed by the container"]
LIS -->|throws| EH["DefaultErrorHandler"]
HDR --> EH
EH --> CLS{"Retryable exception"}
CLS -->|yes and attempts left| BO["Back-off wait on the consumer thread"]
BO --> SEEK["Seek back so the record is polled again"]
SEEK --> POLL
CLS -->|no or attempts used up| REC["DeadLetterPublishingRecoverer"]
REC --> RES["Destination resolver picks topic and partition"]
RES --> DLT["orders.order-placed-dlt"]
DLT -->|publish acknowledged| ACK
DLT -->|publish failed| SEEK- There are exactly two exits: the offset is committed because the listener returned, or because the dead-letter publish was acknowledged. Nothing else moves the partition forward.
- The deserialization branch never touches the listener. That is why validation of field values still belongs in the listener, and why a failed validation should throw a type you have marked as not retryable.
- The loop on the left is the blocking retry. While a record is in it, the container is not delivering anything else from that partition.
- The arrow from a failed publish back to the seek is the safety property: if the dead-letter topic is unavailable, the consumer stalls on the record and lag grows, which is visible, instead of losing it, which is not.
Message flow
Drawing diagram…
Sequence diagram. The container polls two records. The listener throws on record 41. The error handler waits, seeks back to offset 41 and the record is polled and delivered again. After the back-off is exhausted the recoverer sends the record to the dead-letter topic, the broker acknowledges, the container commits offset 42 and delivers record 42.
sequenceDiagram participant K as Broker participant C as Listener container participant L as OrderPlacedListener participant E as DefaultErrorHandler participant R as DLT recoverer C->>K: poll K-->>C: offsets 41 and 42 of partition 3 C->>L: record 41 L--xC: throws PaymentProviderUnavailableException C->>E: record 41 and exception, attempt 1 E->>E: wait 1 s E->>C: seek partition 3 to offset 41 C->>K: poll K-->>C: offsets 41 and 42 C->>L: record 41 again L--xC: throws again Note over C,E: repeated with waits of 2 s, 4 s and 8 s C->>E: record 41, back-off exhausted E->>R: recover record 41 R->>K: send to orders.order-placed-dlt partition 3 K-->>R: acknowledged R-->>E: recovered E-->>C: handled, commit offset 42 C->>L: record 42
- Record 42 was already fetched in the first poll, but it is fetched again after each seek and is not delivered until 41 has either succeeded or been recovered.
- The committed offset after recovery is 42: the position of the next record to read, not an acknowledgement of 41. If the service crashes before that commit, 41 is delivered again and may be published to the dead-letter topic twice.
- If the exception had been
InvalidOrderException, marked not retryable, the diagram would jump from the first failure straight to the recoverer. - Had the broker not acknowledged the dead-letter send, the recoverer would throw and the handler would seek to 41 once more.
In a real system
In the academy's example project the payment service consumes orders.order-placed (12 partitions, keyed by orderId) in group payment-service. Three different things go wrong there, and each needs a different answer:
| Failure | Exception | Policy | Why |
|---|---|---|---|
| The card provider times out or returns 503 | PaymentProviderUnavailableException | Blocking retries: 1 s, 2 s, 4 s, 8 s, then dead-letter | The next order would hit the same outage. Keeping order per partition is worth fifteen seconds. |
| Amount is zero or the currency is unknown | InvalidOrderException | Not retryable: dead-letter on the first failure | The record will be exactly as wrong on the tenth attempt. |
A producer published bytes that are not an OrderPlaced | DeserializationException | Fatal by default: dead-letter on the first failure, as raw bytes | The listener cannot even be called. |
The notification service is different. It consumes payments.payment-result and sends an email. If one recipient's mail provider rejects a message for ten minutes, there is no reason to hold back every other customer's confirmation in that partition, and the order of emails between customers carries no meaning. That listener uses @RetryableTopic with delays of 10 seconds, 1 minute and 6 minutes.
Java and Spring Boot
The payment service's error handling, complete, followed by the notification service's non-blocking variant and an integration test. It relies on Spring Boot's auto-configured consumer factory, producer factory, template and listener container factory, and only adds beans. The full services are in the example project at /kafka/project.
spring: kafka: bootstrap-servers: localhost:9092 consumer: group-id: payment-service auto-offset-reset: earliest key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: "[spring.deserializer.key.delegate.class]": org.apache.kafka.common.serialization.StringDeserializer "[spring.deserializer.value.delegate.class]": org.springframework.kafka.support.serializer.JsonDeserializer "[spring.json.value.default.type]": com.example.payments.OrderPlaced "[spring.json.use.type.headers]": false "[spring.json.trusted.packages]": com.example.payments producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer # The value serializer is set in KafkaErrorHandlingConfig (JSON for objects, raw bytes for dead letters). listener: ack-mode: recordWhy it is written this way
- Both deserializers are
ErrorHandlingDeserializer. The real ones are named by the twodelegate.classproperties, which are the values ofErrorHandlingDeserializer.KEY_DESERIALIZER_CLASSandVALUE_DESERIALIZER_CLASS. - The square brackets with quotes stop Spring Boot from splitting the dotted Kafka property names into nested YAML keys.
spring.json.use.type.headers: falsemakes the deserializer ignore the type header the order service's serializer may add and always buildcom.example.payments.OrderPlaced. Without it, a header naming the producer's own class causes a failure that has nothing to do with the payload.ack-mode: recordcommits after every record. The default,batch, commits once per poll. Either works with the error handler;recordshortens what is redelivered after a crash.- Spring for Apache Kafka sets
enable.auto.committo false unless you set it yourself, so offsets move only when the container commits them.
package com.example.payments;import java.nio.charset.StandardCharsets;import java.time.Duration;import java.util.Map;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.TopicPartition;import org.apache.kafka.common.config.TopicConfig;import org.apache.kafka.common.header.Headers;import org.apache.kafka.common.header.internals.RecordHeaders;import org.apache.kafka.common.serialization.ByteArraySerializer;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.beans.factory.annotation.Value;import org.springframework.boot.autoconfigure.kafka.DefaultKafkaProducerFactoryCustomizer;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;import org.springframework.kafka.core.DefaultKafkaProducerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;import org.springframework.kafka.listener.DefaultErrorHandler;import org.springframework.kafka.listener.RetryListener;import org.springframework.kafka.support.ExponentialBackOffWithMaxRetries;import org.springframework.kafka.support.serializer.DelegatingByTypeSerializer;import org.springframework.kafka.support.serializer.JsonSerializer;@Configurationpublic class KafkaErrorHandlingConfig { public static final String ORDERS_TOPIC = "orders.order-placed"; public static final String ORDERS_DLT = "orders.order-placed-dlt"; private static final Logger log = LoggerFactory.getLogger(KafkaErrorHandlingConfig.class); /** Same partition count as the source topic, because the resolver keeps the partition number. */ @Bean NewTopic ordersDeadLetterTopic(@Value("${payments.topics.replicas:3}") int replicas) { return TopicBuilder.name(ORDERS_DLT) .partitions(12) .replicas(replicas) .config(TopicConfig.RETENTION_MS_CONFIG, String.valueOf(Duration.ofDays(30).toMillis())) .build(); } /** * One producer for everything this service sends: events as JSON, and the untouched * bytes of a record that could not be deserialized. */ @Bean DefaultKafkaProducerFactoryCustomizer jsonOrRawBytesValueSerializer() { return KafkaErrorHandlingConfig::useJsonOrRawBytes; } @SuppressWarnings("unchecked") private static void useJsonOrRawBytes(DefaultKafkaProducerFactory<?, ?> factory) { DelegatingByTypeSerializer serializer = new DelegatingByTypeSerializer( Map.of(byte[].class, new ByteArraySerializer(), Object.class, new JsonSerializer<Object>()), true); ((DefaultKafkaProducerFactory<Object, Object>) factory).setValueSerializer(serializer); } @Bean DeadLetterPublishingRecoverer deadLetterPublishingRecoverer(KafkaTemplate<Object, Object> template) { DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template, KafkaErrorHandlingConfig::deadLetterDestination); // Wait for the broker's answer. If the publish fails, the recoverer throws and // the source record is retried instead of being skipped. recoverer.setFailIfSendResultIsError(true); recoverer.setHeadersFunction((record, exception) -> { Headers extra = new RecordHeaders(); extra.add("x-failed-in", "payment-service".getBytes(StandardCharsets.UTF_8)); return extra; }); return recoverer; } /** The topic name is decided here, in this repository, and not by a library default. */ private static TopicPartition deadLetterDestination(ConsumerRecord<?, ?> record, Exception exception) { return new TopicPartition(record.topic() + "-dlt", record.partition()); } /** Spring Boot attaches a single CommonErrorHandler bean to the default container factory. */ @Bean DefaultErrorHandler kafkaErrorHandler(DeadLetterPublishingRecoverer recoverer) { // First delivery plus four redeliveries, after 1 s, 2 s, 4 s and 8 s: 15 s in total. ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(4); backOff.setInitialInterval(1_000L); backOff.setMultiplier(2.0); backOff.setMaxInterval(10_000L); DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff); // Wrong for ever: do not spend fifteen seconds of the partition on them. handler.addNotRetryableExceptions(InvalidOrderException.class); handler.setRetryListeners(new RetryListener() { @Override public void failedDelivery(ConsumerRecord<?, ?> record, Exception ex, int deliveryAttempt) { log.warn("Delivery {} failed for {}-{}@{}: {}", deliveryAttempt, record.topic(), record.partition(), record.offset(), ex.toString()); } @Override public void recovered(ConsumerRecord<?, ?> record, Exception ex) { log.error("Dead-lettered {}-{}@{}", record.topic(), record.partition(), record.offset(), ex); } @Override public void recoveryFailed(ConsumerRecord<?, ?> record, Exception original, Exception failure) { log.error("Dead-letter publish failed for {}-{}@{}, the record will be retried", record.topic(), record.partition(), record.offset(), failure); } }); return handler; }}Why it is written this way
- No container factory is declared. Spring Boot's auto-configuration adds a
CommonErrorHandlerbean to its defaultConcurrentKafkaListenerContainerFactory. If you declare your own factory, callfactory.setCommonErrorHandler(handler)yourself. DefaultKafkaProducerFactoryCustomizerchanges the auto-configured producer factory instead of replacing it, so Boot's connection settings, including a Testcontainers@ServiceConnection, still apply. Withassignableset to true the most specific matching type wins:byte[]goes out unchanged, every other object as JSON.- The dead-letter topic gets 12 partitions like its source and 30 days of retention, long enough to investigate and replay. The replica count is a property so that a single-broker test can set it to 1;
min.insync.replicasis left to the cluster's default. - The resolver returns the same name the 3.3 default would. Writing it down keeps the name stable across upgrades (the default was
.DLTbefore 3.3) and gives you one place to route by exception type later. ExponentialBackOffWithMaxRetries(4)means four retries after the first delivery. The plainExponentialBackOffretries without limit unless you compute a maximum elapsed time, which is why this subclass exists.- The classifier looks through exception causes, so
InvalidOrderExceptionis recognised even though the container wraps it in aListenerExecutionFailedException. - The
RetryListenerlogs coordinates and the exception, never the payload. It is also the natural place to increment a counter for dead-lettered records. setFailIfSendResultIsError(true)is already the value in the 3.3 source. It is set here because the whole design depends on it.
package com.example.payments;import java.util.Set;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderPlacedListener { private static final Set<String> CURRENCIES = Set.of("EUR", "GBP", "USD"); private final PaymentProcessor processor; public OrderPlacedListener(PaymentProcessor processor) { this.processor = processor; } @KafkaListener(id = "payment-orders", topics = KafkaErrorHandlingConfig.ORDERS_TOPIC, groupId = "payment-service") public void onOrderPlaced(OrderPlaced event) { if (event.amountMinor() <= 0 || !CURRENCIES.contains(event.currency())) { // Permanent: marked not retryable, so it is dead-lettered on the first failure. throw new InvalidOrderException("Order " + event.orderId() + " has an unusable amount or currency"); } // May throw PaymentProviderUnavailableException: transient, retried with back-off. processor.handle(event); }}class InvalidOrderException extends RuntimeException { InvalidOrderException(String message) { super(message); }}class PaymentProviderUnavailableException extends RuntimeException { PaymentProviderUnavailableException(String message, Throwable cause) { super(message, cause); }}Why it is written this way
- The listener contains no try/catch and no acknowledgement. Returning means done; throwing hands the record to the error handler. Catching and logging an exception here would tell the container the record succeeded.
- Two exception types carry the whole policy: one for records that are wrong, one for dependencies that are down. Anything else, such as an unexpected
NullPointerException, gets the retries and then the dead-letter topic. OrderPlacedis the record from the example project:eventId,orderId,customerId,amountMinor,currency,occurredAt.PaymentProcessor.handlemust be idempotent, because every retry calls it again.- The
idnames the listener container. It appears in metrics tags and lets a test find the container in theKafkaListenerEndpointRegistry.
package com.example.notifications;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.annotation.DltHandler;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.annotation.RetryableTopic;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.messaging.handler.annotation.Header;import org.springframework.retry.annotation.Backoff;import org.springframework.scheduling.TaskScheduler;import org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler;import org.springframework.stereotype.Component;@Componentpublic class PaymentResultNotifier { private static final Logger log = LoggerFactory.getLogger(PaymentResultNotifier.class); private final EmailGateway email; public PaymentResultNotifier(EmailGateway email) { this.email = email; } @RetryableTopic( attempts = "4", backoff = @Backoff(delay = 10_000, multiplier = 6.0, maxDelay = 600_000), exclude = UnknownRecipientException.class, numPartitions = "12", replicationFactor = "3") @KafkaListener(id = "notify-payment-result", topics = "payments.payment-result", groupId = "notification-service") public void onPaymentResult(PaymentResult event) { // Throws EmailGatewayException when the provider rejects or times out. email.sendPaymentConfirmation(event); } @DltHandler public void onDeadLetter(PaymentResult event, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic, @Header(KafkaHeaders.OFFSET) long offset) { log.error("No confirmation sent for order {}; parked at {}@{}", event.orderId(), topic, offset); }}@Configurationclass RetryTopicSchedulerConfig { /** The retry containers resume paused partitions through a TaskScheduler bean. */ @Bean TaskScheduler retryTopicTaskScheduler() { return new ThreadPoolTaskScheduler(); }}Why it is written this way
- Four attempts with delays of 10 s, 60 s and 360 s create
payments.payment-result-retry-10000,-retry-60000,-retry-360000andpayments.payment-result-dlt. Retry topics are suffixed with the delay value by default. @Backoffisorg.springframework.retry.annotation.Backofffrom Spring Retry in the 3.3 line, sospring-retrymust be on the classpath. The default maximum delay for an exponential policy is 30 seconds;maxDelayraises it.numPartitionsdefaults to 1. A main topic with 12 partitions and retry topics with one makes the framework's default (same partition as the source record) point at partitions that do not exist, so set it to the source topic's count.excludesends that exception type to the DLT without passing through the retry topics. Deserialization and conversion failures already do that.- The framework forwards records with a
KafkaTemplate: the bean named in thekafkaTemplateattribute, else one calleddefaultRetryTopicKafkaTemplate, else the only template in the context, which is the case here with Spring Boot. - In the 3.3 source the retry infrastructure requires a
TaskScheduler(or aRetryTopicSchedulerWrapper) bean. Spring Boot only creates a scheduler when scheduling is enabled, so one is declared explicitly. - The retry and DLT listeners get group ids derived from the
groupIdgiven here plus the topic suffix. Always setgroupId; without it they share one group and a rebalance on a retry topic disturbs the main topic. - Each retry delivery is a new record in another topic.
email.sendPaymentConfirmationmust be safe to call twice for the same payment.
package com.example.payments;import static org.assertj.core.api.Assertions.assertThat;import static org.mockito.Mockito.timeout;import static org.mockito.Mockito.verify;import java.nio.charset.StandardCharsets;import java.time.Duration;import java.time.Instant;import java.util.Map;import java.util.UUID;import org.apache.kafka.clients.consumer.Consumer;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.serialization.ByteArrayDeserializer;import org.apache.kafka.common.serialization.StringDeserializer;import org.junit.jupiter.api.Test;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.boot.test.context.SpringBootTest;import org.springframework.kafka.core.DefaultKafkaConsumerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.kafka.test.EmbeddedKafkaBroker;import org.springframework.kafka.test.context.EmbeddedKafka;import org.springframework.kafka.test.utils.KafkaTestUtils;import org.springframework.test.annotation.DirtiesContext;import org.springframework.test.context.bean.override.mockito.MockitoBean;@SpringBootTest@DirtiesContext@EmbeddedKafka(kraft = true, partitions = 12, topics = { KafkaErrorHandlingConfig.ORDERS_TOPIC, KafkaErrorHandlingConfig.ORDERS_DLT })class DeadLetterRoutingTest { @Autowired private EmbeddedKafkaBroker broker; @Autowired private KafkaTemplate<Object, Object> template; @MockitoBean private PaymentProcessor processor; @Test void undecodableRecordIsDeadLetteredAndTheNextRecordIsProcessed() { byte[] garbage = "{not json".getBytes(StandardCharsets.UTF_8); OrderPlaced good = new OrderPlaced(UUID.randomUUID(), UUID.randomUUID(), "customer-7", 4_500L, "EUR", Instant.parse("2026-01-15T10:00:00Z")); // Partition 3 for both, so the good record is behind the bad one. template.send(KafkaErrorHandlingConfig.ORDERS_TOPIC, 3, "order-bad", garbage); template.send(KafkaErrorHandlingConfig.ORDERS_TOPIC, 3, good.orderId().toString(), good); Map<String, Object> props = KafkaTestUtils.consumerProps(broker.getBrokersAsString(), "dlt-test"); props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); try (Consumer<String, byte[]> dltConsumer = new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new ByteArrayDeserializer()).createConsumer()) { broker.consumeFromAnEmbeddedTopic(dltConsumer, KafkaErrorHandlingConfig.ORDERS_DLT); ConsumerRecord<String, byte[]> dead = KafkaTestUtils.getSingleRecord( dltConsumer, KafkaErrorHandlingConfig.ORDERS_DLT, Duration.ofSeconds(20)); assertThat(dead.key()).isEqualTo("order-bad"); assertThat(dead.value()).isEqualTo(garbage); assertThat(dead.partition()).isEqualTo(3); assertThat(new String(dead.headers().lastHeader(KafkaHeaders.DLT_ORIGINAL_TOPIC).value(), StandardCharsets.UTF_8)).isEqualTo(KafkaErrorHandlingConfig.ORDERS_TOPIC); assertThat(dead.headers().lastHeader(KafkaHeaders.DLT_EXCEPTION_MESSAGE)).isNotNull(); } // The partition was not blocked: the record behind the bad one reached the processor. verify(processor, timeout(20_000)).handle(good); }}Why it is written this way
- The test sends real bytes through a real broker, so the deserializer wrapper, the error handler, the recoverer and the serializer delegation are all exercised. A unit test of the listener cannot cover any of them.
- Both topics are created by the embedded broker with 12 partitions, so the application's
NewTopicbean for the dead-letter topic finds it already present and is ignored. @EmbeddedKafka(kraft = true)selects the KRaft-based embedded broker. In the 3.3 line the annotation's default is still the ZooKeeper-based one. Since 3.0.10 the embedded broker setsspring.kafka.bootstrap-serversfor the Spring Boot application by itself.- The same
KafkaTemplatethe application uses sends both the garbage and the valid event, which also proves theDelegatingByTypeSerializerwiring. getSingleRecordwith a timeout polls until the record arrives or the time is up, and fails if there is not exactly one record. NoThread.sleep.verify(..., timeout(...))waits for the asynchronous listener. It asserts the second half of the claim: after the dead-letter publish the container moved on.@MockitoBeanis the Spring Framework 6.2 replacement for the deprecated@MockBean. For a second test, make the mock throwPaymentProviderUnavailableExceptionand asserttimes(5)before the dead letter appears; lower the back-off through a test property to keep it fast.
Configuration
The settings that shape the error path. Spring behaviour is from the Spring for Apache Kafka 3.3 reference and Javadoc; consumer defaults are those of kafka-clients 3.9, the client Spring Boot 3.5 manages.
DefaultErrorHandler back-off (constructor argument)Default FixedBackOff(0L, 9): ten deliveries, no delay, then log at ERROR and skip
- What it does
- How many times a failed record is redelivered and how long the consumer thread waits before each redelivery.
- When to change it
- Always replace it. Use a bounded exponential back-off for transient failures and keep every wait well below
max.poll.interval.ms. For waits longer than that, useContainerPausingBackOffHandleror retry topics.
DefaultErrorHandler.addNotRetryableExceptionsDefault DeserializationException, MessageConversionException, ConversionException, MethodArgumentResolutionException, NoSuchMethodException, ClassCastException
- What it does
- Exception types for which the recoverer is called on the first failure.
- When to change it
- Add your own validation and business-rule exceptions. Do not add exceptions that mean a dependency is down; those are the ones retries exist for.
DeadLetterPublishingRecoverer destination resolverDefault <source topic>-dlt, same partition (the suffix was .DLT before 3.3)
- What it does
- A function from the failed record and the exception to the topic and partition of the dead-letter record.
- When to change it
- Provide it explicitly. Give the dead-letter topic at least as many partitions as the source, or return a negative partition and let the producer choose.
spring.deserializer.value.delegate.classDefault none
- What it does
- The real value deserializer that
ErrorHandlingDeserializerwraps. There is a matchingspring.deserializer.key.delegate.class. - When to change it
- Set both whenever the deserializer can fail, which includes every JSON, Avro or Protobuf deserializer. Then make sure the dead-letter producer can serialize
byte[].
spring.kafka.listener.ack-modeDefault batch
- What it does
- When the container commits offsets: after all records of a poll (
batch), after each record (record), or under your control (manual,manual_immediate). - When to change it
recordreduces redelivery after a crash at the cost of more commits. Withmanual_immediate, setcommitRecoveredon the error handler so that a recovered record's offset is committed.
max.poll.interval.ms (consumer)Default 300000 (5 minutes)
- What it does
- The longest allowed gap between two calls to
poll()before the consumer is removed from the group. - When to change it
- It is the ceiling for blocking retries. Prefer shorter back-offs to raising it; a larger value also delays the detection of a truly stuck consumer.
ContainerProperties.deliveryAttemptHeaderDefault false
- What it does
- Adds the
kafka_deliveryAttemptheader (KafkaHeaders.DELIVERY_ATTEMPT), counting from 1, to records given to a record listener. - When to change it
- Turn it on if the listener or a log line needs the attempt number. It counts blocking attempts in this consumer instance only; it resets after a rebalance or restart.
@RetryableTopic attempts / backoffDefault attempts 3 (including the first), fixed delay of 1000 ms
- What it does
- The total number of deliveries across the main and retry topics, and the delay before each retry delivery.
- When to change it
- Size the delays to the outage you expect to ride out. The same values are available without the annotation through
spring.kafka.retry.topic.*(disabled by default).
@RetryableTopic numPartitions / replicationFactorDefault 1 / -1 (the broker default)
- What it does
- How the retry and dead-letter topics are created when
autoCreateTopicsis true (the default). - When to change it
- Match the source topic's partition count. In production many teams set
autoCreateTopicsto false and create the topics with the same tooling as every other topic.
@RetryableTopic dltStrategyDefault ALWAYS_RETRY_ON_ERROR
- What it does
- What happens when the
@DltHandlermethod itself throws: the record is sent to the DLT again, the listener fails (FAIL_ON_ERROR), or there is no DLT at all (NO_DLT). - When to change it
- Keep the DLT handler trivial (log, count, store) so that the default does not turn into a loop. Fatal exceptions such as a deserialization failure are not sent back.
Hands-on exercise
Predict the path of three records, then check in the lab
Work out from the configuration alone where each failing record ends up and how long the partition waits, then confirm the behaviour in the poison-message lab.
Steps · 0 of 6 done
What you should see
The transient failure is delivered 5 times over 15 seconds (waits of 1, 2, 4 and 8 seconds) and then appears in orders.order-placed-dlt on the same partition number. The invalid amount is delivered once and dead-lettered at once. The undecodable record never reaches the listener and arrives in the dead-letter topic as its original bytes. In the lab, only the poison record's partition builds lag, and after the fix that lag is a short bump instead of a line that keeps rising.
What can go wrong
Catching the exception in the listener
- Symptom
- Errors appear in the log, no record ever reaches the dead-letter topic, and downstream data is missing for exactly the records that logged an error.
- Cause
- A try/catch around the listener body logs and returns. To the container a normal return means success, so the offset is committed.
- Fix
- Let exceptions leave the listener. Translate them into your transient or permanent exception types if you need to, but rethrow.
Dead letters that are not the original message
- Symptom
- Records in the dead-letter topic for deserialization failures are quoted Base64 strings, or the dead-letter publish itself fails with a serialization error and the consumer stalls.
- Cause
- The recoverer restores the value to the raw
byte[], but the template's only value serializer is a JSON or Avro serializer. - Fix
- Use
DelegatingByTypeSerializerwith aByteArraySerializerforbyte[], or give the recoverer a second template forbyte[]values.
Back-off longer than the poll interval
- Symptom
- During a dependency outage the group rebalances every few minutes, the same records are processed by different instances, and the dead-letter topic receives records after far more attempts than configured.
- Cause
- The consumer thread sleeps in the back-off and misses
max.poll.interval.ms. The partition moves to another instance, which starts the attempt count again. - Fix
- Shorten the blocking back-off, or move long waits to
@RetryableTopicor aContainerPausingBackOffHandler, which keeps polling while paused.
Retry topics for events that must stay in order
- Symptom
- An order is marked SHIPPED and then, minutes later, PAID. The later state overwrote the newer one.
- Cause
@RetryableTopicmoved the failed PAID event to a retry topic and the listener went on to SHIPPED for the same key.- Fix
- Use blocking retries where per-key order matters, or make the consumer reject stale events by version. See ordering, shutdown and recovery.
Retry and dead-letter topics with one partition
- Symptom
- Retried records for a 12-partition topic all land in partition 0 of the retry topics; retry lag is high while most consumer threads sit idle.
- Cause
@RetryableTopicauto-creates its topics withnumPartitions = 1unless told otherwise.- Fix
- Set
numPartitionsto the source topic's partition count, or create the topics yourself and setautoCreateTopics = "false".
A whole batch in the dead-letter topic
- Symptom
- Hundreds of perfectly valid records appear in the dead-letter topic at the same second as one invalid one.
- Cause
- A batch listener threw an ordinary exception. The handler retried the complete batch and then recovered every record in it.
- Fix
- Catch per record inside the loop and throw
BatchListenerFailedExceptionwith the index of the failing record.
Expecting the error handler in a transactional container
- Symptom
- After adding
transaction-id-prefix, failed records are redelivered ten times and dropped; theDefaultErrorHandlerbean seems to be ignored. - Cause
- With a Kafka transaction manager on the container, a failure rolls the transaction back and the
AfterRollbackProcessordecides what happens next. Its default is also ten attempts, then log. - Fix
- Configure a
DefaultAfterRollbackProcessorwith the same recoverer and back-off. The transactions lesson shows it.
How to troubleshoot
One partition's lag grows while the others stay near zero
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- A
CURRENT-OFFSETthat stands still on one partition whileLOG-END-OFFSETrises means one record is being retried or cannot be recovered. Look for the service's log lines for that partition and offset.
The same stack trace repeats and the offset never moves, even though a dead-letter recoverer is configured
Check
Service log: the RetryListener line `Dead-letter publish failed for ...`, or a producer error such as TimeoutException or UnknownTopicOrPartitionException close to it
- How to read it
- The recoverer is throwing, so the record is kept. Usual causes: the dead-letter topic does not exist and auto-creation is off, the principal has no write permission on it, or the value cannot be serialized.
A record was skipped and nobody knows why
Check
Service log at ERROR from the framework's failure tracker: a line starting `Backoff` and ending `exhausted for <topic>-<partition>@<offset>`
- How to read it
- That is the default recoverer: the record was logged and skipped. Either no
DefaultErrorHandlerbean with a recoverer exists, or the listener uses a container factory the bean was never attached to.
You need to know why a record was dead-lettered
Check
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-placed-dlt --from-beginning --formatter-property print.headers=true --formatter-property print.key=true
- How to read it
- Read
kafka_dlt-original-topic,kafka_dlt-exception-fqcnandkafka_dlt-exception-message. The original partition, offset and timestamp headers are binary numbers and print as raw bytes; the dead-letter record normally sits in the same partition number as its source.
The consumer logs a deserialization error for ever and the listener is never called
Check
spring.kafka.consumer.value-deserializer in the effective configuration, and the log for RecordDeserializationException
- How to read it
- The deserializer is not wrapped. The failure happens inside
poll(), where no error handler can act on the record. ConfigureErrorHandlingDeserializerwith the real deserializer as its delegate.
Key takeaways
- Configure nothing and a failing record is tried ten times without delay, logged and skipped. Always declare a
DefaultErrorHandlerwith a back-off and a recoverer. - Classify exceptions: transient ones get a bounded back-off, permanent ones are marked not retryable and go to the dead-letter topic at once.
ErrorHandlingDeserializerturns an undecodable payload into a record the error handler can route. The dead-letter producer must then be able to send raw bytes.- Name the dead-letter topic in your own destination resolver, give it the source topic's partition count, and wait for the publish result before the offset moves.
- Blocking retries keep order and block the partition for seconds.
@RetryableTopickeeps the partition flowing for minutes or hours and gives up order. - In a batch listener, throw
BatchListenerFailedExceptionwith the index. In a transactional container, the after-rollback processor takes the error handler's role. - Test the error path against a broker: publish a bad record, read it from the dead-letter topic, and assert that the next record was processed.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A payment listener uses the default Spring configuration: no error handler bean. The card provider is unreachable for 40 seconds during a deploy. Afterwards 310 orders from that window have no payment row, lag is zero, and the log holds 3,100 stack traces followed by 310 ERROR lines naming a topic, partition and offset.
What explains the missing payments?
Scenario 2 of 3
A service adds ErrorHandlingDeserializer around its JsonDeserializer and a DeadLetterPublishingRecoverer using the application's KafkaTemplate, whose value serializer is JsonSerializer. A producer bug then writes 20 truncated payloads. The consumer does not stall, but support cannot replay the dead letters: each one is a quoted string of letters and digits.
Which change makes the dead letters replayable?
Scenario 3 of 3
An inventory listener consumes orders.order-placed (12 partitions, keyed by orderId) and must apply events for one order in sequence. The warehouse API fails for 20 to 30 minutes about once a month. With a 15 second blocking back-off, each outage sends around 9,000 records to the dead-letter topic, which are replayed by hand.
Which approach fits both the ordering need and the outage length?
References
References: Spring for Apache Kafka 3.3 reference: Handling Exceptions (DefaultErrorHandler, batch error handling, after-rollback processor, publishing dead-letter records) ↗ · Spring for Apache Kafka 3.3 reference: Serialization, Deserialization, and Message Conversion (Using ErrorHandlingDeserializer, DelegatingByTypeSerializer) ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries (how the pattern works) ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries, features (back-off defaults, exception classifier, topic auto-creation) ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries, topic naming (default suffixes, delay-suffixed retry topics) ↗ · Spring for Apache Kafka 3.3 Javadoc: DeadLetterPublishingRecoverer ↗ · Spring for Apache Kafka 3.3 Javadoc: RetryableTopic (attribute names and defaults) ↗ · Spring Boot 3.5 reference: Apache Kafka support (beans attached to the auto-configured container factory) ↗ · Spring Boot 3.5: Managed dependency coordinates (spring-kafka 3.3.x, kafka-clients 3.9.x) ↗ · Apache Kafka 3.9 consumer configuration reference (max.poll.interval.ms) ↗ · Report a problem