Path 6 · lesson 4 of 5
Dead letter topics: park, inspect, replay
What belongs in a dead letter topic, what Spring records with each failed record, how to size and alert on it, and how to replay it without creating new damage.
What you will learn
- Decide which failures go to a dead letter topic and which must never be sent there
- Configure DefaultErrorHandler with a DeadLetterPublishingRecoverer and predict the topic, partition and headers of a dead letter record
- Explain what the consumer does when publishing to the dead letter topic itself fails
- Size a dead letter topic, set its retention and alert on the right signal
- Build a replay path that is safe to run twice and state what it does to ordering
Before you start
The concept
Message brokers such as RabbitMQ or SQS have a dead letter queue built in. Kafka does not offer one to an ordinary consumer. In Kafka the same idea is a dead letter topic (DLT): a normal topic that your consumer application publishes a record to when it has given up on processing it. The broker treats it like any other topic. Every rule in this lesson is therefore something your code and your team enforce, not something Kafka enforces for you.
The purpose follows from how a partition is consumed. A consumer reads a partition in offset order and its committed offset is one number: the next record to read. It cannot acknowledge record 41 and leave record 40 open. So when record 40 can never be processed, there are only three choices: retry it forever and block everything behind it, skip it and lose it, or move it somewhere else and then advance. A dead letter topic is the third choice. It turns a stuck partition into a recorded, recoverable exception.
| Failure | Goes to the DLT? | Why |
|---|---|---|
| Payload cannot be deserialized (wrong schema, truncated bytes) | Yes, immediately | Retrying reads the same bytes and fails the same way. |
| Record fails business validation (negative amount, unknown currency) | Yes, immediately | The record is wrong, and time will not fix it. |
| A bug in the consumer throws for this record shape | Yes, after a few retries | Permanent until a fix is deployed. Park it, fix, then replay. |
| Retries exhausted on an error that looked transient | Yes | You have spent your retry budget. Blocking the partition longer costs more than parking one record. |
| Database, downstream API or broker is unavailable | No | Every record will fail. The DLT fills with healthy records and the outage becomes a replay project. |
| Record already processed (a duplicate) | No | That is a success. An idempotent consumer ignores it and commits. |
A dead letter record is only useful if someone can later answer three questions from it: where did it come from, why did it fail, and what should be done with it. So the record must carry its origin and the failure with it, and the team must own a path that reads the topic. A DLT nobody reads is a slower way of losing data.
Why it exists
Before a team adds a dead letter topic, a single bad record usually shows up in one of two ways. Either the consumer retries it in a loop, the partition stops moving and lag on that partition grows without bound (the Break card the message that kills its reader is a real case). Or someone wraps the listener in a try/catch that logs and continues, the partition keeps moving, and the failed record now exists only as a log line that will be rotated away.
Without a DLT
- Retry forever: one record blocks every key that shares its partition.
- Catch and log: the record is skipped, and nothing can replay it.
- The only evidence is an application log with a limited lifetime.
- Recovery means resetting offsets and reprocessing thousands of good records to reach one bad one.
With a DLT
- The partition advances after a bounded number of attempts.
- The failed record is stored with the same durability as any topic.
- Origin and exception travel with the record as headers.
- Recovery handles exactly the failed records, when you choose.
The cost is real and should be accepted knowingly. Parking a record means later records for the same key are processed before it, so strict per-key ordering is given up for that key. And the consumer now writes to Kafka as well as reading, so it needs a producer, permissions on the dead letter topic and a decision about what to do when that write fails.
How it works inside
How Spring decides to give up
In Spring for Apache Kafka the listener container calls a CommonErrorHandler when your @KafkaListener method throws. The standard one is DefaultErrorHandler. It is built from two parts: a BackOff that says how many times to redeliver and how long to wait, and a recoverer that is called once the back off is exhausted. If you configure nothing, the back off is FixedBackOff(0L, 9), which is ten deliveries with no delay, and the recoverer only logs the record. That default silently skips the record after ten fast attempts, which is rarely what a team intends.
Retries here are blocking: after a failure the handler seeks the consumer back so that the failed record and everything after it in the poll is fetched again, and waits out the back off on the consumer thread. Some exceptions skip the retries and go straight to the recoverer, because repeating them cannot help. By default these are DeserializationException, MessageConversionException, ConversionException, MethodArgumentResolutionException, NoSuchMethodException and ClassCastException. Add your own permanent failures with addNotRetryableExceptions(...).
What DeadLetterPublishingRecoverer does
- Resolves the destination. A destination resolver function receives the failed
ConsumerRecordand the exception and returns aTopicPartition. The default returns the original topic name with the suffix-dltand the same partition number as the failed record. - Checks the partition. Because of that default, the dead letter topic needs at least as many partitions as the source topic. If the chosen partition does not exist, the recoverer clears the partition and lets the producer choose one (the
verifyPartitioncheck, on by default). A resolver that returns a negative partition asks for the same thing deliberately. - Builds the record. Key and value are the original ones. The original headers are kept and the dead letter headers are added.
- Sends and waits. It publishes with the
KafkaTemplateyou gave it and, withfailIfSendResultIsError(true in the 3.3 source), waits for the broker's answer. The wait is the producer'sdelivery.timeout.msplus a 5 second buffer. - Returns, and the offset moves. Only after the recoverer returns normally does the container treat the record as handled and commit its offset.
The headers Spring adds
| Header name on the wire | KafkaHeaders constant | Content |
|---|---|---|
kafka_dlt-original-topic | DLT_ORIGINAL_TOPIC | Source topic, UTF-8 text |
kafka_dlt-original-partition | DLT_ORIGINAL_PARTITION | Source partition, 4-byte integer |
kafka_dlt-original-offset | DLT_ORIGINAL_OFFSET | Source offset, 8-byte long |
kafka_dlt-original-timestamp | DLT_ORIGINAL_TIMESTAMP | Source record timestamp, 8-byte long |
kafka_dlt-original-timestamp-type | DLT_ORIGINAL_TIMESTAMP_TYPE | CreateTime or LogAppendTime |
kafka_dlt-original-consumer-group | DLT_ORIGINAL_CONSUMER_GROUP | The group whose listener failed |
kafka_dlt-exception-fqcn | DLT_EXCEPTION_FQCN | Exception class, usually ListenerExecutionFailedException |
kafka_dlt-exception-cause-fqcn | DLT_EXCEPTION_CAUSE_FQCN | Class of the cause: your own exception |
kafka_dlt-exception-message | DLT_EXCEPTION_MESSAGE | Exception message |
kafka_dlt-exception-stacktrace | DLT_EXCEPTION_STACKTRACE | Full stack trace as text |
The stack trace header can be several kilobytes. That matters when a record fails more than once, for example after a replay. Two properties keep it bounded: stripPreviousExceptionHeaders (default true) replaces older exception headers, and appendOriginalHeaders (default true) keeps the history of original topics. You can drop a header entirely with excludeHeader(HeaderNames.HeadersToAdd.EX_STACKTRACE).
When publishing to the dead letter topic fails
This is the case that decides whether you lose data. If the send fails (the topic does not exist and cannot be auto-created, the principal has no write permission, the record plus headers is larger than the topic allows, the brokers are unreachable), the recoverer throws. The error handler then treats the record as not recovered: it is included in the seeks, fetched again, and by default the back off starts over (resetStateOnRecoveryFailure). The offset is not committed. The partition is blocked again, which is the correct outcome: a blocked partition is visible as lag, a record dropped between two topics is not.
Deserialization failures need one extra piece. A record that cannot be deserialized never reaches your listener, so the consumer's deserializer must be wrapped in ErrorHandlingDeserializer, which hands the failure to the error handler as a DeserializationException. The recoverer then publishes the original bytes as the value. Your dead letter producer must therefore be able to serialize both your normal type and a raw byte[].
With retry topics
@RetryableTopic (non-blocking retries, covered in the previous lesson) uses the same recoverer underneath and ends in the same kind of topic, <topic>-dlt. A method annotated with @DltHandler in the same class consumes it. Without one, a default handler only logs each dead letter record. If that handler itself throws, the default strategy ALWAYS_RETRY_ON_ERROR sends the record back to the end of the dead letter topic so it does not block the others, except for the fatal exception types listed above. Set autoStartDltHandler = "false" when the application should publish to the DLT but not consume it.
Retry topics & DLQ
Simulation, not a real clusterAn animated walk through a record moving from the main topic through retry topics into the dead letter topic.
Step 1 · The pattern
One at a time, in order
A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.
- message
- processed
Sizing, retention and alerting
- Partitions. With the default resolver, use the same partition count as the source topic. Then every failed record for a key lands in one dead letter partition, in the order the failures happened. If you add partitions to the source topic, add them to the DLT first.
- Replication. Use the same replication factor and
min.insync.replicasas the source topic. Dead letter records are the ones you have already failed once; they deserve the same durability. - Retention. A topic keeps data for
retention.ms, 7 days by default. Choose a value longer than the slowest realistic path from alert to fix to replay, including a long weekend. Many teams use 14 to 30 days. Keepcleanup.policy=delete: with compaction, a later failure for the same key would replace an earlier one. - Size. In a healthy system the DLT is nearly empty, so disk is not the concern. The concern is a burst: a bad deployment can send a large share of traffic there in minutes. A cap with
retention.bytesprotects the brokers but deletes the oldest dead letters first, so prefer an alert to a tight cap. - Alerting. The useful signal is the rate of new records, which is the growth of the topic's end offset. Any record in a topic that should be empty deserves a ticket; a sustained rate deserves a page, because it means a bug or a misclassified outage. Lag of the replay group is not a useful alert, because that group is normally stopped.
The reprocessing path
Publishing to a DLT is half of the feature. The other half is the way back, and it has four steps.
- Inspect. Read the records with their headers and group them by
kafka_dlt-exception-cause-fqcn. Ten thousand records usually collapse into two or three causes. - Decide per cause. A consumer bug: deploy the fix, then replay. Bad data from a producer: the producer republishes a corrected event, and the dead letter is discarded. Obsolete (the order was cancelled since): discard, and write down why.
- Replay. Republish to the original topic with the original key, so the record follows the normal path through the normal consumer. Do not build a second code path that processes dead letters directly; it will drift from the real one.
- Verify. Watch the DLT during the replay. Records that fail again return with a replay counter one higher, and a limit on that counter stops an endless loop.
Architecture
Drawing diagram…
Flowchart. The payment listener reads the orders topic. On an exception the DefaultErrorHandler either seeks back and redelivers, or hands the record to the DeadLetterPublishingRecoverer, which publishes it to orders-dlt. The offset is committed only when that send is confirmed. From the dead letter topic an alert fires, a person inspects and fixes, and a replay listener republishes records to the orders topic with the same key.
flowchart TD
SRC["orders topic, 6 partitions"] --> L["Payment listener"]
L -->|"processed"| C["Commit next offset"]
L -->|"throws"| EH["DefaultErrorHandler"]
EH --> Q{"Not retryable or retries exhausted"}
Q -->|"no"| RB["Seek back, wait for back off, redeliver"]
RB --> L
Q -->|"yes"| REC["DeadLetterPublishingRecoverer"]
REC --> DLT["orders-dlt topic, 6 partitions"]
REC -->|"send confirmed"| C
REC -->|"send failed"| RB
DLT --> AL["Alert on new records"]
DLT --> INS["Inspect headers, group by cause"]
INS --> FIX["Fix the consumer or the data"]
FIX --> RP["Replay listener, started by an operator"]
RP -->|"same key, replay counter"| SRC- There are two exits from the listener: success, and a confirmed write to the dead letter topic. Both end in the same commit. A failed write to the dead letter topic is not an exit; it loops back to redelivery.
- The dead letter topic has the same number of partitions as the source, so the default resolver can keep the partition number and failures for one key stay together.
- The return path goes through the source topic, not straight into the service. The replayed record is handled by the same listener, the same validation and the same idempotency check as any other record.
- The replay listener is a separate consumer group that is stopped by default. A person starts it after the fix is deployed, which is why its lag is not an alert.
Message flow
Drawing diagram…
Sequence diagram. The container polls offsets 40 to 44 of partition 2. The listener fails on offset 40. The error handler seeks back and redelivers three more times with growing delays. After the fourth failure the recoverer publishes the record to orders-dlt partition 2 and waits for the acknowledgement. The container then commits offset 41 and continues. Later an operator reads the dead letter topic and replays the record to the orders topic.
sequenceDiagram participant B as Broker participant C as Listener container participant L as Payment listener participant E as DefaultErrorHandler participant R as DLT recoverer participant O as Operator B->>C: poll returns orders-2 offsets 40 to 44 C->>L: offset 40, attempt 1 L-->>C: throws IllegalStateException C->>E: handle failure E->>C: seek orders-2 back to 40 Note over C,E: Back off 500 ms, then 1 s, then 2 s C->>L: offset 40, attempt 4 L-->>C: throws again E->>R: retries exhausted, recover R->>B: send to orders-dlt partition 2 with headers B-->>R: acknowledged C->>B: commit offset 41 for orders-2 C->>L: offset 41 and onwards O->>B: read orders-dlt, group by cause O->>B: after the fix, replay to orders with the same key
- The seek is why later records are not processed during the retries: offsets 41 to 44 were fetched, but the consumer's position is moved back to 40, so they are fetched again afterwards.
- The whole retry sequence happens on the consumer thread between polls of new data. Its total duration, here about 3.5 seconds plus four processing attempts, must stay well under
max.poll.interval.ms(5 minutes by default), or the group will consider the consumer failed and rebalance. - The commit is for offset 41, the next record to read. Nothing in Kafka marks offset 40 as failed. The only record of the failure is the dead letter record and its headers.
- If the process crashes after the dead letter write is acknowledged but before the commit, offset 40 is delivered again after restart, fails again and is published a second time. Dead letter topics can contain duplicates, so the replay must tolerate them.
In a real system
A typical payment service consumes the topic orders (6 partitions, replication factor 3, key = order id) in the consumer group payment-service, and charges the customer for each OrderPlaced event. Its dead letter topic is orders-dlt: 6 partitions, replication factor 3, retention.ms of 30 days, cleanup.policy=delete.
A new version of the order service starts sending amounts for a three-decimal currency in a shape the payment service does not expect. The payment listener throws IllegalStateException for those orders, about 40 a minute out of 9,000. After four attempts each record goes to orders-dlt, and every other order keeps flowing. The alert on new dead letter records fires within five minutes. An engineer reads ten records with their headers: all carry kafka_dlt-exception-cause-fqcn = java.lang.IllegalStateException, and the same currency.
The fix is deployed two hours later, by which time about 4,800 records are parked. The engineer starts the replay listener. Each record is republished to orders with its order id as key and x-replay-count = 1. The payment service handles them like any other order. Because it stores one payment per order id, the 30 orders that customers had already retried and paid through a second order event are recognised and skipped. Three records come back to orders-dlt with a different cause, a genuinely invalid amount, and are handed to the order team.
Java and Spring Boot
A payment service with blocking retries, a dead letter topic and an operator-started replay listener. The code follows the Spring for Apache Kafka 3.3 reference; it was written without a compiler at hand, so treat it as a careful template.
spring: application: name: payment-service kafka: bootstrap-servers: localhost:9092 consumer: group-id: payment-service auto-offset-reset: earliest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer # Wrap the real deserializer so a bad payload reaches the error handler # instead of failing inside poll() forever. value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer spring.json.value.default.type: com.example.payments.OrderEvent spring.json.use.type.headers: false spring.json.trusted.packages: com.example.payments listener: ack-mode: recordpayments: dlt: max-replays: 3Why it is written this way
ErrorHandlingDeserializerturns a deserialization failure into a record the error handler can route. Without it the consumer throws frompoll()at the same offset on every attempt and no dead letter is ever written.spring.json.use.type.headers: falsewith a default type makes the listener independent of type headers. A replayed record is then read the same way whether or not it carries them.ack-mode: recordcommits after each record, so a crash re-delivers at most the record in flight. The container default isBATCH, which commits after all records of a poll.
package com.example.payments.config;import java.nio.charset.StandardCharsets;import java.time.Duration;import java.util.Map;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.common.TopicPartition;import org.apache.kafka.common.config.TopicConfig;import org.apache.kafka.common.header.Header;import org.apache.kafka.common.header.Headers;import org.apache.kafka.common.header.internals.RecordHeaders;import org.apache.kafka.common.serialization.ByteArrayDeserializer;import org.apache.kafka.common.serialization.ByteArraySerializer;import org.apache.kafka.common.serialization.StringDeserializer;import org.apache.kafka.common.serialization.StringSerializer;import org.springframework.beans.factory.annotation.Value;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;import org.springframework.kafka.config.ContainerCustomizer;import org.springframework.kafka.config.TopicBuilder;import org.springframework.kafka.core.DefaultKafkaConsumerFactory;import org.springframework.kafka.core.DefaultKafkaProducerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.core.ProducerFactory;import org.springframework.kafka.listener.CommonContainerStoppingErrorHandler;import org.springframework.kafka.listener.ConcurrentMessageListenerContainer;import org.springframework.kafka.listener.ContainerProperties;import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;import org.springframework.kafka.listener.DefaultErrorHandler;import org.springframework.kafka.support.ExponentialBackOffWithMaxRetries;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.kafka.support.serializer.DelegatingByTypeSerializer;import org.springframework.kafka.support.serializer.JsonSerializer;import com.example.payments.InvalidOrderException;import com.example.payments.OrderEvent;@Configurationpublic class DeadLetterConfig { public static final String SOURCE_TOPIC = "orders"; public static final String DEAD_LETTER_TOPIC = "orders-dlt"; public static final String ATTEMPTS_HEADER = "x-delivery-attempts"; /** Same partition count and replication as the source topic; 30 days to inspect and replay. */ @Bean NewTopic ordersDeadLetterTopic() { return TopicBuilder.name(DEAD_LETTER_TOPIC) .partitions(6) .replicas(3) .config(TopicConfig.RETENTION_MS_CONFIG, String.valueOf(Duration.ofDays(30).toMillis())) .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") .build(); } /** One producer for normal events (JSON) and for payloads that never deserialized (raw bytes). */ @Bean ProducerFactory<String, Object> producerFactory( @Value("${spring.kafka.bootstrap-servers}") String bootstrapServers) { Map<String, Object> props = Map.of(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); return new DefaultKafkaProducerFactory<>(props, new StringSerializer(), new DelegatingByTypeSerializer(Map.of( byte[].class, new ByteArraySerializer(), OrderEvent.class, new JsonSerializer<Object>()))); } @Bean KafkaTemplate<String, Object> kafkaTemplate(ProducerFactory<String, Object> producerFactory) { return new KafkaTemplate<>(producerFactory); } @Bean DeadLetterPublishingRecoverer deadLetterPublishingRecoverer(KafkaTemplate<String, Object> template) { DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template, (record, exception) -> new TopicPartition(record.topic() + "-dlt", record.partition())); // Wait for the broker's answer; a failed publish must not let the offset advance. recoverer.setFailIfSendResultIsError(true); // Spring adds origin and exception headers, but no attempt count. Carry it over ourselves. recoverer.setHeadersFunction((record, exception) -> { Headers extra = new RecordHeaders(); Header attempt = record.headers().lastHeader(KafkaHeaders.DELIVERY_ATTEMPT); if (attempt != null) { extra.add(ATTEMPTS_HEADER, attempt.value()); } extra.add("x-failed-in", "payment-service".getBytes(StandardCharsets.UTF_8)); return extra; }); return recoverer; } /** Spring Boot applies a single CommonErrorHandler bean to the default listener container factory. */ @Bean DefaultErrorHandler errorHandler(DeadLetterPublishingRecoverer recoverer) { ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(3); backOff.setInitialInterval(500L); backOff.setMultiplier(2.0); backOff.setMaxInterval(5_000L); DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff); // Permanent by definition: do not spend retries, park at once. handler.addNotRetryableExceptions(InvalidOrderException.class); return handler; } /** Puts kafka_deliveryAttempt (a 4-byte int) on each record given to the listener. */ @Bean ContainerCustomizer<Object, Object, ConcurrentMessageListenerContainer<Object, Object>> deliveryAttempts() { return container -> container.getContainerProperties().setDeliveryAttemptHeader(true); } /** Separate factory for the replay listener: raw bytes in, stopped by default, stops on any error. */ @Bean ConcurrentKafkaListenerContainerFactory<String, byte[]> dltReplayContainerFactory( @Value("${spring.kafka.bootstrap-servers}") String bootstrapServers) { Map<String, Object> props = Map.of( ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers, ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest", ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 50); ConcurrentKafkaListenerContainerFactory<String, byte[]> factory = new ConcurrentKafkaListenerContainerFactory<>(); factory.setConsumerFactory( new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new ByteArrayDeserializer())); factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.RECORD); factory.setCommonErrorHandler(new CommonContainerStoppingErrorHandler()); factory.setAutoStartup(false); return factory; }}Why it is written this way
- The destination resolver is written out even though it matches the 3.3 default. The topic name is now fixed in your code and survives a library upgrade; before 3.3 the default suffix was
.DLT. ExponentialBackOffWithMaxRetries(3)gives four deliveries in total: the first plus three retries after 0.5 s, 1 s and 2 s. Keep the sum far belowmax.poll.interval.ms, because the consumer thread is busy for the whole sequence.InvalidOrderExceptionis marked not retryable, so a record that fails validation is parked on the first failure. Exceptions that mean a dependency is down should not be handled by this recoverer at all; give them long or unlimited retries, or pause the container.- The value serializer delegates by type:
OrderEventas JSON for ordinary failures,byte[]unchanged for records that failed deserialization and for the replay tool. A producer with only a JSON serializer would write a deserialization failure as a Base64 string. - Declaring your own
ProducerFactoryandKafkaTemplatemakes Spring Boot's auto-configured ones back off. The replay consumer factory is created inline and not as a bean, so Boot's defaultConsumerFactoryfor the main listener stays in place. - The replay factory uses
CommonContainerStoppingErrorHandler: if a replayed record cannot be republished, the container stops with the offset uncommitted. A replay that skips records on error would defeat its purpose.
package com.example.payments;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderEventListener { private final PaymentService payments; public OrderEventListener(PaymentService payments) { this.payments = payments; } @KafkaListener(id = "payment-orders", topics = "orders") public void onOrder(OrderEvent event) { if (event.amountMinor() <= 0 || event.currency() == null) { // Permanent: this exact record can never be charged. Goes to orders-dlt at once. throw new InvalidOrderException("Order " + event.orderId() + " has no chargeable amount"); } // Idempotent on orderId: a redelivery or a replay of an order that is already paid is a no-op. // Anything else it throws is retried by the error handler, then parked. payments.chargeOnce(event.orderId(), event.version(), event.amountMinor(), event.currency()); }}// OrderEvent.java// public record OrderEvent(String orderId, long version, long amountMinor, String currency) {}// InvalidOrderException.java// public class InvalidOrderException extends RuntimeException {// public InvalidOrderException(String message) { super(message); }// }Why it is written this way
- The listener contains no try/catch and no dead letter code. It classifies failures by the exception type it throws; routing is the error handler's job.
chargeOncemust be idempotent on the order id (a unique constraint on the payments table, or an idempotency key sent to the payment provider). That single property makes redelivery, a duplicate dead letter and a repeated replay all harmless.- The event carries a
version. The consumer uses it to ignore a replayed event that is older than the state it already has.
package com.example.payments.dlt;import java.nio.ByteBuffer;import java.nio.charset.StandardCharsets;import java.util.concurrent.TimeUnit;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.common.header.Header;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.beans.factory.annotation.Value;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.config.KafkaListenerEndpointRegistry;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.listener.MessageListenerContainer;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.stereotype.Component;import org.springframework.web.bind.annotation.PostMapping;import org.springframework.web.bind.annotation.RequestMapping;import org.springframework.web.bind.annotation.RestController;@Componentpublic class OrdersDltReplayer { static final String LISTENER_ID = "orders-dlt-replay"; static final String REPLAY_COUNT = "x-replay-count"; private static final Logger log = LoggerFactory.getLogger(OrdersDltReplayer.class); private final KafkaTemplate<String, Object> template; private final int maxReplays; public OrdersDltReplayer(KafkaTemplate<String, Object> template, @Value("${payments.dlt.max-replays}") int maxReplays) { this.template = template; this.maxReplays = maxReplays; } @KafkaListener(id = LISTENER_ID, topics = "orders-dlt", groupId = "orders-dlt-replay", containerFactory = "dltReplayContainerFactory", autoStartup = "false") public void replay(ConsumerRecord<String, byte[]> dead) throws Exception { Header origin = dead.headers().lastHeader(KafkaHeaders.DLT_ORIGINAL_TOPIC); Header counter = dead.headers().lastHeader(REPLAY_COUNT); int replays = counter == null ? 0 : ByteBuffer.wrap(counter.value()).getInt(); if (origin == null || replays >= maxReplays) { // Not replayable by a tool: leave it for a person. It stays in the DLT until retention removes it. log.warn("Not replaying {}-{}@{}: origin={}, replays={}", dead.topic(), dead.partition(), dead.offset(), origin != null, replays); return; } String originalTopic = new String(origin.value(), StandardCharsets.UTF_8); // No partition given: the producer partitions by key, exactly like the original producer did. ProducerRecord<String, Object> out = new ProducerRecord<>(originalTopic, null, dead.key(), dead.value()); out.headers().add(REPLAY_COUNT, ByteBuffer.allocate(Integer.BYTES).putInt(replays + 1).array()); out.headers().add("x-replayed-from", (dead.topic() + "-" + dead.partition() + "@" + dead.offset()).getBytes(StandardCharsets.UTF_8)); // Block until the broker confirms. Only then does the container commit this dead letter offset. template.send(out).get(30, TimeUnit.SECONDS); } /** Operator switch. Protect it like any other admin endpoint. */ @RestController @RequestMapping("/admin/dlt-replay") static class ReplayController { private final KafkaListenerEndpointRegistry registry; ReplayController(KafkaListenerEndpointRegistry registry) { this.registry = registry; } @PostMapping("/start") String start() { container().start(); return "replay started"; } @PostMapping("/stop") String stop() { container().stop(); return "replay stopped"; } private MessageListenerContainer container() { return registry.getListenerContainer(LISTENER_ID); } }}Why it is written this way
- The replay listener has
autoStartup = "false"and its own group,orders-dlt-replay. It runs only when an operator starts it after the fix is deployed; its committed offset remembers how far the previous replay got, so a second run does not start from the beginning. - Records are republished to the topic named in
kafka_dlt-original-topicwith the original key and no explicit partition. The default partitioner then picks the partition the key maps to today, which is still correct if the source topic has gained partitions since the failure. - The value is passed through as raw bytes, so the replayed payload is byte-for-byte what the original producer sent.
x-replay-counttravels with the record. If the record fails again, the recoverer keeps the original headers, so the counter comes back to the DLT and the next replay sees it. At the limit the tool stops touching the record.- If the process dies between the confirmed send and the commit, the same dead letter is replayed twice. That is at-least-once, and the reason the payment listener must be idempotent.
- This replays everything in the topic. A production tool usually adds a filter (one exception class, one time window) and a rate limit, so a large replay does not starve live traffic.
package com.example.payments;import org.springframework.kafka.annotation.DltHandler;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.annotation.RetryableTopic;import org.springframework.kafka.retrytopic.DltStrategy;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.messaging.handler.annotation.Header;import org.springframework.retry.annotation.Backoff;import org.springframework.stereotype.Component;@Componentpublic class RetryingOrderListener { @RetryableTopic(attempts = "4", backoff = @Backoff(delay = 1000, multiplier = 2.0), exclude = InvalidOrderException.class, dltStrategy = DltStrategy.FAIL_ON_ERROR) @KafkaListener(id = "payment-orders-retrying", topics = "orders", groupId = "payment-service") public void onOrder(OrderEvent event) { // ... same processing as above } @DltHandler public void onDeadLetter(OrderEvent event, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic, @Header(KafkaHeaders.OFFSET) long offset) { // Record a metric and an audit row. Do not attempt the business processing again here. }}Why it is written this way
- Use either this or the
DefaultErrorHandlerconfiguration above for a given listener, not both. Here failed records move throughorders-retry-*topics and end inorders-dlt; aKafkaTemplatebean nameddefaultRetryTopicKafkaTemplate(or the one named in the annotation) does the forwarding. excludesendsInvalidOrderExceptionstraight to the dead letter topic without visiting the retry topics.- With
DltStrategy.FAIL_ON_ERRORa failure inside the@DltHandleris not sent back to the DLT. The default,ALWAYS_RETRY_ON_ERROR, republishes it to the DLT, which can loop if the handler has a bug. - The Spring reference states plainly that this pattern gives up Kafka's ordering guarantee for the topic, because a retried record is processed after newer ones.
Configuration
The settings that decide what reaches the dead letter topic, how long it stays, and what happens when the write fails. Spring values are from the Spring for Apache Kafka 3.3 reference; topic and consumer values from the Kafka 4.1 configuration reference.
DefaultErrorHandler back offDefault FixedBackOff(0L, 9): 10 deliveries, no delay
- What it does
- How often a failed record is redelivered, and with what delay, before the recoverer is called.
- When to change it
- Always set it. Use a few attempts with a growing delay for failures that might clear in seconds, and keep the total well below max.poll.interval.ms.
DefaultErrorHandler recovererDefault logs the failed record
- What it does
- What happens to a record once the back off is exhausted. With the default it is logged and skipped.
- When to change it
- Pass a DeadLetterPublishingRecoverer for any topic whose records you are not willing to lose.
DeadLetterPublishingRecoverer destination resolverDefault <original topic>-dlt, same partition (.DLT before 3.3)
- What it does
- Function from the failed record and exception to the target topic and partition. A negative partition lets the producer choose.
- When to change it
- Set it explicitly. Route by exception type if different teams own different failures, for example schema failures to one topic and business failures to another.
DeadLetterPublishingRecoverer.failIfSendResultIsErrorDefault true (3.3 source)
- What it does
- Wait for the send result and throw if publishing failed, so the record is redelivered instead of committed.
- When to change it
- Leave it on. Turning it off trades a visible blocked partition for a silent lost record.
DeadLetterPublishingRecoverer.verifyPartitionDefault true
- What it does
- Check that the resolved partition exists; if not, clear it and let the producer pick one.
- When to change it
- Keep it on, and still create the DLT with the same partition count so the fallback is never used.
appendOriginalHeaders / stripPreviousExceptionHeadersDefault true / true
- What it does
- When a record is dead-lettered again, keep the history of original-topic headers and replace the earlier exception headers.
- When to change it
- Leave as is. Exception headers that accumulate can grow the record until the broker rejects it as too large.
ContainerProperties.deliveryAttemptHeaderDefault false
- What it does
- Add kafka_deliveryAttempt, a 4-byte integer starting at 1, to each record handed to a record listener.
- When to change it
- Turn it on when you want the attempt count in logs or on the dead letter record.
retention.ms (on the dead letter topic)Default 604800000 (7 days)
- What it does
- How long the broker keeps dead letter records before deleting old segments.
- When to change it
- Raise it to cover your slowest fix-and-replay cycle; 14 to 30 days is common. After that the record is gone for good.
cleanup.policy (on the dead letter topic)Default delete
- What it does
- Whether old data is deleted by age and size, compacted by key, or both.
- When to change it
- Keep delete. With compact, a second failure for an order id would eventually replace the first.
max.poll.interval.ms (consumer)Default 300000 (5 minutes)
- What it does
- Longest allowed gap between two calls to poll() before the group treats the consumer as failed and rebalances.
- When to change it
- Blocking retries run between polls. If attempts multiplied by processing time plus back off approaches this value, shorten the retries or use retry topics.
@RetryableTopic dltStrategyDefault ALWAYS_RETRY_ON_ERROR
- What it does
- What to do when the @DltHandler method throws: send the record to the DLT again, stop processing it, or have no DLT at all (NO_DLT).
- When to change it
- Use FAIL_ON_ERROR when the handler only records metrics, so a handler bug cannot cycle records through the DLT.
Hands-on exercise
Break a partition with one bad record, then get it back
Send a poison record through the simulated order pipeline, watch the partition stall, add a dead letter topic, and replay the parked record after fixing the consumer.
Steps · 0 of 7 done
What you should see
Without the dead letter topic one partition's lag climbs and the same offset is attempted repeatedly. With it, the bad record appears in the dead letter topic with its origin in the headers, the committed offset moves past it and lag returns to zero. The second replay changes nothing when the consumer is idempotent. In the outage lab, parking failures moves healthy records out of the source topic, while waiting keeps them there in order.
What can go wrong
Dead-lettering an outage
- Symptom
- The database fails for ten minutes. Afterwards the main topic shows no lag, the dead letter topic holds tens of thousands of records, and all of them have a connection or timeout exception as cause.
- Cause
- Transient infrastructure errors go through the same short retry budget and the same recoverer as bad records, so every record is parked a few seconds after the dependency goes down.
- Fix
- Classify exceptions. Keep retrying dependency failures with a long or unlimited back off, or pause the listener container until a health check passes, so records wait in the source topic. Reserve the recoverer for failures that belong to the record.
A dead letter topic with no reader and no alert
- Symptom
- A customer reports a missing payment from three weeks ago. The dead letter topic is empty, and nobody remembers ever looking at it.
- Cause
- The topic had the default 7 day retention and no alert. The record was parked correctly, waited a week and was deleted by retention.
- Fix
- Alert on the growth of the dead letter topic's end offset, set retention.ms longer than your worst fix-and-replay time, and give the topic an owner with a runbook.
The dead letter write fails and the partition stalls
- Symptom
- Lag grows on one partition and the log repeats
Dead-letter publication to orders-dlt failed for: orders-2@40. - Cause
- The topic does not exist and auto-creation is disabled, the service principal has no write permission on it, or the record plus the stack trace header exceeds the topic's maximum message size. The recoverer throws, so the record is redelivered.
- Fix
- Create the topic and the ACL as part of deployment and test the path with a deliberately bad record. For oversize records, exclude the stack trace header or raise the limit on the dead letter topic.
Replay resurrects old state
- Symptom
- After a replay, orders that customers cancelled days ago are shown as open again and some are shipped.
- Cause
- The replayed events were appended after newer events for the same keys, and the consumer applies every event it receives as the latest truth.
- Fix
- Put a version or sequence number in each event and have the consumer ignore events that are not newer than its stored state. Until that exists, review a replay per cause instead of replaying the whole topic.
Upgrade silently changes the topic name
- Symptom
- After moving to Spring Boot 3.4 or later, new failures appear in
orders-dltwhile dashboards and the replay tool still watchorders.DLT. - Cause
- Spring for Apache Kafka 3.3 changed the default suffix of DeadLetterPublishingRecoverer from
.DLTto-dlt, and the application relied on the default. - Fix
- Set the destination resolver explicitly to the name you operate, and migrate deliberately if you want the new name.
Deserialization failures arrive unreadable
- Symptom
- Dead letter records for malformed payloads contain a quoted Base64 string instead of the original bytes.
- Cause
- The recoverer restores the original byte array as the value, and the only configured serializer is a JSON serializer, which encodes a byte array as a Base64 string.
- Fix
- Give the recoverer a producer that writes byte arrays unchanged: a DelegatingByTypeSerializer, or a second KafkaTemplate keyed by byte[] in the recoverer's template map.
How to troubleshoot
You need to know whether new records are arriving in the dead letter topic.
Check
kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders-dlt
- How to read it
- Prints topic:partition:end-offset for each partition. Run it twice a few minutes apart: any increase is new dead letters. This difference, exported as a metric, is the alert to build.
You need to see why records were dead-lettered.
Check
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders-dlt --from-beginning --max-messages 20 --property print.headers=true --property print.key=true --property print.partition=true --property print.offset=true
- How to read it
- Read kafka_dlt-exception-cause-fqcn and kafka_dlt-exception-message first, then group by cause. The partition and offset headers are binary and print as raw bytes; decode them in code if you need them. Kafka 4.2 and later print a deprecation notice for --property and accept --formatter-property instead.
One partition of the source topic has growing lag and the others are at zero.
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- If CURRENT-OFFSET for that partition does not move between two runs while LOG-END-OFFSET grows, a record is being retried without end. Either no recoverer is configured or the dead letter write is failing; the application log tells you which.
Records skip the retries and go to the dead letter topic on the first failure.
Check
The kafka_dlt-exception-cause-fqcn header of one of those records
- How to read it
- If it is DeserializationException, MessageConversionException, ClassCastException or a class you added with addNotRetryableExceptions, that is the configured behaviour. If it is a class that should be retried, your classification is too broad.
The consumer is removed from the group while it is retrying a record.
Check
Application log: a commit failure or rebalance message that mentions max.poll.interval.ms
- How to read it
- The blocking retry sequence took longer than max.poll.interval.ms. Reduce attempts or delays, reduce max.poll.records, or move long waits to retry topics.
The replay listener is started but nothing is republished.
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-dlt-replay
- How to read it
- Lag of zero means an earlier replay already consumed these records and committed. Records at the replay limit are logged and skipped. To replay a range again on purpose, stop the listener and reset this group's offsets.
Key takeaways
- Kafka gives an ordinary consumer no dead letter queue. A dead letter topic is a normal topic your consumer writes to, so its rules are yours to define and enforce.
- Park records that would fail on a healthy system: undeserializable, invalid, or out of retries. Never park records because a dependency is down.
- Spring's recoverer writes to <topic>-dlt on the same partition by default (.DLT before 3.3) and adds origin and exception headers, but no attempt count.
- The offset advances only after the dead letter write is confirmed. If that write fails the record is redelivered and the partition blocks, which is the safe failure.
- Match the source topic's partitions and replication, set retention longer than your fix-and-replay time, and alert on new records.
- A dead letter topic without a tested replay path is deferred data loss. Replay through the original topic with the original key.
- Replay reorders and duplicates. It is only safe against an idempotent consumer that ignores events older than its current state.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
At 14:02 the payments database fails over and is unreachable for 6 minutes. Your listener uses DefaultErrorHandler with 3 retries one second apart, then DeadLetterPublishingRecoverer. Traffic is 150 records per second. At 14:10 the main topic has no lag and orders-dlt holds about 50,000 records, all with a JDBC connection exception as cause.
What should change so the next failover does not end this way?
Scenario 2 of 3
An order update for order 7781 (version 3, status PAID) was dead-lettered on Monday because of a consumer bug. On Tuesday the customer cancelled, and version 5 (status CANCELLED) was processed normally. On Wednesday the bug is fixed and you are about to replay orders-dlt into the orders topic. The consumer currently applies each event's status directly to the order row.
What has to be true before this replay is safe?
Scenario 3 of 3
After a deployment, partition 4 of orders stops advancing. The log shows, every few seconds: Dead-letter publication to orders-dlt failed for: orders-4@9120, with a TopicAuthorizationException. Other partitions are healthy. A teammate proposes calling setFailIfSendResultIsError(false) on the recoverer to get the partition moving.
What is the right response?
References
References: Spring for Apache Kafka 3.3 reference: Handling Exceptions (DefaultErrorHandler, Publishing Dead-letter Records, header management) ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries, DLT Strategies ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries, How the Pattern Works ↗ · Spring for Apache Kafka 3.3 API: DeadLetterPublishingRecoverer ↗ · Spring for Apache Kafka: change history (3.3 standardises the -dlt suffix) ↗ · Apache Kafka 4.1 documentation: Topic configs (retention.ms, cleanup.policy) ↗ · Apache Kafka 4.1 documentation: Consumer configs (max.poll.interval.ms) ↗ · Report a problem