Skip to content

Path 6 · lesson 4 of 5

Intermediate30 minApache Kafka 4.x brokers · Spring Boot 3.5 with Spring for Apache Kafka 3.3 (kafka-clients 3.9) · Java 21. Topic and consumer defaults read from the Kafka 4.1 configuration reference.

Dead letter topics: park, inspect, replay

What belongs in a dead letter topic, what Spring records with each failed record, how to size and alert on it, and how to replay it without creating new damage.

What you will learn

  • Decide which failures go to a dead letter topic and which must never be sent there
  • Configure DefaultErrorHandler with a DeadLetterPublishingRecoverer and predict the topic, partition and headers of a dead letter record
  • Explain what the consumer does when publishing to the dead letter topic itself fails
  • Size a dead letter topic, set its retention and alert on the right signal
  • Build a replay path that is safe to run twice and state what it does to ordering

Before you start

The concept

Message brokers such as RabbitMQ or SQS have a dead letter queue built in. Kafka does not offer one to an ordinary consumer. In Kafka the same idea is a dead letter topic (DLT): a normal topic that your consumer application publishes a record to when it has given up on processing it. The broker treats it like any other topic. Every rule in this lesson is therefore something your code and your team enforce, not something Kafka enforces for you.

The purpose follows from how a partition is consumed. A consumer reads a partition in offset order and its committed offset is one number: the next record to read. It cannot acknowledge record 41 and leave record 40 open. So when record 40 can never be processed, there are only three choices: retry it forever and block everything behind it, skip it and lose it, or move it somewhere else and then advance. A dead letter topic is the third choice. It turns a stuck partition into a recorded, recoverable exception.

FailureGoes to the DLT?Why
Payload cannot be deserialized (wrong schema, truncated bytes)Yes, immediatelyRetrying reads the same bytes and fails the same way.
Record fails business validation (negative amount, unknown currency)Yes, immediatelyThe record is wrong, and time will not fix it.
A bug in the consumer throws for this record shapeYes, after a few retriesPermanent until a fix is deployed. Park it, fix, then replay.
Retries exhausted on an error that looked transientYesYou have spent your retry budget. Blocking the partition longer costs more than parking one record.
Database, downstream API or broker is unavailableNoEvery record will fail. The DLT fills with healthy records and the outage becomes a replay project.
Record already processed (a duplicate)NoThat is a success. An idempotent consumer ignores it and commits.
The test: would this record fail again if it were the only record in the system and every dependency were healthy?

A dead letter record is only useful if someone can later answer three questions from it: where did it come from, why did it fail, and what should be done with it. So the record must carry its origin and the failure with it, and the team must own a path that reads the topic. A DLT nobody reads is a slower way of losing data.

Why it exists

Before a team adds a dead letter topic, a single bad record usually shows up in one of two ways. Either the consumer retries it in a loop, the partition stops moving and lag on that partition grows without bound (the Break card the message that kills its reader is a real case). Or someone wraps the listener in a try/catch that logs and continues, the partition keeps moving, and the failed record now exists only as a log line that will be rotated away.

Without a DLT

  • Retry forever: one record blocks every key that shares its partition.
  • Catch and log: the record is skipped, and nothing can replay it.
  • The only evidence is an application log with a limited lifetime.
  • Recovery means resetting offsets and reprocessing thousands of good records to reach one bad one.

With a DLT

  • The partition advances after a bounded number of attempts.
  • The failed record is stored with the same durability as any topic.
  • Origin and exception travel with the record as headers.
  • Recovery handles exactly the failed records, when you choose.

The cost is real and should be accepted knowingly. Parking a record means later records for the same key are processed before it, so strict per-key ordering is given up for that key. And the consumer now writes to Kafka as well as reading, so it needs a producer, permissions on the dead letter topic and a decision about what to do when that write fails.

How it works inside

How Spring decides to give up

In Spring for Apache Kafka the listener container calls a CommonErrorHandler when your @KafkaListener method throws. The standard one is DefaultErrorHandler. It is built from two parts: a BackOff that says how many times to redeliver and how long to wait, and a recoverer that is called once the back off is exhausted. If you configure nothing, the back off is FixedBackOff(0L, 9), which is ten deliveries with no delay, and the recoverer only logs the record. That default silently skips the record after ten fast attempts, which is rarely what a team intends.

Retries here are blocking: after a failure the handler seeks the consumer back so that the failed record and everything after it in the poll is fetched again, and waits out the back off on the consumer thread. Some exceptions skip the retries and go straight to the recoverer, because repeating them cannot help. By default these are DeserializationException, MessageConversionException, ConversionException, MethodArgumentResolutionException, NoSuchMethodException and ClassCastException. Add your own permanent failures with addNotRetryableExceptions(...).

What DeadLetterPublishingRecoverer does

  1. Resolves the destination. A destination resolver function receives the failed ConsumerRecord and the exception and returns a TopicPartition. The default returns the original topic name with the suffix -dlt and the same partition number as the failed record.
  2. Checks the partition. Because of that default, the dead letter topic needs at least as many partitions as the source topic. If the chosen partition does not exist, the recoverer clears the partition and lets the producer choose one (the verifyPartition check, on by default). A resolver that returns a negative partition asks for the same thing deliberately.
  3. Builds the record. Key and value are the original ones. The original headers are kept and the dead letter headers are added.
  4. Sends and waits. It publishes with the KafkaTemplate you gave it and, with failIfSendResultIsError (true in the 3.3 source), waits for the broker's answer. The wait is the producer's delivery.timeout.ms plus a 5 second buffer.
  5. Returns, and the offset moves. Only after the recoverer returns normally does the container treat the record as handled and commit its offset.

The headers Spring adds

Header name on the wireKafkaHeaders constantContent
kafka_dlt-original-topicDLT_ORIGINAL_TOPICSource topic, UTF-8 text
kafka_dlt-original-partitionDLT_ORIGINAL_PARTITIONSource partition, 4-byte integer
kafka_dlt-original-offsetDLT_ORIGINAL_OFFSETSource offset, 8-byte long
kafka_dlt-original-timestampDLT_ORIGINAL_TIMESTAMPSource record timestamp, 8-byte long
kafka_dlt-original-timestamp-typeDLT_ORIGINAL_TIMESTAMP_TYPECreateTime or LogAppendTime
kafka_dlt-original-consumer-groupDLT_ORIGINAL_CONSUMER_GROUPThe group whose listener failed
kafka_dlt-exception-fqcnDLT_EXCEPTION_FQCNException class, usually ListenerExecutionFailedException
kafka_dlt-exception-cause-fqcnDLT_EXCEPTION_CAUSE_FQCNClass of the cause: your own exception
kafka_dlt-exception-messageDLT_EXCEPTION_MESSAGEException message
kafka_dlt-exception-stacktraceDLT_EXCEPTION_STACKTRACEFull stack trace as text
Three more headers (kafka_dlt-key-exception-fqcn, -message, -stacktrace) appear only when the key failed to deserialize.

The stack trace header can be several kilobytes. That matters when a record fails more than once, for example after a replay. Two properties keep it bounded: stripPreviousExceptionHeaders (default true) replaces older exception headers, and appendOriginalHeaders (default true) keeps the history of original topics. You can drop a header entirely with excludeHeader(HeaderNames.HeadersToAdd.EX_STACKTRACE).

When publishing to the dead letter topic fails

This is the case that decides whether you lose data. If the send fails (the topic does not exist and cannot be auto-created, the principal has no write permission, the record plus headers is larger than the topic allows, the brokers are unreachable), the recoverer throws. The error handler then treats the record as not recovered: it is included in the seeks, fetched again, and by default the back off starts over (resetStateOnRecoveryFailure). The offset is not committed. The partition is blocked again, which is the correct outcome: a blocked partition is visible as lag, a record dropped between two topics is not.

Deserialization failures need one extra piece. A record that cannot be deserialized never reaches your listener, so the consumer's deserializer must be wrapped in ErrorHandlingDeserializer, which hands the failure to the error handler as a DeserializationException. The recoverer then publishes the original bytes as the value. Your dead letter producer must therefore be able to serialize both your normal type and a raw byte[].

With retry topics

@RetryableTopic (non-blocking retries, covered in the previous lesson) uses the same recoverer underneath and ends in the same kind of topic, <topic>-dlt. A method annotated with @DltHandler in the same class consumes it. Without one, a default handler only logs each dead letter record. If that handler itself throws, the default strategy ALWAYS_RETRY_ON_ERROR sends the record back to the end of the dead letter topic so it does not block the others, except for the fatal exception types listed above. Set autoStartDltHandler = "false" when the application should publish to the DLT but not consume it.

Retry topics & DLQ

Simulation, not a real cluster

An animated walk through a record moving from the main topic through retry topics into the dead letter topic.

Step 1 · The pattern

One at a time, in order

A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.

  • message
  • processed
Open the full animated lesson

Sizing, retention and alerting

  • Partitions. With the default resolver, use the same partition count as the source topic. Then every failed record for a key lands in one dead letter partition, in the order the failures happened. If you add partitions to the source topic, add them to the DLT first.
  • Replication. Use the same replication factor and min.insync.replicas as the source topic. Dead letter records are the ones you have already failed once; they deserve the same durability.
  • Retention. A topic keeps data for retention.ms, 7 days by default. Choose a value longer than the slowest realistic path from alert to fix to replay, including a long weekend. Many teams use 14 to 30 days. Keep cleanup.policy=delete: with compaction, a later failure for the same key would replace an earlier one.
  • Size. In a healthy system the DLT is nearly empty, so disk is not the concern. The concern is a burst: a bad deployment can send a large share of traffic there in minutes. A cap with retention.bytes protects the brokers but deletes the oldest dead letters first, so prefer an alert to a tight cap.
  • Alerting. The useful signal is the rate of new records, which is the growth of the topic's end offset. Any record in a topic that should be empty deserves a ticket; a sustained rate deserves a page, because it means a bug or a misclassified outage. Lag of the replay group is not a useful alert, because that group is normally stopped.

The reprocessing path

Publishing to a DLT is half of the feature. The other half is the way back, and it has four steps.

  1. Inspect. Read the records with their headers and group them by kafka_dlt-exception-cause-fqcn. Ten thousand records usually collapse into two or three causes.
  2. Decide per cause. A consumer bug: deploy the fix, then replay. Bad data from a producer: the producer republishes a corrected event, and the dead letter is discarded. Obsolete (the order was cancelled since): discard, and write down why.
  3. Replay. Republish to the original topic with the original key, so the record follows the normal path through the normal consumer. Do not build a second code path that processes dead letters directly; it will drift from the real one.
  4. Verify. Watch the DLT during the replay. Records that fail again return with a replay counter one higher, and a limit on that counter stops an endless loop.

Architecture

Drawing diagram…

Flowchart. The payment listener reads the orders topic. On an exception the DefaultErrorHandler either seeks back and redelivers, or hands the record to the DeadLetterPublishingRecoverer, which publishes it to orders-dlt. The offset is committed only when that send is confirmed. From the dead letter topic an alert fires, a person inspects and fixes, and a replay listener republishes records to the orders topic with the same key.

flowchart TD
  SRC["orders topic, 6 partitions"] --> L["Payment listener"]
  L -->|"processed"| C["Commit next offset"]
  L -->|"throws"| EH["DefaultErrorHandler"]
  EH --> Q{"Not retryable or retries exhausted"}
  Q -->|"no"| RB["Seek back, wait for back off, redeliver"]
  RB --> L
  Q -->|"yes"| REC["DeadLetterPublishingRecoverer"]
  REC --> DLT["orders-dlt topic, 6 partitions"]
  REC -->|"send confirmed"| C
  REC -->|"send failed"| RB
  DLT --> AL["Alert on new records"]
  DLT --> INS["Inspect headers, group by cause"]
  INS --> FIX["Fix the consumer or the data"]
  FIX --> RP["Replay listener, started by an operator"]
  RP -->|"same key, replay counter"| SRC
The dead letter loop: a record leaves the partition only after it is safely in the DLT, and returns through the normal topic.
  • There are two exits from the listener: success, and a confirmed write to the dead letter topic. Both end in the same commit. A failed write to the dead letter topic is not an exit; it loops back to redelivery.
  • The dead letter topic has the same number of partitions as the source, so the default resolver can keep the partition number and failures for one key stay together.
  • The return path goes through the source topic, not straight into the service. The replayed record is handled by the same listener, the same validation and the same idempotency check as any other record.
  • The replay listener is a separate consumer group that is stopped by default. A person starts it after the fix is deployed, which is why its lag is not an alert.

Message flow

Drawing diagram…

Sequence diagram. The container polls offsets 40 to 44 of partition 2. The listener fails on offset 40. The error handler seeks back and redelivers three more times with growing delays. After the fourth failure the recoverer publishes the record to orders-dlt partition 2 and waits for the acknowledgement. The container then commits offset 41 and continues. Later an operator reads the dead letter topic and replays the record to the orders topic.

sequenceDiagram
  participant B as Broker
  participant C as Listener container
  participant L as Payment listener
  participant E as DefaultErrorHandler
  participant R as DLT recoverer
  participant O as Operator
  B->>C: poll returns orders-2 offsets 40 to 44
  C->>L: offset 40, attempt 1
  L-->>C: throws IllegalStateException
  C->>E: handle failure
  E->>C: seek orders-2 back to 40
  Note over C,E: Back off 500 ms, then 1 s, then 2 s
  C->>L: offset 40, attempt 4
  L-->>C: throws again
  E->>R: retries exhausted, recover
  R->>B: send to orders-dlt partition 2 with headers
  B-->>R: acknowledged
  C->>B: commit offset 41 for orders-2
  C->>L: offset 41 and onwards
  O->>B: read orders-dlt, group by cause
  O->>B: after the fix, replay to orders with the same key
One poison record: four attempts, a confirmed dead letter write, then the commit that lets the partition move on.
  1. The seek is why later records are not processed during the retries: offsets 41 to 44 were fetched, but the consumer's position is moved back to 40, so they are fetched again afterwards.
  2. The whole retry sequence happens on the consumer thread between polls of new data. Its total duration, here about 3.5 seconds plus four processing attempts, must stay well under max.poll.interval.ms (5 minutes by default), or the group will consider the consumer failed and rebalance.
  3. The commit is for offset 41, the next record to read. Nothing in Kafka marks offset 40 as failed. The only record of the failure is the dead letter record and its headers.
  4. If the process crashes after the dead letter write is acknowledged but before the commit, offset 40 is delivered again after restart, fails again and is published a second time. Dead letter topics can contain duplicates, so the replay must tolerate them.

In a real system

A typical payment service consumes the topic orders (6 partitions, replication factor 3, key = order id) in the consumer group payment-service, and charges the customer for each OrderPlaced event. Its dead letter topic is orders-dlt: 6 partitions, replication factor 3, retention.ms of 30 days, cleanup.policy=delete.

A new version of the order service starts sending amounts for a three-decimal currency in a shape the payment service does not expect. The payment listener throws IllegalStateException for those orders, about 40 a minute out of 9,000. After four attempts each record goes to orders-dlt, and every other order keeps flowing. The alert on new dead letter records fires within five minutes. An engineer reads ten records with their headers: all carry kafka_dlt-exception-cause-fqcn = java.lang.IllegalStateException, and the same currency.

The fix is deployed two hours later, by which time about 4,800 records are parked. The engineer starts the replay listener. Each record is republished to orders with its order id as key and x-replay-count = 1. The payment service handles them like any other order. Because it stores one payment per order id, the 30 orders that customers had already retried and paid through a second order event are recognised and skipped. Three records come back to orders-dlt with a different cause, a genuinely invalid amount, and are handed to the order team.

Java and Spring Boot

A payment service with blocking retries, a dead letter topic and an operator-started replay listener. The code follows the Spring for Apache Kafka 3.3 reference; it was written without a compiler at hand, so treat it as a careful template.

src/main/resources/application.ymlYAML
spring:  application:    name: payment-service  kafka:    bootstrap-servers: localhost:9092    consumer:      group-id: payment-service      auto-offset-reset: earliest      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer      # Wrap the real deserializer so a bad payload reaches the error handler      # instead of failing inside poll() forever.      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer      properties:        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer        spring.json.value.default.type: com.example.payments.OrderEvent        spring.json.use.type.headers: false        spring.json.trusted.packages: com.example.payments    listener:      ack-mode: recordpayments:  dlt:    max-replays: 3

Why it is written this way

  • ErrorHandlingDeserializer turns a deserialization failure into a record the error handler can route. Without it the consumer throws from poll() at the same offset on every attempt and no dead letter is ever written.
  • spring.json.use.type.headers: false with a default type makes the listener independent of type headers. A replayed record is then read the same way whether or not it carries them.
  • ack-mode: record commits after each record, so a crash re-delivers at most the record in flight. The container default is BATCH, which commits after all records of a poll.

Configuration

The settings that decide what reaches the dead letter topic, how long it stays, and what happens when the write fails. Spring values are from the Spring for Apache Kafka 3.3 reference; topic and consumer values from the Kafka 4.1 configuration reference.

DefaultErrorHandler back offDefault FixedBackOff(0L, 9): 10 deliveries, no delay
What it does
How often a failed record is redelivered, and with what delay, before the recoverer is called.
When to change it
Always set it. Use a few attempts with a growing delay for failures that might clear in seconds, and keep the total well below max.poll.interval.ms.
DefaultErrorHandler recovererDefault logs the failed record
What it does
What happens to a record once the back off is exhausted. With the default it is logged and skipped.
When to change it
Pass a DeadLetterPublishingRecoverer for any topic whose records you are not willing to lose.
DeadLetterPublishingRecoverer destination resolverDefault <original topic>-dlt, same partition (.DLT before 3.3)
What it does
Function from the failed record and exception to the target topic and partition. A negative partition lets the producer choose.
When to change it
Set it explicitly. Route by exception type if different teams own different failures, for example schema failures to one topic and business failures to another.
DeadLetterPublishingRecoverer.failIfSendResultIsErrorDefault true (3.3 source)
What it does
Wait for the send result and throw if publishing failed, so the record is redelivered instead of committed.
When to change it
Leave it on. Turning it off trades a visible blocked partition for a silent lost record.
DeadLetterPublishingRecoverer.verifyPartitionDefault true
What it does
Check that the resolved partition exists; if not, clear it and let the producer pick one.
When to change it
Keep it on, and still create the DLT with the same partition count so the fallback is never used.
appendOriginalHeaders / stripPreviousExceptionHeadersDefault true / true
What it does
When a record is dead-lettered again, keep the history of original-topic headers and replace the earlier exception headers.
When to change it
Leave as is. Exception headers that accumulate can grow the record until the broker rejects it as too large.
ContainerProperties.deliveryAttemptHeaderDefault false
What it does
Add kafka_deliveryAttempt, a 4-byte integer starting at 1, to each record handed to a record listener.
When to change it
Turn it on when you want the attempt count in logs or on the dead letter record.
retention.ms (on the dead letter topic)Default 604800000 (7 days)
What it does
How long the broker keeps dead letter records before deleting old segments.
When to change it
Raise it to cover your slowest fix-and-replay cycle; 14 to 30 days is common. After that the record is gone for good.
cleanup.policy (on the dead letter topic)Default delete
What it does
Whether old data is deleted by age and size, compacted by key, or both.
When to change it
Keep delete. With compact, a second failure for an order id would eventually replace the first.
max.poll.interval.ms (consumer)Default 300000 (5 minutes)
What it does
Longest allowed gap between two calls to poll() before the group treats the consumer as failed and rebalances.
When to change it
Blocking retries run between polls. If attempts multiplied by processing time plus back off approaches this value, shorten the retries or use retry topics.
@RetryableTopic dltStrategyDefault ALWAYS_RETRY_ON_ERROR
What it does
What to do when the @DltHandler method throws: send the record to the DLT again, stop processing it, or have no DLT at all (NO_DLT).
When to change it
Use FAIL_ON_ERROR when the handler only records metrics, so a handler bug cannot cycle records through the DLT.

Hands-on exercise

Break a partition with one bad record, then get it back

Send a poison record through the simulated order pipeline, watch the partition stall, add a dead letter topic, and replay the parked record after fixing the consumer.

Steps · 0 of 7 done

Take it further in the lab: Poison message and dead-letter queue

What can go wrong

Dead-lettering an outage
Symptom
The database fails for ten minutes. Afterwards the main topic shows no lag, the dead letter topic holds tens of thousands of records, and all of them have a connection or timeout exception as cause.
Cause
Transient infrastructure errors go through the same short retry budget and the same recoverer as bad records, so every record is parked a few seconds after the dependency goes down.
Fix
Classify exceptions. Keep retrying dependency failures with a long or unlimited back off, or pause the listener container until a health check passes, so records wait in the source topic. Reserve the recoverer for failures that belong to the record.
A dead letter topic with no reader and no alert
Symptom
A customer reports a missing payment from three weeks ago. The dead letter topic is empty, and nobody remembers ever looking at it.
Cause
The topic had the default 7 day retention and no alert. The record was parked correctly, waited a week and was deleted by retention.
Fix
Alert on the growth of the dead letter topic's end offset, set retention.ms longer than your worst fix-and-replay time, and give the topic an owner with a runbook.
The dead letter write fails and the partition stalls
Symptom
Lag grows on one partition and the log repeats Dead-letter publication to orders-dlt failed for: orders-2@40.
Cause
The topic does not exist and auto-creation is disabled, the service principal has no write permission on it, or the record plus the stack trace header exceeds the topic's maximum message size. The recoverer throws, so the record is redelivered.
Fix
Create the topic and the ACL as part of deployment and test the path with a deliberately bad record. For oversize records, exclude the stack trace header or raise the limit on the dead letter topic.
Replay resurrects old state
Symptom
After a replay, orders that customers cancelled days ago are shown as open again and some are shipped.
Cause
The replayed events were appended after newer events for the same keys, and the consumer applies every event it receives as the latest truth.
Fix
Put a version or sequence number in each event and have the consumer ignore events that are not newer than its stored state. Until that exists, review a replay per cause instead of replaying the whole topic.
Upgrade silently changes the topic name
Symptom
After moving to Spring Boot 3.4 or later, new failures appear in orders-dlt while dashboards and the replay tool still watch orders.DLT.
Cause
Spring for Apache Kafka 3.3 changed the default suffix of DeadLetterPublishingRecoverer from .DLT to -dlt, and the application relied on the default.
Fix
Set the destination resolver explicitly to the name you operate, and migrate deliberately if you want the new name.
Deserialization failures arrive unreadable
Symptom
Dead letter records for malformed payloads contain a quoted Base64 string instead of the original bytes.
Cause
The recoverer restores the original byte array as the value, and the only configured serializer is a JSON serializer, which encodes a byte array as a Base64 string.
Fix
Give the recoverer a producer that writes byte arrays unchanged: a DelegatingByTypeSerializer, or a second KafkaTemplate keyed by byte[] in the recoverer's template map.

How to troubleshoot

You need to know whether new records are arriving in the dead letter topic.

Check

kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic orders-dlt
How to read it
Prints topic:partition:end-offset for each partition. Run it twice a few minutes apart: any increase is new dead letters. This difference, exported as a metric, is the alert to build.
You need to see why records were dead-lettered.

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders-dlt --from-beginning --max-messages 20 --property print.headers=true --property print.key=true --property print.partition=true --property print.offset=true
How to read it
Read kafka_dlt-exception-cause-fqcn and kafka_dlt-exception-message first, then group by cause. The partition and offset headers are binary and print as raw bytes; decode them in code if you need them. Kafka 4.2 and later print a deprecation notice for --property and accept --formatter-property instead.
One partition of the source topic has growing lag and the others are at zero.

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
If CURRENT-OFFSET for that partition does not move between two runs while LOG-END-OFFSET grows, a record is being retried without end. Either no recoverer is configured or the dead letter write is failing; the application log tells you which.
Records skip the retries and go to the dead letter topic on the first failure.

Check

The kafka_dlt-exception-cause-fqcn header of one of those records
How to read it
If it is DeserializationException, MessageConversionException, ClassCastException or a class you added with addNotRetryableExceptions, that is the configured behaviour. If it is a class that should be retried, your classification is too broad.
The consumer is removed from the group while it is retrying a record.

Check

Application log: a commit failure or rebalance message that mentions max.poll.interval.ms
How to read it
The blocking retry sequence took longer than max.poll.interval.ms. Reduce attempts or delays, reduce max.poll.records, or move long waits to retry topics.
The replay listener is started but nothing is republished.

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group orders-dlt-replay
How to read it
Lag of zero means an earlier replay already consumed these records and committed. Records at the replay limit are logged and skipped. To replay a range again on purpose, stop the listener and reset this group's offsets.

Key takeaways

  • Kafka gives an ordinary consumer no dead letter queue. A dead letter topic is a normal topic your consumer writes to, so its rules are yours to define and enforce.
  • Park records that would fail on a healthy system: undeserializable, invalid, or out of retries. Never park records because a dependency is down.
  • Spring's recoverer writes to <topic>-dlt on the same partition by default (.DLT before 3.3) and adds origin and exception headers, but no attempt count.
  • The offset advances only after the dead letter write is confirmed. If that write fails the record is redelivered and the partition blocks, which is the safe failure.
  • Match the source topic's partitions and replication, set retention longer than your fix-and-replay time, and alert on new records.
  • A dead letter topic without a tested replay path is deferred data loss. Replay through the original topic with the original key.
  • Replay reorders and duplicates. It is only safe against an idempotent consumer that ignores events older than its current state.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

At 14:02 the payments database fails over and is unreachable for 6 minutes. Your listener uses DefaultErrorHandler with 3 retries one second apart, then DeadLetterPublishingRecoverer. Traffic is 150 records per second. At 14:10 the main topic has no lag and orders-dlt holds about 50,000 records, all with a JDBC connection exception as cause.

What should change so the next failover does not end this way?

Scenario 2 of 3

An order update for order 7781 (version 3, status PAID) was dead-lettered on Monday because of a consumer bug. On Tuesday the customer cancelled, and version 5 (status CANCELLED) was processed normally. On Wednesday the bug is fixed and you are about to replay orders-dlt into the orders topic. The consumer currently applies each event's status directly to the order row.

What has to be true before this replay is safe?

Scenario 3 of 3

After a deployment, partition 4 of orders stops advancing. The log shows, every few seconds: Dead-letter publication to orders-dlt failed for: orders-4@9120, with a TopicAuthorizationException. Other partitions are healthy. A teammate proposes calling setFailIfSendResultIsError(false) on the recoverer to get the partition moving.

What is the right response?

References

References: Spring for Apache Kafka 3.3 reference: Handling Exceptions (DefaultErrorHandler, Publishing Dead-letter Records, header management) ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries, DLT Strategies ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries, How the Pattern Works ↗ · Spring for Apache Kafka 3.3 API: DeadLetterPublishingRecoverer ↗ · Spring for Apache Kafka: change history (3.3 standardises the -dlt suffix) ↗ · Apache Kafka 4.1 documentation: Topic configs (retention.ms, cleanup.policy) ↗ · Apache Kafka 4.1 documentation: Consumer configs (max.poll.interval.ms) ↗ · Report a problem