Skip to content

Path 9 · lesson 4 of 6

Advanced26 minJava 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 · kafka-clients 3.9 (the versions Spring Boot 3.5 manages: spring-kafka 3.3.x, kafka-clients 3.9.x) against Apache Kafka 4.x brokers

Error handling, retries and dead letters in Spring

One production-shaped Spring configuration: DefaultErrorHandler with a back-off, a dead-letter recoverer, ErrorHandlingDeserializer, and @RetryableTopic when retries must not block.

What you will learn

  • State what a Spring listener does with a failing record when you configure nothing, and why that is rarely what a payment flow wants
  • Configure a DefaultErrorHandler with a bounded back-off, not-retryable exceptions and a DeadLetterPublishingRecoverer with an explicit destination
  • Wrap deserializers in ErrorHandlingDeserializer so a malformed payload reaches the error handler and is published as its original bytes
  • Choose between blocking retries and @RetryableTopic, and say what each costs in ordering and throughput
  • Handle a failure inside a batch listener without sending the whole batch to the dead-letter topic
  • Write an integration test that proves a bad record ends in the dead-letter topic and the listener keeps going

Before you start

The concept

The reliability path explained the ideas: what a poison message is, why retries need a back-off and what a dead-letter topic is for. This lesson is the Spring implementation guide. It names the classes, shows where each one sits, and ends with one configuration you can adapt.

In Spring for Apache Kafka a listener method never talks to the consumer. A listener container owns the consumer, runs the poll loop and calls your method once per record. When the method throws, the exception goes back to the container, and the container hands it to a CommonErrorHandler. That one object decides everything that follows: redeliver now, wait and redeliver, or give up and call a recoverer. After a recovery the container commits the record's offset and the partition moves on.

PieceClassWhat it decides
Error handlerDefaultErrorHandlerWhether a failed record is delivered again, and when it stops trying
Back-offFixedBackOff, ExponentialBackOffWithMaxRetriesHow many redeliveries and how long to wait before each
ClassificationaddNotRetryableExceptions(...)Which exceptions skip the retries and go straight to recovery
RecovererDeadLetterPublishingRecovererWhat happens to a record that is given up on: publish it elsewhere, with failure headers
Destination resolvera BiFunction of record and exceptionWhich topic and partition the dead-letter record goes to
Deserializer wrapperErrorHandlingDeserializerThat a payload which cannot be decoded becomes a record the error handler can see
Non-blocking retries@RetryableTopic, @DltHandlerThat a failed record waits in a retry topic while the partition keeps flowing
Transactional listenersDefaultAfterRollbackProcessorThe same decisions when the container runs a Kafka transaction (see the next lesson)
Class names from the Spring for Apache Kafka 3.3 reference. SeekToCurrentErrorHandler and the other legacy handlers were replaced by DefaultErrorHandler in 2.8.

Why it exists

A raw KafkaConsumer loop gives you no error policy at all. If your code throws and you poll again, the consumer's position has already moved past the record, so it is skipped. If you seek back, you loop on it for ever. Every team that wrote its own loop had to write the same things by hand: seek back to the failed offset, count attempts per record, sleep without missing max.poll.interval.ms, publish the record somewhere when giving up, and commit past it only when that publish succeeded.

Spring moved all of that into the container so that the policy is configuration and the listener stays business code. The price is that the policy has defaults, and the defaults were chosen so that a demo never hangs. Two of them matter in production:

  • Log and skip after ten immediate attempts. A database that is down for thirty seconds turns every record polled in that time into ten stack traces and a skipped offset.
  • No protection against undecodable bytes. A deserializer runs inside poll(), before the container has a record to give to anyone. The Spring reference says it plainly: when a deserializer fails, Spring has no way to handle the problem, because it occurs before poll() returns. Without the wrapper shown below, the consumer fails on the same offset again and again and the partition stops.

Retry topics & DLQ

Simulation, not a real cluster

The animated Build lesson shows a poison record blocking a partition, then retry topics and a dead-letter topic releasing it. It is a simulation, not a real broker.

Step 1 · The pattern

One at a time, in order

A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.

  • message
  • processed
Open the full animated lesson

A third reason to be explicit is naming. Up to Spring for Apache Kafka 3.2 the default dead-letter topic was the source topic plus .DLT. From 3.3 it is the source topic plus -dlt. A project that relied on the default started writing to a different topic after an upgrade. A destination resolver in your own code removes that surprise.

How it works inside

What DefaultErrorHandler does with one failed record

  1. The listener throws. The container wraps the exception in a ListenerExecutionFailedException and, before calling the error handler, commits the offsets of the records that already succeeded.
  2. The handler classifies the exception, looking through the causes. Six types are fatal by default and are never retried: DeserializationException, MessageConversionException, ConversionException, MethodArgumentResolutionException, NoSuchMethodException and ClassCastException. addNotRetryableExceptions adds your own.
  3. For a retryable exception it asks the BackOff for the next wait. It sleeps on the consumer thread, then seeks the consumer back so that the failed record and everything after it from that poll are fetched again. The next poll() returns the failed record first.
  4. When the back-off is exhausted, or the exception is not retryable, it calls the recoverer. If the recoverer returns normally the record counts as handled, its offset is committed and the listener receives the next record.
  5. If the recoverer throws (the dead-letter publish failed, for example), the record is not skipped. It is sought again and, by default, goes through the whole back-off sequence once more.

How a bad payload gets to the handler

ErrorHandlingDeserializer is a deserializer that delegates to the real one. When the delegate throws, the wrapper returns null as the value and puts a DeserializationException, containing the cause and the raw bytes, in a record header. poll() therefore succeeds. The container sees the header, does not call your listener, and passes the record to the error handler. DeserializationException is in the fatal list, so there are no retries: the recoverer runs on the first failure.

DeadLetterPublishingRecoverer then restores the record's value to the original bytes that failed to decode. That has a consequence on the producer side: the template it publishes with must be able to send both your normal objects and a raw byte[]. A producer configured only with a JSON serializer would write the bytes as a JSON-encoded string, which is no longer the message that arrived. The configuration below uses DelegatingByTypeSerializer for that.

What the dead-letter record contains

AspectBehaviour in 3.3Why it matters
Default destination<source topic>-dlt, same partition number as the failed recordThe dead-letter topic needs at least as many partitions as the source. If the resolver returns a negative partition, the producer chooses one.
Partition checkIf the chosen partition does not exist, the partition is left unsetRecords still arrive, but no longer line up with the source partition.
Key and valueCopied from the failed record; the original bytes for a deserialization failureThe record can be replayed to the source topic unchanged.
Origin headersKafkaHeaders.DLT_ORIGINAL_TOPIC, DLT_ORIGINAL_PARTITION, DLT_ORIGINAL_OFFSET, DLT_ORIGINAL_TIMESTAMP, DLT_ORIGINAL_CONSUMER_GROUPYou can find the exact source record and the group that gave up on it.
Exception headersDLT_EXCEPTION_FQCN, DLT_EXCEPTION_CAUSE_FQCN, DLT_EXCEPTION_MESSAGE, DLT_EXCEPTION_STACKTRACEEnough to triage without the service logs. The stack trace can be large; excludeHeader removes it.
Publish failureWith failIfSendResultIsError the recoverer waits for the broker's answer and throws if the send failedA record must never be skipped on the strength of a publish that did not happen.

Blocking retries or retry topics

DefaultErrorHandler (blocking)

  • The failed record is retried in place; later records of the partition wait.
  • Order within the partition is kept.
  • Total wait is bounded by max.poll.interval.ms, so think seconds.
  • No extra topics. Works with batch listeners and with transactions (through the after-rollback processor).
  • Fits failures where the next record would fail too: database down, dependency unreachable.

@RetryableTopic (non-blocking)

  • The failed record is published to a retry topic with a due time; the main partition moves on.
  • Order is lost for that record. The reference says so directly: you lose Kafka's ordering guarantees for that topic.
  • Delays of minutes or hours are fine; the retry consumer pauses its partition until the record is due.
  • Creates retry topics and a DLT, with extra listener containers. Not supported for batch listeners, and cannot be combined with container transactions.
  • Fits failures tied to one record or one customer: a rate-limited account, a recipient's mail server.

With @RetryableTopic the defaults are 3 attempts in total (the first delivery counts) with a fixed 1 second delay, retry topics suffixed -retry and a dead-letter topic suffixed -dlt, each auto-created with one partition and the broker's default replication factor. If the class has no @DltHandler method, a default handler only logs what arrives in the DLT. The same six fatal exception types skip the retry topics and go straight to the DLT.

Batch listeners, briefly

A batch listener receives a list, so the container cannot know which element failed unless you say so. Throw BatchListenerFailedException with the index of the failing record: the handler commits the offsets before that index, and retries or recovers only that record, redelivering the rest. Any other exception makes the handler retry the complete batch from memory, and when the back-off is exhausted the recoverer is called for every record in the batch. One bad record then sends a whole poll of good records to the dead-letter topic.

Architecture

Drawing diagram…

Flowchart. A polled record either fails deserialization and goes to the error handler with an exception header, or reaches the listener. A listener exception also goes to the error handler, which either waits and seeks back for another delivery, or calls the dead-letter recoverer. A successful dead-letter publish lets the offset be committed; a failed publish causes another seek.

flowchart TD
  POLL["Container polls orders.order-placed"] --> DES{"Value deserializes"}
  DES -->|no| HDR["ErrorHandlingDeserializer returns null value plus exception header"]
  DES -->|yes| LIS["OrderPlacedListener"]
  LIS -->|returns| ACK["Offset committed by the container"]
  LIS -->|throws| EH["DefaultErrorHandler"]
  HDR --> EH
  EH --> CLS{"Retryable exception"}
  CLS -->|yes and attempts left| BO["Back-off wait on the consumer thread"]
  BO --> SEEK["Seek back so the record is polled again"]
  SEEK --> POLL
  CLS -->|no or attempts used up| REC["DeadLetterPublishingRecoverer"]
  REC --> RES["Destination resolver picks topic and partition"]
  RES --> DLT["orders.order-placed-dlt"]
  DLT -->|publish acknowledged| ACK
  DLT -->|publish failed| SEEK
Every path a record can take through the payment service's listener container.
  • There are exactly two exits: the offset is committed because the listener returned, or because the dead-letter publish was acknowledged. Nothing else moves the partition forward.
  • The deserialization branch never touches the listener. That is why validation of field values still belongs in the listener, and why a failed validation should throw a type you have marked as not retryable.
  • The loop on the left is the blocking retry. While a record is in it, the container is not delivering anything else from that partition.
  • The arrow from a failed publish back to the seek is the safety property: if the dead-letter topic is unavailable, the consumer stalls on the record and lag grows, which is visible, instead of losing it, which is not.

Message flow

Drawing diagram…

Sequence diagram. The container polls two records. The listener throws on record 41. The error handler waits, seeks back to offset 41 and the record is polled and delivered again. After the back-off is exhausted the recoverer sends the record to the dead-letter topic, the broker acknowledges, the container commits offset 42 and delivers record 42.

sequenceDiagram
  participant K as Broker
  participant C as Listener container
  participant L as OrderPlacedListener
  participant E as DefaultErrorHandler
  participant R as DLT recoverer
  C->>K: poll
  K-->>C: offsets 41 and 42 of partition 3
  C->>L: record 41
  L--xC: throws PaymentProviderUnavailableException
  C->>E: record 41 and exception, attempt 1
  E->>E: wait 1 s
  E->>C: seek partition 3 to offset 41
  C->>K: poll
  K-->>C: offsets 41 and 42
  C->>L: record 41 again
  L--xC: throws again
  Note over C,E: repeated with waits of 2 s, 4 s and 8 s
  C->>E: record 41, back-off exhausted
  E->>R: recover record 41
  R->>K: send to orders.order-placed-dlt partition 3
  K-->>R: acknowledged
  R-->>E: recovered
  E-->>C: handled, commit offset 42
  C->>L: record 42
A transient failure on offset 41: four timed redeliveries, then the dead-letter topic, then the next record.
  1. Record 42 was already fetched in the first poll, but it is fetched again after each seek and is not delivered until 41 has either succeeded or been recovered.
  2. The committed offset after recovery is 42: the position of the next record to read, not an acknowledgement of 41. If the service crashes before that commit, 41 is delivered again and may be published to the dead-letter topic twice.
  3. If the exception had been InvalidOrderException, marked not retryable, the diagram would jump from the first failure straight to the recoverer.
  4. Had the broker not acknowledged the dead-letter send, the recoverer would throw and the handler would seek to 41 once more.

In a real system

In the academy's example project the payment service consumes orders.order-placed (12 partitions, keyed by orderId) in group payment-service. Three different things go wrong there, and each needs a different answer:

FailureExceptionPolicyWhy
The card provider times out or returns 503PaymentProviderUnavailableExceptionBlocking retries: 1 s, 2 s, 4 s, 8 s, then dead-letterThe next order would hit the same outage. Keeping order per partition is worth fifteen seconds.
Amount is zero or the currency is unknownInvalidOrderExceptionNot retryable: dead-letter on the first failureThe record will be exactly as wrong on the tenth attempt.
A producer published bytes that are not an OrderPlacedDeserializationExceptionFatal by default: dead-letter on the first failure, as raw bytesThe listener cannot even be called.

The notification service is different. It consumes payments.payment-result and sends an email. If one recipient's mail provider rejects a message for ten minutes, there is no reason to hold back every other customer's confirmation in that partition, and the order of emails between customers carries no meaning. That listener uses @RetryableTopic with delays of 10 seconds, 1 minute and 6 minutes.

Java and Spring Boot

The payment service's error handling, complete, followed by the notification service's non-blocking variant and an integration test. It relies on Spring Boot's auto-configured consumer factory, producer factory, template and listener container factory, and only adds beans. The full services are in the example project at /kafka/project.

payment-service/src/main/resources/application.ymlYAML
spring:  kafka:    bootstrap-servers: localhost:9092    consumer:      group-id: payment-service      auto-offset-reset: earliest      key-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer      properties:        "[spring.deserializer.key.delegate.class]": org.apache.kafka.common.serialization.StringDeserializer        "[spring.deserializer.value.delegate.class]": org.springframework.kafka.support.serializer.JsonDeserializer        "[spring.json.value.default.type]": com.example.payments.OrderPlaced        "[spring.json.use.type.headers]": false        "[spring.json.trusted.packages]": com.example.payments    producer:      key-serializer: org.apache.kafka.common.serialization.StringSerializer      # The value serializer is set in KafkaErrorHandlingConfig (JSON for objects, raw bytes for dead letters).    listener:      ack-mode: record

Why it is written this way

  • Both deserializers are ErrorHandlingDeserializer. The real ones are named by the two delegate.class properties, which are the values of ErrorHandlingDeserializer.KEY_DESERIALIZER_CLASS and VALUE_DESERIALIZER_CLASS.
  • The square brackets with quotes stop Spring Boot from splitting the dotted Kafka property names into nested YAML keys.
  • spring.json.use.type.headers: false makes the deserializer ignore the type header the order service's serializer may add and always build com.example.payments.OrderPlaced. Without it, a header naming the producer's own class causes a failure that has nothing to do with the payload.
  • ack-mode: record commits after every record. The default, batch, commits once per poll. Either works with the error handler; record shortens what is redelivered after a crash.
  • Spring for Apache Kafka sets enable.auto.commit to false unless you set it yourself, so offsets move only when the container commits them.

Configuration

The settings that shape the error path. Spring behaviour is from the Spring for Apache Kafka 3.3 reference and Javadoc; consumer defaults are those of kafka-clients 3.9, the client Spring Boot 3.5 manages.

DefaultErrorHandler back-off (constructor argument)Default FixedBackOff(0L, 9): ten deliveries, no delay, then log at ERROR and skip
What it does
How many times a failed record is redelivered and how long the consumer thread waits before each redelivery.
When to change it
Always replace it. Use a bounded exponential back-off for transient failures and keep every wait well below max.poll.interval.ms. For waits longer than that, use ContainerPausingBackOffHandler or retry topics.
DefaultErrorHandler.addNotRetryableExceptionsDefault DeserializationException, MessageConversionException, ConversionException, MethodArgumentResolutionException, NoSuchMethodException, ClassCastException
What it does
Exception types for which the recoverer is called on the first failure.
When to change it
Add your own validation and business-rule exceptions. Do not add exceptions that mean a dependency is down; those are the ones retries exist for.
DeadLetterPublishingRecoverer destination resolverDefault <source topic>-dlt, same partition (the suffix was .DLT before 3.3)
What it does
A function from the failed record and the exception to the topic and partition of the dead-letter record.
When to change it
Provide it explicitly. Give the dead-letter topic at least as many partitions as the source, or return a negative partition and let the producer choose.
spring.deserializer.value.delegate.classDefault none
What it does
The real value deserializer that ErrorHandlingDeserializer wraps. There is a matching spring.deserializer.key.delegate.class.
When to change it
Set both whenever the deserializer can fail, which includes every JSON, Avro or Protobuf deserializer. Then make sure the dead-letter producer can serialize byte[].
spring.kafka.listener.ack-modeDefault batch
What it does
When the container commits offsets: after all records of a poll (batch), after each record (record), or under your control (manual, manual_immediate).
When to change it
record reduces redelivery after a crash at the cost of more commits. With manual_immediate, set commitRecovered on the error handler so that a recovered record's offset is committed.
max.poll.interval.ms (consumer)Default 300000 (5 minutes)
What it does
The longest allowed gap between two calls to poll() before the consumer is removed from the group.
When to change it
It is the ceiling for blocking retries. Prefer shorter back-offs to raising it; a larger value also delays the detection of a truly stuck consumer.
ContainerProperties.deliveryAttemptHeaderDefault false
What it does
Adds the kafka_deliveryAttempt header (KafkaHeaders.DELIVERY_ATTEMPT), counting from 1, to records given to a record listener.
When to change it
Turn it on if the listener or a log line needs the attempt number. It counts blocking attempts in this consumer instance only; it resets after a rebalance or restart.
@RetryableTopic attempts / backoffDefault attempts 3 (including the first), fixed delay of 1000 ms
What it does
The total number of deliveries across the main and retry topics, and the delay before each retry delivery.
When to change it
Size the delays to the outage you expect to ride out. The same values are available without the annotation through spring.kafka.retry.topic.* (disabled by default).
@RetryableTopic numPartitions / replicationFactorDefault 1 / -1 (the broker default)
What it does
How the retry and dead-letter topics are created when autoCreateTopics is true (the default).
When to change it
Match the source topic's partition count. In production many teams set autoCreateTopics to false and create the topics with the same tooling as every other topic.
@RetryableTopic dltStrategyDefault ALWAYS_RETRY_ON_ERROR
What it does
What happens when the @DltHandler method itself throws: the record is sent to the DLT again, the listener fails (FAIL_ON_ERROR), or there is no DLT at all (NO_DLT).
When to change it
Keep the DLT handler trivial (log, count, store) so that the default does not turn into a loop. Fatal exceptions such as a deserialization failure are not sent back.

Hands-on exercise

Predict the path of three records, then check in the lab

Work out from the configuration alone where each failing record ends up and how long the partition waits, then confirm the behaviour in the poison-message lab.

Steps · 0 of 6 done

Take it further in the lab: Poison message and dead-letter queue

What can go wrong

Catching the exception in the listener
Symptom
Errors appear in the log, no record ever reaches the dead-letter topic, and downstream data is missing for exactly the records that logged an error.
Cause
A try/catch around the listener body logs and returns. To the container a normal return means success, so the offset is committed.
Fix
Let exceptions leave the listener. Translate them into your transient or permanent exception types if you need to, but rethrow.
Dead letters that are not the original message
Symptom
Records in the dead-letter topic for deserialization failures are quoted Base64 strings, or the dead-letter publish itself fails with a serialization error and the consumer stalls.
Cause
The recoverer restores the value to the raw byte[], but the template's only value serializer is a JSON or Avro serializer.
Fix
Use DelegatingByTypeSerializer with a ByteArraySerializer for byte[], or give the recoverer a second template for byte[] values.
Back-off longer than the poll interval
Symptom
During a dependency outage the group rebalances every few minutes, the same records are processed by different instances, and the dead-letter topic receives records after far more attempts than configured.
Cause
The consumer thread sleeps in the back-off and misses max.poll.interval.ms. The partition moves to another instance, which starts the attempt count again.
Fix
Shorten the blocking back-off, or move long waits to @RetryableTopic or a ContainerPausingBackOffHandler, which keeps polling while paused.
Retry topics for events that must stay in order
Symptom
An order is marked SHIPPED and then, minutes later, PAID. The later state overwrote the newer one.
Cause
@RetryableTopic moved the failed PAID event to a retry topic and the listener went on to SHIPPED for the same key.
Fix
Use blocking retries where per-key order matters, or make the consumer reject stale events by version. See ordering, shutdown and recovery.
Retry and dead-letter topics with one partition
Symptom
Retried records for a 12-partition topic all land in partition 0 of the retry topics; retry lag is high while most consumer threads sit idle.
Cause
@RetryableTopic auto-creates its topics with numPartitions = 1 unless told otherwise.
Fix
Set numPartitions to the source topic's partition count, or create the topics yourself and set autoCreateTopics = "false".
A whole batch in the dead-letter topic
Symptom
Hundreds of perfectly valid records appear in the dead-letter topic at the same second as one invalid one.
Cause
A batch listener threw an ordinary exception. The handler retried the complete batch and then recovered every record in it.
Fix
Catch per record inside the loop and throw BatchListenerFailedException with the index of the failing record.
Expecting the error handler in a transactional container
Symptom
After adding transaction-id-prefix, failed records are redelivered ten times and dropped; the DefaultErrorHandler bean seems to be ignored.
Cause
With a Kafka transaction manager on the container, a failure rolls the transaction back and the AfterRollbackProcessor decides what happens next. Its default is also ten attempts, then log.
Fix
Configure a DefaultAfterRollbackProcessor with the same recoverer and back-off. The transactions lesson shows it.

How to troubleshoot

One partition's lag grows while the others stay near zero

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
A CURRENT-OFFSET that stands still on one partition while LOG-END-OFFSET rises means one record is being retried or cannot be recovered. Look for the service's log lines for that partition and offset.
The same stack trace repeats and the offset never moves, even though a dead-letter recoverer is configured

Check

Service log: the RetryListener line `Dead-letter publish failed for ...`, or a producer error such as TimeoutException or UnknownTopicOrPartitionException close to it
How to read it
The recoverer is throwing, so the record is kept. Usual causes: the dead-letter topic does not exist and auto-creation is off, the principal has no write permission on it, or the value cannot be serialized.
A record was skipped and nobody knows why

Check

Service log at ERROR from the framework's failure tracker: a line starting `Backoff` and ending `exhausted for <topic>-<partition>@<offset>`
How to read it
That is the default recoverer: the record was logged and skipped. Either no DefaultErrorHandler bean with a recoverer exists, or the listener uses a container factory the bean was never attached to.
You need to know why a record was dead-lettered

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-placed-dlt --from-beginning --formatter-property print.headers=true --formatter-property print.key=true
How to read it
Read kafka_dlt-original-topic, kafka_dlt-exception-fqcn and kafka_dlt-exception-message. The original partition, offset and timestamp headers are binary numbers and print as raw bytes; the dead-letter record normally sits in the same partition number as its source.
The consumer logs a deserialization error for ever and the listener is never called

Check

spring.kafka.consumer.value-deserializer in the effective configuration, and the log for RecordDeserializationException
How to read it
The deserializer is not wrapped. The failure happens inside poll(), where no error handler can act on the record. Configure ErrorHandlingDeserializer with the real deserializer as its delegate.

Key takeaways

  • Configure nothing and a failing record is tried ten times without delay, logged and skipped. Always declare a DefaultErrorHandler with a back-off and a recoverer.
  • Classify exceptions: transient ones get a bounded back-off, permanent ones are marked not retryable and go to the dead-letter topic at once.
  • ErrorHandlingDeserializer turns an undecodable payload into a record the error handler can route. The dead-letter producer must then be able to send raw bytes.
  • Name the dead-letter topic in your own destination resolver, give it the source topic's partition count, and wait for the publish result before the offset moves.
  • Blocking retries keep order and block the partition for seconds. @RetryableTopic keeps the partition flowing for minutes or hours and gives up order.
  • In a batch listener, throw BatchListenerFailedException with the index. In a transactional container, the after-rollback processor takes the error handler's role.
  • Test the error path against a broker: publish a bad record, read it from the dead-letter topic, and assert that the next record was processed.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A payment listener uses the default Spring configuration: no error handler bean. The card provider is unreachable for 40 seconds during a deploy. Afterwards 310 orders from that window have no payment row, lag is zero, and the log holds 3,100 stack traces followed by 310 ERROR lines naming a topic, partition and offset.

What explains the missing payments?

Scenario 2 of 3

A service adds ErrorHandlingDeserializer around its JsonDeserializer and a DeadLetterPublishingRecoverer using the application's KafkaTemplate, whose value serializer is JsonSerializer. A producer bug then writes 20 truncated payloads. The consumer does not stall, but support cannot replay the dead letters: each one is a quoted string of letters and digits.

Which change makes the dead letters replayable?

Scenario 3 of 3

An inventory listener consumes orders.order-placed (12 partitions, keyed by orderId) and must apply events for one order in sequence. The warehouse API fails for 20 to 30 minutes about once a month. With a 15 second blocking back-off, each outage sends around 9,000 records to the dead-letter topic, which are replayed by hand.

Which approach fits both the ordering need and the outage length?

References

References: Spring for Apache Kafka 3.3 reference: Handling Exceptions (DefaultErrorHandler, batch error handling, after-rollback processor, publishing dead-letter records) ↗ · Spring for Apache Kafka 3.3 reference: Serialization, Deserialization, and Message Conversion (Using ErrorHandlingDeserializer, DelegatingByTypeSerializer) ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries (how the pattern works) ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries, features (back-off defaults, exception classifier, topic auto-creation) ↗ · Spring for Apache Kafka 3.3 reference: Non-Blocking Retries, topic naming (default suffixes, delay-suffixed retry topics) ↗ · Spring for Apache Kafka 3.3 Javadoc: DeadLetterPublishingRecoverer ↗ · Spring for Apache Kafka 3.3 Javadoc: RetryableTopic (attribute names and defaults) ↗ · Spring Boot 3.5 reference: Apache Kafka support (beans attached to the auto-configured container factory) ↗ · Spring Boot 3.5: Managed dependency coordinates (spring-kafka 3.3.x, kafka-clients 3.9.x) ↗ · Apache Kafka 3.9 consumer configuration reference (max.poll.interval.ms) ↗ · Report a problem