Skip to content

Path 6 · lesson 2 of 5

Intermediate28 minApache Kafka 4.x · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 (kafka-clients 3.9). Class names and defaults checked against the Spring for Apache Kafka 3.3 reference

Poison messages and error handling

Why one record that can never be processed blocks a partition under at-least-once delivery, and how to classify, skip, dead-letter or stop in Spring for Apache Kafka.

What you will learn

  • Recognise a poison message from its symptoms: lag on one partition, the same offset failing repeatedly, or a consumer in a restart loop
  • Explain why at-least-once delivery turns one unprocessable record into a blocked partition
  • Configure ErrorHandlingDeserializer so that a payload that cannot be deserialised reaches the error handler instead of failing inside poll()
  • Classify exceptions as retryable or not in DefaultErrorHandler, so permanent failures skip the retries
  • Choose between skipping, dead-lettering and stopping the container for a given topic, and implement the choice

Before you start

The concept

A poison message is a record that this consumer can never process successfully, however many times it is delivered. The failure belongs to the record (or to the code that meets it), not to the environment. Waiting does not help and retrying does not help.

KindExampleWhere it fails
Cannot be deserialisedA producer wrote plain text to a JSON topic, the payload is truncated, or the schema changed in an incompatible wayIn the deserialiser, inside poll(), before your listener is called
Deserialises, but the data is invalidA payment with a negative amount, an unknown currency code, a reference to an order that does not existIn your listener's validation or in a database constraint
Valid data that hits a bugA null field the code did not expect, an integer overflow on an unusually large valueAnywhere in your listener, as an unexpected runtime exception

Under at-least-once delivery the consumer commits an offset only after the record was processed. A committed offset is the position of the next record to read, so the group cannot move past a record without committing beyond it. If the record fails and nothing commits beyond it, the next fetch returns the same record again. Because a partition is read in order, every record behind it waits.

Blocked partition

  • The exception is caught by the framework and the record is redelivered
  • The same offset fails again and again, usually many times per second
  • Lag grows on that one partition while the others are healthy
  • The log fills with the same stack trace

Crash loop

  • The record makes the process die: out of memory, a fatal error, a failed health check
  • The orchestrator restarts the instance, it reads the same offset, and dies again
  • After the rebalance, the next instance receives the partition and dies too
  • One record takes down the whole consumer group, one member at a time

The Break card The message that kills its reader is a crash loop of the second kind in a real system. Both shapes have the same root: the delivery guarantee is doing exactly what it promises, and the application has no rule for a record it cannot handle.

Why it exists

Before there is an explicit rule for bad records, a team has two bad choices. Catch every exception in the listener and log it, which silently turns at-least-once into at-most-once for anything that fails, including records that failed only because the database was restarting. Or let every exception propagate, which is safe for transient failures and fatal for permanent ones.

Error handling in a Kafka consumer exists to give a third path: decide per failure. A transient failure is retried in place. A permanent failure leaves the partition quickly, in a way that keeps the record and the reason for later repair. To do that, three things have to be true.

  1. Every failure must reach one place that can make the decision. Deserialisation failures do not, by default: they happen inside the Kafka client before any application code runs.
  2. The decision must be driven by the kind of failure. A parse error and a connection timeout need opposite treatment, so they must be distinguishable by exception type.
  3. Moving on must be safe. If the record is set aside, it has to be stored somewhere durable before the offset is committed past it, or it is lost.

Spring for Apache Kafka provides a part for each: ErrorHandlingDeserializer for the first, the exception classification in DefaultErrorHandler for the second, and DeadLetterPublishingRecoverer for the third.

How it works inside

Why a deserialisation failure is special

The Kafka consumer deserialises keys and values inside poll(). If the configured deserialiser throws, poll() throws, and the listener container never receives a record it could hand to an error handler. The consumer's position has not moved, so the next poll() fetches the same bytes and throws again. The Spring reference puts it plainly: when a deserialiser fails, Spring has no way to handle the problem, because it occurs before poll() returns.

ErrorHandlingDeserializer is a wrapper that removes this special case. You configure it as the deserialiser and tell it which real deserialiser to delegate to. When the delegate throws, the wrapper does not throw. It returns a null value and stores a DeserializationException, containing the cause and the raw bytes, in a record header. poll() returns normally. The container sees the header, does not call your listener for that record, and passes the record and the exception straight to the error handler.

What DefaultErrorHandler does with a failure

  1. The listener throws (or the record arrives with a deserialisation error header). The container first commits the offsets of the records it had already processed from this poll.
  2. The handler classifies the exception. If it is on the not-retryable list, the retries are skipped and the recoverer is called on this first failure.
  3. Otherwise it asks its BackOff for the next delay. While attempts remain, it waits, then seeks the partition back to the failed offset (and other partitions to their first unprocessed record), so the next poll() returns the failed record again.
  4. When the back-off is exhausted, it calls the recoverer with the record and the exception. The default recoverer only logs.
  5. If the recoverer returns normally, the record counts as handled and its offset is committed. The partition moves on.
  6. If the recoverer throws, for example because the dead letter topic cannot be reached, the record is included in the seeks and delivered again. Nothing is lost.

The default back-off is FixedBackOff(0, 9): no delay, 9 retries, 10 deliveries in total. The retry counter is kept in memory per topic, partition and offset by the consumer that owns the partition, so it starts again after a restart or a rebalance.

Classifying exceptions

DefaultErrorHandler treats these exception types as fatal by default and does not retry them: DeserializationException, MessageConversionException, ConversionException, MethodArgumentResolutionException, NoSuchMethodException and ClassCastException. All of them describe a record or a listener signature that will be equally wrong on the next delivery. Everything else is retried.

That default knows nothing about your business rules. Add your own permanent failures with addNotRetryableExceptions(...). The cleanest way is to throw a dedicated exception type from validation, for example InvalidPaymentException, and register that type. Be careful with broad types: registering IllegalArgumentException or NullPointerException also catches the ones thrown by libraries for reasons you did not plan for.

Skip, dead-letter or stop

ChoiceHowWhat happens to the recordUse when
SkipDefaultErrorHandler with its default logging recovererLogged and dropped from the flowThe record has no lasting value: metrics, cache refresh hints, anything a later record supersedes
Dead-letterDefaultErrorHandler with a DeadLetterPublishingRecovererPublished to another topic with headers that record the origin and the exception, then the offset is committedThe record matters, and later records for other keys should not wait for it. This is the common choice
StopCommonContainerStoppingErrorHandlerStays where it is. The container stops and nothing more is consumed until someone restarts itProcessing later records without this one would be wrong: a ledger, a strict sequence per partition

DeadLetterPublishingRecoverer publishes, by default, to a topic named after the original with the suffix -dlt, and to the same partition number as the original record. The dead letter topic therefore needs at least as many partitions as the source topic, unless you supply your own destination resolver. It adds headers with the original topic, partition, offset and timestamp, the consumer group, and the exception class, message and stack trace.

For a record that failed deserialisation, the recoverer publishes the original bytes, so the dead letter record is exactly what the producer sent. This has a consequence for configuration: the KafkaTemplate used by the recoverer must be able to write both a normal object and a raw byte[]. The reference recommends a DelegatingByTypeSerializer for that, as in the code below.

Retry topics & DLQ

Simulation, not a real cluster

The animated lesson shows one failing record blocking a partition, then the same stream with a retry topic and a dead letter topic.

Step 1 · The pattern

One at a time, in order

A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.

  • message
  • processed
Open the full animated lesson

Architecture

Drawing diagram…

A flowchart that follows a polled record through deserialisation, the listener, exception classification, retries by seeking back to the failed offset, and the recoverer, ending either in a committed offset or in a redelivery.

flowchart TD
  POLL["poll returns a record"] --> DESER{"Deserialised?"}
  DESER -- "No" --> HDR["Null value plus DeserializationException header"]
  HDR --> EH["DefaultErrorHandler"]
  DESER -- "Yes" --> LIS["Listener method"]
  LIS -- "Returns" --> COMMIT["Offset committed, next record"]
  LIS -- "Throws" --> EH
  EH --> CLASS{"Retryable exception?"}
  CLASS -- "No" --> REC["Recoverer"]
  CLASS -- "Yes" --> LEFT{"Attempts left?"}
  LEFT -- "Yes" --> SEEK["Back off, seek to failed offset"]
  SEEK --> POLL
  LEFT -- "No" --> REC
  REC --> PUB{"Dead letter publish succeeded?"}
  PUB -- "Yes" --> COMMIT
  PUB -- "No" --> SEEK
The decision path of one record through ErrorHandlingDeserializer, DefaultErrorHandler and the recoverer.
  • There are two ways into the error handler: a deserialisation failure, which arrives without the listener ever being called, and an exception thrown by the listener. From there on they are treated by the same rules.
  • The only exits to the next record are a listener that returns and a recoverer that succeeds. Every other path leads back to the same offset. That is what makes the flow safe: a record is passed over only after it was processed or stored elsewhere.
  • The loop through the seek is where a partition blocks. A permanent failure that is classified as retryable goes round it until the attempts run out; with an unlimited back-off it goes round for ever.

Message flow

Drawing diagram…

A sequence diagram in which the listener container polls a malformed record, the ErrorHandlingDeserializer returns a null value with an exception header, the DefaultErrorHandler passes it to the DeadLetterPublishingRecoverer, the record is written to the dead letter topic, and the container commits offset 413.

sequenceDiagram
  participant K as Topic payments.requested p2
  participant C as Listener container
  participant D as ErrorHandlingDeserializer
  participant E as DefaultErrorHandler
  participant R as DeadLetterPublishingRecoverer
  participant T as Topic payments.requested-dlt p2
  C->>K: poll
  K-->>C: Record at offset 412, raw bytes
  C->>D: Deserialise value
  D-->>C: Null value and exception in header
  Note over C: Listener is not called for offset 412
  C->>E: Handle record 412 with DeserializationException
  Note over E: Not retryable, go to recovery now
  E->>R: Recover record 412
  R->>T: Send original bytes with origin and error headers
  T-->>R: Ack
  R-->>E: Recovered
  E-->>C: Handled
  C->>K: Commit offset 413
  C->>K: poll
  K-->>C: Record at offset 413
A malformed payload at offset 412 is dead-lettered on its first delivery and the partition continues at 413.
  1. The deserialiser fails but does not throw, so poll() completes and the container has a record object to work with. Without the wrapper the diagram would end at the third arrow, repeated for ever.
  2. DeserializationException is on the default not-retryable list, so there are no retries. One delivery, one dead letter record.
  3. The commit of offset 413 comes after the dead letter topic acknowledged the write. If that send fails, the recoverer throws and offset 412 is delivered again.
  4. The dead letter record carries the bytes the producer sent, in the same partition number, with headers that name the original topic, partition and offset. Someone can later find out exactly what arrived and why it was rejected.

In a real system

A typical payment flow: the Order Service publishes PaymentRequested as JSON to payments.requested, 6 partitions, keyed by order id. The Payment Service consumes it in the group payment-service with three instances and charges the customer. The dead letter topic is payments.requested-dlt, also with 6 partitions.

Three different poison messages arrive in the same month, and each needs a different part of the configuration.

What arrivedWithout handlingWith the configuration below
A test tool writes the text hello to the production topicpoll() throws on partition 2 for ever. Lag on partition 2 grows; restarts do not helpErrorHandlingDeserializer captures the failure. The bytes go to the dead letter topic on the first delivery
A new Order Service version sends amountMinor: -1500 for refunds, which this service does not supportThe listener throws on every delivery. Ten fast attempts, then the default handler logs the record and skips itThe listener throws InvalidPaymentException, registered as not retryable. Dead-lettered on the first delivery, with the message in a header
The payments database fails over for 90 secondsEvery record fails ten times in milliseconds and is skippedNot a poison message. A different exception type, retried with back-off. See the next lesson

The third row is there on purpose. The most damaging mistake in this area is to build a good dead letter flow and then send transient failures down it. The classification is the design; the dead letter topic is only where one branch ends.

Java and Spring Boot

A complete Spring Boot 3 configuration for the Payment Service: the wrapped deserialiser in application.yml, an error handler with classification and a dead letter recoverer, and a listener that throws a dedicated exception for invalid data. Spring Boot adds a CommonErrorHandler bean to the auto-configured listener container factory by itself.

payment-service/src/main/resources/application.ymlYAML
spring:  kafka:    bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092    consumer:      group-id: payment-service      auto-offset-reset: earliest      enable-auto-commit: false      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer      properties:        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer        spring.json.use.type.headers: false        spring.json.value.default.type: com.example.payments.PaymentRequested        spring.json.trusted.packages: com.example.payments    listener:      ack-mode: record

Why it is written this way

  • value-deserializer is the wrapper. The real deserialiser is named in spring.deserializer.value.delegate.class. Swapping these two lines is the usual mistake and brings the endless poll() failure back.
  • spring.json.use.type.headers: false with spring.json.value.default.type makes JsonDeserializer ignore the type header a Spring producer adds and always create this service's own class, so the consumer does not depend on the producer's class names.
  • ack-mode: record commits after each record. The Spring default is BATCH, which is also correct here: the container commits the already processed records before it calls the error handler.
  • enable-auto-commit: false leaves offset commits to the listener container, which is what makes commit-after-processing hold.

Configuration

The settings that decide what happens to a record that fails. Defaults are those of Spring for Apache Kafka 3.3 and Spring Boot 3.5.

spring.kafka.consumer.value-deserializerDefault org.apache.kafka.common.serialization.StringDeserializer (Spring Boot's default)
What it does
The class the consumer uses to turn value bytes into an object. It runs inside poll().
When to change it
Set it to ErrorHandlingDeserializer whenever the real deserialiser can throw: JSON, Avro, Protobuf. Do the same for the key if the key is not a plain string.
spring.deserializer.value.delegate.classDefault none
What it does
Consumer property read by ErrorHandlingDeserializer: the real value deserialiser it delegates to. The key variant is spring.deserializer.key.delegate.class.
When to change it
Required when the wrapper is configured by class name. In Spring Boot it goes under spring.kafka.consumer.properties.
spring.json.trusted.packagesDefault java.util, java.lang
What it does
Packages from which JsonDeserializer may create classes named in type headers.
When to change it
Add your event package. A class outside the list fails deserialisation, which with the wrapper becomes a dead letter record instead of a blocked partition.
DefaultErrorHandler back-offDefault FixedBackOff(0, 9): 9 retries, no delay
What it does
How many times and how far apart a failed record is redelivered before the recoverer is called.
When to change it
Always set it deliberately. Ten instant attempts are too many for a bad record and far too short for a dependency outage.
DefaultErrorHandler recovererDefault Logs the record at ERROR level
What it does
What happens to a record when retries are exhausted or the exception is not retryable.
When to change it
Pass a DeadLetterPublishingRecoverer unless losing the record is acceptable for this topic.
DefaultErrorHandler.addNotRetryableExceptions(...)Default DeserializationException, MessageConversionException, ConversionException, MethodArgumentResolutionException, NoSuchMethodException, ClassCastException
What it does
Exception types for which retries are skipped and the recoverer is called on the first failure.
When to change it
Add your own validation exception types. Keep dependency failures off this list.
DeadLetterPublishingRecoverer destinationDefault <original topic>-dlt, same partition as the original record
What it does
Where the failed record is published. A resolver function passed to the constructor can choose another topic or partition per record and exception.
When to change it
Create the dead letter topic with at least as many partitions as the source, or supply a resolver. A resolver that returns a negative partition lets the producer choose.
spring.kafka.listener.ack-modeDefault BATCH (the container default when the property is not set)
What it does
When the container commits offsets: after each record, after each poll's records, or under manual control.
When to change it
RECORD or BATCH both work with DefaultErrorHandler. With manual immediate acknowledgement, set commitRecovered on the handler so recovered records are committed.

Hands-on exercise

Poison a partition, then give the record somewhere to go

Inject a record that can never be processed, watch it block one partition while the others keep working, then add error handling and send the same record again.

Steps · 0 of 6 done

Take it further in the lab: Poison message and dead-letter queue

What can go wrong

The wrapper is configured, but as the delegate
Symptom
The consumer logs the same deserialisation error many times per second and lag grows on one partition, although the project uses ErrorHandlingDeserializer.
Cause
value-deserializer is still JsonDeserializer. The wrapper is named somewhere it has no effect, or only the key is wrapped. The delegate throws inside poll() as before.
Fix
Set value-deserializer to ErrorHandlingDeserializer and name the real deserialiser in spring.deserializer.value.delegate.class.
Catching everything in the listener
Symptom
No errors in the consumer group, no lag, and payments are missing. The log has warnings that nobody reads.
Cause
A try/catch (Exception e) around the listener body logs and returns. The container sees a successful call and commits. Every failure, transient or permanent, is skipped.
Fix
Let exceptions leave the listener. Translate them to meaningful types if needed, and let the error handler retry or dead-letter.
Relying on the default handler
Symptom
A record fails ten times within a second, an ERROR line appears, and processing continues. Days later someone asks where that payment went.
Cause
No error handler bean was defined. The default retries 9 times with no delay and then logs and skips.
Fix
Define a DefaultErrorHandler bean with an explicit back-off and a DeadLetterPublishingRecoverer.
Dead letter template cannot write the failed record
Symptom
Records that fail deserialisation are redelivered for ever, with a serialisation or class cast error from the recoverer in the log.
Cause
The recoverer publishes the original byte[] for a deserialisation failure, and the template's value serialiser accepts only the event class. The recoverer throws, so the record is redelivered.
Fix
Give the recoverer a template whose value serialiser is a DelegatingByTypeSerializer with ByteArraySerializer for byte[] and the normal serialiser for everything else.
Dead letter topic with fewer partitions than the source
Symptom
Dead letter records from high partition numbers land in unexpected partitions, or publishing fails on older Spring versions.
Cause
The default resolver targets the same partition number as the original record. Since version 2.7 the recoverer checks that the partition exists and lets the producer choose when it does not.
Fix
Create the dead letter topic with the same partition count as the source, or pass a destination resolver that sets the partition you want.
Unlimited retries on a permanent failure
Symptom
One partition has been stuck on one offset for hours. The exception is a validation error.
Cause
The back-off is unlimited, or the exception type is not on the not-retryable list, so a failure that cannot pass is retried indefinitely.
Fix
Bound the back-off and register permanent exception types as not retryable. Reserve long or unlimited retries for failures of a dependency.

How to troubleshoot

Lag grows on one partition only

Check

kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group payment-service
How to read it
Run it twice, a minute apart. A partition whose CURRENT-OFFSET is identical both times while LOG-END-OFFSET rises, with a consumer assigned, is stuck on the record at that offset. Other partitions moving normally rules out a general slowdown.
You need to see the record that blocks the partition

Check

kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic payments.requested --partition 2 --offset 412 --max-messages 1
How to read it
Prints the raw value at the stuck offset, taken from the CURRENT-OFFSET column. Plain text in a JSON topic, a truncated document or an unexpected shape tells you which kind of poison message it is.
The same exception repeats in the consumer log

Check

The exception class and the topic, partition and offset in the listener container's ERROR lines
How to read it
The same offset with a deserialisation error means the failure is in poll() or was captured by the wrapper. The same offset with your own exception means the listener rejects it and the classification or back-off lets it retry too long.
Records are missing and the dead letter topic is empty

Check

Search the consumer log for the ERROR line the default recoverer writes when retries are exhausted, and check whether a `DefaultErrorHandler` bean with a recoverer exists
How to read it
If such lines exist, records are being skipped by the default logging recoverer. If there are none, look for a catch block in the listener that swallows the exception.
You want to know why a record was dead-lettered

Check

kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic payments.requested-dlt --from-beginning --formatter-property print.headers=true
How to read it
The headers kafka_dlt-original-topic, kafka_dlt-original-partition, kafka_dlt-original-offset, kafka_dlt-exception-fqcn and kafka_dlt-exception-message identify the source record and the failure. Tools older than Kafka 4.2 take --property print.headers=true instead.

Key takeaways

  • A poison message is a record that fails on every delivery. Under at-least-once delivery it blocks its partition, or crashes each consumer that reads it, until something commits past it.
  • Deserialisation happens inside poll(), before your code. ErrorHandlingDeserializer turns that failure into a record the error handler can deal with.
  • The default DefaultErrorHandler makes 10 fast attempts and then logs and skips. A dead letter topic exists only if you configure a recoverer.
  • Classify by exception type: permanent failures go to the recoverer on the first delivery, transient ones are retried. Do not let a dependency outage flow into the dead letter topic.
  • Skip when the record has no lasting value, dead-letter when it matters and others can proceed, stop when nothing after it may be processed without it.
  • A record is passed over only after the recoverer succeeded. If the dead letter publish fails, the record is delivered again.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A Spring Boot consumer reads payments.requested with value-deserializer set to JsonDeserializer and a DefaultErrorHandler bean with a DeadLetterPublishingRecoverer. At 10:12 lag starts to grow on partition 4 only. The log shows a deserialisation error for partition 4, offset 9001, repeated several times per second. The dead letter topic is empty.

Why does the dead letter flow not catch this record?

Scenario 2 of 3

An inventory consumer validates each StockAdjusted event and throws IllegalStateException when the warehouse id is unknown. The error handler is new DefaultErrorHandler(recoverer, new FixedBackOff(5000L, 12L)). Operations reports that a single bad event delays every later event on its partition by about a minute before it reaches the dead letter topic. Unknown warehouse ids are never fixed by waiting.

What is the right change?

Scenario 3 of 3

A ledger service applies AccountEntry events from a topic keyed by account id. Each entry's balance check depends on all earlier entries for that account in the partition. One entry fails validation because of a bug in the producing service that will be fixed and the entry re-sent correctly within the hour.

Which handling fits this topic?

References

References: Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler, container stopping error handlers, publishing dead-letter records) ↗ · Spring for Apache Kafka 3.3 reference: Serialization, deserialization and message conversion (Using ErrorHandlingDeserializer) ↗ · Spring for Apache Kafka 3.3 Javadoc: DefaultErrorHandler ↗ · Apache Kafka documentation: Consumer configs (enable.auto.commit, key and value deserializers) ↗ · Apache Kafka documentation: Basic Kafka operations (kafka-consumer-groups.sh) ↗ · Spring Boot 3.5 reference: Common application properties (spring.kafka.*) ↗ · Report a problem