Path 6 · lesson 2 of 5
Poison messages and error handling
Why one record that can never be processed blocks a partition under at-least-once delivery, and how to classify, skip, dead-letter or stop in Spring for Apache Kafka.
What you will learn
- Recognise a poison message from its symptoms: lag on one partition, the same offset failing repeatedly, or a consumer in a restart loop
- Explain why at-least-once delivery turns one unprocessable record into a blocked partition
- Configure
ErrorHandlingDeserializerso that a payload that cannot be deserialised reaches the error handler instead of failing insidepoll() - Classify exceptions as retryable or not in
DefaultErrorHandler, so permanent failures skip the retries - Choose between skipping, dead-lettering and stopping the container for a given topic, and implement the choice
Before you start
The concept
A poison message is a record that this consumer can never process successfully, however many times it is delivered. The failure belongs to the record (or to the code that meets it), not to the environment. Waiting does not help and retrying does not help.
| Kind | Example | Where it fails |
|---|---|---|
| Cannot be deserialised | A producer wrote plain text to a JSON topic, the payload is truncated, or the schema changed in an incompatible way | In the deserialiser, inside poll(), before your listener is called |
| Deserialises, but the data is invalid | A payment with a negative amount, an unknown currency code, a reference to an order that does not exist | In your listener's validation or in a database constraint |
| Valid data that hits a bug | A null field the code did not expect, an integer overflow on an unusually large value | Anywhere in your listener, as an unexpected runtime exception |
Under at-least-once delivery the consumer commits an offset only after the record was processed. A committed offset is the position of the next record to read, so the group cannot move past a record without committing beyond it. If the record fails and nothing commits beyond it, the next fetch returns the same record again. Because a partition is read in order, every record behind it waits.
Blocked partition
- The exception is caught by the framework and the record is redelivered
- The same offset fails again and again, usually many times per second
- Lag grows on that one partition while the others are healthy
- The log fills with the same stack trace
Crash loop
- The record makes the process die: out of memory, a fatal error, a failed health check
- The orchestrator restarts the instance, it reads the same offset, and dies again
- After the rebalance, the next instance receives the partition and dies too
- One record takes down the whole consumer group, one member at a time
The Break card The message that kills its reader is a crash loop of the second kind in a real system. Both shapes have the same root: the delivery guarantee is doing exactly what it promises, and the application has no rule for a record it cannot handle.
Why it exists
Before there is an explicit rule for bad records, a team has two bad choices. Catch every exception in the listener and log it, which silently turns at-least-once into at-most-once for anything that fails, including records that failed only because the database was restarting. Or let every exception propagate, which is safe for transient failures and fatal for permanent ones.
Error handling in a Kafka consumer exists to give a third path: decide per failure. A transient failure is retried in place. A permanent failure leaves the partition quickly, in a way that keeps the record and the reason for later repair. To do that, three things have to be true.
- Every failure must reach one place that can make the decision. Deserialisation failures do not, by default: they happen inside the Kafka client before any application code runs.
- The decision must be driven by the kind of failure. A parse error and a connection timeout need opposite treatment, so they must be distinguishable by exception type.
- Moving on must be safe. If the record is set aside, it has to be stored somewhere durable before the offset is committed past it, or it is lost.
Spring for Apache Kafka provides a part for each: ErrorHandlingDeserializer for the first, the exception classification in DefaultErrorHandler for the second, and DeadLetterPublishingRecoverer for the third.
How it works inside
Why a deserialisation failure is special
The Kafka consumer deserialises keys and values inside poll(). If the configured deserialiser throws, poll() throws, and the listener container never receives a record it could hand to an error handler. The consumer's position has not moved, so the next poll() fetches the same bytes and throws again. The Spring reference puts it plainly: when a deserialiser fails, Spring has no way to handle the problem, because it occurs before poll() returns.
ErrorHandlingDeserializer is a wrapper that removes this special case. You configure it as the deserialiser and tell it which real deserialiser to delegate to. When the delegate throws, the wrapper does not throw. It returns a null value and stores a DeserializationException, containing the cause and the raw bytes, in a record header. poll() returns normally. The container sees the header, does not call your listener for that record, and passes the record and the exception straight to the error handler.
What DefaultErrorHandler does with a failure
- The listener throws (or the record arrives with a deserialisation error header). The container first commits the offsets of the records it had already processed from this poll.
- The handler classifies the exception. If it is on the not-retryable list, the retries are skipped and the recoverer is called on this first failure.
- Otherwise it asks its
BackOfffor the next delay. While attempts remain, it waits, then seeks the partition back to the failed offset (and other partitions to their first unprocessed record), so the nextpoll()returns the failed record again. - When the back-off is exhausted, it calls the recoverer with the record and the exception. The default recoverer only logs.
- If the recoverer returns normally, the record counts as handled and its offset is committed. The partition moves on.
- If the recoverer throws, for example because the dead letter topic cannot be reached, the record is included in the seeks and delivered again. Nothing is lost.
The default back-off is FixedBackOff(0, 9): no delay, 9 retries, 10 deliveries in total. The retry counter is kept in memory per topic, partition and offset by the consumer that owns the partition, so it starts again after a restart or a rebalance.
Classifying exceptions
DefaultErrorHandler treats these exception types as fatal by default and does not retry them: DeserializationException, MessageConversionException, ConversionException, MethodArgumentResolutionException, NoSuchMethodException and ClassCastException. All of them describe a record or a listener signature that will be equally wrong on the next delivery. Everything else is retried.
That default knows nothing about your business rules. Add your own permanent failures with addNotRetryableExceptions(...). The cleanest way is to throw a dedicated exception type from validation, for example InvalidPaymentException, and register that type. Be careful with broad types: registering IllegalArgumentException or NullPointerException also catches the ones thrown by libraries for reasons you did not plan for.
Skip, dead-letter or stop
| Choice | How | What happens to the record | Use when |
|---|---|---|---|
| Skip | DefaultErrorHandler with its default logging recoverer | Logged and dropped from the flow | The record has no lasting value: metrics, cache refresh hints, anything a later record supersedes |
| Dead-letter | DefaultErrorHandler with a DeadLetterPublishingRecoverer | Published to another topic with headers that record the origin and the exception, then the offset is committed | The record matters, and later records for other keys should not wait for it. This is the common choice |
| Stop | CommonContainerStoppingErrorHandler | Stays where it is. The container stops and nothing more is consumed until someone restarts it | Processing later records without this one would be wrong: a ledger, a strict sequence per partition |
DeadLetterPublishingRecoverer publishes, by default, to a topic named after the original with the suffix -dlt, and to the same partition number as the original record. The dead letter topic therefore needs at least as many partitions as the source topic, unless you supply your own destination resolver. It adds headers with the original topic, partition, offset and timestamp, the consumer group, and the exception class, message and stack trace.
For a record that failed deserialisation, the recoverer publishes the original bytes, so the dead letter record is exactly what the producer sent. This has a consequence for configuration: the KafkaTemplate used by the recoverer must be able to write both a normal object and a raw byte[]. The reference recommends a DelegatingByTypeSerializer for that, as in the code below.
Retry topics & DLQ
Simulation, not a real clusterThe animated lesson shows one failing record blocking a partition, then the same stream with a retry topic and a dead letter topic.
Step 1 · The pattern
One at a time, in order
A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.
- message
- processed
Architecture
Drawing diagram…
A flowchart that follows a polled record through deserialisation, the listener, exception classification, retries by seeking back to the failed offset, and the recoverer, ending either in a committed offset or in a redelivery.
flowchart TD
POLL["poll returns a record"] --> DESER{"Deserialised?"}
DESER -- "No" --> HDR["Null value plus DeserializationException header"]
HDR --> EH["DefaultErrorHandler"]
DESER -- "Yes" --> LIS["Listener method"]
LIS -- "Returns" --> COMMIT["Offset committed, next record"]
LIS -- "Throws" --> EH
EH --> CLASS{"Retryable exception?"}
CLASS -- "No" --> REC["Recoverer"]
CLASS -- "Yes" --> LEFT{"Attempts left?"}
LEFT -- "Yes" --> SEEK["Back off, seek to failed offset"]
SEEK --> POLL
LEFT -- "No" --> REC
REC --> PUB{"Dead letter publish succeeded?"}
PUB -- "Yes" --> COMMIT
PUB -- "No" --> SEEK- There are two ways into the error handler: a deserialisation failure, which arrives without the listener ever being called, and an exception thrown by the listener. From there on they are treated by the same rules.
- The only exits to the next record are a listener that returns and a recoverer that succeeds. Every other path leads back to the same offset. That is what makes the flow safe: a record is passed over only after it was processed or stored elsewhere.
- The loop through the seek is where a partition blocks. A permanent failure that is classified as retryable goes round it until the attempts run out; with an unlimited back-off it goes round for ever.
Message flow
Drawing diagram…
A sequence diagram in which the listener container polls a malformed record, the ErrorHandlingDeserializer returns a null value with an exception header, the DefaultErrorHandler passes it to the DeadLetterPublishingRecoverer, the record is written to the dead letter topic, and the container commits offset 413.
sequenceDiagram participant K as Topic payments.requested p2 participant C as Listener container participant D as ErrorHandlingDeserializer participant E as DefaultErrorHandler participant R as DeadLetterPublishingRecoverer participant T as Topic payments.requested-dlt p2 C->>K: poll K-->>C: Record at offset 412, raw bytes C->>D: Deserialise value D-->>C: Null value and exception in header Note over C: Listener is not called for offset 412 C->>E: Handle record 412 with DeserializationException Note over E: Not retryable, go to recovery now E->>R: Recover record 412 R->>T: Send original bytes with origin and error headers T-->>R: Ack R-->>E: Recovered E-->>C: Handled C->>K: Commit offset 413 C->>K: poll K-->>C: Record at offset 413
- The deserialiser fails but does not throw, so
poll()completes and the container has a record object to work with. Without the wrapper the diagram would end at the third arrow, repeated for ever. DeserializationExceptionis on the default not-retryable list, so there are no retries. One delivery, one dead letter record.- The commit of offset 413 comes after the dead letter topic acknowledged the write. If that send fails, the recoverer throws and offset 412 is delivered again.
- The dead letter record carries the bytes the producer sent, in the same partition number, with headers that name the original topic, partition and offset. Someone can later find out exactly what arrived and why it was rejected.
In a real system
A typical payment flow: the Order Service publishes PaymentRequested as JSON to payments.requested, 6 partitions, keyed by order id. The Payment Service consumes it in the group payment-service with three instances and charges the customer. The dead letter topic is payments.requested-dlt, also with 6 partitions.
Three different poison messages arrive in the same month, and each needs a different part of the configuration.
| What arrived | Without handling | With the configuration below |
|---|---|---|
A test tool writes the text hello to the production topic | poll() throws on partition 2 for ever. Lag on partition 2 grows; restarts do not help | ErrorHandlingDeserializer captures the failure. The bytes go to the dead letter topic on the first delivery |
A new Order Service version sends amountMinor: -1500 for refunds, which this service does not support | The listener throws on every delivery. Ten fast attempts, then the default handler logs the record and skips it | The listener throws InvalidPaymentException, registered as not retryable. Dead-lettered on the first delivery, with the message in a header |
| The payments database fails over for 90 seconds | Every record fails ten times in milliseconds and is skipped | Not a poison message. A different exception type, retried with back-off. See the next lesson |
The third row is there on purpose. The most damaging mistake in this area is to build a good dead letter flow and then send transient failures down it. The classification is the design; the dead letter topic is only where one branch ends.
Java and Spring Boot
A complete Spring Boot 3 configuration for the Payment Service: the wrapped deserialiser in application.yml, an error handler with classification and a dead letter recoverer, and a listener that throws a dedicated exception for invalid data. Spring Boot adds a CommonErrorHandler bean to the auto-configured listener container factory by itself.
spring: kafka: bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092 consumer: group-id: payment-service auto-offset-reset: earliest enable-auto-commit: false key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer spring.json.use.type.headers: false spring.json.value.default.type: com.example.payments.PaymentRequested spring.json.trusted.packages: com.example.payments listener: ack-mode: recordWhy it is written this way
value-deserializeris the wrapper. The real deserialiser is named inspring.deserializer.value.delegate.class. Swapping these two lines is the usual mistake and brings the endlesspoll()failure back.spring.json.use.type.headers: falsewithspring.json.value.default.typemakesJsonDeserializerignore the type header a Spring producer adds and always create this service's own class, so the consumer does not depend on the producer's class names.ack-mode: recordcommits after each record. The Spring default isBATCH, which is also correct here: the container commits the already processed records before it calls the error handler.enable-auto-commit: falseleaves offset commits to the listener container, which is what makes commit-after-processing hold.
package com.example.payments;import java.util.Map;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.common.serialization.ByteArraySerializer;import org.apache.kafka.common.serialization.StringSerializer;import org.springframework.beans.factory.annotation.Value;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.core.DefaultKafkaProducerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.core.ProducerFactory;import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;import org.springframework.kafka.listener.DefaultErrorHandler;import org.springframework.kafka.support.serializer.DelegatingByTypeSerializer;import org.springframework.kafka.support.serializer.JsonSerializer;import org.springframework.util.backoff.FixedBackOff;@Configurationpublic class KafkaErrorHandlingConfig { @Bean public ProducerFactory<String, Object> producerFactory( @Value("${spring.kafka.bootstrap-servers}") String bootstrapServers) { Map<String, Object> config = Map.of( ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers, ProducerConfig.ACKS_CONFIG, "all", ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); return new DefaultKafkaProducerFactory<>(config, new StringSerializer(), new DelegatingByTypeSerializer(Map.of( byte[].class, new ByteArraySerializer(), PaymentRequested.class, new JsonSerializer<Object>()))); } @Bean public KafkaTemplate<String, Object> kafkaTemplate(ProducerFactory<String, Object> producerFactory) { return new KafkaTemplate<>(producerFactory); } @Bean public DeadLetterPublishingRecoverer deadLetterRecoverer(KafkaTemplate<String, Object> kafkaTemplate) { // Default destination: <original topic>-dlt, same partition number as the failed record. return new DeadLetterPublishingRecoverer(kafkaTemplate); } @Bean public DefaultErrorHandler errorHandler(DeadLetterPublishingRecoverer deadLetterRecoverer) { // Two retries, one second apart, for failures that might pass. Then dead-letter. DefaultErrorHandler handler = new DefaultErrorHandler(deadLetterRecoverer, new FixedBackOff(1000L, 2L)); // Permanent business failures: no retries, recover on the first delivery. handler.addNotRetryableExceptions(InvalidPaymentException.class); return handler; }}Why it is written this way
- The template's value serialiser chooses by type:
byte[]for records that failed deserialisation (the recoverer publishes the original bytes), JSON for records that were deserialised and failed later. A plainJsonSerializerwould wrap the raw bytes in JSON and the dead letter record would no longer match what the producer sent. - Defining a
ProducerFactoryand aKafkaTemplatereplaces the ones Spring Boot would auto-configure. That is intended here: one template that can write both kinds of value. FixedBackOff(1000L, 2L)means 2 retries, so 3 deliveries. The second argument counts retries, not attempts.DeserializationExceptiondoes not need to be registered. It is on the handler's default not-retryable list.- The recoverer verifies the send by default and throws if it fails, so an unavailable dead letter topic causes a redelivery, not a lost record.
package com.example.payments;import java.util.Set;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class PaymentListener { private static final Set<String> SUPPORTED_CURRENCIES = Set.of("EUR", "GBP", "USD"); private final PaymentService paymentService; public PaymentListener(PaymentService paymentService) { this.paymentService = paymentService; } @KafkaListener(topics = "payments.requested") public void onPaymentRequested(PaymentRequested request) { validate(request); paymentService.charge(request); } private void validate(PaymentRequested request) { if (request.paymentId() == null || request.orderId() == null) { throw new InvalidPaymentException("Missing payment id or order id"); } if (request.amountMinor() <= 0) { throw new InvalidPaymentException("Amount must be positive, got " + request.amountMinor()); } if (!SUPPORTED_CURRENCIES.contains(request.currency())) { throw new InvalidPaymentException("Unsupported currency " + request.currency()); } }}Why it is written this way
- Validation runs first and throws one dedicated type. The listener states that the record is permanently wrong; the error handler configuration decides what follows from that.
- There is no
try/catcharoundcharge. An exception from the database or the payment provider propagates and is retried by the handler. Catching and logging it here would commit past a payment that was never made. - The listener is never called for a record that failed deserialisation, so it does not need a null check for that case.
package com.example.payments;public record PaymentRequested(String paymentId, String orderId, long amountMinor, String currency) {}Why it is written this way
paymentIdis the idempotency key for the charge. Error handling redelivers records, soPaymentService.chargemust be safe to run twice for the same id.
package com.example.payments;public class InvalidPaymentException extends RuntimeException { public InvalidPaymentException(String message) { super(message); }}Why it is written this way
- A type of your own is safer to register as not retryable than
IllegalArgumentException, which libraries also throw for reasons that may be transient or unrelated.
Configuration
The settings that decide what happens to a record that fails. Defaults are those of Spring for Apache Kafka 3.3 and Spring Boot 3.5.
spring.kafka.consumer.value-deserializerDefault org.apache.kafka.common.serialization.StringDeserializer (Spring Boot's default)
- What it does
- The class the consumer uses to turn value bytes into an object. It runs inside
poll(). - When to change it
- Set it to
ErrorHandlingDeserializerwhenever the real deserialiser can throw: JSON, Avro, Protobuf. Do the same for the key if the key is not a plain string.
spring.deserializer.value.delegate.classDefault none
- What it does
- Consumer property read by
ErrorHandlingDeserializer: the real value deserialiser it delegates to. The key variant isspring.deserializer.key.delegate.class. - When to change it
- Required when the wrapper is configured by class name. In Spring Boot it goes under
spring.kafka.consumer.properties.
spring.json.trusted.packagesDefault java.util, java.lang
- What it does
- Packages from which
JsonDeserializermay create classes named in type headers. - When to change it
- Add your event package. A class outside the list fails deserialisation, which with the wrapper becomes a dead letter record instead of a blocked partition.
DefaultErrorHandler back-offDefault FixedBackOff(0, 9): 9 retries, no delay
- What it does
- How many times and how far apart a failed record is redelivered before the recoverer is called.
- When to change it
- Always set it deliberately. Ten instant attempts are too many for a bad record and far too short for a dependency outage.
DefaultErrorHandler recovererDefault Logs the record at ERROR level
- What it does
- What happens to a record when retries are exhausted or the exception is not retryable.
- When to change it
- Pass a
DeadLetterPublishingRecovererunless losing the record is acceptable for this topic.
DefaultErrorHandler.addNotRetryableExceptions(...)Default DeserializationException, MessageConversionException, ConversionException, MethodArgumentResolutionException, NoSuchMethodException, ClassCastException
- What it does
- Exception types for which retries are skipped and the recoverer is called on the first failure.
- When to change it
- Add your own validation exception types. Keep dependency failures off this list.
DeadLetterPublishingRecoverer destinationDefault <original topic>-dlt, same partition as the original record
- What it does
- Where the failed record is published. A resolver function passed to the constructor can choose another topic or partition per record and exception.
- When to change it
- Create the dead letter topic with at least as many partitions as the source, or supply a resolver. A resolver that returns a negative partition lets the producer choose.
spring.kafka.listener.ack-modeDefault BATCH (the container default when the property is not set)
- What it does
- When the container commits offsets: after each record, after each poll's records, or under manual control.
- When to change it
RECORDorBATCHboth work withDefaultErrorHandler. With manual immediate acknowledgement, setcommitRecoveredon the handler so recovered records are committed.
Hands-on exercise
Poison a partition, then give the record somewhere to go
Inject a record that can never be processed, watch it block one partition while the others keep working, then add error handling and send the same record again.
Steps · 0 of 6 done
What you should see
Before the fix, the same offset fails over and over, the committed offset of that partition does not move, its lag climbs while the other partitions stay near zero, and a restart changes nothing. After the fix, the bad record appears once in the dead letter topic with its original topic, partition and offset, the committed offset moves past it, and the lag drains.
What can go wrong
The wrapper is configured, but as the delegate
- Symptom
- The consumer logs the same deserialisation error many times per second and lag grows on one partition, although the project uses
ErrorHandlingDeserializer. - Cause
value-deserializeris stillJsonDeserializer. The wrapper is named somewhere it has no effect, or only the key is wrapped. The delegate throws insidepoll()as before.- Fix
- Set
value-deserializertoErrorHandlingDeserializerand name the real deserialiser inspring.deserializer.value.delegate.class.
Catching everything in the listener
- Symptom
- No errors in the consumer group, no lag, and payments are missing. The log has warnings that nobody reads.
- Cause
- A
try/catch (Exception e)around the listener body logs and returns. The container sees a successful call and commits. Every failure, transient or permanent, is skipped. - Fix
- Let exceptions leave the listener. Translate them to meaningful types if needed, and let the error handler retry or dead-letter.
Relying on the default handler
- Symptom
- A record fails ten times within a second, an ERROR line appears, and processing continues. Days later someone asks where that payment went.
- Cause
- No error handler bean was defined. The default retries 9 times with no delay and then logs and skips.
- Fix
- Define a
DefaultErrorHandlerbean with an explicit back-off and aDeadLetterPublishingRecoverer.
Dead letter template cannot write the failed record
- Symptom
- Records that fail deserialisation are redelivered for ever, with a serialisation or class cast error from the recoverer in the log.
- Cause
- The recoverer publishes the original
byte[]for a deserialisation failure, and the template's value serialiser accepts only the event class. The recoverer throws, so the record is redelivered. - Fix
- Give the recoverer a template whose value serialiser is a
DelegatingByTypeSerializerwithByteArraySerializerforbyte[]and the normal serialiser for everything else.
Dead letter topic with fewer partitions than the source
- Symptom
- Dead letter records from high partition numbers land in unexpected partitions, or publishing fails on older Spring versions.
- Cause
- The default resolver targets the same partition number as the original record. Since version 2.7 the recoverer checks that the partition exists and lets the producer choose when it does not.
- Fix
- Create the dead letter topic with the same partition count as the source, or pass a destination resolver that sets the partition you want.
Unlimited retries on a permanent failure
- Symptom
- One partition has been stuck on one offset for hours. The exception is a validation error.
- Cause
- The back-off is unlimited, or the exception type is not on the not-retryable list, so a failure that cannot pass is retried indefinitely.
- Fix
- Bound the back-off and register permanent exception types as not retryable. Reserve long or unlimited retries for failures of a dependency.
How to troubleshoot
Lag grows on one partition only
Check
kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group payment-service
- How to read it
- Run it twice, a minute apart. A partition whose CURRENT-OFFSET is identical both times while LOG-END-OFFSET rises, with a consumer assigned, is stuck on the record at that offset. Other partitions moving normally rules out a general slowdown.
You need to see the record that blocks the partition
Check
kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic payments.requested --partition 2 --offset 412 --max-messages 1
- How to read it
- Prints the raw value at the stuck offset, taken from the CURRENT-OFFSET column. Plain text in a JSON topic, a truncated document or an unexpected shape tells you which kind of poison message it is.
The same exception repeats in the consumer log
Check
The exception class and the topic, partition and offset in the listener container's ERROR lines
- How to read it
- The same offset with a deserialisation error means the failure is in
poll()or was captured by the wrapper. The same offset with your own exception means the listener rejects it and the classification or back-off lets it retry too long.
Records are missing and the dead letter topic is empty
Check
Search the consumer log for the ERROR line the default recoverer writes when retries are exhausted, and check whether a `DefaultErrorHandler` bean with a recoverer exists
- How to read it
- If such lines exist, records are being skipped by the default logging recoverer. If there are none, look for a
catchblock in the listener that swallows the exception.
You want to know why a record was dead-lettered
Check
kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic payments.requested-dlt --from-beginning --formatter-property print.headers=true
- How to read it
- The headers
kafka_dlt-original-topic,kafka_dlt-original-partition,kafka_dlt-original-offset,kafka_dlt-exception-fqcnandkafka_dlt-exception-messageidentify the source record and the failure. Tools older than Kafka 4.2 take--property print.headers=trueinstead.
Key takeaways
- A poison message is a record that fails on every delivery. Under at-least-once delivery it blocks its partition, or crashes each consumer that reads it, until something commits past it.
- Deserialisation happens inside
poll(), before your code.ErrorHandlingDeserializerturns that failure into a record the error handler can deal with. - The default
DefaultErrorHandlermakes 10 fast attempts and then logs and skips. A dead letter topic exists only if you configure a recoverer. - Classify by exception type: permanent failures go to the recoverer on the first delivery, transient ones are retried. Do not let a dependency outage flow into the dead letter topic.
- Skip when the record has no lasting value, dead-letter when it matters and others can proceed, stop when nothing after it may be processed without it.
- A record is passed over only after the recoverer succeeded. If the dead letter publish fails, the record is delivered again.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A Spring Boot consumer reads payments.requested with value-deserializer set to JsonDeserializer and a DefaultErrorHandler bean with a DeadLetterPublishingRecoverer. At 10:12 lag starts to grow on partition 4 only. The log shows a deserialisation error for partition 4, offset 9001, repeated several times per second. The dead letter topic is empty.
Why does the dead letter flow not catch this record?
Scenario 2 of 3
An inventory consumer validates each StockAdjusted event and throws IllegalStateException when the warehouse id is unknown. The error handler is new DefaultErrorHandler(recoverer, new FixedBackOff(5000L, 12L)). Operations reports that a single bad event delays every later event on its partition by about a minute before it reaches the dead letter topic. Unknown warehouse ids are never fixed by waiting.
What is the right change?
Scenario 3 of 3
A ledger service applies AccountEntry events from a topic keyed by account id. Each entry's balance check depends on all earlier entries for that account in the partition. One entry fails validation because of a bug in the producing service that will be fixed and the entry re-sent correctly within the hour.
Which handling fits this topic?
References
References: Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler, container stopping error handlers, publishing dead-letter records) ↗ · Spring for Apache Kafka 3.3 reference: Serialization, deserialization and message conversion (Using ErrorHandlingDeserializer) ↗ · Spring for Apache Kafka 3.3 Javadoc: DefaultErrorHandler ↗ · Apache Kafka documentation: Consumer configs (enable.auto.commit, key and value deserializers) ↗ · Apache Kafka documentation: Basic Kafka operations (kafka-consumer-groups.sh) ↗ · Spring Boot 3.5 reference: Common application properties (spring.kafka.*) ↗ · Report a problem