Skip to content

Kafka / streaming · 6 of 9

Step 1 · The pattern

One at a time, in order

A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.

  • message
  • processed

In short

Retry topics & DLQ: how it works and how it breaks

A consumer reads a partition strictly in order. If it hits a message it cannot process, such as malformed data or a record that triggers a bug, the cautious behaviour is to retry. But the same message fails the same way every time, and nothing behind it can be read until it succeeds. One bad record stops the whole partition. This is a poison message. The remedy is to limit how long any single message may hold up the line. After a few failed attempts, publish it to a separate retry topic, where another consumer tries again after a delay, and carry on. If it still fails once the retries are used up, move it to a dead-letter queue: a parking place where engineers can inspect it, fix the cause and replay it. You give up strict ordering for that one message and keep everything else moving.

In Spring Boot

Retry topics and a dead letter topic in Spring Kafka

PaymentListener.java
@RetryableTopic(attempts = "4")
@KafkaListener(topics = "payments", groupId = "ledger")
public void handle(PaymentEvent event) {
  ledger.apply(event); // throw to send the record to the next retry topic
}

@DltHandler
public void parked(PaymentEvent event) {
  alerts.poisonMessage(event); // four attempts failed
}

A record that fails is published to a retry topic and the main partition carries on. After four attempts in total it goes to the dead letter topic, payments-dlt by default, where the @DltHandler method receives it. Spring Kafka creates the retry and dead letter topics.

Gotcha: Retry topics give up Kafka's ordering for that topic: a later record can be handled before an earlier one that is still retrying. Without @RetryableTopic the default error handler retries in place instead, ten attempts with no delay, which blocks the partition while it does so.

How this comes up in interviews

"One malformed message has stopped a partition and lag is growing without limit. How should the consumer have been designed?"

Then they ask

  1. What ordering guarantee do you give up when you set a failing message aside and carry on?
  2. How do you tell a poison message apart from a downstream outage that makes every message fail?
  3. What do you do with the messages that end up in the dead-letter queue?

A strong answer covers

  • A partition is read strictly in order, so a message that always fails blocks everything behind it.
  • Limit how long any single message may hold up the line: a few attempts, then move on.
  • Publish failures to a retry topic, where another consumer tries again after a delay.
  • When retries are used up, move the message to a dead-letter queue to inspect, fix and replay.
  • The cost is strict ordering for that one message. Hold back later messages for that key if order matters.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Engineering write-up

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.