Step 1 · The pattern
One at a time, in order
A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.
- message
- processed
In short
Retry topics & DLQ: how it works and how it breaks
A consumer reads a partition strictly in order. If it hits a message it cannot process, such as malformed data or a record that triggers a bug, the cautious behaviour is to retry. But the same message fails the same way every time, and nothing behind it can be read until it succeeds. One bad record stops the whole partition. This is a poison message. The remedy is to limit how long any single message may hold up the line. After a few failed attempts, publish it to a separate retry topic, where another consumer tries again after a delay, and carry on. If it still fails once the retries are used up, move it to a dead-letter queue: a parking place where engineers can inspect it, fix the cause and replay it. You give up strict ordering for that one message and keep everything else moving.
In Spring Boot
Retry topics and a dead letter topic in Spring Kafka
@RetryableTopic(attempts = "4")
@KafkaListener(topics = "payments", groupId = "ledger")
public void handle(PaymentEvent event) {
ledger.apply(event); // throw to send the record to the next retry topic
}
@DltHandler
public void parked(PaymentEvent event) {
alerts.poisonMessage(event); // four attempts failed
}A record that fails is published to a retry topic and the main partition carries on. After four attempts in total it goes to the dead letter topic, payments-dlt by default, where the @DltHandler method receives it. Spring Kafka creates the retry and dead letter topics.
Gotcha: Retry topics give up Kafka's ordering for that topic: a later record can be handled before an earlier one that is still retrying. Without @RetryableTopic the default error handler retries in place instead, ten attempts with no delay, which blocks the partition while it does so.
Names checked against the official docs on 15 Nov 2026:
How this comes up in interviews
"One malformed message has stopped a partition and lag is growing without limit. How should the consumer have been designed?"
Then they ask
- What ordering guarantee do you give up when you set a failing message aside and carry on?
- How do you tell a poison message apart from a downstream outage that makes every message fail?
- What do you do with the messages that end up in the dead-letter queue?
A strong answer covers
- A partition is read strictly in order, so a message that always fails blocks everything behind it.
- Limit how long any single message may hold up the line: a few attempts, then move on.
- Publish failures to a retry topic, where another consumer tries again after a delay.
- When retries are used up, move the message to a dead-letter queue to inspect, fix and replay.
- The cost is strict ordering for that one message. Hold back later messages for that key if order matters.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Uber: Building reliable reprocessing and dead letter queues with Kafka ↗ · Apache Kafka documentation: consumers ↗ · AWS Builders' Library: Avoiding insurmountable queue backlogs ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.