Path 7 · lesson 6 of 8
Retry topics and dead letter topics as architecture
Design the retry and dead letter topology for a consumer: how many topics, who owns them, what happens to ordering, how big they get and what to watch.
What you will learn
- Choose between a chain of delay topics and a single retry topic, and explain what a mixed-delay topic does to waiting records
- Give each consumer group its own retry and dead letter topics and justify that against sharing them per source topic
- Decide what to do about ordering when one event of a key is in retry: block the key, or accept reordering and rely on versions
- Size the retry topics for an outage, set their retention, and predict when a retry design turns an outage into a full dead letter topic
- Define the metrics and alerts that show a retry flow is healthy
Before you start
The concept
The reliability path covers the mechanics: how a failed record is redelivered with backoff and how a dead letter topic is written, inspected and replayed. This lesson is the level above. It treats the retry flow as a small architecture with its own topics, consumers, capacity and failure modes, and goes through the decisions you make once per consumer and live with for years.
The pattern is this. A consumer that cannot process a record does not keep the partition waiting. It republishes the record to another topic, commits past it and continues. A second consumer reads that topic after a delay and tries again. After a bounded number of attempts the record goes to a dead letter topic (DLT), where it waits for a person. Five decisions shape it:
- Topology: one topic per delay, or one retry topic for all delays.
- Ownership: retry and dead letter topics per consumer group, or shared per source topic.
- Ordering: what happens to later events of a key while an earlier one is in retry.
- Capacity and retention: how much flows through the retry topics during an outage, and how long they keep it.
- Observability: how you tell a healthy trickle of retries from a consumer that is failing everything.
Advantages
- One slow or failing record does not stall its partition and every key behind it
- Long delays (minutes) are possible without holding a consumer thread or risking a rebalance
- Each attempt and its error are visible as records with headers, not as log lines
- A dead letter topic turns silent skipping into a queue that someone owns
- Retry load is isolated: retry consumers can be scaled and throttled separately
Disadvantages
- Per-key ordering is lost unless you add key parking or version checks
- More topics, consumer groups and ACLs per consumer: a chain of four delays is five extra topics
- Every republish can produce a duplicate, so the handler must be idempotent
- During an outage the whole stream flows through the retry topics, multiplied by the attempts
- A dead letter topic without an owner and an alert is a place where events disappear
- Delays are approximate, and never shorter than configured
Retry topics & DLQ
Simulation, not a real clusterThe animated lesson follows one record from the main topic through the retry topics into the dead letter topic.
Step 1 · The pattern
One at a time, in order
A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.
- message
- processed
Why it exists
A Kafka consumer reads a partition in offset order and its position is one number. That is what gives ordering, and it is also the problem: there is no way to say "I have handled 41 and 43 but not 42". When record 42 fails, the consumer has three options, and each plain one is bad.
| Option | Effect on the partition | Effect on the business |
|---|---|---|
| Retry in place until it works | Blocked. Lag grows for every key in that partition. | One customer's failing payment delays thousands of other orders. A record that can never succeed blocks for ever. |
| Skip it and commit | Moves on | The event is lost. An order is never paid and nobody knows. |
| Sleep longer between attempts | Blocked for longer, and a sleep beyond max.poll.interval.ms gets the consumer removed from the group | A rebalance, redelivery to another instance, and the same failure there. |
Brokers offer no per-record delay or per-record acknowledgement for a consumer group. So the delay has to be built from what Kafka does have: topics and consumers. Moving the failed record to another topic frees the partition. A separate consumer that waits until the record is due provides the delay. A final topic that nothing retries provides the place to stop. That is the retry topic pattern, and everything difficult about it follows from the fact that the record has left its original position in the log.
How it works inside
Decision 1: a chain of delay topics or a single retry topic
A retry consumer cannot pick records out of the middle of a partition. It reads the next record, sees from a header when it is due, and if that time has not come it pauses the partition and waits. Everything behind that record waits too. This is harmless when every record in the topic has the same delay, because then the record at the head is always the one that is due first. It is the reason topologies differ.
| Topology | How it works | Good for | Problem |
|---|---|---|---|
| Chain of delay topics | One topic per delay: -retry-1000, -retry-2000, -retry-4000, then the DLT. A failure in one tier publishes to the next. | Exponential backoff. Each topic has one delay, so its head is always the earliest due. | A topic and a consumer per tier. More tiers means more to create, secure, monitor and size. |
| Single retry topic, fixed delay | One topic. A record that fails again is appended to the same topic with a new due time. | Fixed backoff. Few topics, and the simplest base for key parking. | No growing delay: the tenth attempt comes as quickly as the second. |
| Single retry topic, mixed delays in a due-time header | One topic, each record carries its own due time, which differs by attempt. | Looks economical | Head-of-line blocking: a record due in 10 minutes at the head holds back one due in 1 second behind it. Avoid, unless the consumer moves not-yet-due records into a store. |
| Delay store | Failed records go to a database table with a due_at column. A scheduler republishes them when due. | Long delays (hours), arbitrary schedules, per-key ordering decisions made with SQL. | A database and a scheduler in the hot path of error handling. One more dual write to get right. |
Spring for Apache Kafka's @RetryableTopic implements the first two. Each forwarded record carries a back off timestamp, the retry consumer pauses the partition until that time, and the reference guarantees only that a record is never processed before it is due: precision is best effort, because one consumer thread does the waiting and the processing. Since version 3.2, attempts that share the same delay reuse one topic by default (SameIntervalTopicReuseStrategy.SINGLE_TOPIC), so an exponential policy capped at a maximum delay no longer needs one topic per attempt at the cap.
Decision 2: retry and dead letter topics belong to a consumer group
orders.events is read by payment-service, inventory-service and notification-service. Suppose the payment provider is down and the Payment Service cannot process an event. Inventory and Notification handled the same event without trouble. The failure is a fact about one consumer's attempt, not about the event. So the retry and dead letter topics must belong to the consumer group that failed.
Shared: orders.events-retry for everyone
- Every group reads the retry topic, so a record that failed only in Payment is delivered again to Inventory and Notification
- Each group must filter out records it did not fail, or process them twice
- One team's bug fills a dead letter topic that three teams are alerted on
- Replaying the DLT replays the event for every consumer
- Retention and partition count cannot fit three different workloads
Per group: orders.events-payment-retry
- Only the group that failed sees the record again
- The team that owns the consumer owns its retry and dead letter topics, their ACLs and their alerts
- Delays and attempts match that consumer's dependency: a payment provider and an email service fail differently
- A replay affects one consumer
- More topics in total, named by a convention that includes the group
Decision 3: what happens to ordering
Order o-1001 emits OrderCreated (version 1) and then OrderCancelled (version 2). Version 1 fails in the Payment Service and goes to a 30 second retry topic. Version 2 arrives a moment later and succeeds: the payment is cancelled, although none exists yet. Thirty seconds later version 1 succeeds and creates a payment for a cancelled order. There are three ways to deal with this, and you should pick one deliberately per consumer.
| Tactic | How | Use when | Cost |
|---|---|---|---|
| Accept reordering, check versions | Events carry an aggregate version. The handler applies an event only if it is newer than what it has stored, and handles "cancel for something I do not have yet" explicitly. | Events carry state, and the consumer keeps a row per key. This is the default choice. | Every handler must be written for out-of-order arrival. |
| Block the key | Record which keys have an event in retry. Any later event for such a key is sent to the retry topic as well, without being processed, so it stays behind the first. | Events are deltas or commands whose order matters, and only a few keys are in retry at a time. | A shared store for parked keys, a lookup on every record, and whole keys in the DLT when the first event is dead-lettered. |
| Do not leave the partition | Blocking retries only: the partition waits until the record succeeds or is dead-lettered. | Strict order across keys matters, or the failure is one where every record would fail anyway. | The partition stalls. Suitable for short retries and for dependency outages, where nothing behind the record could be processed either. |
Blocking the key has two details that are easy to miss. First, later events must queue behind the failed one, which a chain of tiers does not give: an event that enters the 1 second tier can become due before an earlier one sitting in the 4 second tier. Use one fixed-delay retry topic, and do not rely on position alone: check on every attempt whether an earlier version of the same key is still parked. Second, decide what happens when the first event is dead-lettered. If the key stays blocked, its later events follow it into the DLT and wait for the same fix, which keeps the order intact for the replay. If the key is released, the later events run against state the missing event never created.
Decision 4: capacity and retention
In normal operation the retry topics carry a tiny share of the traffic. They must be sized for the abnormal case. Take orders.events at 400 events per second and a payment provider that is down for 10 minutes. Every event fails, so 240,000 records enter the retry flow. With 6 attempts 30 seconds apart, a record gives up 2.5 minutes after its first failure. The outage lasts longer than that, so nearly all 240,000 records end in the dead letter topic, after each was written to the retry topic five times: about 1.2 million writes.
- Throughput: a retry topic can receive up to the main topic's full rate, and a single reused retry topic receives it once per attempt. Give it the same partition count and brokers that can take the load.
- Partitions: by default the forwarded record goes to the same partition number as the original, so the retry and dead letter topics need at least as many partitions as the main topic. Spring auto-creates them with one partition unless you say otherwise.
- Retention: a retry topic must keep a record for at least its delay plus the longest time the retry consumer might be down. A retention shorter than that deletes records that were never retried, without any error. Keep the 7 day default unless you have a reason.
- Total retry window: choose attempts and delays so that the window is longer than the outages you expect the consumer to ride through. A window of 2.5 minutes dead-letters every outage longer than that.
- Outages are not retried record by record: when everything fails, stop consuming (pause the container, or let blocking retries hold the partition) and let the main topic be the buffer. It is already sized and retained for that.
Decision 5: observability
| Signal | Where it comes from | Healthy | Act when |
|---|---|---|---|
| Inflow to each retry topic | Growth of the topic's end offsets | A small fraction of the main topic's rate | It approaches the main topic's rate: the consumer fails everything, likely an outage or a bad deployment |
| Retry group lag | kafka-consumer-groups.sh --describe on the retry group | Non-zero: records are waiting for their due time | It exceeds inflow rate times delay, or keeps growing: the retry consumer is stuck or too slow |
| Attempts per record | The retry_topic-attempts header, as a metric by attempt number | Most records succeed on the first retry | Most reach the last attempt: the delays are too short for this failure |
| Dead letter inflow | Growth of the DLT's end offsets | Zero | Any record: open a ticket. A sustained rate: page someone. |
| Parked keys | Row count and oldest row of the parking table | A handful, minutes old | Old rows: a key is blocked behind a dead letter and needs a decision |
| Time to final outcome | Original timestamp header against the time of success or dead letter | Within the retry window | It exceeds what the business accepts for an unpaid order |
Failure scenarios
| Crash point | Resulting state | Recovery |
|---|---|---|
| Consumer, after the handler failed, before the record is forwarded | Nothing forwarded, main offset not committed | The record is redelivered from the main topic, fails again and is forwarded then. No loss. |
| Consumer, after the retry topic acknowledged, before the main offset commit | The record is in the retry topic and will also be redelivered from the main topic | Two copies are processed. The handler's idempotency absorbs the second. |
| Forwarding fails because the retry topic is unavailable | The record cannot be moved | The error stays with the main consumer: the record is redelivered and the partition waits. The flow degrades to blocking, which is safe. |
| Retry consumer, while waiting for the due time | Offset not committed, record still in the retry topic | After restart or rebalance the record is read again. The due time is an absolute timestamp in a header, so the wait is not restarted. |
| Retry consumer down longer than the retry topic's retention | Waiting records are deleted by the broker | They are gone. Only a replay of the main topic for that period recovers them, which is why retention must exceed any plausible downtime. |
| Dead letter handler throws | Depends on the DLT strategy | With Spring's default, ALWAYS_RETRY_ON_ERROR, the record is sent to the DLT again so it does not block the others. Keep the handler trivial. |
| Service crashes between parking a key and forwarding the record | Key marked as parked, record not yet in the retry topic | The record is redelivered from the main topic. Parking is an idempotent insert, the handler fails or is blocked again, and the record is forwarded. |
Architecture
Drawing diagram…
Flowchart: the orders.events topic is read by three consumer groups. The payment group forwards failed records, and records whose key is parked, to its own retry topic with a fixed 30 second delay. A retry consumer reads it, sends records that fail again back to the same topic, and sends exhausted or non-retryable records to the payment dead letter topic, which leads to alerting and replay. Both payment consumers use a parked_event table. Inventory and notification each have their own separate retry and dead letter topics.
flowchart TB M["orders.events, 12 partitions, key orderId"] M --> PG["Group payment-service"] M --> IG["Group inventory-service"] M --> NG["Group notification-service"] PG -->|"failed, or key is parked"| PR["orders.events-payment-retry, fixed 30 s"] PR --> PRC["Payment retry consumer"] PRC -->|"failed again"| PR PRC -->|"attempts exhausted, or not retryable"| PD["orders.events-payment-dlt"] PG -->|"not retryable"| PD PG --> GATE["parked_event table"] PRC --> GATE PD --> OPS["Alert, inspect, replay"] IG -->|"failed"| IR["orders.events-inventory-retry"] IR --> ID["orders.events-inventory-dlt"] NG -->|"failed"| NR["orders.events-notification-retry"] NR --> ND["orders.events-notification-dlt"]
- The three groups share the source topic and nothing else. A payment failure creates no work for Inventory or Notification.
- Payment's retry topic loops back to itself: one topic, one delay, so the record at the head is always the next one due.
- Two arrows lead into the retry topic from the main consumer: records that failed, and records that were not attempted because an earlier event of their order is parked. Both carry the same payload, and only the reason differs.
- The dead letter topic has exactly one exit, to people. Nothing consumes it automatically except a handler that records and alerts.
Message flow
Drawing diagram…
Sequence diagram: the payment consumer fails to charge for order o-1001 version 1, parks the key, forwards the record to the retry topic and commits. The cancellation, version 2 of the same order, arrives next. The consumer finds version 1 parked and forwards version 2 to the retry topic without processing it. An event for another order is processed normally. Thirty seconds later the retry consumer processes version 1, releases it, then processes version 2. A note says that after six failed attempts a record goes to the dead letter topic and its key stays parked.
sequenceDiagram participant K as orders.events participant P as Payment consumer participant G as parked_event table participant R as payment-retry topic participant RC as Retry consumer participant D as payment-dlt topic K->>P: OrderCreated o-1001 version 1 P->>P: charge fails, provider timeout P->>G: park o-1001 version 1 P->>R: forward version 1, due in 30 s P->>K: commit offset K->>P: OrderCancelled o-1001 version 2 P->>G: earlier version parked for o-1001 G-->>P: yes, version 1 P->>G: park o-1001 version 2 P->>R: forward version 2 unprocessed, due in 30 s K->>P: OrderCreated o-1002 version 1 P->>P: processed normally R->>RC: version 1 is due RC->>RC: charge succeeds RC->>G: release o-1001 version 1 R->>RC: version 2 is due RC->>G: earlier version parked for o-1001 G-->>RC: no RC->>RC: cancel payment, release version 2 Note over RC,D: after 6 failed attempts a record goes to the DLT and its key stays parked
- The main consumer commits past both events of o-1001 within milliseconds, so the partition keeps moving. That is the benefit over a blocking retry.
- Version 2 is forwarded with no attempt to process it. Its trip through the retry topic costs one of its attempts, which is why a blocked event can reach the DLT shortly after the event it was waiting for.
- The check "is an earlier version parked" runs on every attempt, in the main consumer and in the retry consumer. Position in the retry topic alone is not trusted, because a record that fails again is appended behind the records that were waiting for it.
In a real system
A typical setup for the order system. orders.events has 12 partitions, replication factor 3 and the order id as key. Three services consume it, and each made a different choice, because their failures differ.
| Consumer group | Typical failure | Topology | Ordering tactic | Topics |
|---|---|---|---|---|
payment-service | Payment provider timeouts, seconds to minutes | A few blocking retries, then one fixed-delay retry topic, 6 attempts 30 s apart | Block the key: a cancel must not overtake a create | orders.events-payment-retry, orders.events-payment-dlt |
inventory-service | Database deadlocks and lock timeouts, milliseconds | Blocking retries only, then DLT | Order preserved because the record never leaves the partition | orders.events-inventory-dlt |
notification-service | Email and SMS provider rate limits, minutes | Chain with exponential delays: 10 s, 1 min, 5 min, then DLT | Accept reordering: each notification is independent, deduplicated by event id | Three retry topics and orders.events-notification-dlt |
All retry and dead letter topics have 12 partitions and the same replication settings as the source topic. Retry topics keep the default 7 days. Dead letter topics keep 14 days so that a record parked before a long weekend is still there when someone looks. Each team's alerting watches its own DLT.
When to use retry topics
- Failures are per record and transient: one customer's card processor is slow, one row is locked, one downstream lookup is not ready yet.
- The wait you need is longer than you can hold a partition: tens of seconds to minutes.
- Either the handler tolerates out-of-order events for a key, or you are prepared to build and operate key parking.
- The handler is idempotent, because forwarding and redelivery both produce duplicates.
- Someone owns the dead letter topic: an alert, a runbook and a replay tool exist.
When not to
- The dependency is down for everyone. Every record would fail, so moving them all through retry topics achieves nothing except filling the DLT. Pause the consumer and let the main topic buffer.
- The record can never succeed (it cannot be deserialized, it fails validation). Retrying is waste: send it straight to the DLT.
- The stream needs strict order and events are deltas, and you do not want a parking store. Use blocking retries and accept the stall.
- The listener uses Kafka transactions or is a batch listener. Spring's non-blocking retries support neither.
- The failures are in fact fast and rare. Two or three blocking retries with a short backoff are simpler and keep order.
Java and Spring Boot
The Payment Service's retry flow on Spring Boot 3: per-group topic names, one fixed-delay retry topic, explicit topics, and a key gate so that later events of an order wait behind one that is in retry. The mechanics of @RetryableTopic are explained in the reliability path; here the interest is in the design choices. Keys and values are JSON strings.
-- Payment Service database. One row per event of a key that is currently in the retry flow.CREATE TABLE parked_event ( consumer varchar(64) NOT NULL, -- 'payment-service' record_key varchar(255) NOT NULL, -- the Kafka record key: the order id version bigint NOT NULL, -- aggregate version carried in the event reason text NOT NULL, parked_at timestamptz NOT NULL DEFAULT now(), dead boolean NOT NULL DEFAULT false, -- true once the record reached the DLT PRIMARY KEY (consumer, record_key, version));-- Gate, before every attempt (main topic and retry topic):-- SELECT EXISTS (SELECT 1 FROM parked_event-- WHERE consumer = ? AND record_key = ? AND version < ?);-- true -> do not process. Park this version too and send the record to the retry topic.-- false -> process.-- Park (its own short transaction, so it survives the rollback of the failed business work):-- INSERT INTO parked_event (consumer, record_key, version, reason) VALUES (?, ?, ?, ?)-- ON CONFLICT (consumer, record_key, version) DO UPDATE SET reason = EXCLUDED.reason;-- Success (same transaction as the business effect):-- BEGIN;-- ... payment rows ...-- DELETE FROM parked_event WHERE consumer = ? AND record_key = ? AND version = ?;-- COMMIT;-- Dead letter handler:-- UPDATE parked_event SET dead = true WHERE consumer = ? AND record_key = ? AND version = ?;-- The row stays, so the key stays blocked until an operator replays or discards the event:-- DELETE FROM parked_event WHERE consumer = ? AND record_key = ?;-- Dashboard and alert:-- SELECT count(DISTINCT record_key) AS parked_keys,-- count(*) FILTER (WHERE dead) AS dead_events,-- min(parked_at) AS oldest-- FROM parked_event WHERE consumer = 'payment-service';Why it is written this way
- The gate compares the event's own
version, set by the Order Service, not Kafka offsets. Offsets of the main topic and of the retry topic cannot be compared with each other. - The parking insert must commit even though the business transaction rolls back, so it runs outside that transaction. The release runs inside it, so a key is never released for work that did not commit.
- The primary key leads with
consumer, record_key, which makes the gate lookup an index range scan on one key. - Rows with
dead = trueare the list of orders that need a human. Their age is the most useful alert in this design.
package com.example.payments.retry;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.stereotype.Component;/** Tracks which order ids have an event in the retry flow, so later events can wait behind it. */@Componentpublic class KeyGate { private static final String CONSUMER = "payment-service"; private final JdbcTemplate jdbc; public KeyGate(JdbcTemplate jdbc) { this.jdbc = jdbc; } public boolean earlierVersionParked(String key, long version) { Boolean parked = jdbc.queryForObject(""" SELECT EXISTS (SELECT 1 FROM parked_event WHERE consumer = ? AND record_key = ? AND version < ?)""", Boolean.class, CONSUMER, key, version); return Boolean.TRUE.equals(parked); } /** Idempotent: parking the same version again only refreshes the reason. */ public void park(String key, long version, String reason) { jdbc.update(""" INSERT INTO parked_event (consumer, record_key, version, reason) VALUES (?, ?, ?, ?) ON CONFLICT (consumer, record_key, version) DO UPDATE SET reason = EXCLUDED.reason""", CONSUMER, key, version, reason); } /** Call inside the business transaction, so the release commits with the effect. */ public void release(String key, long version) { jdbc.update("DELETE FROM parked_event WHERE consumer = ? AND record_key = ? AND version = ?", CONSUMER, key, version); } public void markDead(String key, long version) { jdbc.update("UPDATE parked_event SET dead = true WHERE consumer = ? AND record_key = ? AND version = ?", CONSUMER, key, version); }}Why it is written this way
parkis called when no transaction is active, soJdbcTemplatecommits it immediately.releaseis called inside aTransactionTemplatecallback and joins that transaction.- The table is shared by all instances of the service. An in-memory set would be lost on restart and would not be seen by the instance that owns the retry topic's partition.
- The gate adds one indexed lookup per record. For most keys the table has no rows and the lookup is cheap.
package com.example.payments.retry;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import java.nio.charset.StandardCharsets;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.DltHandler;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.annotation.RetryableTopic;import org.springframework.kafka.retrytopic.SameIntervalTopicReuseStrategy;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.retry.annotation.Backoff;import org.springframework.stereotype.Component;import org.springframework.transaction.support.TransactionTemplate;@Componentpublic class OrderEventsRetryListener { private static final Logger log = LoggerFactory.getLogger(OrderEventsRetryListener.class); /** State of an order after a change. version increases with every change of that order. */ public record OrderEvent(String orderId, long version, String status, long totalCents) {} /** The business work. It must be idempotent: see the idempotent consumer pattern. */ public interface OrderEventHandler { void handle(OrderEvent event); } /** Permanent failure: retrying cannot help. */ public static class InvalidOrderEventException extends RuntimeException { public InvalidOrderEventException(String message, Throwable cause) { super(message, cause); } } /** Not a failure of this record: an earlier event of the same order is still in retry. */ public static class KeyBlockedException extends RuntimeException { public KeyBlockedException(String key) { super("An earlier event of " + key + " is in retry"); } } private final KeyGate gate; private final OrderEventHandler handler; private final TransactionTemplate tx; private final ObjectMapper json; public OrderEventsRetryListener(KeyGate gate, OrderEventHandler handler, TransactionTemplate tx, ObjectMapper json) { this.gate = gate; this.handler = handler; this.tx = tx; this.json = json; } @RetryableTopic( attempts = "6", // the first delivery plus 5 retries backoff = @Backoff(delay = 30_000), // fixed delay sameIntervalTopicReuseStrategy = SameIntervalTopicReuseStrategy.SINGLE_TOPIC, retryTopicSuffix = "-payment-retry", // orders.events-payment-retry dltTopicSuffix = "-payment-dlt", // orders.events-payment-dlt exclude = InvalidOrderEventException.class, // straight to the DLT autoCreateTopics = "false") // topics are declared in RetryTopicsConfig @KafkaListener(id = "payment-orders", topics = "orders.events", groupId = "payment-service") public void onOrderEvent(ConsumerRecord<String, String> record) { OrderEvent event = parse(record); String key = record.key(); if (gate.earlierVersionParked(key, event.version())) { gate.park(key, event.version(), "waiting for an earlier event of this order"); throw new KeyBlockedException(key); // forwarded to the retry topic unprocessed } try { tx.executeWithoutResult(status -> { handler.handle(event); gate.release(key, event.version()); // commits with the business effect }); } catch (RuntimeException e) { gate.park(key, event.version(), e.toString()); // the business transaction has rolled back throw e; // forwarded to the retry topic, or to the DLT } } @DltHandler public void onDeadLetter(ConsumerRecord<String, String> record) { Header cause = record.headers().lastHeader(KafkaHeaders.DLT_EXCEPTION_MESSAGE); String error = cause == null ? "unknown" : new String(cause.value(), StandardCharsets.UTF_8); log.error("Dead letter from {}-{}@{} key={} error={}", record.topic(), record.partition(), record.offset(), record.key(), error); try { OrderEvent event = json.readValue(record.value(), OrderEvent.class); gate.markDead(record.key(), event.version()); // the key stays blocked until resolved } catch (JsonProcessingException e) { // An unreadable record was never parked: nothing to mark. } } private OrderEvent parse(ConsumerRecord<String, String> record) { try { return json.readValue(record.value(), OrderEvent.class); } catch (JsonProcessingException e) { throw new InvalidOrderEventException("Unreadable order event at offset " + record.offset(), e); } }}Why it is written this way
- The same method handles deliveries from the main topic and from the retry topic. Spring creates the retry and dead letter consumers from this one annotation, in their own consumer groups derived from
payment-service. - The suffixes carry the consumer's name. With the defaults, every service that listens to
orders.eventswould shareorders.events-retryandorders.events-dlt. - A fixed delay with
SINGLE_TOPICgives one retry topic. That keeps the number of topics down and avoids the overtaking that tiers with different delays allow. KeyBlockedExceptionis an ordinary retryable exception, so a blocked record travels the same path as a failed one and uses up attempts while it waits. With 6 attempts it reaches the DLT soon after the event that blocks it.excludesendsInvalidOrderEventExceptionto the DLT without retries. An unreadable event is not parked, because without a version it cannot take part in the ordering.- The DLT handler only records and alerts. It never calls the business handler. Spring's default DLT strategy sends a record back to the DLT if this method throws.
attemptsand the delay give a retry window of 2.5 minutes. An outage longer than that dead-letters every record, so pair this with pausing the listener when the provider is down for everyone.
package com.example.payments.retry;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;/** * The retry and dead letter topics are part of this service's design, so they are declared * here and not left to auto-creation defaults (one partition, broker default replication). */@Configurationpublic class RetryTopicsConfig { private static final int PARTITIONS = 12; // same as orders.events private static final String FOURTEEN_DAYS_MS = "1209600000"; @Bean public NewTopic paymentRetryTopic() { return TopicBuilder.name("orders.events-payment-retry") .partitions(PARTITIONS) .replicas(3) .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") .build(); // retention: broker default, 7 days } @Bean public NewTopic paymentDeadLetterTopic() { return TopicBuilder.name("orders.events-payment-dlt") .partitions(PARTITIONS) .replicas(3) .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") .config(TopicConfig.RETENTION_MS_CONFIG, FOURTEEN_DAYS_MS) .build(); }}Why it is written this way
- Forwarded records keep their partition number by default, so both topics have the main topic's 12 partitions. All events of one order then stay in one partition of the retry topic as well.
- Replication and
min.insync.replicasmatch the source topic. A record in retry is as valuable as it was in the main topic. - The retry topic keeps the default retention on purpose. Shortening it because the data is "temporary" risks deleting records while the retry consumer is down.
- In a cluster where applications may not create topics, the same definitions belong in your topic provisioning, and these beans can be dropped.
Configuration
Attributes of @RetryableTopic with their defaults in Spring for Apache Kafka 3.3 (from the annotation's Javadoc and the reference), and the topic setting that most often breaks a retry flow.
attemptsDefault 3
- What it does
- Total deliveries before the record goes to the DLT. The first delivery counts, so 3 means two retries.
- When to change it
- Derive it from the retry window you need: attempts and delays together must outlast the failures you intend to ride through.
backoff (@Backoff delay, multiplier, maxDelay)Default fixed, 1000 ms
- What it does
- The delay before each retry. With a multiplier it grows per attempt, capped at
maxDelay, which defaults to 30 seconds for an exponential policy. - When to change it
- Use a fixed delay for a single retry topic. For exponential delays raise
maxDelaydeliberately, since each distinct delay is a topic.
retryTopicSuffix / dltTopicSuffixDefault -retry / -dlt
- What it does
- Appended to the main topic's name to form the retry and dead letter topic names.
- When to change it
- Include the consumer's name whenever more than one application consumes the source topic, so each group gets its own topics.
sameIntervalTopicReuseStrategyDefault SINGLE_TOPIC (since 3.2)
- What it does
- Whether attempts with the same delay share one retry topic or get one topic each (
MULTIPLE_TOPICS). - When to change it
- Keep the default. One topic per attempt only multiplies what you have to operate.
numPartitions / replicationFactorDefault 1 / -1 (broker default)
- What it does
- Partition count and replication factor used when the framework creates the retry and dead letter topics.
- When to change it
- Set the partitions to the main topic's count, or declare the topics yourself. One partition serialises all retries through one consumer thread.
concurrencyDefault same as the main listener container
- What it does
- Number of consumer threads for the retry and DLT containers.
- When to change it
- Lower it to throttle retries against a recovering dependency, without slowing the main consumer.
timeoutDefault none
- What it does
- Overall time limit for retrying a record. Once exceeded, the next failure sends it to the DLT whatever the attempt count.
- When to change it
- Set it when the business has a deadline after which a late success is worse than a dead letter, such as a payment for an order that will be auto-cancelled.
dltStrategyDefault ALWAYS_RETRY_ON_ERROR
- What it does
- What happens when the DLT handler fails: send the record to the DLT again, stop (
FAIL_ON_ERROR), or have no DLT at all (NO_DLT). - When to change it
NO_DLTdrops the record after the last attempt. Choose it only for data you have decided you can lose.
retention.ms (retry and dead letter topics)Default 604800000 (7 days)
- What it does
- How long the broker keeps records in the topic under the delete policy.
- When to change it
- Never below the longest delay plus the longest plausible downtime of the retry consumer. For the DLT, longer than the slowest path from alert to replay.
Hands-on exercise
Watch a cancel overtake a create, then stop it
See a partition stall behind one failing record, free it with a retry topic, observe the ordering problem that this creates for one order, and compare the tactics that deal with it.
Steps · 0 of 5 done
What you should see
With in-place retries the whole partition's lag grows until the record succeeds. With a retry topic the lag stays near zero, and the second event of the order is applied before the first. Without protection the final state is wrong. With a version check the late first event is recognised and handled as stale. With key parking both events are applied in order about one delay later, and the second has used one attempt without having failed.
What can go wrong
Two services share a retry topic by accident
- Symptom
- Inventory processes some order events twice, always shortly after the Payment Service logs a failure for the same order.
- Cause
- Both services use
@RetryableTopicwith default suffixes onorders.events, so both read and writeorders.events-retry. A record that failed in Payment is delivered to Inventory's retry consumer too. - Fix
- Give each consumer its own suffixes, and check with
kafka-topics.sh --listthat the retry and dead letter topic names contain the consumer's name.
Dead-lettering an outage
- Symptom
- After a 20 minute provider incident the dead letter topic holds hundreds of thousands of records, and the main topic shows no lag.
- Cause
- The retry window (attempts times delay) was a few minutes. Every record ran through it and gave up while the provider was still down.
- Fix
- Stop consuming when the dependency is down for everyone: pause the listener container behind a circuit breaker, and resume when the provider recovers. Size the retry window for per-record failures.
A cancel that overtakes its create
- Symptom
- Payments exist for orders that were cancelled seconds after creation.
- Cause
- The create went to a retry topic, and the cancel was processed first from the main topic. A retry topic does not preserve order per key.
- Fix
- Apply events by version and handle a cancel for a payment that does not exist yet, or park the key so later events wait behind the one in retry.
Retry topics created with one partition
- Symptom
- Retries are slow under load, and all retry traffic goes through one consumer thread although the main topic has 12 partitions.
- Cause
- The topics were auto-created with the framework default of one partition, while the framework by default publishes a retried record to the same partition number it came from.
- Fix
- Set
numPartitionsto the main topic's count or declare the topics explicitly, and add partitions to retry and dead letter topics before adding them to the main topic.
Short retention on a retry topic
- Symptom
- After a weekend during which the retry consumer was crash-looping, some orders were never paid and are in neither the retry topic nor the DLT.
- Cause
- The retry topic's retention was set to an hour because its data was considered temporary. The broker deleted records that had not been retried yet.
- Fix
- Keep retention far above the delay, and alert on the retry group's lag and on its consumers being absent.
Mixed delays in one topic
- Symptom
- Records with a 5 second backoff are retried after 10 minutes.
- Cause
- A hand-built single retry topic holds records with different due times. The consumer pauses at a record due in 10 minutes, and the 5 second records behind it wait.
- Fix
- Use one topic per distinct delay, or a single topic with one fixed delay.
A key that stays parked for ever
- Symptom
- All events of one order go straight to retry and then to the DLT for days.
- Cause
- The first event of that key was dead-lettered and its parking row is still there, as designed, but nobody was alerted or assigned to resolve it.
- Fix
- Alert on the age of
deadrows in the parking table. Resolving means replaying the dead letters of that key in version order, or discarding them, and then deleting the rows.
How to troubleshoot
Not sure which retry and dead letter topics exist for a source topic
Check
kafka-topics.sh --bootstrap-server broker:9092 --list | grep orders.events
- How to read it
- There should be one set per consumer, with the consumer's name in it. A bare
orders.events-retryororders.events-dltmeans some service uses the default suffixes.
Records seem stuck in a retry topic
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group payment-service-payment-retry
- How to read it
- Find the retry group's exact name with
--list: Spring derives it from the listener's group id and the topic suffix. Some lag is normal, since records wait for their due time. Lag with no active members means the retry consumer is not running. Lag that only grows means it is slower than the inflow.
Need to know why a record was retried and how often
Check
kafka-console-consumer.sh --bootstrap-server broker:9092 --topic orders.events-payment-retry --from-beginning --property print.key=true --property print.headers=true
- How to read it
- The headers include
retry_topic-attempts,retry_topic-backoff-timestampand the exception class and message. The attempt and timestamp values are binary and may not print readably. Recent 4.x releases deprecate--propertyhere in favour of--formatter-property.
Retry and dead letter topics do not match the main topic
Check
kafka-topics.sh --bootstrap-server broker:9092 --describe --topic orders.events-payment-retry
- How to read it
- Compare PartitionCount and ReplicationFactor with the main topic, and look at
retention.msunder Configs. One partition, or a short retention, is the auto-creation default or a past shortcut.
An order makes no progress and nothing is failing now
Check
SELECT record_key, version, dead, reason, parked_at FROM parked_event WHERE consumer = 'payment-service' ORDER BY parked_at LIMIT 20;
- How to read it
- A row with
dead = trueis an event in the DLT whose key is still blocked. Rows that are minutes old and not dead are in the retry topic and will resolve or die on their own.
Key takeaways
- A retry topic frees the partition by moving the failed record elsewhere. That move is also what breaks per-key ordering.
- Use one topic per distinct delay, or one topic with one fixed delay. Mixed delays in one topic block the head of the line.
- Retry and dead letter topics belong to a consumer group, not to a source topic. Put the consumer's name in the topic names.
- When order matters, choose a tactic on purpose: version checks, parking the key, or staying in the partition with blocking retries.
- Size retry topics for an outage: full main-topic rate times the attempts, same partitions, and retention longer than any downtime.
- Retry per-record failures. For a dependency that is down for everyone, stop consuming.
- Watch inflow per tier, the age of waiting records, dead letter inflow and parked keys.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
orders.events is consumed by payment-service and inventory-service. Both use @RetryableTopic with default settings. During a payment provider slowdown, Inventory's metrics show a few hundred OrderCreated events handled twice, and its dead letter alert fires for records that Inventory never failed.
What is the cause?
Scenario 2 of 3
A payment consumer uses 4 attempts with a fixed 15 second delay, then a DLT. The provider has a full outage for 12 minutes at 300 events per second. Afterwards the main topic has no lag, and the on-call engineer faces about 216,000 dead letters to replay.
Which change best prevents this next time?
Scenario 3 of 3
An inventory consumer applies StockAdjusted events that say "add 5" or "remove 2" for a SKU, keyed by SKU. To avoid stalls, a developer adds a three-tier retry chain (1 s, 10 s, 60 s). A week later some SKUs show negative stock, although every event was eventually processed exactly once.
What should the design have been?
References
References: Spring for Apache Kafka 3.3 reference: Non-blocking retries, how the pattern works (ordering is lost) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, topic naming (suffixes, single topic for same intervals) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, multiple listeners on the same topic ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, configuration (retry group ids, partitions, NewTopic overrides) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, features (back off defaults, timeout, topic auto-creation) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, back off delay precision ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, DLT strategies ↗ · Spring for Apache Kafka 3.3 Javadoc: @RetryableTopic (attribute defaults) ↗ · Apache Kafka 4.1: topic configuration reference (retention.ms) ↗ · Apache Kafka 4.1: consumer configuration reference (max.poll.interval.ms) ↗ · Report a problem