Skip to content

Path 7 · lesson 6 of 8

Advanced30 minApache Kafka 4.x (configuration reference read for 4.1) · Spring for Apache Kafka 3.3 on Spring Boot 3.x · Java 21 · PostgreSQL

Retry topics and dead letter topics as architecture

Design the retry and dead letter topology for a consumer: how many topics, who owns them, what happens to ordering, how big they get and what to watch.

What you will learn

  • Choose between a chain of delay topics and a single retry topic, and explain what a mixed-delay topic does to waiting records
  • Give each consumer group its own retry and dead letter topics and justify that against sharing them per source topic
  • Decide what to do about ordering when one event of a key is in retry: block the key, or accept reordering and rely on versions
  • Size the retry topics for an outage, set their retention, and predict when a retry design turns an outage into a full dead letter topic
  • Define the metrics and alerts that show a retry flow is healthy

Before you start

The concept

The reliability path covers the mechanics: how a failed record is redelivered with backoff and how a dead letter topic is written, inspected and replayed. This lesson is the level above. It treats the retry flow as a small architecture with its own topics, consumers, capacity and failure modes, and goes through the decisions you make once per consumer and live with for years.

The pattern is this. A consumer that cannot process a record does not keep the partition waiting. It republishes the record to another topic, commits past it and continues. A second consumer reads that topic after a delay and tries again. After a bounded number of attempts the record goes to a dead letter topic (DLT), where it waits for a person. Five decisions shape it:

  1. Topology: one topic per delay, or one retry topic for all delays.
  2. Ownership: retry and dead letter topics per consumer group, or shared per source topic.
  3. Ordering: what happens to later events of a key while an earlier one is in retry.
  4. Capacity and retention: how much flows through the retry topics during an outage, and how long they keep it.
  5. Observability: how you tell a healthy trickle of retries from a consumer that is failing everything.

Advantages

  • One slow or failing record does not stall its partition and every key behind it
  • Long delays (minutes) are possible without holding a consumer thread or risking a rebalance
  • Each attempt and its error are visible as records with headers, not as log lines
  • A dead letter topic turns silent skipping into a queue that someone owns
  • Retry load is isolated: retry consumers can be scaled and throttled separately

Disadvantages

  • Per-key ordering is lost unless you add key parking or version checks
  • More topics, consumer groups and ACLs per consumer: a chain of four delays is five extra topics
  • Every republish can produce a duplicate, so the handler must be idempotent
  • During an outage the whole stream flows through the retry topics, multiplied by the attempts
  • A dead letter topic without an owner and an alert is a place where events disappear
  • Delays are approximate, and never shorter than configured

Retry topics & DLQ

Simulation, not a real cluster

The animated lesson follows one record from the main topic through the retry topics into the dead letter topic.

Step 1 · The pattern

One at a time, in order

A consumer works through a partition in order. It finishes one message before it starts the next. That is what makes the ordering guarantee possible.

  • message
  • processed
Open the full animated lesson

Why it exists

A Kafka consumer reads a partition in offset order and its position is one number. That is what gives ordering, and it is also the problem: there is no way to say "I have handled 41 and 43 but not 42". When record 42 fails, the consumer has three options, and each plain one is bad.

OptionEffect on the partitionEffect on the business
Retry in place until it worksBlocked. Lag grows for every key in that partition.One customer's failing payment delays thousands of other orders. A record that can never succeed blocks for ever.
Skip it and commitMoves onThe event is lost. An order is never paid and nobody knows.
Sleep longer between attemptsBlocked for longer, and a sleep beyond max.poll.interval.ms gets the consumer removed from the groupA rebalance, redelivery to another instance, and the same failure there.

Brokers offer no per-record delay or per-record acknowledgement for a consumer group. So the delay has to be built from what Kafka does have: topics and consumers. Moving the failed record to another topic frees the partition. A separate consumer that waits until the record is due provides the delay. A final topic that nothing retries provides the place to stop. That is the retry topic pattern, and everything difficult about it follows from the fact that the record has left its original position in the log.

How it works inside

Decision 1: a chain of delay topics or a single retry topic

A retry consumer cannot pick records out of the middle of a partition. It reads the next record, sees from a header when it is due, and if that time has not come it pauses the partition and waits. Everything behind that record waits too. This is harmless when every record in the topic has the same delay, because then the record at the head is always the one that is due first. It is the reason topologies differ.

TopologyHow it worksGood forProblem
Chain of delay topicsOne topic per delay: -retry-1000, -retry-2000, -retry-4000, then the DLT. A failure in one tier publishes to the next.Exponential backoff. Each topic has one delay, so its head is always the earliest due.A topic and a consumer per tier. More tiers means more to create, secure, monitor and size.
Single retry topic, fixed delayOne topic. A record that fails again is appended to the same topic with a new due time.Fixed backoff. Few topics, and the simplest base for key parking.No growing delay: the tenth attempt comes as quickly as the second.
Single retry topic, mixed delays in a due-time headerOne topic, each record carries its own due time, which differs by attempt.Looks economicalHead-of-line blocking: a record due in 10 minutes at the head holds back one due in 1 second behind it. Avoid, unless the consumer moves not-yet-due records into a store.
Delay storeFailed records go to a database table with a due_at column. A scheduler republishes them when due.Long delays (hours), arbitrary schedules, per-key ordering decisions made with SQL.A database and a scheduler in the hot path of error handling. One more dual write to get right.

Spring for Apache Kafka's @RetryableTopic implements the first two. Each forwarded record carries a back off timestamp, the retry consumer pauses the partition until that time, and the reference guarantees only that a record is never processed before it is due: precision is best effort, because one consumer thread does the waiting and the processing. Since version 3.2, attempts that share the same delay reuse one topic by default (SameIntervalTopicReuseStrategy.SINGLE_TOPIC), so an exponential policy capped at a maximum delay no longer needs one topic per attempt at the cap.

Decision 2: retry and dead letter topics belong to a consumer group

orders.events is read by payment-service, inventory-service and notification-service. Suppose the payment provider is down and the Payment Service cannot process an event. Inventory and Notification handled the same event without trouble. The failure is a fact about one consumer's attempt, not about the event. So the retry and dead letter topics must belong to the consumer group that failed.

Shared: orders.events-retry for everyone

  • Every group reads the retry topic, so a record that failed only in Payment is delivered again to Inventory and Notification
  • Each group must filter out records it did not fail, or process them twice
  • One team's bug fills a dead letter topic that three teams are alerted on
  • Replaying the DLT replays the event for every consumer
  • Retention and partition count cannot fit three different workloads

Per group: orders.events-payment-retry

  • Only the group that failed sees the record again
  • The team that owns the consumer owns its retry and dead letter topics, their ACLs and their alerts
  • Delays and attempts match that consumer's dependency: a payment provider and an email service fail differently
  • A replay affects one consumer
  • More topics in total, named by a convention that includes the group

Decision 3: what happens to ordering

Order o-1001 emits OrderCreated (version 1) and then OrderCancelled (version 2). Version 1 fails in the Payment Service and goes to a 30 second retry topic. Version 2 arrives a moment later and succeeds: the payment is cancelled, although none exists yet. Thirty seconds later version 1 succeeds and creates a payment for a cancelled order. There are three ways to deal with this, and you should pick one deliberately per consumer.

TacticHowUse whenCost
Accept reordering, check versionsEvents carry an aggregate version. The handler applies an event only if it is newer than what it has stored, and handles "cancel for something I do not have yet" explicitly.Events carry state, and the consumer keeps a row per key. This is the default choice.Every handler must be written for out-of-order arrival.
Block the keyRecord which keys have an event in retry. Any later event for such a key is sent to the retry topic as well, without being processed, so it stays behind the first.Events are deltas or commands whose order matters, and only a few keys are in retry at a time.A shared store for parked keys, a lookup on every record, and whole keys in the DLT when the first event is dead-lettered.
Do not leave the partitionBlocking retries only: the partition waits until the record succeeds or is dead-lettered.Strict order across keys matters, or the failure is one where every record would fail anyway.The partition stalls. Suitable for short retries and for dependency outages, where nothing behind the record could be processed either.

Blocking the key has two details that are easy to miss. First, later events must queue behind the failed one, which a chain of tiers does not give: an event that enters the 1 second tier can become due before an earlier one sitting in the 4 second tier. Use one fixed-delay retry topic, and do not rely on position alone: check on every attempt whether an earlier version of the same key is still parked. Second, decide what happens when the first event is dead-lettered. If the key stays blocked, its later events follow it into the DLT and wait for the same fix, which keeps the order intact for the replay. If the key is released, the later events run against state the missing event never created.

Decision 4: capacity and retention

In normal operation the retry topics carry a tiny share of the traffic. They must be sized for the abnormal case. Take orders.events at 400 events per second and a payment provider that is down for 10 minutes. Every event fails, so 240,000 records enter the retry flow. With 6 attempts 30 seconds apart, a record gives up 2.5 minutes after its first failure. The outage lasts longer than that, so nearly all 240,000 records end in the dead letter topic, after each was written to the retry topic five times: about 1.2 million writes.

  • Throughput: a retry topic can receive up to the main topic's full rate, and a single reused retry topic receives it once per attempt. Give it the same partition count and brokers that can take the load.
  • Partitions: by default the forwarded record goes to the same partition number as the original, so the retry and dead letter topics need at least as many partitions as the main topic. Spring auto-creates them with one partition unless you say otherwise.
  • Retention: a retry topic must keep a record for at least its delay plus the longest time the retry consumer might be down. A retention shorter than that deletes records that were never retried, without any error. Keep the 7 day default unless you have a reason.
  • Total retry window: choose attempts and delays so that the window is longer than the outages you expect the consumer to ride through. A window of 2.5 minutes dead-letters every outage longer than that.
  • Outages are not retried record by record: when everything fails, stop consuming (pause the container, or let blocking retries hold the partition) and let the main topic be the buffer. It is already sized and retained for that.

Decision 5: observability

SignalWhere it comes fromHealthyAct when
Inflow to each retry topicGrowth of the topic's end offsetsA small fraction of the main topic's rateIt approaches the main topic's rate: the consumer fails everything, likely an outage or a bad deployment
Retry group lagkafka-consumer-groups.sh --describe on the retry groupNon-zero: records are waiting for their due timeIt exceeds inflow rate times delay, or keeps growing: the retry consumer is stuck or too slow
Attempts per recordThe retry_topic-attempts header, as a metric by attempt numberMost records succeed on the first retryMost reach the last attempt: the delays are too short for this failure
Dead letter inflowGrowth of the DLT's end offsetsZeroAny record: open a ticket. A sustained rate: page someone.
Parked keysRow count and oldest row of the parking tableA handful, minutes oldOld rows: a key is blocked behind a dead letter and needs a decision
Time to final outcomeOriginal timestamp header against the time of success or dead letterWithin the retry windowIt exceeds what the business accepts for an unpaid order

Failure scenarios

Crash pointResulting stateRecovery
Consumer, after the handler failed, before the record is forwardedNothing forwarded, main offset not committedThe record is redelivered from the main topic, fails again and is forwarded then. No loss.
Consumer, after the retry topic acknowledged, before the main offset commitThe record is in the retry topic and will also be redelivered from the main topicTwo copies are processed. The handler's idempotency absorbs the second.
Forwarding fails because the retry topic is unavailableThe record cannot be movedThe error stays with the main consumer: the record is redelivered and the partition waits. The flow degrades to blocking, which is safe.
Retry consumer, while waiting for the due timeOffset not committed, record still in the retry topicAfter restart or rebalance the record is read again. The due time is an absolute timestamp in a header, so the wait is not restarted.
Retry consumer down longer than the retry topic's retentionWaiting records are deleted by the brokerThey are gone. Only a replay of the main topic for that period recovers them, which is why retention must exceed any plausible downtime.
Dead letter handler throwsDepends on the DLT strategyWith Spring's default, ALWAYS_RETRY_ON_ERROR, the record is sent to the DLT again so it does not block the others. Keep the handler trivial.
Service crashes between parking a key and forwarding the recordKey marked as parked, record not yet in the retry topicThe record is redelivered from the main topic. Parking is an idempotent insert, the handler fails or is blocked again, and the record is forwarded.
Crash or fault at each point of the retry flow, what is left behind, and how it recovers.

Architecture

Drawing diagram…

Flowchart: the orders.events topic is read by three consumer groups. The payment group forwards failed records, and records whose key is parked, to its own retry topic with a fixed 30 second delay. A retry consumer reads it, sends records that fail again back to the same topic, and sends exhausted or non-retryable records to the payment dead letter topic, which leads to alerting and replay. Both payment consumers use a parked_event table. Inventory and notification each have their own separate retry and dead letter topics.

flowchart TB
  M["orders.events, 12 partitions, key orderId"]
  M --> PG["Group payment-service"]
  M --> IG["Group inventory-service"]
  M --> NG["Group notification-service"]
  PG -->|"failed, or key is parked"| PR["orders.events-payment-retry, fixed 30 s"]
  PR --> PRC["Payment retry consumer"]
  PRC -->|"failed again"| PR
  PRC -->|"attempts exhausted, or not retryable"| PD["orders.events-payment-dlt"]
  PG -->|"not retryable"| PD
  PG --> GATE["parked_event table"]
  PRC --> GATE
  PD --> OPS["Alert, inspect, replay"]
  IG -->|"failed"| IR["orders.events-inventory-retry"]
  IR --> ID["orders.events-inventory-dlt"]
  NG -->|"failed"| NR["orders.events-notification-retry"]
  NR --> ND["orders.events-notification-dlt"]
Each consumer group has its own retry and dead letter topics. Payment uses a single fixed-delay retry topic and a table of parked keys.
  • The three groups share the source topic and nothing else. A payment failure creates no work for Inventory or Notification.
  • Payment's retry topic loops back to itself: one topic, one delay, so the record at the head is always the next one due.
  • Two arrows lead into the retry topic from the main consumer: records that failed, and records that were not attempted because an earlier event of their order is parked. Both carry the same payload, and only the reason differs.
  • The dead letter topic has exactly one exit, to people. Nothing consumes it automatically except a handler that records and alerts.

Message flow

Drawing diagram…

Sequence diagram: the payment consumer fails to charge for order o-1001 version 1, parks the key, forwards the record to the retry topic and commits. The cancellation, version 2 of the same order, arrives next. The consumer finds version 1 parked and forwards version 2 to the retry topic without processing it. An event for another order is processed normally. Thirty seconds later the retry consumer processes version 1, releases it, then processes version 2. A note says that after six failed attempts a record goes to the dead letter topic and its key stays parked.

sequenceDiagram
  participant K as orders.events
  participant P as Payment consumer
  participant G as parked_event table
  participant R as payment-retry topic
  participant RC as Retry consumer
  participant D as payment-dlt topic
  K->>P: OrderCreated o-1001 version 1
  P->>P: charge fails, provider timeout
  P->>G: park o-1001 version 1
  P->>R: forward version 1, due in 30 s
  P->>K: commit offset
  K->>P: OrderCancelled o-1001 version 2
  P->>G: earlier version parked for o-1001
  G-->>P: yes, version 1
  P->>G: park o-1001 version 2
  P->>R: forward version 2 unprocessed, due in 30 s
  K->>P: OrderCreated o-1002 version 1
  P->>P: processed normally
  R->>RC: version 1 is due
  RC->>RC: charge succeeds
  RC->>G: release o-1001 version 1
  R->>RC: version 2 is due
  RC->>G: earlier version parked for o-1001
  G-->>RC: no
  RC->>RC: cancel payment, release version 2
  Note over RC,D: after 6 failed attempts a record goes to the DLT and its key stays parked
Version 2 is not processed while version 1 is in retry. Order o-1002 is not affected at all.
  1. The main consumer commits past both events of o-1001 within milliseconds, so the partition keeps moving. That is the benefit over a blocking retry.
  2. Version 2 is forwarded with no attempt to process it. Its trip through the retry topic costs one of its attempts, which is why a blocked event can reach the DLT shortly after the event it was waiting for.
  3. The check "is an earlier version parked" runs on every attempt, in the main consumer and in the retry consumer. Position in the retry topic alone is not trusted, because a record that fails again is appended behind the records that were waiting for it.

In a real system

A typical setup for the order system. orders.events has 12 partitions, replication factor 3 and the order id as key. Three services consume it, and each made a different choice, because their failures differ.

Consumer groupTypical failureTopologyOrdering tacticTopics
payment-servicePayment provider timeouts, seconds to minutesA few blocking retries, then one fixed-delay retry topic, 6 attempts 30 s apartBlock the key: a cancel must not overtake a createorders.events-payment-retry, orders.events-payment-dlt
inventory-serviceDatabase deadlocks and lock timeouts, millisecondsBlocking retries only, then DLTOrder preserved because the record never leaves the partitionorders.events-inventory-dlt
notification-serviceEmail and SMS provider rate limits, minutesChain with exponential delays: 10 s, 1 min, 5 min, then DLTAccept reordering: each notification is independent, deduplicated by event idThree retry topics and orders.events-notification-dlt

All retry and dead letter topics have 12 partitions and the same replication settings as the source topic. Retry topics keep the default 7 days. Dead letter topics keep 14 days so that a record parked before a long weekend is still there when someone looks. Each team's alerting watches its own DLT.

When to use retry topics

  • Failures are per record and transient: one customer's card processor is slow, one row is locked, one downstream lookup is not ready yet.
  • The wait you need is longer than you can hold a partition: tens of seconds to minutes.
  • Either the handler tolerates out-of-order events for a key, or you are prepared to build and operate key parking.
  • The handler is idempotent, because forwarding and redelivery both produce duplicates.
  • Someone owns the dead letter topic: an alert, a runbook and a replay tool exist.

When not to

  • The dependency is down for everyone. Every record would fail, so moving them all through retry topics achieves nothing except filling the DLT. Pause the consumer and let the main topic buffer.
  • The record can never succeed (it cannot be deserialized, it fails validation). Retrying is waste: send it straight to the DLT.
  • The stream needs strict order and events are deltas, and you do not want a parking store. Use blocking retries and accept the stall.
  • The listener uses Kafka transactions or is a batch listener. Spring's non-blocking retries support neither.
  • The failures are in fact fast and rare. Two or three blocking retries with a short backoff are simpler and keep order.

Java and Spring Boot

The Payment Service's retry flow on Spring Boot 3: per-group topic names, one fixed-delay retry topic, explicit topics, and a key gate so that later events of an order wait behind one that is in retry. The mechanics of @RetryableTopic are explained in the reliability path; here the interest is in the design choices. Keys and values are JSON strings.

payment-service/src/main/resources/db/migration/V4__parked_event.sqlSQL
-- Payment Service database. One row per event of a key that is currently in the retry flow.CREATE TABLE parked_event (    consumer    varchar(64)  NOT NULL,           -- 'payment-service'    record_key  varchar(255) NOT NULL,           -- the Kafka record key: the order id    version     bigint       NOT NULL,           -- aggregate version carried in the event    reason      text         NOT NULL,    parked_at   timestamptz  NOT NULL DEFAULT now(),    dead        boolean      NOT NULL DEFAULT false,   -- true once the record reached the DLT    PRIMARY KEY (consumer, record_key, version));-- Gate, before every attempt (main topic and retry topic):--   SELECT EXISTS (SELECT 1 FROM parked_event--                  WHERE consumer = ? AND record_key = ? AND version < ?);--   true  -> do not process. Park this version too and send the record to the retry topic.--   false -> process.-- Park (its own short transaction, so it survives the rollback of the failed business work):--   INSERT INTO parked_event (consumer, record_key, version, reason) VALUES (?, ?, ?, ?)--   ON CONFLICT (consumer, record_key, version) DO UPDATE SET reason = EXCLUDED.reason;-- Success (same transaction as the business effect):--   BEGIN;--   ... payment rows ...--   DELETE FROM parked_event WHERE consumer = ? AND record_key = ? AND version = ?;--   COMMIT;-- Dead letter handler:--   UPDATE parked_event SET dead = true WHERE consumer = ? AND record_key = ? AND version = ?;--   The row stays, so the key stays blocked until an operator replays or discards the event:--   DELETE FROM parked_event WHERE consumer = ? AND record_key = ?;-- Dashboard and alert:--   SELECT count(DISTINCT record_key) AS parked_keys,--          count(*) FILTER (WHERE dead) AS dead_events,--          min(parked_at) AS oldest--   FROM parked_event WHERE consumer = 'payment-service';

Why it is written this way

  • The gate compares the event's own version, set by the Order Service, not Kafka offsets. Offsets of the main topic and of the retry topic cannot be compared with each other.
  • The parking insert must commit even though the business transaction rolls back, so it runs outside that transaction. The release runs inside it, so a key is never released for work that did not commit.
  • The primary key leads with consumer, record_key, which makes the gate lookup an index range scan on one key.
  • Rows with dead = true are the list of orders that need a human. Their age is the most useful alert in this design.

Configuration

Attributes of @RetryableTopic with their defaults in Spring for Apache Kafka 3.3 (from the annotation's Javadoc and the reference), and the topic setting that most often breaks a retry flow.

attemptsDefault 3
What it does
Total deliveries before the record goes to the DLT. The first delivery counts, so 3 means two retries.
When to change it
Derive it from the retry window you need: attempts and delays together must outlast the failures you intend to ride through.
backoff (@Backoff delay, multiplier, maxDelay)Default fixed, 1000 ms
What it does
The delay before each retry. With a multiplier it grows per attempt, capped at maxDelay, which defaults to 30 seconds for an exponential policy.
When to change it
Use a fixed delay for a single retry topic. For exponential delays raise maxDelay deliberately, since each distinct delay is a topic.
retryTopicSuffix / dltTopicSuffixDefault -retry / -dlt
What it does
Appended to the main topic's name to form the retry and dead letter topic names.
When to change it
Include the consumer's name whenever more than one application consumes the source topic, so each group gets its own topics.
sameIntervalTopicReuseStrategyDefault SINGLE_TOPIC (since 3.2)
What it does
Whether attempts with the same delay share one retry topic or get one topic each (MULTIPLE_TOPICS).
When to change it
Keep the default. One topic per attempt only multiplies what you have to operate.
numPartitions / replicationFactorDefault 1 / -1 (broker default)
What it does
Partition count and replication factor used when the framework creates the retry and dead letter topics.
When to change it
Set the partitions to the main topic's count, or declare the topics yourself. One partition serialises all retries through one consumer thread.
concurrencyDefault same as the main listener container
What it does
Number of consumer threads for the retry and DLT containers.
When to change it
Lower it to throttle retries against a recovering dependency, without slowing the main consumer.
timeoutDefault none
What it does
Overall time limit for retrying a record. Once exceeded, the next failure sends it to the DLT whatever the attempt count.
When to change it
Set it when the business has a deadline after which a late success is worse than a dead letter, such as a payment for an order that will be auto-cancelled.
dltStrategyDefault ALWAYS_RETRY_ON_ERROR
What it does
What happens when the DLT handler fails: send the record to the DLT again, stop (FAIL_ON_ERROR), or have no DLT at all (NO_DLT).
When to change it
NO_DLT drops the record after the last attempt. Choose it only for data you have decided you can lose.
retention.ms (retry and dead letter topics)Default 604800000 (7 days)
What it does
How long the broker keeps records in the topic under the delete policy.
When to change it
Never below the longest delay plus the longest plausible downtime of the retry consumer. For the DLT, longer than the slowest path from alert to replay.

Hands-on exercise

Watch a cancel overtake a create, then stop it

See a partition stall behind one failing record, free it with a retry topic, observe the ordering problem that this creates for one order, and compare the tactics that deal with it.

Steps · 0 of 5 done

Take it further in the lab: Retry and backoff strategy

What can go wrong

Two services share a retry topic by accident
Symptom
Inventory processes some order events twice, always shortly after the Payment Service logs a failure for the same order.
Cause
Both services use @RetryableTopic with default suffixes on orders.events, so both read and write orders.events-retry. A record that failed in Payment is delivered to Inventory's retry consumer too.
Fix
Give each consumer its own suffixes, and check with kafka-topics.sh --list that the retry and dead letter topic names contain the consumer's name.
Dead-lettering an outage
Symptom
After a 20 minute provider incident the dead letter topic holds hundreds of thousands of records, and the main topic shows no lag.
Cause
The retry window (attempts times delay) was a few minutes. Every record ran through it and gave up while the provider was still down.
Fix
Stop consuming when the dependency is down for everyone: pause the listener container behind a circuit breaker, and resume when the provider recovers. Size the retry window for per-record failures.
A cancel that overtakes its create
Symptom
Payments exist for orders that were cancelled seconds after creation.
Cause
The create went to a retry topic, and the cancel was processed first from the main topic. A retry topic does not preserve order per key.
Fix
Apply events by version and handle a cancel for a payment that does not exist yet, or park the key so later events wait behind the one in retry.
Retry topics created with one partition
Symptom
Retries are slow under load, and all retry traffic goes through one consumer thread although the main topic has 12 partitions.
Cause
The topics were auto-created with the framework default of one partition, while the framework by default publishes a retried record to the same partition number it came from.
Fix
Set numPartitions to the main topic's count or declare the topics explicitly, and add partitions to retry and dead letter topics before adding them to the main topic.
Short retention on a retry topic
Symptom
After a weekend during which the retry consumer was crash-looping, some orders were never paid and are in neither the retry topic nor the DLT.
Cause
The retry topic's retention was set to an hour because its data was considered temporary. The broker deleted records that had not been retried yet.
Fix
Keep retention far above the delay, and alert on the retry group's lag and on its consumers being absent.
Mixed delays in one topic
Symptom
Records with a 5 second backoff are retried after 10 minutes.
Cause
A hand-built single retry topic holds records with different due times. The consumer pauses at a record due in 10 minutes, and the 5 second records behind it wait.
Fix
Use one topic per distinct delay, or a single topic with one fixed delay.
A key that stays parked for ever
Symptom
All events of one order go straight to retry and then to the DLT for days.
Cause
The first event of that key was dead-lettered and its parking row is still there, as designed, but nobody was alerted or assigned to resolve it.
Fix
Alert on the age of dead rows in the parking table. Resolving means replaying the dead letters of that key in version order, or discarding them, and then deleting the rows.

How to troubleshoot

Not sure which retry and dead letter topics exist for a source topic

Check

kafka-topics.sh --bootstrap-server broker:9092 --list | grep orders.events
How to read it
There should be one set per consumer, with the consumer's name in it. A bare orders.events-retry or orders.events-dlt means some service uses the default suffixes.
Records seem stuck in a retry topic

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group payment-service-payment-retry
How to read it
Find the retry group's exact name with --list: Spring derives it from the listener's group id and the topic suffix. Some lag is normal, since records wait for their due time. Lag with no active members means the retry consumer is not running. Lag that only grows means it is slower than the inflow.
Need to know why a record was retried and how often

Check

kafka-console-consumer.sh --bootstrap-server broker:9092 --topic orders.events-payment-retry --from-beginning --property print.key=true --property print.headers=true
How to read it
The headers include retry_topic-attempts, retry_topic-backoff-timestamp and the exception class and message. The attempt and timestamp values are binary and may not print readably. Recent 4.x releases deprecate --property here in favour of --formatter-property.
Retry and dead letter topics do not match the main topic

Check

kafka-topics.sh --bootstrap-server broker:9092 --describe --topic orders.events-payment-retry
How to read it
Compare PartitionCount and ReplicationFactor with the main topic, and look at retention.ms under Configs. One partition, or a short retention, is the auto-creation default or a past shortcut.
An order makes no progress and nothing is failing now

Check

SELECT record_key, version, dead, reason, parked_at FROM parked_event WHERE consumer = 'payment-service' ORDER BY parked_at LIMIT 20;
How to read it
A row with dead = true is an event in the DLT whose key is still blocked. Rows that are minutes old and not dead are in the retry topic and will resolve or die on their own.

Key takeaways

  • A retry topic frees the partition by moving the failed record elsewhere. That move is also what breaks per-key ordering.
  • Use one topic per distinct delay, or one topic with one fixed delay. Mixed delays in one topic block the head of the line.
  • Retry and dead letter topics belong to a consumer group, not to a source topic. Put the consumer's name in the topic names.
  • When order matters, choose a tactic on purpose: version checks, parking the key, or staying in the partition with blocking retries.
  • Size retry topics for an outage: full main-topic rate times the attempts, same partitions, and retention longer than any downtime.
  • Retry per-record failures. For a dependency that is down for everyone, stop consuming.
  • Watch inflow per tier, the age of waiting records, dead letter inflow and parked keys.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

orders.events is consumed by payment-service and inventory-service. Both use @RetryableTopic with default settings. During a payment provider slowdown, Inventory's metrics show a few hundred OrderCreated events handled twice, and its dead letter alert fires for records that Inventory never failed.

What is the cause?

Scenario 2 of 3

A payment consumer uses 4 attempts with a fixed 15 second delay, then a DLT. The provider has a full outage for 12 minutes at 300 events per second. Afterwards the main topic has no lag, and the on-call engineer faces about 216,000 dead letters to replay.

Which change best prevents this next time?

Scenario 3 of 3

An inventory consumer applies StockAdjusted events that say "add 5" or "remove 2" for a SKU, keyed by SKU. To avoid stalls, a developer adds a three-tier retry chain (1 s, 10 s, 60 s). A week later some SKUs show negative stock, although every event was eventually processed exactly once.

What should the design have been?

References

References: Spring for Apache Kafka 3.3 reference: Non-blocking retries, how the pattern works (ordering is lost) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, topic naming (suffixes, single topic for same intervals) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, multiple listeners on the same topic ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, configuration (retry group ids, partitions, NewTopic overrides) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, features (back off defaults, timeout, topic auto-creation) ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, back off delay precision ↗ · Spring for Apache Kafka 3.3 reference: Non-blocking retries, DLT strategies ↗ · Spring for Apache Kafka 3.3 Javadoc: @RetryableTopic (attribute defaults) ↗ · Apache Kafka 4.1: topic configuration reference (retention.ms) ↗ · Apache Kafka 4.1: consumer configuration reference (max.poll.interval.ms) ↗ · Report a problem