Path 3 · lesson 3 of 5
Acks, retries and the idempotent producer
What an acknowledgement proves, how retries are bounded by the delivery timeout, how they duplicate and reorder records, and exactly what producer idempotence fixes.
What you will learn
- State what the producer knows after
acks=0,acks=1andacks=all, and whatmin.insync.replicasadds toacks=all - Work out how long a record can stay unresolved from
delivery.timeout.ms,request.timeout.ms,linger.msand the retry backoff - Explain how a retry creates a duplicate and how it reorders two batches when several requests are in flight
- Describe how the broker uses producer id, epoch and sequence numbers to reject duplicate and out-of-order batches
- Say precisely which duplicates producer idempotence removes and which it cannot
- Tell a retriable error from a final one and decide what the application does when a send fails for good
Before you start
The concept
A produce request ends in one of three ways: the broker says it succeeded, the broker says it failed, or the producer hears nothing. Three groups of settings decide what happens in each case. acks defines what the broker must have done before it answers. Retries, bounded by delivery.timeout.ms, define what the producer does when the answer is an error or does not come. Idempotence makes those retries safe.
What an acknowledgement proves
| acks | The leader answers when | A record can be lost if | The producer learns |
|---|---|---|---|
0 | It does not answer. The record counts as sent once it is in the socket buffer | Anything goes wrong after that: connection drop, leader crash, full disk | Nothing. The returned offset is always -1 and retries have no effect |
1 | It has written the record to its own log | The leader fails before any follower copied the record | That one broker had it at that moment |
all (same as -1) | Every replica currently in the in-sync set has the record | All in-sync replicas are lost, which can be a single broker if the set had shrunk to one | That the record was committed to the in-sync replicas of that moment |
all. Java clients before 3.0 defaulted to 1.min.insync.replicas is a topic setting (with a broker-wide default) that is enforced only for acks=all writes. If the in-sync set is smaller than it, the leader refuses the write with NotEnoughReplicasException. The producer then gets an honest failure instead of a weak success. You trade availability of writes for the certainty that an acknowledged record exists on at least that many brokers.
Retries, and their real limit
When a request fails with a transient error or no response arrives within request.timeout.ms, the producer puts the batch back and sends it again after a backoff. retries defaults to 2,147,483,647, so the count is not what stops it. The limit is delivery.timeout.ms (default 2 minutes): the maximum time between send() returning and the future completing. It covers time waiting in the buffer, time waiting for the response and every retry. When it runs out, the batch fails with a TimeoutException.
Idempotence in one paragraph
A retry resends a batch the broker may already have written. With enable.idempotence=true (the default) every batch carries the producer's id and a per-partition sequence number, and the broker refuses to append a sequence it has already appended. Retries then cannot duplicate or reorder records in the log. The scope of that statement is narrow, and the internals section spells it out.
Why it exists
Networks lose responses. From the producer's side, a request that timed out is ambiguous: the broker may never have received it, or it may have written the batch and the reply was lost. There are only two things to do with an ambiguous request.
Give up (no retries)
- A record that was not written is lost: at-most-once
- The application sees an error for something that may in fact be in the log
- No duplicates from the producer
Send again (retries)
- A record that was not written gets another chance: at-least-once
- A record that was written is written a second time
- With several requests in flight, a retried batch can land behind a later one
Before Kafka 0.11 those were the only choices, and the safe configuration was retries with one request in flight at a time, which protected ordering at the cost of throughput and still allowed duplicates. The idempotent producer (KIP-98) removed the trade-off by letting the broker recognise a resend. Since Kafka 3.0 it is on by default together with acks=all, because a default that quietly loses or duplicates data surprised too many people.
The retry count turned out to be the wrong knob as well. Nobody knows whether 3 or 10 retries is right; what an application can reason about is how long it is willing to wait for a definite answer. delivery.timeout.ms expresses that directly, which is why the client documentation recommends leaving retries alone and setting the timeout.
How it works inside
The time budget of one record
The clock starts when send() returns. The record waits up to linger.ms in its batch, then the batch is sent. Each attempt waits at most request.timeout.ms (30 s) for a response. A failed attempt is followed by a backoff that starts at retry.backoff.ms (100 ms) and grows exponentially to retry.backoff.max.ms (1 s), with jitter. All of it must fit inside delivery.timeout.ms, which is therefore required to be at least linger.ms + request.timeout.ms. With the defaults, a batch whose requests keep timing out gets about four attempts before it expires.
How a retry duplicates
The leader appends the batch, the response is lost, the producer resends, the leader appends again. Without idempotence the log now has the records twice, at different offsets, and every consumer reads both. The same happens with NotEnoughReplicasAfterAppendException: the leader wrote the batch, then found the in-sync set too small to acknowledge it, and the retry writes a second copy.
How a retry reorders
The producer may have up to max.in.flight.requests.per.connection (default 5) unanswered requests to a broker. Suppose batch 1 and batch 2 for the same partition are both in flight. Batch 1 fails with a transient error, batch 2 succeeds, and batch 1 succeeds on retry. The log now holds batch 2 before batch 1. For records sharing a key, per-key order is broken inside the partition. This can only happen when retries are enabled, more than one request is in flight and idempotence is off.
The idempotent producer
- When it starts, the producer asks the cluster for a producer id and an epoch. Without a
transactional.id, every new producer instance gets a fresh id. - For each partition it writes to, the producer numbers its records with a sequence starting at 0. Each batch header carries the producer id, the epoch and the batch's base sequence. A retry resends the batch with the same numbers.
- The partition leader remembers, per producer id, the sequences of the most recent batches (five of them). A batch whose sequence is the next expected one is appended.
- A batch whose sequences were already appended is recognised as a duplicate. The leader does not write it again and the producer treats the request as successful.
- A batch whose sequence is ahead of the expected one means an earlier batch is missing. The leader rejects it, so a later batch can never be written before an earlier one. The producer resends in order. This is why ordering holds with up to 5 requests in flight, and why idempotence requires
max.in.flight.requests.per.connectionto be 5 or less. - The id, epoch and sequence are stored in the log with the batch, so they are replicated. A follower that becomes leader can continue the check.
| Situation | Does producer idempotence prevent a duplicate? |
|---|---|
| The producer's own retry of a batch after a timeout or transient error | Yes. This is the case it exists for |
| A retry that lands on a new leader after a failover | Yes. The sequence state travels with the replicated log |
Your code calls send() twice for the same business event | No. Each call gets a new sequence number and is a new record |
| The application restarts and republishes events it is unsure about | No. The new producer instance has a new producer id |
| Two service instances publish the same event | No. Different producers, different ids |
| A consumer processes a record twice after a rebalance or crash | No. This has nothing to do with the producer |
Idempotence can switch itself off
Idempotence requires acks=all, retries greater than 0 and at most 5 in-flight requests. If you set a conflicting value and did not set enable.idempotence yourself, the client disables idempotence and logs one INFO line such as Idempotence will be disabled because acks is set to 1, not set to 'all'. If you set enable.idempotence=true explicitly, the same conflict fails at startup with a ConfigException. Stating it explicitly turns a silent downgrade into a visible error.
Retriable and final errors
| Error | Retried by the producer? | Typical cause |
|---|---|---|
NotLeaderOrFollowerException, LeaderNotAvailableException | Yes, after a metadata refresh | Leader moved or is being elected |
NetworkException, request timeout | Yes | Connection dropped, broker slow or restarting |
NotEnoughReplicasException, NotEnoughReplicasAfterAppendException | Yes | In-sync set smaller than min.insync.replicas |
UnknownTopicOrPartitionException | Yes | Topic still being created, or metadata not propagated yet |
RecordTooLargeException | No | Record exceeds a size limit; sending it again cannot help |
TopicAuthorizationException | No | Missing ACL for the topic |
SerializationException | No, thrown by send() itself | Value does not match the serialiser |
TimeoutException with Expiring N record(s) | No, this is the retries giving up | delivery.timeout.ms passed without a successful attempt |
RetriableException are retried.Whatever reaches your callback is final. The producer has already retried for as long as it was allowed. An expired batch is the awkward case: the outcome is unknown, because the last attempt may have been written without the response arriving. The client documentation's advice for this case is to treat it seriously: if you resend from the application, the broker cannot recognise the resend, so you may duplicate the record, and records for the same key that were sent later may already be in the log ahead of it.
The ordering guarantee that results
- With idempotence on: records one producer sends to one partition are appended in send order, without duplicates from retries, with up to 5 requests in flight.
- With idempotence off, retries on and more than one request in flight: order within a partition can change after a failed send.
- With idempotence off and
max.in.flight.requests.per.connection=1: order holds, duplicates from retries remain. - In every case, a send that finally fails leaves a hole. Later records of the same key that succeeded are in the log, and anything you publish afterwards to fill the hole lands behind them.
Break ordering and durability, then repair them
Simulation · not a real clusterThe starting settings are a producer with acks=1 and idempotence off. Inject a lost response and a leader failure, then change one setting at a time.
ISR / acks
Simulation, not a real clusterThe animated Build lesson shows the broker side of the same story: the in-sync set shrinking and what each acks level then accepts.
Step 1 · The pattern
A leader and its followers
Every partition has a leader that takes writes and followers that copy them. If the leader dies, a follower that is caught up takes over.
- message
- being copied to follower
- confirmed
Architecture
Drawing diagram…
State diagram of a producer batch. It starts Buffered, becomes InFlight when full or when linger has passed, and ends Acked on a success response. A retriable error or request timeout moves it to Backoff and then back to InFlight with the same sequence numbers. A non-retriable error, or the delivery timeout passing in any state, moves it to Failed.
stateDiagram-v2 [*] --> Buffered Buffered --> InFlight: batch full or linger passed InFlight --> Acked: success response InFlight --> Backoff: retriable error or request timeout Backoff --> InFlight: resend with the same sequence numbers InFlight --> Failed: error that is not retriable Buffered --> Failed: delivery timeout passed InFlight --> Failed: delivery timeout passed Backoff --> Failed: delivery timeout passed Acked --> [*] Failed --> [*]
- The loop between InFlight and Backoff is the retry mechanism. It has no counter that matters in practice; the only exit besides success is a final error or the delivery timeout.
- The delivery timeout applies in every state, including Buffered. A batch can expire without ever being sent, for example when the partition has no leader for two minutes.
- A resend carries the same producer id, epoch and sequence numbers as the first attempt. That sameness is what lets the broker recognise it.
- Acked means what
ackssays it means. Withacks=1a batch can reach Acked and still be lost when the leader fails.
Message flow
Drawing diagram…
Sequence diagram. The producer sends batch A with producer id 7000, epoch 0 and sequences 0 to 2. The leader appends it and the follower replicates it, but the response to the producer is lost. After the request timeout the producer resends batch A with the same identifiers. The leader sees the sequences were already appended, writes nothing and returns success with the original base offset. Batch B with sequences 3 to 5 is then appended normally.
sequenceDiagram participant P as Producer participant L as Leader broker 1 participant F as Follower broker 2 P->>L: Produce batch A with producer id 7000, epoch 0, sequence 0 to 2 L->>L: append A at offsets 100 to 102 F->>L: fetch L-->>F: batch A L--xP: success response is lost Note over P: request.timeout.ms passes and batch A is retried P->>L: Produce batch A again with the same id, epoch and sequence L->>L: sequence 0 to 2 already appended, nothing written L-->>P: success, base offset 100 P->>L: Produce batch B with sequence 3 to 5 L->>L: sequence follows 2, append at offsets 103 to 105 L-->>P: success, base offset 103
- The first five messages are the ambiguous case: the batch is safely in the log, but the producer cannot know that.
- The retry is byte-for-byte the same batch with the same sequence numbers. Without idempotence the leader would append it at offsets 103 to 105 and consumers would read the three records twice.
- With idempotence the leader answers the duplicate with success. The application's future completes normally and never learns there was a retry.
- Batch B is accepted because its first sequence, 3, is exactly one after the last appended sequence. Had batch B arrived while A was missing, it would have been rejected and resent after A.
In a real system
A typical payment service publishes PaymentCaptured after the card network confirms a capture. Losing the event means an order that is paid but never shipped. Writing it twice means downstream services see two captures unless they deduplicate. Writing PaymentRefunded ahead of PaymentCaptured for the same payment breaks the ledger.
| Setting | Value | Why |
|---|---|---|
Topic payments.events.v1 | 12 partitions, replication factor 3, min.insync.replicas=2 | An acknowledged event is on at least two brokers; one broker can be down and writes continue |
| Key | paymentId | All events of one payment share a partition |
acks | all | Only committed events count as published |
enable.idempotence | true, set explicitly | Retries cannot duplicate or reorder; a conflicting setting fails at startup instead of silently downgrading |
delivery.timeout.ms | 120000 | A rolling broker restart takes seconds; two minutes rides through it. After that the service wants a definite failure |
max.in.flight.requests.per.connection | 5 (default) | Safe with idempotence, no reason to give up the throughput |
During a rolling restart, the leader of a partition stops. Requests in flight to it time out or fail with NotLeaderOrFollowerException. The producer refreshes metadata, finds the new leader and resends the same batches with the same sequence numbers. Batches the old leader had already replicated are recognised as duplicates; the rest are appended in order. The application sees slightly slower futures and nothing else.
If two of the three brokers are down, the in-sync set drops to one and every write fails with NotEnoughReplicasException until a second replica returns. The producer keeps retrying for two minutes, then the futures fail. This is the design working: the service is told it cannot publish durably. What it does next (mark the payment event as unpublished and let an outbox relay publish it later) is the application's decision, and because that later publish is a new send, consumers must still tolerate a duplicate.
Java and Spring Boot
Reliability settings stated explicitly in Spring Boot, a topic that makes acks=all meaningful, and a publisher that separates the two kinds of final failure: definitely not written, and unknown.
spring: kafka: bootstrap-servers: broker-1:9092,broker-2:9092,broker-3:9092 producer: client-id: payment-service key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer acks: all properties: spring.json.add.type.headers: false # Explicit, so a conflicting setting fails at startup instead of disabling it quietly. enable.idempotence: true # The real retry limit. 'retries' is deliberately not set. delivery.timeout.ms: 120000 request.timeout.ms: 30000 max.in.flight.requests.per.connection: 5 # Fail a send() that cannot even be buffered after 10 s. max.block.ms: 10000Why it is written this way
acks: allandenable.idempotence: trueare the client defaults. They are written down so the intent survives a later edit and so a conflict is an error, not a log line.spring.kafka.producer.retriesexists as a Boot property and is left unset on purpose. Setting it to 0 would disable idempotence; setting it to a small number makes the producer give up long before the delivery timeout.request.timeout.msstays at 30 s. The reference recommends keeping it above the broker'sreplica.lag.time.max.ms(30 s by default) so the producer does not retry while the leader is still waiting on a slow follower.- Three bootstrap servers, so the application can start while one broker is down.
package com.example.payments.events;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class PaymentTopicConfig { public static final String PAYMENT_EVENTS = "payments.events.v1"; @Bean NewTopic paymentEventsTopic() { return TopicBuilder.name(PAYMENT_EVENTS) .partitions(12) .replicas(3) .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") .build(); }}Why it is written this way
min.insync.replicas=2with three replicas is what turnsacks=allinto: acknowledged means on at least two brokers.- Setting it to 3 would block all writes whenever one broker is down, including during every rolling restart.
- The producer cannot set this. It is a property of the topic, so durability is a contract between whoever owns the topic and whoever produces to it.
package com.example.payments.events;import java.util.concurrent.CompletableFuture;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.clients.producer.RecordMetadata;import org.apache.kafka.common.errors.TimeoutException;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.core.KafkaProducerException;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.support.SendResult;import org.springframework.stereotype.Component;@Componentpublic class PaymentCapturedPublisher { private static final Logger log = LoggerFactory.getLogger(PaymentCapturedPublisher.class); private final KafkaTemplate<String, PaymentCaptured> kafkaTemplate; private final UnpublishedEvents unpublishedEvents; public PaymentCapturedPublisher(KafkaTemplate<String, PaymentCaptured> kafkaTemplate, UnpublishedEvents unpublishedEvents) { this.kafkaTemplate = kafkaTemplate; this.unpublishedEvents = unpublishedEvents; } public void publish(PaymentCaptured event) { ProducerRecord<String, PaymentCaptured> record = new ProducerRecord<>(PaymentTopicConfig.PAYMENT_EVENTS, event.paymentId(), event); CompletableFuture<SendResult<String, PaymentCaptured>> future; try { future = kafkaTemplate.send(record); } catch (RuntimeException ex) { // Never buffered: serialisation error, no metadata, buffer full, record too large. unpublishedEvents.record(event, Outcome.NOT_WRITTEN, ex); return; } future.whenComplete((result, ex) -> { if (ex == null) { RecordMetadata meta = result.getRecordMetadata(); log.info("PaymentCaptured committed paymentId={} partition={} offset={}", event.paymentId(), meta.partition(), meta.offset()); return; } // The producer has already retried for delivery.timeout.ms. Do NOT call send() again here. Throwable cause = ex instanceof KafkaProducerException ? ex.getCause() : ex; Outcome outcome = cause instanceof TimeoutException ? Outcome.UNKNOWN : Outcome.NOT_WRITTEN; unpublishedEvents.record(event, outcome, cause); }); } public enum Outcome { /** The broker rejected it or it never left the process. */ NOT_WRITTEN, /** Retries ran out. The last attempt may or may not be in the log. */ UNKNOWN } /** Hands a failed event to whatever republishes later (an outbox table, an alert, a job). Must be quick. */ public interface UnpublishedEvents { void record(PaymentCaptured event, Outcome outcome, Throwable cause); } public record PaymentCaptured(String eventId, String paymentId, String orderId, long amountMinor, String currency) { }}Why it is written this way
- A send can fail at the call or later.
KafkaTemplate.send()throws when the record cannot be buffered; everything after that arrives through the future as aKafkaProducerExceptionwrapping the Kafka cause. - There is no retry loop in this class. Retrying is the producer's job, and it already did it with sequence numbers the broker understands. A resend from here would be a new record that can duplicate and that lands behind later events of the same payment.
- A
TimeoutExceptionafter the delivery timeout is recorded as UNKNOWN, not as a clean failure. The republishing path and the consumers have to cope with the event possibly existing already, which is why the event carries aneventId. UnpublishedEvents.recordruns on the producer's I/O thread, so its implementation must hand off quickly, for example to a queue or an executor, rather than write to a database inline.- With a transactional outbox the event row is written in the same database transaction as the payment, and this class is driven by the relay. That closes the gap this code can only report.
Configuration
Defaults are from the Apache Kafka 4.x configuration reference and are identical in the 3.9 client that Spring Boot 3.5 manages, except linger.ms. The last two entries are set on the topic or broker, not on the producer.
acksDefault all
- What it does
- What the leader must have before it answers: nothing (
0), its own write (1), or the write on every in-sync replica (all, same as-1). - When to change it
- Keep
allfor anything you would mind losing. Lowering it also disables idempotence.0is only for data where loss is acceptable, such as debug-level telemetry.
enable.idempotenceDefault true
- What it does
- Adds producer id, epoch and sequence numbers so the broker rejects duplicate and out-of-order batches caused by this producer's retries.
- When to change it
- Leave it on and set it explicitly, so that a conflicting
acks,retriesor in-flight value raises aConfigExceptioninstead of turning it off.
retriesDefault 2147483647
- What it does
- Maximum number of resends of a batch after a transient error.
- When to change it
- Leave unset and control retry behaviour with
delivery.timeout.ms, as the reference recommends.0disables retries and idempotence.
delivery.timeout.msDefault 120000 (2 minutes)
- What it does
- Upper bound on the time from
send()returning to success or failure being reported, including buffering, waiting for the response and retries. - When to change it
- Set it to how long the business can wait for a definite answer. Must be at least
linger.ms + request.timeout.ms. Longer rides out longer outages but keeps callers uncertain for longer.
request.timeout.msDefault 30000 (30 seconds)
- What it does
- How long the client waits for the response to one request before it resends or fails it.
- When to change it
- Keep it above the broker's
replica.lag.time.max.ms(default 30 s). Lowering it far below that causes retries of requests that were about to succeed.
retry.backoff.ms / retry.backoff.max.msDefault 100 / 1000 (1 second)
- What it does
- Initial wait before retrying a failed request to a partition, and the cap of its exponential growth. A jitter of 20% is applied.
- When to change it
- Rarely changed. Raise the maximum if many producers hammer a recovering cluster.
max.in.flight.requests.per.connectionDefault 5
- What it does
- Unanswered requests the client may have on one connection before it stops sending more.
- When to change it
- Keep 5 with idempotence; ordering is preserved for any value up to 5. Only a producer with idempotence off needs 1 to keep order across retries.
linger.msDefault 5 (0 in clients before 4.0)
- What it does
- Time a batch waits for more records. It counts against the delivery timeout.
- When to change it
- Not a reliability setting, but it is part of the constraint
delivery.timeout.ms >= linger.ms + request.timeout.ms.
spring.kafka.producer.acks / spring.kafka.producer.retriesDefault none (the client default applies)
- What it does
- Spring Boot properties that set the Kafka
acksandretriesproducer configs. - When to change it
- Set
ackshere if you want it stated. Avoidretries; putenable.idempotenceand the timeouts underspring.kafka.producer.properties.
min.insync.replicas (topic, broker default)Default 1
- What it does
- Smallest in-sync set, leader included, for which an
acks=allwrite is accepted. Below it the producer getsNotEnoughReplicasException. - When to change it
- Set to 2 with replication factor 3 for data that matters. The default of 1 lets
acks=allsucceed with a single copy.
producer.id.expiration.ms (broker)Default 86400000 (1 day)
- What it does
- How long a partition leader keeps the sequence state of a producer id that has stopped writing.
- When to change it
- Leave the default. It matters only if it were set below
delivery.timeout.ms, where a late retry might no longer be recognised.
Hands-on exercise
Lose a response, lose a leader
Use the producer simulation to produce a duplicate, a reordering and a lost acknowledged record, and find the one setting that removes each.
Steps · 0 of 6 done
Producer reliability settings
Simulation · not a real clusterChange acks, idempotence, retries and in-flight requests, then inject a lost response or a leader failure. This is a deterministic simulation, not a real broker.
What you should see
With idempotence off, a lost response gives the batch twice and a failed first batch ends up behind the second. One request in flight fixes the order but not the duplicate. Idempotence with acks all fixes both at 5 in flight. With acks 1 a record acknowledged just before the leader stops can be missing afterwards; with acks all it is on a follower and survives.
What can go wrong
Lowering acks and losing idempotence with it
- Symptom
- After a change made for latency, consumers report occasional duplicate records and, around broker restarts, events of one key in the wrong order.
- Cause
acks=1(orretries=0) conflicts with idempotence. Becauseenable.idempotencewas not set explicitly, the client disabled it and logged a single INFO line at startup.- Fix
- Remove the override. Set
enable.idempotence=trueexplicitly so the conflict becomes a startup error. Measure before assumingacks=allis the latency problem.
acks=all with min.insync.replicas left at 1
- Symptom
- Records that were acknowledged with
acks=allare gone after a broker failure that followed a period of degraded replication. - Cause
- The in-sync set had shrunk to the leader alone. With the default
min.insync.replicas=1the leader kept acknowledging writes that existed on one disk. - Fix
- Use replication factor 3 with
min.insync.replicas=2and alert on partitions below it. Keep unclean leader election disabled, which is the default.
Treating enable.idempotence as exactly-once
- Symptom
- The same business event appears twice on the topic with different offsets, although idempotence is on.
- Cause
- The application sent it twice: a retry loop around
send(), a restart that republished, or two instances doing the same work. Eachsend()is a new record to the broker. - Fix
- Remove application-level resends where the producer's own retries suffice. Where republishing is unavoidable, carry an event id and make consumers idempotent.
Resending from the failure callback
- Symptom
- After an outage, some events are duplicated and older events of a key appear after newer ones.
- Cause
- The callback called
send()again when a batch expired. The outcome of the expired batch was unknown, and the new send was queued behind records sent in the meantime. - Fix
- Record the failure and republish through a controlled path. If order per key matters, hold later events of that key until the earlier one is resolved, which is what an outbox relay reading in order does.
A small retries value from an old configuration
- Symptom
- Sends fail within a second or two of a leader election with
NotLeaderOrFollowerException, long before the delivery timeout. - Cause
retries=3was copied from an old configuration or tutorial. Three quick retries are used up before the new leader is elected.- Fix
- Delete the
retriessetting and letdelivery.timeout.msbound the retries.
min.insync.replicas equal to the replication factor
- Symptom
- Every rolling restart or single broker failure makes producers fail with
NotEnoughReplicasException. - Cause
- With all replicas required, losing any one of them blocks
acks=allwrites. - Fix
- Keep
min.insync.replicasone below the replication factor, so one broker can be absent.
How to troubleshoot
Sends fail with NotEnoughReplicasException or NotEnoughReplicasAfterAppendException
Check
kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-min-isr-partitions
- How to read it
- Lists partitions whose in-sync set is smaller than
min.insync.replicas. Compare theIsrcolumn withReplicasto see which broker is missing, then look at that broker. The producer is behaving correctly; the cluster cannot give the durability the topic asks for.
Futures fail with TimeoutException: Expiring N record(s) for <topic>-<partition>:120000 ms has passed since batch creation
Check
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic payments.events.v1
- How to read it
- The batch could not be delivered within
delivery.timeout.ms. A partition listed without a leader had none to send to. If leaders are present, check the producer'srequest-latency-avgand whether the brokers are reachable from the application's network.
You need to know whether idempotence is actually on
Check
Application startup log: the `ProducerConfig values` block, line `enable.idempotence`, and any line starting `Idempotence will be disabled because`
- How to read it
- The logged value is the effective one after conflict resolution.
falsetogether with thedisabled becauseline names the setting that caused it.
Suspected retry storms or a struggling broker
Check
Producer metrics `record-retry-rate` and `record-error-rate` (MBean `kafka.producer:type=producer-metrics,client-id=payment-service`)
- How to read it
- A retry rate above zero with an error rate of zero means transient failures that the producer absorbed. An error rate above zero means sends failed for good and the application was told through the future.
Durability is at risk before anything has failed
Check
Broker metrics `kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount` and `name=UnderReplicatedPartitions`
- How to read it
- Both should be 0. Under-replicated partitions mean a follower is behind and the margin is gone; under-min-ISR partitions are already rejecting
acks=allwrites.
Duplicates on the topic and you need to tell producer retries from application resends
Check
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments.events.v1 --from-beginning --formatter-property print.offset=true --formatter-property print.key=true --formatter-property print.headers=true
- How to read it
- With idempotence on, two records with the same event id at different offsets were sent twice by an application, since a producer retry would not have been appended. Tools before Kafka 4.2 spell the option
--property.
Key takeaways
acksdefines what success means.allwaits for every in-sync replica, and is a durability guarantee only withmin.insync.replicasof 2 or more and a replication factor above it.- Retries are effectively unlimited by count.
delivery.timeout.msis the limit, and it covers buffering, waiting and all retries. - A retry can duplicate a batch, and with several requests in flight it can reorder batches. Idempotence prevents both for the producer's own retries.
- Idempotence works through a producer id, an epoch and per-partition sequence numbers checked by the leader. Its scope is one producer session and one partition.
- It does not cover application resends, restarts, other producers or anything a consumer does. Consumer idempotency is separate application logic.
- Setting
acksbelowallorretriesto 0 silently disables idempotence unless you enabled it explicitly. - An error in the callback is final, and an expired batch has an unknown outcome. Record it and republish deliberately; do not resend in the callback.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A topic has replication factor 3, min.insync.replicas=1, and producers use acks=all. During an incident two followers fall out of the in-sync set for ten minutes; producers see no errors. Then the leader's disk fails. After recovery, about ten minutes of acknowledged records are missing.
Which change would have prevented acknowledged records from being lost?
Scenario 2 of 3
A service's configuration contains spring.kafka.producer.acks: 1, added a year ago with the comment faster. Nothing else about the producer is set. After each rolling broker restart, a downstream team reports a handful of duplicate records and, twice, two events of the same key in swapped order.
What explains both symptoms, and what should change?
Scenario 3 of 3
With idempotence on, a producer's callback receives a TimeoutException for event A of payment 501 after 120 seconds of broker trouble. The callback calls send() for A again. Meanwhile event B of the same payment, sent 30 seconds after A, was acknowledged. Later a consumer reads B, then A, and in a few cases A twice.
What is the right way to handle the expired send?
References
References: Apache Kafka documentation: Producer Configs (acks, enable.idempotence, retries, delivery.timeout.ms, max.in.flight.requests.per.connection) ↗ · Apache Kafka documentation: Topic Configs (min.insync.replicas) ↗ · Apache Kafka documentation: Design, Message Delivery Semantics ↗ · KafkaProducer Javadoc (idempotent producer, retries, error handling in send) ↗ · KIP-98: Exactly Once Delivery and Transactional Messaging (producer id, epoch, sequence numbers) ↗ · KIP-679: Producer will enable the strongest delivery guarantee by default ↗ · Apache Kafka documentation: Message Format (record batch header fields) ↗ · RetriableException Javadoc (which errors the client treats as transient) ↗ · Spring for Apache Kafka 3.3 reference: Sending Messages ↗ · Report a problem