Skip to content

Path 3 · lesson 3 of 5

Intermediate28 minApache Kafka 4.x (producer, topic and broker configs checked against the 4.0, 4.1 and 4.4 reference) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 (kafka-clients 3.9)

Acks, retries and the idempotent producer

What an acknowledgement proves, how retries are bounded by the delivery timeout, how they duplicate and reorder records, and exactly what producer idempotence fixes.

What you will learn

  • State what the producer knows after acks=0, acks=1 and acks=all, and what min.insync.replicas adds to acks=all
  • Work out how long a record can stay unresolved from delivery.timeout.ms, request.timeout.ms, linger.ms and the retry backoff
  • Explain how a retry creates a duplicate and how it reorders two batches when several requests are in flight
  • Describe how the broker uses producer id, epoch and sequence numbers to reject duplicate and out-of-order batches
  • Say precisely which duplicates producer idempotence removes and which it cannot
  • Tell a retriable error from a final one and decide what the application does when a send fails for good

Before you start

The concept

A produce request ends in one of three ways: the broker says it succeeded, the broker says it failed, or the producer hears nothing. Three groups of settings decide what happens in each case. acks defines what the broker must have done before it answers. Retries, bounded by delivery.timeout.ms, define what the producer does when the answer is an error or does not come. Idempotence makes those retries safe.

What an acknowledgement proves

acksThe leader answers whenA record can be lost ifThe producer learns
0It does not answer. The record counts as sent once it is in the socket bufferAnything goes wrong after that: connection drop, leader crash, full diskNothing. The returned offset is always -1 and retries have no effect
1It has written the record to its own logThe leader fails before any follower copied the recordThat one broker had it at that moment
all (same as -1)Every replica currently in the in-sync set has the recordAll in-sync replicas are lost, which can be a single broker if the set had shrunk to oneThat the record was committed to the in-sync replicas of that moment
The default is all. Java clients before 3.0 defaulted to 1.

min.insync.replicas is a topic setting (with a broker-wide default) that is enforced only for acks=all writes. If the in-sync set is smaller than it, the leader refuses the write with NotEnoughReplicasException. The producer then gets an honest failure instead of a weak success. You trade availability of writes for the certainty that an acknowledged record exists on at least that many brokers.

Retries, and their real limit

When a request fails with a transient error or no response arrives within request.timeout.ms, the producer puts the batch back and sends it again after a backoff. retries defaults to 2,147,483,647, so the count is not what stops it. The limit is delivery.timeout.ms (default 2 minutes): the maximum time between send() returning and the future completing. It covers time waiting in the buffer, time waiting for the response and every retry. When it runs out, the batch fails with a TimeoutException.

Idempotence in one paragraph

A retry resends a batch the broker may already have written. With enable.idempotence=true (the default) every batch carries the producer's id and a per-partition sequence number, and the broker refuses to append a sequence it has already appended. Retries then cannot duplicate or reorder records in the log. The scope of that statement is narrow, and the internals section spells it out.

Why it exists

Networks lose responses. From the producer's side, a request that timed out is ambiguous: the broker may never have received it, or it may have written the batch and the reply was lost. There are only two things to do with an ambiguous request.

Give up (no retries)

  • A record that was not written is lost: at-most-once
  • The application sees an error for something that may in fact be in the log
  • No duplicates from the producer

Send again (retries)

  • A record that was not written gets another chance: at-least-once
  • A record that was written is written a second time
  • With several requests in flight, a retried batch can land behind a later one

Before Kafka 0.11 those were the only choices, and the safe configuration was retries with one request in flight at a time, which protected ordering at the cost of throughput and still allowed duplicates. The idempotent producer (KIP-98) removed the trade-off by letting the broker recognise a resend. Since Kafka 3.0 it is on by default together with acks=all, because a default that quietly loses or duplicates data surprised too many people.

The retry count turned out to be the wrong knob as well. Nobody knows whether 3 or 10 retries is right; what an application can reason about is how long it is willing to wait for a definite answer. delivery.timeout.ms expresses that directly, which is why the client documentation recommends leaving retries alone and setting the timeout.

How it works inside

The time budget of one record

The clock starts when send() returns. The record waits up to linger.ms in its batch, then the batch is sent. Each attempt waits at most request.timeout.ms (30 s) for a response. A failed attempt is followed by a backoff that starts at retry.backoff.ms (100 ms) and grows exponentially to retry.backoff.max.ms (1 s), with jitter. All of it must fit inside delivery.timeout.ms, which is therefore required to be at least linger.ms + request.timeout.ms. With the defaults, a batch whose requests keep timing out gets about four attempts before it expires.

How a retry duplicates

The leader appends the batch, the response is lost, the producer resends, the leader appends again. Without idempotence the log now has the records twice, at different offsets, and every consumer reads both. The same happens with NotEnoughReplicasAfterAppendException: the leader wrote the batch, then found the in-sync set too small to acknowledge it, and the retry writes a second copy.

How a retry reorders

The producer may have up to max.in.flight.requests.per.connection (default 5) unanswered requests to a broker. Suppose batch 1 and batch 2 for the same partition are both in flight. Batch 1 fails with a transient error, batch 2 succeeds, and batch 1 succeeds on retry. The log now holds batch 2 before batch 1. For records sharing a key, per-key order is broken inside the partition. This can only happen when retries are enabled, more than one request is in flight and idempotence is off.

The idempotent producer

  1. When it starts, the producer asks the cluster for a producer id and an epoch. Without a transactional.id, every new producer instance gets a fresh id.
  2. For each partition it writes to, the producer numbers its records with a sequence starting at 0. Each batch header carries the producer id, the epoch and the batch's base sequence. A retry resends the batch with the same numbers.
  3. The partition leader remembers, per producer id, the sequences of the most recent batches (five of them). A batch whose sequence is the next expected one is appended.
  4. A batch whose sequences were already appended is recognised as a duplicate. The leader does not write it again and the producer treats the request as successful.
  5. A batch whose sequence is ahead of the expected one means an earlier batch is missing. The leader rejects it, so a later batch can never be written before an earlier one. The producer resends in order. This is why ordering holds with up to 5 requests in flight, and why idempotence requires max.in.flight.requests.per.connection to be 5 or less.
  6. The id, epoch and sequence are stored in the log with the batch, so they are replicated. A follower that becomes leader can continue the check.
SituationDoes producer idempotence prevent a duplicate?
The producer's own retry of a batch after a timeout or transient errorYes. This is the case it exists for
A retry that lands on a new leader after a failoverYes. The sequence state travels with the replicated log
Your code calls send() twice for the same business eventNo. Each call gets a new sequence number and is a new record
The application restarts and republishes events it is unsure aboutNo. The new producer instance has a new producer id
Two service instances publish the same eventNo. Different producers, different ids
A consumer processes a record twice after a rebalance or crashNo. This has nothing to do with the producer
Idempotence covers one producer session, one partition, and the producer's internal retries. Nothing else.

Idempotence can switch itself off

Idempotence requires acks=all, retries greater than 0 and at most 5 in-flight requests. If you set a conflicting value and did not set enable.idempotence yourself, the client disables idempotence and logs one INFO line such as Idempotence will be disabled because acks is set to 1, not set to 'all'. If you set enable.idempotence=true explicitly, the same conflict fails at startup with a ConfigException. Stating it explicitly turns a silent downgrade into a visible error.

Retriable and final errors

ErrorRetried by the producer?Typical cause
NotLeaderOrFollowerException, LeaderNotAvailableExceptionYes, after a metadata refreshLeader moved or is being elected
NetworkException, request timeoutYesConnection dropped, broker slow or restarting
NotEnoughReplicasException, NotEnoughReplicasAfterAppendExceptionYesIn-sync set smaller than min.insync.replicas
UnknownTopicOrPartitionExceptionYesTopic still being created, or metadata not propagated yet
RecordTooLargeExceptionNoRecord exceeds a size limit; sending it again cannot help
TopicAuthorizationExceptionNoMissing ACL for the topic
SerializationExceptionNo, thrown by send() itselfValue does not match the serialiser
TimeoutException with Expiring N record(s)No, this is the retries giving updelivery.timeout.ms passed without a successful attempt
Classification follows the client's exception hierarchy: subclasses of RetriableException are retried.

Whatever reaches your callback is final. The producer has already retried for as long as it was allowed. An expired batch is the awkward case: the outcome is unknown, because the last attempt may have been written without the response arriving. The client documentation's advice for this case is to treat it seriously: if you resend from the application, the broker cannot recognise the resend, so you may duplicate the record, and records for the same key that were sent later may already be in the log ahead of it.

The ordering guarantee that results

  • With idempotence on: records one producer sends to one partition are appended in send order, without duplicates from retries, with up to 5 requests in flight.
  • With idempotence off, retries on and more than one request in flight: order within a partition can change after a failed send.
  • With idempotence off and max.in.flight.requests.per.connection=1: order holds, duplicates from retries remain.
  • In every case, a send that finally fails leaves a hole. Later records of the same key that succeeded are in the log, and anything you publish afterwards to fill the hole lands behind them.

Break ordering and durability, then repair them

Simulation · not a real cluster

The starting settings are a producer with acks=1 and idempotence off. Inject a lost response and a leader failure, then change one setting at a time.

ISR / acks

Simulation, not a real cluster

The animated Build lesson shows the broker side of the same story: the in-sync set shrinking and what each acks level then accepts.

Step 1 · The pattern

A leader and its followers

Every partition has a leader that takes writes and followers that copy them. If the leader dies, a follower that is caught up takes over.

  • message
  • being copied to follower
  • confirmed
Open the full animated lesson

Architecture

Drawing diagram…

State diagram of a producer batch. It starts Buffered, becomes InFlight when full or when linger has passed, and ends Acked on a success response. A retriable error or request timeout moves it to Backoff and then back to InFlight with the same sequence numbers. A non-retriable error, or the delivery timeout passing in any state, moves it to Failed.

stateDiagram-v2
  [*] --> Buffered
  Buffered --> InFlight: batch full or linger passed
  InFlight --> Acked: success response
  InFlight --> Backoff: retriable error or request timeout
  Backoff --> InFlight: resend with the same sequence numbers
  InFlight --> Failed: error that is not retriable
  Buffered --> Failed: delivery timeout passed
  InFlight --> Failed: delivery timeout passed
  Backoff --> Failed: delivery timeout passed
  Acked --> [*]
  Failed --> [*]
Life of one batch inside the producer. Every path to Failed or Acked completes the futures of its records.
  • The loop between InFlight and Backoff is the retry mechanism. It has no counter that matters in practice; the only exit besides success is a final error or the delivery timeout.
  • The delivery timeout applies in every state, including Buffered. A batch can expire without ever being sent, for example when the partition has no leader for two minutes.
  • A resend carries the same producer id, epoch and sequence numbers as the first attempt. That sameness is what lets the broker recognise it.
  • Acked means what acks says it means. With acks=1 a batch can reach Acked and still be lost when the leader fails.

Message flow

Drawing diagram…

Sequence diagram. The producer sends batch A with producer id 7000, epoch 0 and sequences 0 to 2. The leader appends it and the follower replicates it, but the response to the producer is lost. After the request timeout the producer resends batch A with the same identifiers. The leader sees the sequences were already appended, writes nothing and returns success with the original base offset. Batch B with sequences 3 to 5 is then appended normally.

sequenceDiagram
  participant P as Producer
  participant L as Leader broker 1
  participant F as Follower broker 2
  P->>L: Produce batch A with producer id 7000, epoch 0, sequence 0 to 2
  L->>L: append A at offsets 100 to 102
  F->>L: fetch
  L-->>F: batch A
  L--xP: success response is lost
  Note over P: request.timeout.ms passes and batch A is retried
  P->>L: Produce batch A again with the same id, epoch and sequence
  L->>L: sequence 0 to 2 already appended, nothing written
  L-->>P: success, base offset 100
  P->>L: Produce batch B with sequence 3 to 5
  L->>L: sequence follows 2, append at offsets 103 to 105
  L-->>P: success, base offset 103
A lost response and a retry with idempotence on: the log gets batch A once.
  1. The first five messages are the ambiguous case: the batch is safely in the log, but the producer cannot know that.
  2. The retry is byte-for-byte the same batch with the same sequence numbers. Without idempotence the leader would append it at offsets 103 to 105 and consumers would read the three records twice.
  3. With idempotence the leader answers the duplicate with success. The application's future completes normally and never learns there was a retry.
  4. Batch B is accepted because its first sequence, 3, is exactly one after the last appended sequence. Had batch B arrived while A was missing, it would have been rejected and resent after A.

In a real system

A typical payment service publishes PaymentCaptured after the card network confirms a capture. Losing the event means an order that is paid but never shipped. Writing it twice means downstream services see two captures unless they deduplicate. Writing PaymentRefunded ahead of PaymentCaptured for the same payment breaks the ledger.

SettingValueWhy
Topic payments.events.v112 partitions, replication factor 3, min.insync.replicas=2An acknowledged event is on at least two brokers; one broker can be down and writes continue
KeypaymentIdAll events of one payment share a partition
acksallOnly committed events count as published
enable.idempotencetrue, set explicitlyRetries cannot duplicate or reorder; a conflicting setting fails at startup instead of silently downgrading
delivery.timeout.ms120000A rolling broker restart takes seconds; two minutes rides through it. After that the service wants a definite failure
max.in.flight.requests.per.connection5 (default)Safe with idempotence, no reason to give up the throughput

During a rolling restart, the leader of a partition stops. Requests in flight to it time out or fail with NotLeaderOrFollowerException. The producer refreshes metadata, finds the new leader and resends the same batches with the same sequence numbers. Batches the old leader had already replicated are recognised as duplicates; the rest are appended in order. The application sees slightly slower futures and nothing else.

If two of the three brokers are down, the in-sync set drops to one and every write fails with NotEnoughReplicasException until a second replica returns. The producer keeps retrying for two minutes, then the futures fail. This is the design working: the service is told it cannot publish durably. What it does next (mark the payment event as unpublished and let an outbox relay publish it later) is the application's decision, and because that later publish is a new send, consumers must still tolerate a duplicate.

Java and Spring Boot

Reliability settings stated explicitly in Spring Boot, a topic that makes acks=all meaningful, and a publisher that separates the two kinds of final failure: definitely not written, and unknown.

src/main/resources/application.ymlYAML
spring:  kafka:    bootstrap-servers: broker-1:9092,broker-2:9092,broker-3:9092    producer:      client-id: payment-service      key-serializer: org.apache.kafka.common.serialization.StringSerializer      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer      acks: all      properties:        spring.json.add.type.headers: false        # Explicit, so a conflicting setting fails at startup instead of disabling it quietly.        enable.idempotence: true        # The real retry limit. 'retries' is deliberately not set.        delivery.timeout.ms: 120000        request.timeout.ms: 30000        max.in.flight.requests.per.connection: 5        # Fail a send() that cannot even be buffered after 10 s.        max.block.ms: 10000

Why it is written this way

  • acks: all and enable.idempotence: true are the client defaults. They are written down so the intent survives a later edit and so a conflict is an error, not a log line.
  • spring.kafka.producer.retries exists as a Boot property and is left unset on purpose. Setting it to 0 would disable idempotence; setting it to a small number makes the producer give up long before the delivery timeout.
  • request.timeout.ms stays at 30 s. The reference recommends keeping it above the broker's replica.lag.time.max.ms (30 s by default) so the producer does not retry while the leader is still waiting on a slow follower.
  • Three bootstrap servers, so the application can start while one broker is down.

Configuration

Defaults are from the Apache Kafka 4.x configuration reference and are identical in the 3.9 client that Spring Boot 3.5 manages, except linger.ms. The last two entries are set on the topic or broker, not on the producer.

acksDefault all
What it does
What the leader must have before it answers: nothing (0), its own write (1), or the write on every in-sync replica (all, same as -1).
When to change it
Keep all for anything you would mind losing. Lowering it also disables idempotence. 0 is only for data where loss is acceptable, such as debug-level telemetry.
enable.idempotenceDefault true
What it does
Adds producer id, epoch and sequence numbers so the broker rejects duplicate and out-of-order batches caused by this producer's retries.
When to change it
Leave it on and set it explicitly, so that a conflicting acks, retries or in-flight value raises a ConfigException instead of turning it off.
retriesDefault 2147483647
What it does
Maximum number of resends of a batch after a transient error.
When to change it
Leave unset and control retry behaviour with delivery.timeout.ms, as the reference recommends. 0 disables retries and idempotence.
delivery.timeout.msDefault 120000 (2 minutes)
What it does
Upper bound on the time from send() returning to success or failure being reported, including buffering, waiting for the response and retries.
When to change it
Set it to how long the business can wait for a definite answer. Must be at least linger.ms + request.timeout.ms. Longer rides out longer outages but keeps callers uncertain for longer.
request.timeout.msDefault 30000 (30 seconds)
What it does
How long the client waits for the response to one request before it resends or fails it.
When to change it
Keep it above the broker's replica.lag.time.max.ms (default 30 s). Lowering it far below that causes retries of requests that were about to succeed.
retry.backoff.ms / retry.backoff.max.msDefault 100 / 1000 (1 second)
What it does
Initial wait before retrying a failed request to a partition, and the cap of its exponential growth. A jitter of 20% is applied.
When to change it
Rarely changed. Raise the maximum if many producers hammer a recovering cluster.
max.in.flight.requests.per.connectionDefault 5
What it does
Unanswered requests the client may have on one connection before it stops sending more.
When to change it
Keep 5 with idempotence; ordering is preserved for any value up to 5. Only a producer with idempotence off needs 1 to keep order across retries.
linger.msDefault 5 (0 in clients before 4.0)
What it does
Time a batch waits for more records. It counts against the delivery timeout.
When to change it
Not a reliability setting, but it is part of the constraint delivery.timeout.ms >= linger.ms + request.timeout.ms.
spring.kafka.producer.acks / spring.kafka.producer.retriesDefault none (the client default applies)
What it does
Spring Boot properties that set the Kafka acks and retries producer configs.
When to change it
Set acks here if you want it stated. Avoid retries; put enable.idempotence and the timeouts under spring.kafka.producer.properties.
min.insync.replicas (topic, broker default)Default 1
What it does
Smallest in-sync set, leader included, for which an acks=all write is accepted. Below it the producer gets NotEnoughReplicasException.
When to change it
Set to 2 with replication factor 3 for data that matters. The default of 1 lets acks=all succeed with a single copy.
producer.id.expiration.ms (broker)Default 86400000 (1 day)
What it does
How long a partition leader keeps the sequence state of a producer id that has stopped writing.
When to change it
Leave the default. It matters only if it were set below delivery.timeout.ms, where a late retry might no longer be recognised.

Hands-on exercise

Lose a response, lose a leader

Use the producer simulation to produce a duplicate, a reordering and a lost acknowledged record, and find the one setting that removes each.

Steps · 0 of 6 done

Producer reliability settings

Simulation · not a real cluster

Change acks, idempotence, retries and in-flight requests, then inject a lost response or a leader failure. This is a deterministic simulation, not a real broker.

Take it further in the lab: Producer publishing messages

What can go wrong

Lowering acks and losing idempotence with it
Symptom
After a change made for latency, consumers report occasional duplicate records and, around broker restarts, events of one key in the wrong order.
Cause
acks=1 (or retries=0) conflicts with idempotence. Because enable.idempotence was not set explicitly, the client disabled it and logged a single INFO line at startup.
Fix
Remove the override. Set enable.idempotence=true explicitly so the conflict becomes a startup error. Measure before assuming acks=all is the latency problem.
acks=all with min.insync.replicas left at 1
Symptom
Records that were acknowledged with acks=all are gone after a broker failure that followed a period of degraded replication.
Cause
The in-sync set had shrunk to the leader alone. With the default min.insync.replicas=1 the leader kept acknowledging writes that existed on one disk.
Fix
Use replication factor 3 with min.insync.replicas=2 and alert on partitions below it. Keep unclean leader election disabled, which is the default.
Treating enable.idempotence as exactly-once
Symptom
The same business event appears twice on the topic with different offsets, although idempotence is on.
Cause
The application sent it twice: a retry loop around send(), a restart that republished, or two instances doing the same work. Each send() is a new record to the broker.
Fix
Remove application-level resends where the producer's own retries suffice. Where republishing is unavoidable, carry an event id and make consumers idempotent.
Resending from the failure callback
Symptom
After an outage, some events are duplicated and older events of a key appear after newer ones.
Cause
The callback called send() again when a batch expired. The outcome of the expired batch was unknown, and the new send was queued behind records sent in the meantime.
Fix
Record the failure and republish through a controlled path. If order per key matters, hold later events of that key until the earlier one is resolved, which is what an outbox relay reading in order does.
A small retries value from an old configuration
Symptom
Sends fail within a second or two of a leader election with NotLeaderOrFollowerException, long before the delivery timeout.
Cause
retries=3 was copied from an old configuration or tutorial. Three quick retries are used up before the new leader is elected.
Fix
Delete the retries setting and let delivery.timeout.ms bound the retries.
min.insync.replicas equal to the replication factor
Symptom
Every rolling restart or single broker failure makes producers fail with NotEnoughReplicasException.
Cause
With all replicas required, losing any one of them blocks acks=all writes.
Fix
Keep min.insync.replicas one below the replication factor, so one broker can be absent.

How to troubleshoot

Sends fail with NotEnoughReplicasException or NotEnoughReplicasAfterAppendException

Check

kafka-topics.sh --bootstrap-server localhost:9092 --describe --under-min-isr-partitions
How to read it
Lists partitions whose in-sync set is smaller than min.insync.replicas. Compare the Isr column with Replicas to see which broker is missing, then look at that broker. The producer is behaving correctly; the cluster cannot give the durability the topic asks for.
Futures fail with TimeoutException: Expiring N record(s) for <topic>-<partition>:120000 ms has passed since batch creation

Check

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic payments.events.v1
How to read it
The batch could not be delivered within delivery.timeout.ms. A partition listed without a leader had none to send to. If leaders are present, check the producer's request-latency-avg and whether the brokers are reachable from the application's network.
You need to know whether idempotence is actually on

Check

Application startup log: the `ProducerConfig values` block, line `enable.idempotence`, and any line starting `Idempotence will be disabled because`
How to read it
The logged value is the effective one after conflict resolution. false together with the disabled because line names the setting that caused it.
Suspected retry storms or a struggling broker

Check

Producer metrics `record-retry-rate` and `record-error-rate` (MBean `kafka.producer:type=producer-metrics,client-id=payment-service`)
How to read it
A retry rate above zero with an error rate of zero means transient failures that the producer absorbed. An error rate above zero means sends failed for good and the application was told through the future.
Durability is at risk before anything has failed

Check

Broker metrics `kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount` and `name=UnderReplicatedPartitions`
How to read it
Both should be 0. Under-replicated partitions mean a follower is behind and the margin is gone; under-min-ISR partitions are already rejecting acks=all writes.
Duplicates on the topic and you need to tell producer retries from application resends

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments.events.v1 --from-beginning --formatter-property print.offset=true --formatter-property print.key=true --formatter-property print.headers=true
How to read it
With idempotence on, two records with the same event id at different offsets were sent twice by an application, since a producer retry would not have been appended. Tools before Kafka 4.2 spell the option --property.

Key takeaways

  • acks defines what success means. all waits for every in-sync replica, and is a durability guarantee only with min.insync.replicas of 2 or more and a replication factor above it.
  • Retries are effectively unlimited by count. delivery.timeout.ms is the limit, and it covers buffering, waiting and all retries.
  • A retry can duplicate a batch, and with several requests in flight it can reorder batches. Idempotence prevents both for the producer's own retries.
  • Idempotence works through a producer id, an epoch and per-partition sequence numbers checked by the leader. Its scope is one producer session and one partition.
  • It does not cover application resends, restarts, other producers or anything a consumer does. Consumer idempotency is separate application logic.
  • Setting acks below all or retries to 0 silently disables idempotence unless you enabled it explicitly.
  • An error in the callback is final, and an expired batch has an unknown outcome. Record it and republish deliberately; do not resend in the callback.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A topic has replication factor 3, min.insync.replicas=1, and producers use acks=all. During an incident two followers fall out of the in-sync set for ten minutes; producers see no errors. Then the leader's disk fails. After recovery, about ten minutes of acknowledged records are missing.

Which change would have prevented acknowledged records from being lost?

Scenario 2 of 3

A service's configuration contains spring.kafka.producer.acks: 1, added a year ago with the comment faster. Nothing else about the producer is set. After each rolling broker restart, a downstream team reports a handful of duplicate records and, twice, two events of the same key in swapped order.

What explains both symptoms, and what should change?

Scenario 3 of 3

With idempotence on, a producer's callback receives a TimeoutException for event A of payment 501 after 120 seconds of broker trouble. The callback calls send() for A again. Meanwhile event B of the same payment, sent 30 seconds after A, was acknowledged. Later a consumer reads B, then A, and in a few cases A twice.

What is the right way to handle the expired send?

References

References: Apache Kafka documentation: Producer Configs (acks, enable.idempotence, retries, delivery.timeout.ms, max.in.flight.requests.per.connection) ↗ · Apache Kafka documentation: Topic Configs (min.insync.replicas) ↗ · Apache Kafka documentation: Design, Message Delivery Semantics ↗ · KafkaProducer Javadoc (idempotent producer, retries, error handling in send) ↗ · KIP-98: Exactly Once Delivery and Transactional Messaging (producer id, epoch, sequence numbers) ↗ · KIP-679: Producer will enable the strongest delivery guarantee by default ↗ · Apache Kafka documentation: Message Format (record batch header fields) ↗ · RetriableException Javadoc (which errors the client treats as transient) ↗ · Spring for Apache Kafka 3.3 reference: Sending Messages ↗ · Report a problem