Skip to content

Path 5 · lesson 2 of 5

Intermediate25 minApache Kafka 4.x (defaults checked against the 4.4 documentation) · Java 21 · Spring Boot 3.x with Spring for Apache Kafka 3.3 · SQL examples use PostgreSQL syntax

Duplicates: producer idempotence vs consumer idempotency

Every place a duplicate can enter a Kafka pipeline, the single one that producer idempotence removes, and how to make a consumer safe against the rest.

What you will learn

  • List the five sources of duplicates in a Kafka pipeline and place a given duplicate in the right one
  • Explain how the broker uses a producer id and sequence numbers to drop a retried batch, and where that protection ends
  • Keep producer idempotence and consumer idempotency apart when reading a design or an incident report
  • Pick a consumer-side technique for a handler: natural idempotence, a unique business key, a processed-events table, or a conditional update
  • Write a Spring listener that records the event id and applies the business change in one database transaction

Before you start

The concept

A duplicate is the same business event taking effect more than once: one order charged twice, one payment posted to the ledger twice. An operation is idempotent when doing it again with the same input leaves the same result as doing it once. Kafka uses that word for two unrelated mechanisms that sit at opposite ends of the pipeline.

Producer idempotenceConsumer idempotency
What it isA Kafka client and broker featureA property of your handler and your data model
Turned on byenable.idempotence=true, the default since Kafka 3.0Code and schema that you write
Duplicate it removesThe same batch appended twice because the producer retried itThe same event applied twice, whatever the reason it arrived twice
Identity it comparesProducer id, producer epoch and a per-partition sequence numberA business identifier: event id, order id, payment id
Where the memory livesOn the partition leader, replicated with the logIn your database, next to the data it protects
ScopeOne producer session, one partitionAs long as you keep the identifiers
Does not help withAnything the application or the consumer doesNothing on the produce path. The log can still contain the event twice
Same word, different mechanism. One does not replace the other.

Where duplicates come from

SourceWhat happensRemoved by producer idempotence?
1. Producer retryThe broker appended the batch, the acknowledgement was lost, and the client sent the batch againYes
2. Application-level resendYour code calls send again for the same event: a retried HTTP request, a send that timed out and was repeated by a job, a restart that republishesNo. A new call is a new record with a new sequence number
3. Consumer redeliveryThe handler finished, then the consumer crashed or lost the partition in a rebalance before the offset was committedNo. The log holds the record once and it is read twice
4. ReplayOffsets are reset to reprocess, a group's offsets expire, or a record is retried from a retry or dead letter topicNo
5. Upstream duplicateThe event already arrived twice at the producing service, for example a webhook delivered twice or a CDC snapshot re-runNo. They are two distinct records as far as Kafka can tell

Four of the five are outside the reach of any producer setting. That is why a consumer that changes money or stock needs its own defence.

Why it exists

Before Kafka 0.11 a producer that did not receive an acknowledgement had one safe choice: send again. If the first attempt had in fact been written, the log then held the record twice. With several requests in flight a retry could also land after a later batch and reorder records within the partition. The idempotent producer was added to close exactly this gap, and since Kafka 3.0 it is on by default.

It could be solved inside Kafka because the broker sees both attempts and can be given a way to recognise them as the same request. The other four sources cannot be solved there. The broker cannot know that two records with different sequence numbers describe the same customer payment, and it has no view of what a consumer did with a record before it crashed.

So the consumer has to carry its own memory of what it has applied, and that memory has to change atomically with the business effect. If the two are written separately, a crash between them recreates the original problem one level down: either the effect exists without the memory, and a redelivery repeats it, or the memory exists without the effect, and the event is lost.

Dedupe check in a separate store

  • Write the event id to a cache, then update the database
  • Crash after the cache write: the redelivery is skipped and the payment is never recorded
  • Crash after the database write but before the cache write: the redelivery is applied again
  • A cache entry that expires or is evicted lets old duplicates through

Dedupe record in the same transaction

  • Insert the event id and apply the change in one database transaction
  • Crash before commit: neither exists, the redelivery does both
  • Crash after commit: both exist, the redelivery finds the id and does nothing
  • No window in which the two can disagree

How it works inside

How the broker drops a retried batch

  1. When an idempotent producer starts, it asks the cluster for a producer id (PID). The id is internal and never shown to your code.
  2. For each partition it writes to, the producer numbers its records with a sequence number that starts at zero and increases with every record. Each batch carries the PID, the producer epoch and its first sequence number.
  3. The partition leader remembers the latest sequence numbers per producer id. That state is derived from the log itself, so it follows the partition to a new leader.
  4. A batch whose sequence continues from the last one is appended. A batch the leader already has is not appended again: the leader answers it as a duplicate, and the producer treats that as success. A batch that skips ahead is rejected with an out-of-order sequence error, because a gap would mean something was lost.
  5. Because gaps are rejected, retries can no longer reorder a partition even with five requests in flight.

Idempotence requires acks=all, retries greater than zero and max.in.flight.requests.per.connection of at most 5, because the broker retains only the last five batches per producer for this check. If one of those conflicts and you did not set enable.idempotence explicitly, the client switches idempotence off without failing. If you did set it explicitly, the conflict raises a ConfigException at startup.

Where the protection ends

  • A new session. A restarted producer gets a new producer id and starts its sequences again. If the application resends an event after a restart because it never saw the acknowledgement, the broker treats it as a new record. The KafkaProducer documentation states that idempotence is only guaranteed within a single session.
  • A new send call. Each call gets the next sequence number. Two calls with the same payload are two records.
  • After the delivery timeout. When delivery.timeout.ms expires the send fails with an error although the record may be in the log. Code that catches that error and sends again creates source 2.
  • Another partition or topic. Sequences are per partition. The same event sent to two partitions is not compared.

Consumer-side techniques

TechniqueHow it worksUse it whenWatch out for
Natural idempotenceThe operation sets a value instead of changing it: status = PAID, or an upsert of the full row by primary keyThe event carries the complete new stateIncrements, appends and "send an email" are not naturally idempotent. An older event replayed later can overwrite newer state unless you also compare versions.
Unique business keyThe row you insert has a unique constraint on a business identifier, such as payment(order_id) or ledger_entry(payment_id)One event creates one rowThe key must identify the business action, not the Kafka record. Topic, partition and offset do not survive a resend.
Processed-events tableInsert (consumer, event_id) into a table with a primary key, in the same transaction as the business change. If the insert finds the row, skip.The effect is spread over several rows or tables, or is an incrementThe table grows. Delete rows older than the longest replay you allow.
Conditional update or version checkUPDATE ... WHERE status = 'PENDING' or WHERE version = ?, then look at the affected row countThe event is a state transition of an existing aggregateZero rows updated needs a decision: already applied, or arrived out of order.

All four rely on the producing service assigning a stable identifier once, when the business action happens, and carrying it in the event. An id generated inside the publishing code at send time is different on every resend and protects nothing.

Architecture

Drawing diagram…

Flowchart from an upstream caller through the Payment Service code, the Kafka producer client and the partition leader into the payments.captured log, then to a ledger consumer and its database. Arrows are numbered with the five duplicate sources. A box around the producer client and partition leader is labelled producer idempotence, and a box around the consumer and database is labelled consumer idempotency.

flowchart TB
  UP["Upstream caller or webhook"] -->|"5 upstream duplicate"| APP["Payment Service code"]
  APP -->|"2 second send call"| PR["Kafka producer client"]
  PR -->|"1 automatic retry"| LD["Partition leader"]
  LD -->|"PID and sequence check"| LOG["payments.captured log"]
  LOG -->|"3 redelivery after crash or rebalance"| CON["Ledger consumer"]
  LOG -->|"4 replay after offset reset"| CON
  CON -->|"event id checked in the same transaction"| DB["Ledger database"]
  subgraph PI["Producer idempotence covers this"]
    PR
    LD
  end
  subgraph CI["Consumer idempotency covers this"]
    CON
    DB
  end
Five numbered entry points for duplicates. Producer idempotence closes number 1. The consumer's own check has to absorb the other four.
  • Source 1 is the only one that happens between the producer client and the leader, which is the only place the producer id and sequence number are compared.
  • Sources 2 and 5 put two separate records in the log. From then on they are indistinguishable from two real events unless the payload carries a business identifier.
  • Sources 3 and 4 read a single record more than once. The log is correct and the repeat is created on the read side.
  • The consumer's check sits after every source, so it is the one place that catches all five. That is the reason to build it even when the producer side is configured well.

Message flow

Drawing diagram…

Sequence diagram. The producer sends a batch with producer id 7 and sequence 12, the acknowledgement is lost, and the retry with the same sequence is acknowledged without being appended again. Later the ledger consumer reads offset 88, inserts the event id and the ledger entry in one database transaction, and crashes before committing the offset. After restart it reads offset 88 again, the insert of the event id affects zero rows, and it skips the ledger entry and commits offset 89.

sequenceDiagram
  participant P as Payment producer
  participant L as Partition leader
  participant C as Ledger consumer
  participant DB as Ledger database
  P->>L: batch PID 7 seq 12 payment 9001
  L-->>P: ack lost on the network
  P->>L: retry PID 7 seq 12
  L-->>P: ack, already appended, not written again
  Note over P,L: producer idempotence ends here
  C->>L: poll
  L-->>C: offset 88 payment 9001
  C->>DB: begin, insert event id, insert ledger entry, commit
  Note over C: crash before the offset commit
  C->>L: poll after restart
  L-->>C: offset 88 payment 9001 again
  C->>DB: begin, insert event id
  DB-->>C: 0 rows, id already present
  Note over C,DB: skip the entry, then commit offset 89
Two different duplicates of payment 9001, stopped in two different places: the retried batch at the leader, the redelivered record at the database.
  1. The first half is source 1. The leader recognises producer id 7, sequence 12 as a batch it already has and does not append it, so the log holds payment 9001 once.
  2. The second half is source 3. The log is fine. The consumer did the work and died before the group's offset moved, so the same offset is delivered again.
  3. The event id row and the ledger entry were committed together, so the second attempt finds the id and stops before touching the ledger.
  4. Take away either mechanism and one of the two duplicates gets through. Neither mechanism can do the other's job.

In a real system

A typical payments platform has a Payment Service that captures card payments and publishes PaymentCaptured to the topic payments.captured, keyed by paymentId, with 12 partitions. A Ledger Service in consumer group ledger-service posts a double-entry booking for each capture and increases the merchant's pending balance.

What happensSourceWhat stops the double booking
A broker restarts mid-request and the producer resends the batch1 Producer retryProducer idempotence. The leader does not append the batch twice.
The Payment Service times out waiting for the send, its caller retries the capture request, and the event is published again2 Application resendThe eventId was created with the payment row, so both records carry the same id and the ledger skips the second.
A ledger pod is killed after its database commit and before the offset commit3 RedeliveryThe processed-events row committed with the booking.
Finance asks for last Tuesday to be reprocessed and the group's offsets are reset4 ReplayThe same row, provided it has not been purged yet.
The card provider delivers its capture webhook twice5 Upstream duplicateA unique constraint on the provider's reference in the Payment Service, before anything is published.

The balance update is the dangerous part. pending_balance = pending_balance + amount is not naturally idempotent, so it must run inside the transaction that inserts the event id. The booking rows could rely on a unique key on payment_id alone, but the increment could not.

Idempotency keys

Simulation, not a real cluster

The animated Build lesson shows the same idea at an HTTP API: a key stored with the result so a retried request returns the first outcome.

Step 1 · The pattern

One tap, one charge

A shopper taps Pay. The app calls the payments API, the API charges the card, and a success response comes back. One order, one charge.

  • payment request
  • success
Open the full animated lesson

Java and Spring Boot

A ledger consumer that is safe against all five sources. The event id is inserted and the ledger is changed in one JDBC transaction, and the Kafka offset is committed afterwards by the listener container.

src/main/resources/db/migration/V7__ledger_idempotency.sqlSQL
-- One row per event this consumer has applied.CREATE TABLE processed_events (    consumer     VARCHAR(100) NOT NULL,    event_id     UUID         NOT NULL,    processed_at TIMESTAMPTZ  NOT NULL DEFAULT now(),    PRIMARY KEY (consumer, event_id));-- Second line of defence: one booking per payment, whatever the code does.ALTER TABLE ledger_entries    ADD CONSTRAINT uq_ledger_entries_payment UNIQUE (payment_id);-- Used by the nightly purge.CREATE INDEX idx_processed_events_processed_at ON processed_events (processed_at);

Why it is written this way

  • The primary key is what makes the check safe under concurrency. Two transactions inserting the same id cannot both commit, so there is no read-then-write race.
  • consumer is part of the key so several consumers can share the table, each remembering events independently.
  • The unique constraint on payment_id is a business rule in its own right and still holds after old processed_events rows are purged.
  • Purge by processed_at, keeping rows for longer than the topic's retention plus any replay window you support.

Configuration

Only the producer half has configuration. Consumer idempotency is schema and code, and there is no setting that provides it. Defaults are from the Apache Kafka 4.4 documentation.

enable.idempotenceDefault true
What it does
The broker deduplicates this producer's retried batches using its producer id and per-partition sequence numbers.
When to change it
Set it to true explicitly. Left implicit, a conflicting setting disables it silently. Set explicitly, the same conflict fails at startup with a ConfigException.
acksDefault all
What it does
Acknowledgements required before a send counts as complete.
When to change it
Must be all for idempotence. Changing it to 1 for latency is the most common way idempotence gets switched off by accident.
retriesDefault 2147483647
What it does
How many times the client resends a batch after a transient error, bounded in time by delivery.timeout.ms.
When to change it
Leave it unset and control retry duration with delivery.timeout.ms. Must be greater than 0 for idempotence.
max.in.flight.requests.per.connectionDefault 5
What it does
Unacknowledged requests the client may have outstanding on one connection.
When to change it
Keep at 5 or lower. With idempotence on, ordering is preserved for any allowed value, so the old advice to set 1 no longer applies.
delivery.timeout.msDefault 120000 (2 minutes)
What it does
Total time allowed for a record, including retries, before the send fails.
When to change it
When it expires the outcome is unknown. Do not resend blindly from application code. If you must resend, the event needs a stable id so the consumer can drop the copy.
producer.id.expiration.ms (broker)Default 86400000 (1 day)
What it does
How long a partition leader keeps the sequence state of a producer id that has stopped writing.
When to change it
Rarely changed. It matters only if it is set below delivery.timeout.ms, because a retry could then arrive after the state it is compared against has gone.

Hands-on exercise

Stop the double charge

Reproduce a duplicate that producer idempotence cannot prevent, then remove its effect with a consumer-side check and prove it by repeating the same failure.

Steps · 0 of 5 done

Redelivery with and without an idempotent consumer

Simulation · not a real cluster

The consumer charges the card and crashes before the offset is committed. Toggle the idempotent consumer and compare the two runs.

Take it further in the lab: Duplicate message processing

What can go wrong

Using topic, partition and offset as the dedupe key
Symptom
Redeliveries are handled correctly, but an event that the producer published twice is still applied twice.
Cause
The coordinates identify a record in the log, not a business event. A resend, an upstream duplicate or a copy in a retry topic has different coordinates.
Fix
Carry an eventId assigned once when the business action is recorded, and key the dedupe on that or on a natural business identifier.
Generating the event id at send time
Symptom
The dedupe table fills up, yet duplicates from application retries still get through.
Cause
UUID.randomUUID() is called inside the publishing method, so every resend of the same business action carries a fresh id.
Fix
Create the id in the same database transaction as the business change, store it, and publish the stored value. A transactional outbox does this by construction.
Check-then-act outside one transaction
Symptom
Rare double bookings under load or right after a rebalance, although the code does check for the event id.
Cause
The code runs SELECT to see whether the id exists and inserts later, or keeps the id in a cache separate from the database. After a rebalance two consumers can briefly work on the same record and both pass the check.
Fix
Let a primary key or unique constraint decide, with the insert in the same transaction as the business change.
Idempotence switched off by a tuning change
Symptom
Duplicate records with adjacent offsets appear in a topic after broker restarts, sometimes out of order.
Cause
Someone set acks=1 or retries=0. enable.idempotence was never set explicitly, so the client disabled it without an error.
Fix
Set enable.idempotence=true explicitly so a conflicting setting fails at startup, and check the ProducerConfig values block logged when the producer starts.
Purging dedupe rows sooner than a replay can reach
Symptom
A deliberate reprocessing of last month's events applies some of them a second time.
Cause
processed_events rows are deleted after 7 days while the topic retains 30 days of records.
Fix
Keep dedupe rows for at least the topic retention plus the replay window, or rely on a permanent unique business key for effects that must never repeat.

How to troubleshoot

A business effect happened twice and you need to know which source it was

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments.captured --partition 4 --offset 5100 --max-messages 50 --formatter-property print.offset=true --formatter-property print.key=true
How to read it
(On clients before Kafka 4.2 the option is --property.) If the event is in the log once, it was read twice: source 3 or 4. If it is there twice with the same event id, the application or its upstream sent it twice: source 2 or 5. Twice with different ids means the id is generated at send time.
Suspected redelivery after a rebalance

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ledger-service, together with the consumer logs around the time of the duplicate
How to read it
Look for partitions being revoked and assigned, or a commit failing because the member was no longer in the group. Records handled after the last successful commit are delivered again to the new owner.
Not sure whether idempotence is really enabled on a producer

Check

The ProducerConfig values block that the client logs at INFO level on startup: enable.idempotence, acks, retries, max.in.flight.requests.per.connection
How to read it
enable.idempotence = false together with acks = 1 or retries = 0 shows that a conflicting setting disabled it.
The producer logs OutOfOrderSequenceException

Check

Producer application logs, and whether that partition recently lost acknowledged data, for example through an unclean leader election
How to read it
The broker saw a gap in this producer's sequence numbers, so something it expected is missing. With idempotence only, the documentation recommends closing the producer and creating a new one to keep ordering.
Duplicates are being skipped and you want to know how often

Check

A counter in the handler's duplicate branch, for example ledger.events.duplicate, split by topic
How to read it
A small steady rate around deployments and rebalances is normal at-least-once behaviour. A sudden rise points at a rebalance loop, an upstream resend storm or a replay.

Key takeaways

  • Duplicates come from five places: producer retries, application resends, consumer redelivery, replays and upstream duplicates.
  • Producer idempotence removes only the first, by comparing producer id and sequence number at the partition leader, within one producer session and one partition.
  • Consumer idempotency is application logic. No Kafka setting provides it, and it is the only defence that catches all five sources.
  • The record of "already applied" must commit in the same database transaction as the effect, enforced by a primary key or unique constraint.
  • Identify events by an id assigned once with the business action. Offsets and send-time ids do not survive a resend.
  • Pick the lightest technique that fits: a natural overwrite, a unique business key, a conditional update, or a processed-events table for increments and multi-row effects.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

An Order Service calls kafkaTemplate.send(...) and waits up to 2 seconds for the result. On a timeout, its HTTP client upstream retries the whole "place order" request, and the service publishes OrderPlaced again with a freshly generated eventId. The producer has enable.idempotence=true. Payments are occasionally taken twice.

Which change addresses the cause?

Scenario 2 of 3

A stock consumer handles OrderPlaced with UPDATE stock SET available = available - :qty WHERE sku = :sku. After a rebalance during a deploy, 14 SKUs show less stock than the warehouse count. The topic contains each order exactly once.

What is the most direct fix?

Scenario 3 of 3

A ledger handler first checks Redis for the event id, then writes the booking to PostgreSQL, then sets the id in Redis with a 24 hour expiry. In a month it has produced 3 double bookings and 1 missing booking.

What explains both kinds of error?

References

References: KafkaProducer Javadoc: the idempotent producer, application-level re-sends, single-session guarantee ↗ · Apache Kafka documentation: Producer configs (enable.idempotence, acks, retries, max.in.flight.requests.per.connection) ↗ · KIP-98: Exactly Once Delivery and Transactional Messaging (idempotent producer guarantees) ↗ · Apache Kafka documentation: Design, Message Delivery Semantics ↗ · Apache Kafka documentation: Broker configs (producer.id.expiration.ms) ↗ · Spring for Apache Kafka reference: Message listener containers, committing offsets ↗ · Report a problem