Path 5 · lesson 2 of 5
Duplicates: producer idempotence vs consumer idempotency
Every place a duplicate can enter a Kafka pipeline, the single one that producer idempotence removes, and how to make a consumer safe against the rest.
What you will learn
- List the five sources of duplicates in a Kafka pipeline and place a given duplicate in the right one
- Explain how the broker uses a producer id and sequence numbers to drop a retried batch, and where that protection ends
- Keep producer idempotence and consumer idempotency apart when reading a design or an incident report
- Pick a consumer-side technique for a handler: natural idempotence, a unique business key, a processed-events table, or a conditional update
- Write a Spring listener that records the event id and applies the business change in one database transaction
Before you start
The concept
A duplicate is the same business event taking effect more than once: one order charged twice, one payment posted to the ledger twice. An operation is idempotent when doing it again with the same input leaves the same result as doing it once. Kafka uses that word for two unrelated mechanisms that sit at opposite ends of the pipeline.
| Producer idempotence | Consumer idempotency | |
|---|---|---|
| What it is | A Kafka client and broker feature | A property of your handler and your data model |
| Turned on by | enable.idempotence=true, the default since Kafka 3.0 | Code and schema that you write |
| Duplicate it removes | The same batch appended twice because the producer retried it | The same event applied twice, whatever the reason it arrived twice |
| Identity it compares | Producer id, producer epoch and a per-partition sequence number | A business identifier: event id, order id, payment id |
| Where the memory lives | On the partition leader, replicated with the log | In your database, next to the data it protects |
| Scope | One producer session, one partition | As long as you keep the identifiers |
| Does not help with | Anything the application or the consumer does | Nothing on the produce path. The log can still contain the event twice |
Where duplicates come from
| Source | What happens | Removed by producer idempotence? |
|---|---|---|
| 1. Producer retry | The broker appended the batch, the acknowledgement was lost, and the client sent the batch again | Yes |
| 2. Application-level resend | Your code calls send again for the same event: a retried HTTP request, a send that timed out and was repeated by a job, a restart that republishes | No. A new call is a new record with a new sequence number |
| 3. Consumer redelivery | The handler finished, then the consumer crashed or lost the partition in a rebalance before the offset was committed | No. The log holds the record once and it is read twice |
| 4. Replay | Offsets are reset to reprocess, a group's offsets expire, or a record is retried from a retry or dead letter topic | No |
| 5. Upstream duplicate | The event already arrived twice at the producing service, for example a webhook delivered twice or a CDC snapshot re-run | No. They are two distinct records as far as Kafka can tell |
Four of the five are outside the reach of any producer setting. That is why a consumer that changes money or stock needs its own defence.
Why it exists
Before Kafka 0.11 a producer that did not receive an acknowledgement had one safe choice: send again. If the first attempt had in fact been written, the log then held the record twice. With several requests in flight a retry could also land after a later batch and reorder records within the partition. The idempotent producer was added to close exactly this gap, and since Kafka 3.0 it is on by default.
It could be solved inside Kafka because the broker sees both attempts and can be given a way to recognise them as the same request. The other four sources cannot be solved there. The broker cannot know that two records with different sequence numbers describe the same customer payment, and it has no view of what a consumer did with a record before it crashed.
So the consumer has to carry its own memory of what it has applied, and that memory has to change atomically with the business effect. If the two are written separately, a crash between them recreates the original problem one level down: either the effect exists without the memory, and a redelivery repeats it, or the memory exists without the effect, and the event is lost.
Dedupe check in a separate store
- Write the event id to a cache, then update the database
- Crash after the cache write: the redelivery is skipped and the payment is never recorded
- Crash after the database write but before the cache write: the redelivery is applied again
- A cache entry that expires or is evicted lets old duplicates through
Dedupe record in the same transaction
- Insert the event id and apply the change in one database transaction
- Crash before commit: neither exists, the redelivery does both
- Crash after commit: both exist, the redelivery finds the id and does nothing
- No window in which the two can disagree
How it works inside
How the broker drops a retried batch
- When an idempotent producer starts, it asks the cluster for a producer id (PID). The id is internal and never shown to your code.
- For each partition it writes to, the producer numbers its records with a sequence number that starts at zero and increases with every record. Each batch carries the PID, the producer epoch and its first sequence number.
- The partition leader remembers the latest sequence numbers per producer id. That state is derived from the log itself, so it follows the partition to a new leader.
- A batch whose sequence continues from the last one is appended. A batch the leader already has is not appended again: the leader answers it as a duplicate, and the producer treats that as success. A batch that skips ahead is rejected with an out-of-order sequence error, because a gap would mean something was lost.
- Because gaps are rejected, retries can no longer reorder a partition even with five requests in flight.
Idempotence requires acks=all, retries greater than zero and max.in.flight.requests.per.connection of at most 5, because the broker retains only the last five batches per producer for this check. If one of those conflicts and you did not set enable.idempotence explicitly, the client switches idempotence off without failing. If you did set it explicitly, the conflict raises a ConfigException at startup.
Where the protection ends
- A new session. A restarted producer gets a new producer id and starts its sequences again. If the application resends an event after a restart because it never saw the acknowledgement, the broker treats it as a new record. The KafkaProducer documentation states that idempotence is only guaranteed within a single session.
- A new
sendcall. Each call gets the next sequence number. Two calls with the same payload are two records. - After the delivery timeout. When
delivery.timeout.msexpires the send fails with an error although the record may be in the log. Code that catches that error and sends again creates source 2. - Another partition or topic. Sequences are per partition. The same event sent to two partitions is not compared.
Consumer-side techniques
| Technique | How it works | Use it when | Watch out for |
|---|---|---|---|
| Natural idempotence | The operation sets a value instead of changing it: status = PAID, or an upsert of the full row by primary key | The event carries the complete new state | Increments, appends and "send an email" are not naturally idempotent. An older event replayed later can overwrite newer state unless you also compare versions. |
| Unique business key | The row you insert has a unique constraint on a business identifier, such as payment(order_id) or ledger_entry(payment_id) | One event creates one row | The key must identify the business action, not the Kafka record. Topic, partition and offset do not survive a resend. |
| Processed-events table | Insert (consumer, event_id) into a table with a primary key, in the same transaction as the business change. If the insert finds the row, skip. | The effect is spread over several rows or tables, or is an increment | The table grows. Delete rows older than the longest replay you allow. |
| Conditional update or version check | UPDATE ... WHERE status = 'PENDING' or WHERE version = ?, then look at the affected row count | The event is a state transition of an existing aggregate | Zero rows updated needs a decision: already applied, or arrived out of order. |
All four rely on the producing service assigning a stable identifier once, when the business action happens, and carrying it in the event. An id generated inside the publishing code at send time is different on every resend and protects nothing.
Architecture
Drawing diagram…
Flowchart from an upstream caller through the Payment Service code, the Kafka producer client and the partition leader into the payments.captured log, then to a ledger consumer and its database. Arrows are numbered with the five duplicate sources. A box around the producer client and partition leader is labelled producer idempotence, and a box around the consumer and database is labelled consumer idempotency.
flowchart TB
UP["Upstream caller or webhook"] -->|"5 upstream duplicate"| APP["Payment Service code"]
APP -->|"2 second send call"| PR["Kafka producer client"]
PR -->|"1 automatic retry"| LD["Partition leader"]
LD -->|"PID and sequence check"| LOG["payments.captured log"]
LOG -->|"3 redelivery after crash or rebalance"| CON["Ledger consumer"]
LOG -->|"4 replay after offset reset"| CON
CON -->|"event id checked in the same transaction"| DB["Ledger database"]
subgraph PI["Producer idempotence covers this"]
PR
LD
end
subgraph CI["Consumer idempotency covers this"]
CON
DB
end- Source 1 is the only one that happens between the producer client and the leader, which is the only place the producer id and sequence number are compared.
- Sources 2 and 5 put two separate records in the log. From then on they are indistinguishable from two real events unless the payload carries a business identifier.
- Sources 3 and 4 read a single record more than once. The log is correct and the repeat is created on the read side.
- The consumer's check sits after every source, so it is the one place that catches all five. That is the reason to build it even when the producer side is configured well.
Message flow
Drawing diagram…
Sequence diagram. The producer sends a batch with producer id 7 and sequence 12, the acknowledgement is lost, and the retry with the same sequence is acknowledged without being appended again. Later the ledger consumer reads offset 88, inserts the event id and the ledger entry in one database transaction, and crashes before committing the offset. After restart it reads offset 88 again, the insert of the event id affects zero rows, and it skips the ledger entry and commits offset 89.
sequenceDiagram participant P as Payment producer participant L as Partition leader participant C as Ledger consumer participant DB as Ledger database P->>L: batch PID 7 seq 12 payment 9001 L-->>P: ack lost on the network P->>L: retry PID 7 seq 12 L-->>P: ack, already appended, not written again Note over P,L: producer idempotence ends here C->>L: poll L-->>C: offset 88 payment 9001 C->>DB: begin, insert event id, insert ledger entry, commit Note over C: crash before the offset commit C->>L: poll after restart L-->>C: offset 88 payment 9001 again C->>DB: begin, insert event id DB-->>C: 0 rows, id already present Note over C,DB: skip the entry, then commit offset 89
- The first half is source 1. The leader recognises producer id 7, sequence 12 as a batch it already has and does not append it, so the log holds payment 9001 once.
- The second half is source 3. The log is fine. The consumer did the work and died before the group's offset moved, so the same offset is delivered again.
- The event id row and the ledger entry were committed together, so the second attempt finds the id and stops before touching the ledger.
- Take away either mechanism and one of the two duplicates gets through. Neither mechanism can do the other's job.
In a real system
A typical payments platform has a Payment Service that captures card payments and publishes PaymentCaptured to the topic payments.captured, keyed by paymentId, with 12 partitions. A Ledger Service in consumer group ledger-service posts a double-entry booking for each capture and increases the merchant's pending balance.
| What happens | Source | What stops the double booking |
|---|---|---|
| A broker restarts mid-request and the producer resends the batch | 1 Producer retry | Producer idempotence. The leader does not append the batch twice. |
| The Payment Service times out waiting for the send, its caller retries the capture request, and the event is published again | 2 Application resend | The eventId was created with the payment row, so both records carry the same id and the ledger skips the second. |
| A ledger pod is killed after its database commit and before the offset commit | 3 Redelivery | The processed-events row committed with the booking. |
| Finance asks for last Tuesday to be reprocessed and the group's offsets are reset | 4 Replay | The same row, provided it has not been purged yet. |
| The card provider delivers its capture webhook twice | 5 Upstream duplicate | A unique constraint on the provider's reference in the Payment Service, before anything is published. |
The balance update is the dangerous part. pending_balance = pending_balance + amount is not naturally idempotent, so it must run inside the transaction that inserts the event id. The booking rows could rely on a unique key on payment_id alone, but the increment could not.
Idempotency keys
Simulation, not a real clusterThe animated Build lesson shows the same idea at an HTTP API: a key stored with the result so a retried request returns the first outcome.
Step 1 · The pattern
One tap, one charge
A shopper taps Pay. The app calls the payments API, the API charges the card, and a success response comes back. One order, one charge.
- payment request
- success
Java and Spring Boot
A ledger consumer that is safe against all five sources. The event id is inserted and the ledger is changed in one JDBC transaction, and the Kafka offset is committed afterwards by the listener container.
-- One row per event this consumer has applied.CREATE TABLE processed_events ( consumer VARCHAR(100) NOT NULL, event_id UUID NOT NULL, processed_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (consumer, event_id));-- Second line of defence: one booking per payment, whatever the code does.ALTER TABLE ledger_entries ADD CONSTRAINT uq_ledger_entries_payment UNIQUE (payment_id);-- Used by the nightly purge.CREATE INDEX idx_processed_events_processed_at ON processed_events (processed_at);Why it is written this way
- The primary key is what makes the check safe under concurrency. Two transactions inserting the same id cannot both commit, so there is no read-then-write race.
consumeris part of the key so several consumers can share the table, each remembering events independently.- The unique constraint on
payment_idis a business rule in its own right and still holds after oldprocessed_eventsrows are purged. - Purge by
processed_at, keeping rows for longer than the topic's retention plus any replay window you support.
package com.example.ledger;import java.util.UUID;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;@Componentpublic class PaymentCapturedListener { private static final Logger log = LoggerFactory.getLogger(PaymentCapturedListener.class); private static final String CONSUMER = "ledger-service"; private final JdbcTemplate jdbc; public PaymentCapturedListener(JdbcTemplate jdbc) { this.jdbc = jdbc; } public record PaymentCaptured(UUID eventId, String paymentId, String merchantId, long amountMinor, String currency) {} @KafkaListener(topics = "payments.captured", groupId = "ledger-service") @Transactional // database transaction (DataSourceTransactionManager), not a Kafka transaction public void onPaymentCaptured(PaymentCaptured event) { int inserted = jdbc.update(""" INSERT INTO processed_events (consumer, event_id) VALUES (?, ?) ON CONFLICT (consumer, event_id) DO NOTHING """, CONSUMER, event.eventId()); if (inserted == 0) { log.info("Skipping duplicate event {} for payment {}", event.eventId(), event.paymentId()); return; } jdbc.update(""" INSERT INTO ledger_entries (payment_id, merchant_id, amount_minor, currency) VALUES (?, ?, ?, ?) """, event.paymentId(), event.merchantId(), event.amountMinor(), event.currency()); jdbc.update(""" UPDATE merchant_balances SET pending_minor = pending_minor + ? WHERE merchant_id = ? AND currency = ? """, event.amountMinor(), event.merchantId(), event.currency()); }}Why it is written this way
@Transactionalhere opens a database transaction. The three statements commit or roll back together, which is the whole point: the id and the effect can never disagree.ON CONFLICT DO NOTHINGreturns an update count of 0 for a duplicate instead of raising an error. In PostgreSQL a failed statement aborts the surrounding transaction, so catching a duplicate-key exception and continuing would not work.- Returning normally on a duplicate lets the container commit the offset, so the group moves past the record.
- The Kafka offset is committed after the database transaction. A crash in between causes one more delivery, which the first statement absorbs. This is at-least-once delivery with an exactly-once effect in the database. No Kafka transaction is involved.
- The balance update is an increment. It is only safe because it sits behind the
inserted == 0check in the same transaction.
package com.example.orders;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.stereotype.Repository;@Repositorypublic class OrderStatusUpdater { private final JdbcTemplate jdbc; public OrderStatusUpdater(JdbcTemplate jdbc) { this.jdbc = jdbc; } /** * Conditional update: moves the order to PAID only from PENDING. * Returns true if this call made the change, false if it had already * been made (a duplicate) or the order is in another state. */ public boolean markPaid(String orderId, String paymentId) { int updated = jdbc.update(""" UPDATE orders SET status = 'PAID', payment_id = ?, version = version + 1 WHERE order_id = ? AND status = 'PENDING' """, paymentId, orderId); return updated == 1; }}Why it is written this way
- No extra table is needed: the current state of the row is the memory. The second delivery matches zero rows.
- The
WHERE status = 'PENDING'condition and the update are one statement, so two consumers racing on the same order cannot both succeed. - A
falseresult is ambiguous. Read the row to tell "already PAID with this payment id" (a duplicate, safe to ignore) from "CANCELLED" (a real conflict that needs a refund).
Configuration
Only the producer half has configuration. Consumer idempotency is schema and code, and there is no setting that provides it. Defaults are from the Apache Kafka 4.4 documentation.
enable.idempotenceDefault true
- What it does
- The broker deduplicates this producer's retried batches using its producer id and per-partition sequence numbers.
- When to change it
- Set it to
trueexplicitly. Left implicit, a conflicting setting disables it silently. Set explicitly, the same conflict fails at startup with aConfigException.
acksDefault all
- What it does
- Acknowledgements required before a send counts as complete.
- When to change it
- Must be
allfor idempotence. Changing it to1for latency is the most common way idempotence gets switched off by accident.
retriesDefault 2147483647
- What it does
- How many times the client resends a batch after a transient error, bounded in time by
delivery.timeout.ms. - When to change it
- Leave it unset and control retry duration with
delivery.timeout.ms. Must be greater than 0 for idempotence.
max.in.flight.requests.per.connectionDefault 5
- What it does
- Unacknowledged requests the client may have outstanding on one connection.
- When to change it
- Keep at 5 or lower. With idempotence on, ordering is preserved for any allowed value, so the old advice to set 1 no longer applies.
delivery.timeout.msDefault 120000 (2 minutes)
- What it does
- Total time allowed for a record, including retries, before the send fails.
- When to change it
- When it expires the outcome is unknown. Do not resend blindly from application code. If you must resend, the event needs a stable id so the consumer can drop the copy.
producer.id.expiration.ms (broker)Default 86400000 (1 day)
- What it does
- How long a partition leader keeps the sequence state of a producer id that has stopped writing.
- When to change it
- Rarely changed. It matters only if it is set below
delivery.timeout.ms, because a retry could then arrive after the state it is compared against has gone.
Hands-on exercise
Stop the double charge
Reproduce a duplicate that producer idempotence cannot prevent, then remove its effect with a consumer-side check and prove it by repeating the same failure.
Steps · 0 of 5 done
Redelivery with and without an idempotent consumer
Simulation · not a real clusterThe consumer charges the card and crashes before the offset is committed. Toggle the idempotent consumer and compare the two runs.
What you should see
Without the idempotent consumer the redelivered record produces a second charge. This is source 3, and no producer setting touches it. With it, the record is still delivered twice, but the second delivery finds the event id and changes nothing. At-most-once also removes the duplicate, but only by being able to lose the payment entirely.
What can go wrong
Using topic, partition and offset as the dedupe key
- Symptom
- Redeliveries are handled correctly, but an event that the producer published twice is still applied twice.
- Cause
- The coordinates identify a record in the log, not a business event. A resend, an upstream duplicate or a copy in a retry topic has different coordinates.
- Fix
- Carry an
eventIdassigned once when the business action is recorded, and key the dedupe on that or on a natural business identifier.
Generating the event id at send time
- Symptom
- The dedupe table fills up, yet duplicates from application retries still get through.
- Cause
UUID.randomUUID()is called inside the publishing method, so every resend of the same business action carries a fresh id.- Fix
- Create the id in the same database transaction as the business change, store it, and publish the stored value. A transactional outbox does this by construction.
Check-then-act outside one transaction
- Symptom
- Rare double bookings under load or right after a rebalance, although the code does check for the event id.
- Cause
- The code runs
SELECTto see whether the id exists and inserts later, or keeps the id in a cache separate from the database. After a rebalance two consumers can briefly work on the same record and both pass the check. - Fix
- Let a primary key or unique constraint decide, with the insert in the same transaction as the business change.
Idempotence switched off by a tuning change
- Symptom
- Duplicate records with adjacent offsets appear in a topic after broker restarts, sometimes out of order.
- Cause
- Someone set
acks=1orretries=0.enable.idempotencewas never set explicitly, so the client disabled it without an error. - Fix
- Set
enable.idempotence=trueexplicitly so a conflicting setting fails at startup, and check theProducerConfig valuesblock logged when the producer starts.
Purging dedupe rows sooner than a replay can reach
- Symptom
- A deliberate reprocessing of last month's events applies some of them a second time.
- Cause
processed_eventsrows are deleted after 7 days while the topic retains 30 days of records.- Fix
- Keep dedupe rows for at least the topic retention plus the replay window, or rely on a permanent unique business key for effects that must never repeat.
How to troubleshoot
A business effect happened twice and you need to know which source it was
Check
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments.captured --partition 4 --offset 5100 --max-messages 50 --formatter-property print.offset=true --formatter-property print.key=true
- How to read it
- (On clients before Kafka 4.2 the option is
--property.) If the event is in the log once, it was read twice: source 3 or 4. If it is there twice with the same event id, the application or its upstream sent it twice: source 2 or 5. Twice with different ids means the id is generated at send time.
Suspected redelivery after a rebalance
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ledger-service, together with the consumer logs around the time of the duplicate
- How to read it
- Look for partitions being revoked and assigned, or a commit failing because the member was no longer in the group. Records handled after the last successful commit are delivered again to the new owner.
Not sure whether idempotence is really enabled on a producer
Check
The ProducerConfig values block that the client logs at INFO level on startup: enable.idempotence, acks, retries, max.in.flight.requests.per.connection
- How to read it
enable.idempotence = falsetogether withacks = 1orretries = 0shows that a conflicting setting disabled it.
The producer logs OutOfOrderSequenceException
Check
Producer application logs, and whether that partition recently lost acknowledged data, for example through an unclean leader election
- How to read it
- The broker saw a gap in this producer's sequence numbers, so something it expected is missing. With idempotence only, the documentation recommends closing the producer and creating a new one to keep ordering.
Duplicates are being skipped and you want to know how often
Check
A counter in the handler's duplicate branch, for example ledger.events.duplicate, split by topic
- How to read it
- A small steady rate around deployments and rebalances is normal at-least-once behaviour. A sudden rise points at a rebalance loop, an upstream resend storm or a replay.
Key takeaways
- Duplicates come from five places: producer retries, application resends, consumer redelivery, replays and upstream duplicates.
- Producer idempotence removes only the first, by comparing producer id and sequence number at the partition leader, within one producer session and one partition.
- Consumer idempotency is application logic. No Kafka setting provides it, and it is the only defence that catches all five sources.
- The record of "already applied" must commit in the same database transaction as the effect, enforced by a primary key or unique constraint.
- Identify events by an id assigned once with the business action. Offsets and send-time ids do not survive a resend.
- Pick the lightest technique that fits: a natural overwrite, a unique business key, a conditional update, or a processed-events table for increments and multi-row effects.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
An Order Service calls kafkaTemplate.send(...) and waits up to 2 seconds for the result. On a timeout, its HTTP client upstream retries the whole "place order" request, and the service publishes OrderPlaced again with a freshly generated eventId. The producer has enable.idempotence=true. Payments are occasionally taken twice.
Which change addresses the cause?
Scenario 2 of 3
A stock consumer handles OrderPlaced with UPDATE stock SET available = available - :qty WHERE sku = :sku. After a rebalance during a deploy, 14 SKUs show less stock than the warehouse count. The topic contains each order exactly once.
What is the most direct fix?
Scenario 3 of 3
A ledger handler first checks Redis for the event id, then writes the booking to PostgreSQL, then sets the id in Redis with a 24 hour expiry. In a month it has produced 3 double bookings and 1 missing booking.
What explains both kinds of error?
References
References: KafkaProducer Javadoc: the idempotent producer, application-level re-sends, single-session guarantee ↗ · Apache Kafka documentation: Producer configs (enable.idempotence, acks, retries, max.in.flight.requests.per.connection) ↗ · KIP-98: Exactly Once Delivery and Transactional Messaging (idempotent producer guarantees) ↗ · Apache Kafka documentation: Design, Message Delivery Semantics ↗ · Apache Kafka documentation: Broker configs (producer.id.expiration.ms) ↗ · Spring for Apache Kafka reference: Message listener containers, committing offsets ↗ · Report a problem