Skip to content

Path 7 · lesson 7 of 8

Advanced30 minApache Kafka 4.x (configuration reference read for 4.1) · Spring for Apache Kafka 3.3 on Spring Boot 3.x · Java 21 · PostgreSQL

The idempotent consumer pattern

Make a consumer safe to run twice on the same event: pick the right idempotency key, record it in the same transaction as the effect, clean the table up, and cover external calls.

What you will learn

  • Choose an idempotency key for a consumer (event id, business key, or topic-partition-offset) and name the situation in which each one fails
  • Implement insert-first deduplication with a unique constraint, in the same database transaction as the business effect
  • Handle the constraint violation correctly on PostgreSQL and on databases without an insert-or-ignore statement
  • Set a retention for the dedup table from the windows in which a duplicate can still arrive, and clean it up without long locks
  • Explain what happens when two consumers process the same record during a rebalance, and why the constraint settles it
  • Extend the pattern to a call to an external API that cannot join the transaction

Before you start

The concept

A consumer is idempotent when processing the same message twice leaves the system in the same state as processing it once. The delivery path explains where duplicates come from and walks through one complete payment flow. This lesson treats the solution as a reusable pattern with four parts, each of which is a design decision:

  1. An idempotency key: the value that says "this is the same message as before".
  2. A record of what was processed: a table with a unique constraint on that key, or a business table whose own unique key does the same job.
  3. One transaction that contains both the record and the business effect.
  4. A lifetime for the records, so that the table does not grow for ever and is not emptied too early.

The running example is the Inventory Service. It consumes payments.events and, for each PaymentCaptured, takes the ordered quantity out of stock. Removing 2 units is a delta. Applied twice it removes 4. Unlike setting a status, it cannot be made safe by simply writing it again, so the consumer has to remember what it has already applied.

Advantages

  • At-least-once delivery becomes safe: redelivery, replay and upstream duplicates all turn into no-ops
  • Operations get simpler: offsets can be reset and dead letters replayed without fear
  • It needs only a local database transaction, no distributed transaction and no Kafka transaction
  • The unique constraint also settles the race between two instances during a rebalance
  • The dedup table is an audit trail of what was processed and when

Disadvantages

  • One extra insert per message, and one more table to size and clean
  • It depends on the producer supplying a stable identity: a bad key defeats the whole pattern
  • It only covers effects inside the transaction. An email or a card charge needs additional design.
  • Retention is a real trade-off: too short lets old duplicates through, too long grows the table
  • It removes repeats, not reordering: an old event arriving late still needs a version check

Idempotency keys

Simulation, not a real cluster

The animated lesson shows the same idea for an HTTP API: a repeated request with the same key returns the first result.

Step 1 · The pattern

One tap, one charge

A shopper taps Pay. The app calls the payments API, the API charges the card, and a success response comes back. One order, one charge.

  • payment request
  • success
Open the full animated lesson

Why it exists

A Kafka consumer's position is committed separately from its work. With the usual at-least-once setup the consumer processes a record and commits the offset afterwards. Those are two writes to two systems. If the process stops between them, the group resumes from the last committed offset and the record is delivered again. No ordering of the two steps removes this: committing first loses the record instead.

The Inventory Service without the pattern: PaymentCaptured for order o-1001 arrives, the service runs UPDATE stock SET available = available - 2, commits, and is killed by a deployment before the offset commit. The new instance receives the same record and removes 2 more. Stock is now 2 lower than reality, and nothing failed, so nothing was logged. The shop shows "sold out" for units sitting on the shelf.

Source of the second deliverySame topic-partition-offset?Same event id?
Crash after processing, before the offset commitYesYes
Rebalance while a record is in flightYesYes
Offsets reset for a replayYesYes
Record forwarded to a retry topic and read againNoYes
Dead letter record replayed into the main topicNoYes
Outbox relay or application published the event twiceNoYes, if the id was stored with the business fact
Two different events asking for the same business actionNoNo
What is the same between the two copies decides which idempotency key can recognise them.

Attempts to avoid the problem instead of solving it do not hold. Exactly-once delivery to a database does not exist. Checking "did I do this?" with a query before acting has a race. Keeping processed ids in memory or in a separate cache is lost on restart, or commits separately from the effect. What works is to make the second execution harmless, and to let the database, which already has transactions and unique constraints, be the judge.

How it works inside

Choosing the idempotency key

KeyWhat it identifiesWorks forBreaks when
Event id (a UUID in a header or the payload)One business fact, as recorded by the producerEvery source in the table above except the last row. The general-purpose choice.The producer generates the id at send time, so a resend gets a new one. Or two producers use overlapping ids: scope the key by consumer and, if needed, by source.
Business key (order id, payment id, order id plus line number)The business action itself"One reservation per order", "one payment per order". Also catches two different events that ask for the same thing.The same key legitimately has several events (an order is changed three times). Then the key needs the event type or a version added to it.
Topic, partition, offsetOne record in one logRedelivery of that exact record: crash, rebalance, offset reset. Available even when the producer sends no id.Any copy that is a new record: a resend, a retry topic, a dead letter replay, a mirrored topic in another cluster. If a topic is deleted and recreated, offsets restart and new events look already processed.

Prefer a business key when one exists, because it expresses the rule you care about and needs no extra table: UNIQUE (order_id, sku) on the reservation table is the dedup. Use the event id with a processed-message table when the effect has no natural unique row, which is typical for deltas and for effects spread over several tables. Treat topic-partition-offset as a fallback for sources you do not control, and be explicit that it only covers redelivery.

Whatever the key, it must be created once, where the business fact is created, and stored with it. In the order system the Payment Service writes the event id into its outbox row in the transaction that captures the payment. Every publication of that row carries the same id.

Insert first, and let the constraint decide

The processed-message table has a primary key on (consumer, message_id). The consumer's transaction starts by inserting its key. The database guarantees that only one transaction can ever commit that row. If the insert succeeds, this is the first time: do the work in the same transaction. If the row exists, this is a repeat: do nothing and let the offset advance.

  1. BEGIN
  2. INSERT INTO processed_message (consumer, message_id) VALUES (...) ON CONFLICT DO NOTHING
  3. If 0 rows were inserted: COMMIT and return. The message was handled before.
  4. Otherwise apply the business effect: update stock, insert the reservation, insert any outgoing event into the outbox.
  5. COMMIT. The dedup row and the effect become durable together.
  6. The listener container commits the Kafka offset after the method returns.

Inserting first matters. The alternative, SELECT to see whether the key exists and insert at the end, lets two transactions both see "not there" and both do the work. With insert-first there is no check that can go stale: the unique index is the check.

Why the same transaction is not optional

Where the dedup record is writtenCrash between the two writes leavesResult
Before the effect, separate transactionKey recorded, effect missingThe redelivery is skipped. The stock is never reduced: a lost message.
After the effect, separate transactionEffect applied, key missingThe redelivery applies the effect again: the duplicate you were trying to stop.
In a cache or another databaseEither of the above, depending on orderA dual write with the same two failure modes.
Same transaction as the effectBoth or neitherRedelivery either does the work for the first time or skips it. No third state.

Two consumers at once

During a rebalance, an instance that has lost a partition may still be in the middle of a record while the new owner starts the same record. This happens when a consumer exceeds max.poll.interval.ms, or is paused for a long garbage collection and misses its session timeout. Keying by order id puts an order's events in one partition, but it does not stop two processes from briefly working on that partition.

Insert-first resolves it in the database. Both transactions insert the same key. PostgreSQL makes the second insert wait until the first transaction ends. If the first commits, the second sees the conflict, inserts nothing and skips. If the first rolls back, the second insert succeeds and that instance does the work. Exactly one effect is committed. The old owner's later offset commit is rejected by the group coordinator because it no longer owns the partition, which is harmless: the new owner commits the offset.

Retention and cleanup of the dedup table

A dedup row is needed for as long as a duplicate of that message can still arrive. Add up the windows: the retention of the source topic (a replay can reach back that far), the retention of retry and dead letter topics (a replay from there arrives later still), and any margin for manual reprocessing. With 7 days on the topic and 14 days on the dead letter topic, 30 days of dedup rows is a defensible setting. Deleting after 7 days while dead letters live for 14 means a dead letter replayed in the second week is applied again.

  • Delete in small batches on a schedule, using an index on processed_at. One large delete holds locks for a long time and bloats the table.
  • Or partition the table by day and drop whole partitions, which is nearly free. The unique key must then include the partition column, so a duplicate arriving on a different day is only caught if you look across partitions. Know this before choosing it.
  • Size it: one row per message per consumer. At 400 messages per second that is about 35 million rows a day. A UUID key and a timestamp are small, but the index still has to fit in memory to stay fast.
  • Effects that must never repeat should not depend on a table that is cleaned. Protect them with a permanent unique business key, such as UNIQUE (order_id) on payments.

Effects outside the database

A call to an email provider or a payment provider cannot be rolled back with your transaction. Insert-first alone gives the wrong result in both orders: record first and crash, and the email is never sent. Call first and crash, and it is sent again. Split the work so that each step can be repeated:

  1. Claim: in a transaction, insert a row for the delivery with state PENDING, keyed by the idempotency key. If it already exists, read its state.
  2. Call: if the state is not final, call the provider and pass the same key as the provider's idempotency key. A provider that supports this returns the original result for a repeated key instead of acting again.
  3. Complete: in a transaction, move the row to SENT, conditional on it still being PENDING.

A redelivery at any point resumes: a SENT row means nothing to do, and a PENDING row means the call is repeated with the same key. If the provider offers no idempotency key, the call is at-least-once and the honest outcome is "rarely twice": acceptable for a shipping notification, not for a charge. Check how long the provider remembers keys. Stripe's API documentation, for example, says keys can be removed once they are at least 24 hours old, so a retry after that is a new request.

Failure scenarios

Crash pointResulting stateRecovery
After the dedup insert, before the business updateTransaction not committed: neither row existsRedelivered and processed normally, as if for the first time.
After the database commit, before the offset commitDedup row and effect committed, offset not advancedRedelivered. The insert finds the key, the work is skipped, and the offset is committed this time.
Business update fails (deadlock, constraint)Transaction rolled back, including the dedup rowThe exception goes to the error handler, which retries. The key is free, so the retry is not mistaken for a duplicate.
Rebalance during processing, two instances on one recordTwo transactions inserting the same keyThe second waits for the first. One commits the effect, the other skips or takes over after a rollback.
Database unavailableNothing committed, every record failsOffsets do not advance past the failure. When the database returns, processing resumes with no duplicates and no gaps.
External call succeeded, crash before the completion updateDelivery row still PENDING, email already sentRedelivered. The call is repeated with the same idempotency key and the provider returns the first result, then the row is completed.
Dedup rows deleted, then an old dead letter is replayedNo record that the message was processedThe effect is applied a second time. Prevented only by retention that covers the replay window, or by a permanent business key.
Crash at each point of the idempotent consumer, what is left behind, and how it recovers.

Architecture

Drawing diagram…

Flowchart: a record from payments.events reaches the inventory listener, which begins a database transaction and inserts the message id into processed_message with on conflict do nothing. If no row was inserted the work is skipped. If a row was inserted the listener updates stock and inserts a reservation and an outbox row. Either way the transaction commits and the container then commits the Kafka offset. The three tables live in the inventory database, and a cleanup job deletes old processed_message rows in batches.

flowchart TD
  T["payments.events, key orderId, header id"] --> L["Inventory listener, group inventory-service"]
  L --> B["BEGIN transaction"]
  B --> I["INSERT processed_message, ON CONFLICT DO NOTHING"]
  I --> Q{"Row inserted?"}
  Q -->|"no, seen before"| S["Skip the work"]
  Q -->|"yes, first time"| W["UPDATE stock, INSERT reservation and outbox row"]
  W --> C["COMMIT, dedup row and effect together"]
  S --> C2["COMMIT, nothing changed"]
  C --> O["Container commits the Kafka offset"]
  C2 --> O
  subgraph DB["Inventory database"]
    PM["processed_message"]
    ST["stock"]
    RS["stock_reservation"]
  end
  I -.-> PM
  W -.-> ST
  W -.-> RS
  CL["Cleanup job, batched delete"] -.-> PM
The dedup insert and the business writes share one transaction. The offset commit comes after it and may be repeated safely.
  • The decision diamond is answered by the database, not by application code: the affected row count of the insert is the result of a unique index check.
  • Both branches end in a commit and then an offset commit. A duplicate is a normal, successful outcome, not an error path.
  • Everything the consumer changes is inside the box labelled Inventory database. That is what makes one local transaction sufficient.
  • The cleanup job is part of the pattern. A dedup table without one is a slow outage.

Message flow

Drawing diagram…

Sequence diagram: instance A receives a payment captured event with id e7, inserts the id into processed_message and updates stock, then pauses for longer than the max poll interval. The group rebalances and instance B receives the same record. B's insert of e7 waits for A's transaction. A commits, B's insert then reports zero rows, and B commits without changing anything and commits the offset. A's own offset commit is rejected because it no longer owns the partition.

sequenceDiagram
  participant K as payments.events
  participant A as Inventory instance A
  participant B as Inventory instance B
  participant D as Inventory DB
  K->>A: PaymentCaptured e7 for order o-1001
  A->>D: BEGIN, insert e7 into processed_message
  D-->>A: 1 row inserted
  A->>D: UPDATE stock minus 2, insert reservation
  Note over A: long pause, max.poll.interval.ms exceeded
  Note over K,B: rebalance, partition moves to B
  K->>B: PaymentCaptured e7 redelivered
  B->>D: BEGIN, insert e7 into processed_message
  Note over B,D: insert waits for the transaction of A
  A->>D: COMMIT
  D-->>B: 0 rows inserted
  B->>D: COMMIT, nothing changed
  B->>K: commit offset
  A->>K: commit offset
  K-->>A: rejected, A no longer owns the partition
Two instances process the same record during a rebalance. The unique key lets exactly one of them apply the stock change.
  1. Instance B's insert blocks on the uncommitted row of instance A. That wait is the database serialising the two attempts.
  2. Had instance A crashed instead of committing, its transaction would have rolled back, B's insert would have succeeded, and B would have applied the change. Either way stock is reduced once.
  3. The rejected offset commit of instance A is expected. The effect is already safe in the database, and B's commit moves the group forward.

In a real system

In a typical order system each consuming service applies the pattern with the key that fits its effect. payments.events has 12 partitions and the order id as key. Events carry an id header set from the Payment Service's outbox row.

ConsumerEffectIdempotency keyWhere it is enforcedRetention
Payment Service on orders.eventsCreate one payment per orderBusiness key: order idUNIQUE (order_id) on paymentPermanent
Inventory Service on payments.eventsDecrease stock, create reservationEvent idprocessed_message primary key, plus UNIQUE (order_id, sku) on reservations30 days, reservations permanent
Notification Service on orders.eventsSend email and SMSEvent id plus channelnotification_delivery primary key, passed to the provider as its idempotency key30 days
Clickstream importer on a partner's topicAppend page viewsTopic, partition, offsetUnique key on the three columnsTopic retention plus a margin

The Inventory Service uses two layers on purpose. The processed-message table catches every repeat of the same event, including deltas. The unique key on the reservation catches the rarer case of two different events for the same order line, and it stays in force after the dedup rows have been cleaned up.

When to use it

  • Always, for a consumer whose effect is not naturally idempotent: increments, decrements, appends, sending something.
  • For every consumer downstream of an outbox, a retry topic or a dead letter replay, because all three produce duplicates by design.
  • When the effect spans several tables, so that no single unique row represents "done".
  • When you want offset resets and replays to be routine operations and not incidents.

When a dedup table is not needed

  • The effect is an upsert of full state by primary key with a version check. Running it twice already gives the same row. See the projection in CQRS.
  • A unique business key already expresses the rule. Adding a processed-message table on top only adds an insert.
  • The work is read-process-write entirely inside Kafka, from topic to topic. Use Kafka transactions, which make the output and the offset commit atomic.
  • The data tolerates occasional duplicates and the volume is very high, such as metrics samples. Paying an indexed insert per record may cost more than the duplicates do.
  • The producer cannot supply a stable identity and you cannot derive a business key. Then fix the producer first: a dedup table keyed on something unstable protects nothing.

Java and Spring Boot

The Inventory Service on Spring Boot 3 with JdbcTemplate and PostgreSQL: the schema, a small reusable component that runs work at most once per message, the listener that uses it, the cleanup job, and the Notification Service's variant for an external API.

inventory-service/src/main/resources/db/migration/V3__idempotent_consumer.sqlSQL
-- Inventory Service database.CREATE TABLE processed_message (    consumer     varchar(64)  NOT NULL,              -- consumer group: one table can serve several listeners    message_id   varchar(128) NOT NULL,              -- the idempotency key: the event id from the header    processed_at timestamptz  NOT NULL DEFAULT now(),    PRIMARY KEY (consumer, message_id));CREATE INDEX processed_message_age ON processed_message (processed_at);   -- for the cleanup jobCREATE TABLE stock (    sku        varchar(64) PRIMARY KEY,    available  integer     NOT NULL CHECK (available >= 0));CREATE TABLE stock_reservation (    order_id    varchar(36) NOT NULL,    sku         varchar(64) NOT NULL,    quantity    integer     NOT NULL,    reserved_at timestamptz NOT NULL DEFAULT now(),    PRIMARY KEY (order_id, sku)                      -- business key: one reservation per order line, for ever);-- One transaction per message:--   BEGIN;--   INSERT INTO processed_message (consumer, message_id) VALUES (?, ?) ON CONFLICT DO NOTHING;--     -> 0 rows: seen before. COMMIT and stop.--   UPDATE stock SET available = available - ? WHERE sku = ? AND available >= ?;--     -> 0 rows: not enough stock. A business outcome, recorded and published, not retried.--   INSERT INTO stock_reservation (order_id, sku, quantity) VALUES (?, ?, ?);--   COMMIT;-- The Kafka offset is committed after this commit, by the listener container.-- Cleanup, small batches on a schedule. 30 days covers topic retention (7 days),-- dead letter retention (14 days) and a margin for manual replays.--   DELETE FROM processed_message--   WHERE (consumer, message_id) IN (--       SELECT consumer, message_id FROM processed_message--       WHERE processed_at < now() - interval '30 days'--       LIMIT 10000);-- Notification Service database: an external effect needs a state, not only a marker.CREATE TABLE notification_delivery (    event_id   varchar(128) NOT NULL,    channel    varchar(16)  NOT NULL,                -- EMAIL, SMS    status     varchar(16)  NOT NULL,                -- PENDING, SENT    created_at timestamptz  NOT NULL DEFAULT now(),    sent_at    timestamptz,    PRIMARY KEY (event_id, channel));

Why it is written this way

  • consumer is part of the key because two consumer groups may each need to process the same event once. Without it the second group would treat the first group's row as its own.
  • The reservation's primary key is a second, permanent guard. It outlives the cleanup of processed_message and also rejects a different event that tries to reserve the same order line.
  • available >= 0 and the conditional update make overselling impossible even if every other check failed.
  • notification_delivery has a status column because an external call has an in-between state: claimed but not yet confirmed.

Configuration

The pattern is application code, but several Kafka and Spring settings decide how often duplicates arrive and how long they can keep arriving. Kafka defaults are from the 4.1 configuration reference, Spring defaults from the 3.3 reference.

enable.auto.commitDefault true (Kafka client). Spring for Apache Kafka sets it to false unless you configure it.
What it does
Whether the client commits offsets periodically in the background, regardless of whether your code has finished processing.
When to change it
Leave it off in Spring so the container commits after the listener returns. Background commits can advance past a record whose database transaction later rolls back, which loses the message. No dedup table repairs that.
spring.kafka.listener.ack-modeDefault BATCH
What it does
When the container commits offsets. BATCH commits after all records of a poll have been processed, RECORD after each record.
When to change it
With BATCH, a crash mid-batch redelivers the records already processed in that batch. The pattern absorbs them. RECORD reduces duplicates at the cost of more commits.
max.poll.interval.msDefault 300000 (5 minutes)
What it does
Maximum time between polls before the consumer is considered failed and its partitions are reassigned.
When to change it
Exceeding it is the usual cause of two instances working on one record. Keep per-poll work well below it by lowering max.poll.records before raising this.
max.poll.recordsDefault 500
What it does
Maximum records returned by one poll.
When to change it
Lower it when each record includes a slow call, so a batch always finishes inside the poll interval.
retention.ms (source, retry and dead letter topics)Default 604800000 (7 days)
What it does
How long records stay available, and therefore how far back a replay can deliver an old message again.
When to change it
The dedup retention must be longer than the longest of these, plus the time a replay might be started after it.
idempotency.retention-daysDefault none (application setting, 30 in this lesson)
What it does
Age after which processed_message rows are deleted by the cleanup job.
When to change it
Raise it when a topic's retention is raised. Do not lower it to save space without checking every topic that can be replayed into the consumer.
isolation.levelDefault read_uncommitted
What it does
Whether the consumer also receives records from Kafka transactions that were aborted.
When to change it
Set read_committed when the upstream producer uses transactions. Otherwise the consumer processes events from aborted transactions, and no dedup key can tell that they never happened.

Hands-on exercise

Deliver the same event twice and keep the stock right

Produce a duplicate by crashing a consumer between its database commit and its offset commit, see the effect applied twice, then add the processed-message check and repeat the crash.

Steps · 0 of 5 done

Crash between processing and commit

Simulation · not a real cluster

At-least-once consumption with a crash after the work is done. Run it with the idempotent consumer off, then on, and compare the outcomes.

Take it further in the lab: Duplicate message processing

What can go wrong

Dedup record and effect in different transactions
Symptom
Stock is occasionally not reduced for a paid order, and the event's id is present in processed_message.
Cause
The dedup insert ran in its own transaction (auto-commit, REQUIRES_NEW, or a different data source) and committed. The business update then failed or the process died. The redelivery was skipped as a duplicate.
Fix
Run the insert and the business statements on the same connection inside one transaction, and test it by forcing an exception after the insert and checking that the row is gone.
Catching the duplicate-key error inside a PostgreSQL transaction
Symptom
Duplicates produce current transaction is aborted, commands ignored until end of transaction block, and the record is retried until it reaches the dead letter topic.
Cause
The code inserts, catches DuplicateKeyException and carries on in the same transaction. PostgreSQL has already marked that transaction as failed.
Fix
Use INSERT ... ON CONFLICT DO NOTHING and check the row count, or let the exception end the transaction and catch it outside.
Topic, partition and offset as the key
Symptom
Redeliveries are skipped correctly, but after a dead letter replay several orders have stock reduced twice.
Cause
The replayed record is a new record with a new offset, and often a different topic. Its coordinates were never seen before.
Fix
Key on an event id assigned by the producer, or on a business key. Keep coordinates only as a fallback for sources that provide neither.
An event id generated when the message is sent
Symptom
The dedup table fills as expected, yet upstream republications are still applied twice.
Cause
The producer calls UUID.randomUUID() in its publishing code. Each publication of the same business fact gets a fresh id.
Fix
Create the id in the transaction that records the business fact and store it, for example in the outbox row, so that every publication carries the same value.
Dedup rows deleted before the last possible duplicate
Symptom
A replay of 10-day-old dead letters reduces stock again for orders that were fulfilled long ago.
Cause
The cleanup removes rows after 7 days, while the dead letter topic keeps records for 14.
Fix
Set the retention from the longest replay window, and guard effects that must never repeat with a permanent unique business key.
Marking the message before an external call
Symptom
Some customers never receive their confirmation email, and the delivery row exists.
Cause
The consumer recorded the message as processed, then crashed before the provider call. The redelivery saw the record and skipped the send.
Fix
Use states: claim as PENDING, call with a stable idempotency key, then mark SENT. A redelivery that finds PENDING repeats the call.
A dedup table that is never cleaned
Symptom
Consumer throughput falls over months and lag appears at peak. The slowest statement is the insert into processed_message.
Cause
The table has grown to billions of rows and its primary key index no longer fits in memory, so each insert reads from disk.
Fix
Add the scheduled, batched delete with an index on the timestamp, or partition the table by time and drop old partitions.

How to troubleshoot

Suspect that an effect was applied twice for one order

Check

SELECT order_id, sku, count(*) FROM stock_reservation GROUP BY order_id, sku HAVING count(*) > 1; and SELECT * FROM processed_message WHERE message_id = '<event id>';
How to read it
With the unique key in place the first query returns nothing. One processed_message row and a doubled effect means the effect ran outside the dedup transaction. No row at all means the key was cleaned up or never written.
Want to know how often duplicates arrive

Check

The application metric `inventory.duplicate.messages` (a counter incremented when the dedup insert affects 0 rows)
How to read it
A low steady rate after deployments and rebalances is normal. A rate that follows the incoming rate means something upstream publishes everything twice, or the group is rebalancing continuously.
Duplicates arrive in bursts

Check

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group inventory-service --state
How to read it
A state that keeps changing away from Stable means repeated rebalances, each of which redelivers uncommitted records. Look in the application log for a commit failure that mentions max.poll.interval.ms.
Two records with the same event id on the topic

Check

kafka-console-consumer.sh --bootstrap-server broker:9092 --topic payments.events --from-beginning --property print.key=true --property print.headers=true --property print.offset=true
How to read it
The same id header at two offsets is an upstream republication, which the pattern absorbs. Different ids for what is clearly one business action means the producer creates ids at send time. Recent 4.x releases deprecate --property here in favour of --formatter-property.
The dedup table is large or the cleanup is behind

Check

SELECT count(*), min(processed_at) FROM processed_message WHERE consumer = 'inventory-service';
How to read it
The oldest row should be about as old as the configured retention. Much older means the cleanup job is not running or cannot keep up with the insert rate.

Key takeaways

  • At-least-once delivery plus an idempotent consumer is how effects happen once. Kafka does not deduplicate for consumers.
  • The idempotency key must identify the business fact and be created once at the source. Offsets identify a record, not a fact.
  • Insert the key first, under a unique constraint, in the same transaction as the effect. The constraint also settles the race between two instances.
  • On PostgreSQL use ON CONFLICT DO NOTHING. Elsewhere, catch the violation outside the transaction.
  • Keep dedup rows longer than any topic that can be replayed into the consumer, and delete them in batches.
  • For an external call, use claim, call with a stable idempotency key, complete. Marking first loses the effect, calling first repeats it.
  • Prefer a permanent unique business key for anything that must never happen twice.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

An inventory consumer deduplicates on (topic, partition, offset) stored in the same transaction as the stock update. It has run cleanly for months. After a bug, 1,200 records land in the dead letter topic. The fix is deployed, the dead letters are republished to the main topic, and 85 orders end up with stock reduced twice.

Why did the dedup not catch those 85?

Scenario 2 of 3

A listener on PostgreSQL does this inside one @Transactional method: insert the event id into processed_event; if DuplicateKeyException is thrown, catch it and return; otherwise update the balance. In production every duplicate ends with an error about an aborted transaction, is retried ten times and goes to the dead letter topic.

What is the smallest correct change?

Scenario 3 of 3

A notification consumer inserts (event_id, 'EMAIL') into a dedup table and then calls the email provider, all in one database transaction. The provider call takes 300 ms on average. During a provider slowdown the connection pool is exhausted, and afterwards 40 customers report two confirmation emails.

Which redesign addresses both problems?

References

References: Apache Kafka 4.1: design, message delivery semantics ↗ · Apache Kafka 4.1: consumer configuration reference (enable.auto.commit, isolation.level, max.poll.interval.ms, max.poll.records) ↗ · Apache Kafka 4.1: producer configuration reference (enable.idempotence and what it covers) ↗ · Spring for Apache Kafka 3.3 reference: message listener containers (AckMode, committing offsets) ↗ · PostgreSQL documentation: INSERT, the ON CONFLICT clause ↗ · PostgreSQL documentation: index uniqueness checks (a conflicting insert waits for the other transaction) ↗ · Stripe API reference: idempotent requests (an example of provider-side idempotency keys) ↗ · Chris Richardson, microservices.io: Idempotent consumer pattern ↗ · Report a problem