Path 7 · lesson 7 of 8
The idempotent consumer pattern
Make a consumer safe to run twice on the same event: pick the right idempotency key, record it in the same transaction as the effect, clean the table up, and cover external calls.
What you will learn
- Choose an idempotency key for a consumer (event id, business key, or topic-partition-offset) and name the situation in which each one fails
- Implement insert-first deduplication with a unique constraint, in the same database transaction as the business effect
- Handle the constraint violation correctly on PostgreSQL and on databases without an insert-or-ignore statement
- Set a retention for the dedup table from the windows in which a duplicate can still arrive, and clean it up without long locks
- Explain what happens when two consumers process the same record during a rebalance, and why the constraint settles it
- Extend the pattern to a call to an external API that cannot join the transaction
Before you start
The concept
A consumer is idempotent when processing the same message twice leaves the system in the same state as processing it once. The delivery path explains where duplicates come from and walks through one complete payment flow. This lesson treats the solution as a reusable pattern with four parts, each of which is a design decision:
- An idempotency key: the value that says "this is the same message as before".
- A record of what was processed: a table with a unique constraint on that key, or a business table whose own unique key does the same job.
- One transaction that contains both the record and the business effect.
- A lifetime for the records, so that the table does not grow for ever and is not emptied too early.
The running example is the Inventory Service. It consumes payments.events and, for each PaymentCaptured, takes the ordered quantity out of stock. Removing 2 units is a delta. Applied twice it removes 4. Unlike setting a status, it cannot be made safe by simply writing it again, so the consumer has to remember what it has already applied.
Advantages
- At-least-once delivery becomes safe: redelivery, replay and upstream duplicates all turn into no-ops
- Operations get simpler: offsets can be reset and dead letters replayed without fear
- It needs only a local database transaction, no distributed transaction and no Kafka transaction
- The unique constraint also settles the race between two instances during a rebalance
- The dedup table is an audit trail of what was processed and when
Disadvantages
- One extra insert per message, and one more table to size and clean
- It depends on the producer supplying a stable identity: a bad key defeats the whole pattern
- It only covers effects inside the transaction. An email or a card charge needs additional design.
- Retention is a real trade-off: too short lets old duplicates through, too long grows the table
- It removes repeats, not reordering: an old event arriving late still needs a version check
Idempotency keys
Simulation, not a real clusterThe animated lesson shows the same idea for an HTTP API: a repeated request with the same key returns the first result.
Step 1 · The pattern
One tap, one charge
A shopper taps Pay. The app calls the payments API, the API charges the card, and a success response comes back. One order, one charge.
- payment request
- success
Why it exists
A Kafka consumer's position is committed separately from its work. With the usual at-least-once setup the consumer processes a record and commits the offset afterwards. Those are two writes to two systems. If the process stops between them, the group resumes from the last committed offset and the record is delivered again. No ordering of the two steps removes this: committing first loses the record instead.
The Inventory Service without the pattern: PaymentCaptured for order o-1001 arrives, the service runs UPDATE stock SET available = available - 2, commits, and is killed by a deployment before the offset commit. The new instance receives the same record and removes 2 more. Stock is now 2 lower than reality, and nothing failed, so nothing was logged. The shop shows "sold out" for units sitting on the shelf.
| Source of the second delivery | Same topic-partition-offset? | Same event id? |
|---|---|---|
| Crash after processing, before the offset commit | Yes | Yes |
| Rebalance while a record is in flight | Yes | Yes |
| Offsets reset for a replay | Yes | Yes |
| Record forwarded to a retry topic and read again | No | Yes |
| Dead letter record replayed into the main topic | No | Yes |
| Outbox relay or application published the event twice | No | Yes, if the id was stored with the business fact |
| Two different events asking for the same business action | No | No |
Attempts to avoid the problem instead of solving it do not hold. Exactly-once delivery to a database does not exist. Checking "did I do this?" with a query before acting has a race. Keeping processed ids in memory or in a separate cache is lost on restart, or commits separately from the effect. What works is to make the second execution harmless, and to let the database, which already has transactions and unique constraints, be the judge.
How it works inside
Choosing the idempotency key
| Key | What it identifies | Works for | Breaks when |
|---|---|---|---|
| Event id (a UUID in a header or the payload) | One business fact, as recorded by the producer | Every source in the table above except the last row. The general-purpose choice. | The producer generates the id at send time, so a resend gets a new one. Or two producers use overlapping ids: scope the key by consumer and, if needed, by source. |
| Business key (order id, payment id, order id plus line number) | The business action itself | "One reservation per order", "one payment per order". Also catches two different events that ask for the same thing. | The same key legitimately has several events (an order is changed three times). Then the key needs the event type or a version added to it. |
| Topic, partition, offset | One record in one log | Redelivery of that exact record: crash, rebalance, offset reset. Available even when the producer sends no id. | Any copy that is a new record: a resend, a retry topic, a dead letter replay, a mirrored topic in another cluster. If a topic is deleted and recreated, offsets restart and new events look already processed. |
Prefer a business key when one exists, because it expresses the rule you care about and needs no extra table: UNIQUE (order_id, sku) on the reservation table is the dedup. Use the event id with a processed-message table when the effect has no natural unique row, which is typical for deltas and for effects spread over several tables. Treat topic-partition-offset as a fallback for sources you do not control, and be explicit that it only covers redelivery.
Whatever the key, it must be created once, where the business fact is created, and stored with it. In the order system the Payment Service writes the event id into its outbox row in the transaction that captures the payment. Every publication of that row carries the same id.
Insert first, and let the constraint decide
The processed-message table has a primary key on (consumer, message_id). The consumer's transaction starts by inserting its key. The database guarantees that only one transaction can ever commit that row. If the insert succeeds, this is the first time: do the work in the same transaction. If the row exists, this is a repeat: do nothing and let the offset advance.
BEGININSERT INTO processed_message (consumer, message_id) VALUES (...) ON CONFLICT DO NOTHING- If 0 rows were inserted:
COMMITand return. The message was handled before. - Otherwise apply the business effect: update stock, insert the reservation, insert any outgoing event into the outbox.
COMMIT. The dedup row and the effect become durable together.- The listener container commits the Kafka offset after the method returns.
Inserting first matters. The alternative, SELECT to see whether the key exists and insert at the end, lets two transactions both see "not there" and both do the work. With insert-first there is no check that can go stale: the unique index is the check.
Why the same transaction is not optional
| Where the dedup record is written | Crash between the two writes leaves | Result |
|---|---|---|
| Before the effect, separate transaction | Key recorded, effect missing | The redelivery is skipped. The stock is never reduced: a lost message. |
| After the effect, separate transaction | Effect applied, key missing | The redelivery applies the effect again: the duplicate you were trying to stop. |
| In a cache or another database | Either of the above, depending on order | A dual write with the same two failure modes. |
| Same transaction as the effect | Both or neither | Redelivery either does the work for the first time or skips it. No third state. |
Two consumers at once
During a rebalance, an instance that has lost a partition may still be in the middle of a record while the new owner starts the same record. This happens when a consumer exceeds max.poll.interval.ms, or is paused for a long garbage collection and misses its session timeout. Keying by order id puts an order's events in one partition, but it does not stop two processes from briefly working on that partition.
Insert-first resolves it in the database. Both transactions insert the same key. PostgreSQL makes the second insert wait until the first transaction ends. If the first commits, the second sees the conflict, inserts nothing and skips. If the first rolls back, the second insert succeeds and that instance does the work. Exactly one effect is committed. The old owner's later offset commit is rejected by the group coordinator because it no longer owns the partition, which is harmless: the new owner commits the offset.
Retention and cleanup of the dedup table
A dedup row is needed for as long as a duplicate of that message can still arrive. Add up the windows: the retention of the source topic (a replay can reach back that far), the retention of retry and dead letter topics (a replay from there arrives later still), and any margin for manual reprocessing. With 7 days on the topic and 14 days on the dead letter topic, 30 days of dedup rows is a defensible setting. Deleting after 7 days while dead letters live for 14 means a dead letter replayed in the second week is applied again.
- Delete in small batches on a schedule, using an index on
processed_at. One large delete holds locks for a long time and bloats the table. - Or partition the table by day and drop whole partitions, which is nearly free. The unique key must then include the partition column, so a duplicate arriving on a different day is only caught if you look across partitions. Know this before choosing it.
- Size it: one row per message per consumer. At 400 messages per second that is about 35 million rows a day. A UUID key and a timestamp are small, but the index still has to fit in memory to stay fast.
- Effects that must never repeat should not depend on a table that is cleaned. Protect them with a permanent unique business key, such as
UNIQUE (order_id)on payments.
Effects outside the database
A call to an email provider or a payment provider cannot be rolled back with your transaction. Insert-first alone gives the wrong result in both orders: record first and crash, and the email is never sent. Call first and crash, and it is sent again. Split the work so that each step can be repeated:
- Claim: in a transaction, insert a row for the delivery with state
PENDING, keyed by the idempotency key. If it already exists, read its state. - Call: if the state is not final, call the provider and pass the same key as the provider's idempotency key. A provider that supports this returns the original result for a repeated key instead of acting again.
- Complete: in a transaction, move the row to
SENT, conditional on it still beingPENDING.
A redelivery at any point resumes: a SENT row means nothing to do, and a PENDING row means the call is repeated with the same key. If the provider offers no idempotency key, the call is at-least-once and the honest outcome is "rarely twice": acceptable for a shipping notification, not for a charge. Check how long the provider remembers keys. Stripe's API documentation, for example, says keys can be removed once they are at least 24 hours old, so a retry after that is a new request.
Failure scenarios
| Crash point | Resulting state | Recovery |
|---|---|---|
| After the dedup insert, before the business update | Transaction not committed: neither row exists | Redelivered and processed normally, as if for the first time. |
| After the database commit, before the offset commit | Dedup row and effect committed, offset not advanced | Redelivered. The insert finds the key, the work is skipped, and the offset is committed this time. |
| Business update fails (deadlock, constraint) | Transaction rolled back, including the dedup row | The exception goes to the error handler, which retries. The key is free, so the retry is not mistaken for a duplicate. |
| Rebalance during processing, two instances on one record | Two transactions inserting the same key | The second waits for the first. One commits the effect, the other skips or takes over after a rollback. |
| Database unavailable | Nothing committed, every record fails | Offsets do not advance past the failure. When the database returns, processing resumes with no duplicates and no gaps. |
| External call succeeded, crash before the completion update | Delivery row still PENDING, email already sent | Redelivered. The call is repeated with the same idempotency key and the provider returns the first result, then the row is completed. |
| Dedup rows deleted, then an old dead letter is replayed | No record that the message was processed | The effect is applied a second time. Prevented only by retention that covers the replay window, or by a permanent business key. |
Architecture
Drawing diagram…
Flowchart: a record from payments.events reaches the inventory listener, which begins a database transaction and inserts the message id into processed_message with on conflict do nothing. If no row was inserted the work is skipped. If a row was inserted the listener updates stock and inserts a reservation and an outbox row. Either way the transaction commits and the container then commits the Kafka offset. The three tables live in the inventory database, and a cleanup job deletes old processed_message rows in batches.
flowchart TD
T["payments.events, key orderId, header id"] --> L["Inventory listener, group inventory-service"]
L --> B["BEGIN transaction"]
B --> I["INSERT processed_message, ON CONFLICT DO NOTHING"]
I --> Q{"Row inserted?"}
Q -->|"no, seen before"| S["Skip the work"]
Q -->|"yes, first time"| W["UPDATE stock, INSERT reservation and outbox row"]
W --> C["COMMIT, dedup row and effect together"]
S --> C2["COMMIT, nothing changed"]
C --> O["Container commits the Kafka offset"]
C2 --> O
subgraph DB["Inventory database"]
PM["processed_message"]
ST["stock"]
RS["stock_reservation"]
end
I -.-> PM
W -.-> ST
W -.-> RS
CL["Cleanup job, batched delete"] -.-> PM- The decision diamond is answered by the database, not by application code: the affected row count of the insert is the result of a unique index check.
- Both branches end in a commit and then an offset commit. A duplicate is a normal, successful outcome, not an error path.
- Everything the consumer changes is inside the box labelled Inventory database. That is what makes one local transaction sufficient.
- The cleanup job is part of the pattern. A dedup table without one is a slow outage.
Message flow
Drawing diagram…
Sequence diagram: instance A receives a payment captured event with id e7, inserts the id into processed_message and updates stock, then pauses for longer than the max poll interval. The group rebalances and instance B receives the same record. B's insert of e7 waits for A's transaction. A commits, B's insert then reports zero rows, and B commits without changing anything and commits the offset. A's own offset commit is rejected because it no longer owns the partition.
sequenceDiagram participant K as payments.events participant A as Inventory instance A participant B as Inventory instance B participant D as Inventory DB K->>A: PaymentCaptured e7 for order o-1001 A->>D: BEGIN, insert e7 into processed_message D-->>A: 1 row inserted A->>D: UPDATE stock minus 2, insert reservation Note over A: long pause, max.poll.interval.ms exceeded Note over K,B: rebalance, partition moves to B K->>B: PaymentCaptured e7 redelivered B->>D: BEGIN, insert e7 into processed_message Note over B,D: insert waits for the transaction of A A->>D: COMMIT D-->>B: 0 rows inserted B->>D: COMMIT, nothing changed B->>K: commit offset A->>K: commit offset K-->>A: rejected, A no longer owns the partition
- Instance B's insert blocks on the uncommitted row of instance A. That wait is the database serialising the two attempts.
- Had instance A crashed instead of committing, its transaction would have rolled back, B's insert would have succeeded, and B would have applied the change. Either way stock is reduced once.
- The rejected offset commit of instance A is expected. The effect is already safe in the database, and B's commit moves the group forward.
In a real system
In a typical order system each consuming service applies the pattern with the key that fits its effect. payments.events has 12 partitions and the order id as key. Events carry an id header set from the Payment Service's outbox row.
| Consumer | Effect | Idempotency key | Where it is enforced | Retention |
|---|---|---|---|---|
Payment Service on orders.events | Create one payment per order | Business key: order id | UNIQUE (order_id) on payment | Permanent |
Inventory Service on payments.events | Decrease stock, create reservation | Event id | processed_message primary key, plus UNIQUE (order_id, sku) on reservations | 30 days, reservations permanent |
Notification Service on orders.events | Send email and SMS | Event id plus channel | notification_delivery primary key, passed to the provider as its idempotency key | 30 days |
| Clickstream importer on a partner's topic | Append page views | Topic, partition, offset | Unique key on the three columns | Topic retention plus a margin |
The Inventory Service uses two layers on purpose. The processed-message table catches every repeat of the same event, including deltas. The unique key on the reservation catches the rarer case of two different events for the same order line, and it stays in force after the dedup rows have been cleaned up.
When to use it
- Always, for a consumer whose effect is not naturally idempotent: increments, decrements, appends, sending something.
- For every consumer downstream of an outbox, a retry topic or a dead letter replay, because all three produce duplicates by design.
- When the effect spans several tables, so that no single unique row represents "done".
- When you want offset resets and replays to be routine operations and not incidents.
When a dedup table is not needed
- The effect is an upsert of full state by primary key with a version check. Running it twice already gives the same row. See the projection in CQRS.
- A unique business key already expresses the rule. Adding a processed-message table on top only adds an insert.
- The work is read-process-write entirely inside Kafka, from topic to topic. Use Kafka transactions, which make the output and the offset commit atomic.
- The data tolerates occasional duplicates and the volume is very high, such as metrics samples. Paying an indexed insert per record may cost more than the duplicates do.
- The producer cannot supply a stable identity and you cannot derive a business key. Then fix the producer first: a dedup table keyed on something unstable protects nothing.
Java and Spring Boot
The Inventory Service on Spring Boot 3 with JdbcTemplate and PostgreSQL: the schema, a small reusable component that runs work at most once per message, the listener that uses it, the cleanup job, and the Notification Service's variant for an external API.
-- Inventory Service database.CREATE TABLE processed_message ( consumer varchar(64) NOT NULL, -- consumer group: one table can serve several listeners message_id varchar(128) NOT NULL, -- the idempotency key: the event id from the header processed_at timestamptz NOT NULL DEFAULT now(), PRIMARY KEY (consumer, message_id));CREATE INDEX processed_message_age ON processed_message (processed_at); -- for the cleanup jobCREATE TABLE stock ( sku varchar(64) PRIMARY KEY, available integer NOT NULL CHECK (available >= 0));CREATE TABLE stock_reservation ( order_id varchar(36) NOT NULL, sku varchar(64) NOT NULL, quantity integer NOT NULL, reserved_at timestamptz NOT NULL DEFAULT now(), PRIMARY KEY (order_id, sku) -- business key: one reservation per order line, for ever);-- One transaction per message:-- BEGIN;-- INSERT INTO processed_message (consumer, message_id) VALUES (?, ?) ON CONFLICT DO NOTHING;-- -> 0 rows: seen before. COMMIT and stop.-- UPDATE stock SET available = available - ? WHERE sku = ? AND available >= ?;-- -> 0 rows: not enough stock. A business outcome, recorded and published, not retried.-- INSERT INTO stock_reservation (order_id, sku, quantity) VALUES (?, ?, ?);-- COMMIT;-- The Kafka offset is committed after this commit, by the listener container.-- Cleanup, small batches on a schedule. 30 days covers topic retention (7 days),-- dead letter retention (14 days) and a margin for manual replays.-- DELETE FROM processed_message-- WHERE (consumer, message_id) IN (-- SELECT consumer, message_id FROM processed_message-- WHERE processed_at < now() - interval '30 days'-- LIMIT 10000);-- Notification Service database: an external effect needs a state, not only a marker.CREATE TABLE notification_delivery ( event_id varchar(128) NOT NULL, channel varchar(16) NOT NULL, -- EMAIL, SMS status varchar(16) NOT NULL, -- PENDING, SENT created_at timestamptz NOT NULL DEFAULT now(), sent_at timestamptz, PRIMARY KEY (event_id, channel));Why it is written this way
consumeris part of the key because two consumer groups may each need to process the same event once. Without it the second group would treat the first group's row as its own.- The reservation's primary key is a second, permanent guard. It outlives the cleanup of
processed_messageand also rejects a different event that tries to reserve the same order line. available >= 0and the conditional update make overselling impossible even if every other check failed.notification_deliveryhas a status column because an external call has an in-between state: claimed but not yet confirmed.
package com.example.inventory.idempotency;import org.springframework.dao.DuplicateKeyException;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.stereotype.Component;import org.springframework.transaction.support.TransactionTemplate;/** Runs a piece of work at most once per (consumer, messageId), atomically with the dedup record. */@Componentpublic class IdempotentProcessor { private final JdbcTemplate jdbc; private final TransactionTemplate tx; public IdempotentProcessor(JdbcTemplate jdbc, TransactionTemplate tx) { this.jdbc = jdbc; this.tx = tx; } /** * PostgreSQL version. Returns true if the work ran, false if the message was seen before. * If the work throws, the transaction rolls back and the dedup row disappears with it. */ public boolean runOnce(String consumer, String messageId, Runnable work) { Boolean ran = tx.execute(status -> { int inserted = jdbc.update(""" INSERT INTO processed_message (consumer, message_id) VALUES (?, ?) ON CONFLICT DO NOTHING""", consumer, messageId); if (inserted == 0) { return false; // duplicate: commit an empty transaction } work.run(); // same connection, same transaction return true; }); return Boolean.TRUE.equals(ran); } /** * For databases without an insert-or-ignore statement. The violation rolls the whole * transaction back, and the exception is caught outside it. */ public boolean runOncePortable(String consumer, String messageId, Runnable work) { try { tx.executeWithoutResult(status -> { jdbc.update("INSERT INTO processed_message (consumer, message_id) VALUES (?, ?)", consumer, messageId); work.run(); }); return true; } catch (DuplicateKeyException e) { // Thrown by the first statement when the key exists. Nothing was changed. // Caveat: a unique violation raised by the work itself would end up here too. return false; } }}Why it is written this way
ON CONFLICT DO NOTHINGreports a duplicate as zero affected rows, with no error. That matters on PostgreSQL, where an error would abort the transaction.- If another transaction is inserting the same key at the same moment, the insert waits for it to commit or roll back and then answers. That is the whole concurrency control of the pattern.
- The work runs through the same
JdbcTemplateinside theTransactionTemplatecallback, so it uses the same connection. Work that opens its own transaction (REQUIRES_NEW, another data source, an@Asynccall) is outside the guarantee. runOncePortablecannot tell a duplicate message from a unique violation inside the work. If the work can raise one, catch it there and translate it into a specific exception.- The method returns whether the work ran, so the listener can count duplicates in a metric.
package com.example.inventory;import com.example.inventory.idempotency.IdempotentProcessor;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import io.micrometer.core.instrument.Counter;import io.micrometer.core.instrument.MeterRegistry;import java.nio.charset.StandardCharsets;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.header.Header;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class PaymentCapturedListener { private static final String CONSUMER = "inventory-service"; public record PaymentCaptured(String orderId, String sku, int quantity) {} private final IdempotentProcessor idempotent; private final JdbcTemplate jdbc; private final ObjectMapper json; private final Counter duplicates; public PaymentCapturedListener(IdempotentProcessor idempotent, JdbcTemplate jdbc, ObjectMapper json, MeterRegistry metrics) { this.idempotent = idempotent; this.jdbc = jdbc; this.json = json; this.duplicates = metrics.counter("inventory.duplicate.messages"); } @KafkaListener(id = "inventory-payments", topics = "payments.events", groupId = CONSUMER) public void onPaymentEvent(ConsumerRecord<String, String> record) throws JsonProcessingException { if (!"PaymentCaptured".equals(header(record, "eventType"))) { return; } // The event id was assigned by the Payment Service when it captured the payment. String eventId = header(record, "id"); PaymentCaptured event = json.readValue(record.value(), PaymentCaptured.class); boolean ran = idempotent.runOnce(CONSUMER, eventId, () -> reserve(event)); if (!ran) { duplicates.increment(); } // The container commits the offset after this method returns, in both cases. } private void reserve(PaymentCaptured event) { int updated = jdbc.update(""" UPDATE stock SET available = available - ? WHERE sku = ? AND available >= ?""", event.quantity(), event.sku(), event.quantity()); if (updated == 0) { // Not enough stock: a business result. A real service writes an // OutOfStock event to its outbox here, in this same transaction. return; } jdbc.update(""" INSERT INTO stock_reservation (order_id, sku, quantity) VALUES (?, ?, ?)""", event.orderId(), event.sku(), event.quantity()); } private static String header(ConsumerRecord<String, String> record, String name) { Header h = record.headers().lastHeader(name); if (h == null) { throw new IllegalArgumentException("Missing header " + name); } return new String(h.value(), StandardCharsets.UTF_8); }}Why it is written this way
- The idempotency key is the event id from the header, never
record.offset(). A copy that arrives through a retry topic or a dead letter replay has other coordinates and the same id. - The stock update is a delta, which is exactly the kind of effect that needs the dedup row. Without it, a redelivery would remove the quantity twice.
- "Not enough stock" returns normally, so the dedup row is committed and the event is not processed again. Throwing would roll back the dedup row and retry a result that will not change.
- A duplicate is counted, not logged as an error. A steady low rate is normal. A sudden rise points at a rebalance storm or an upstream republication.
- A missing header throws. Configure the container's error handler with a dead letter recoverer so such a record does not block the partition.
package com.example.inventory.idempotency;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.beans.factory.annotation.Value;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;@Componentpublic class ProcessedMessageCleanup { private static final Logger log = LoggerFactory.getLogger(ProcessedMessageCleanup.class); private static final int BATCH = 10_000; private final JdbcTemplate jdbc; private final int retentionDays; public ProcessedMessageCleanup(JdbcTemplate jdbc, @Value("${idempotency.retention-days:30}") int retentionDays) { this.jdbc = jdbc; this.retentionDays = retentionDays; } @Scheduled(cron = "0 */10 * * * *") public void deleteOldRows() { int total = 0; int deleted; do { // Each statement is its own short transaction. deleted = jdbc.update(""" DELETE FROM processed_message WHERE (consumer, message_id) IN ( SELECT consumer, message_id FROM processed_message WHERE processed_at < now() - make_interval(days => ?) LIMIT ?)""", retentionDays, BATCH); total += deleted; } while (deleted == BATCH && total < 20 * BATCH); log.debug("Removed {} processed_message rows older than {} days", total, retentionDays); }}Why it is written this way
- The retention is configuration with a visible default. It must stay longer than the retention of the source topic and of any retry or dead letter topic that can be replayed into this consumer.
- Bounded batches keep each delete short, so it never blocks the listener's inserts for long. The loop stops after 200,000 rows per run and continues on the next schedule.
- The index on
processed_atmakes finding old rows cheap. Without it every run scans the table. @Scheduledneeds@EnableSchedulingon a configuration class. With several instances, each runs the job. That is harmless here because deleting the same old rows twice has no effect.
package com.example.notifications;import java.util.Map;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.stereotype.Component;import org.springframework.web.client.RestClient;/** An effect outside the database: claim, call with an idempotency key, complete. */@Componentpublic class OrderConfirmationSender { private final JdbcTemplate jdbc; private final RestClient emailProvider; public OrderConfirmationSender(JdbcTemplate jdbc, RestClient emailProvider) { this.jdbc = jdbc; this.emailProvider = emailProvider; } /** Safe to call any number of times for the same event. Throws if the provider call fails. */ public void sendOnce(String eventId, String recipient, String orderId) { // 1. Claim. Committed on its own, so a crash after this point leaves a visible PENDING row. jdbc.update(""" INSERT INTO notification_delivery (event_id, channel, status) VALUES (?, 'EMAIL', 'PENDING') ON CONFLICT DO NOTHING""", eventId); String status = jdbc.queryForObject(""" SELECT status FROM notification_delivery WHERE event_id = ? AND channel = 'EMAIL'""", String.class, eventId); if ("SENT".equals(status)) { return; // finished on an earlier delivery } // 2. Call. The same key on every attempt lets the provider recognise a repeat. emailProvider.post() .uri("/v1/messages") .header("Idempotency-Key", eventId + ":EMAIL") .body(Map.of("to", recipient, "template", "order-confirmed", "orderId", orderId)) .retrieve() .toBodilessEntity(); // throws on a 4xx or 5xx response // 3. Complete. Conditional, so two instances finishing at once do not disagree. jdbc.update(""" UPDATE notification_delivery SET status = 'SENT', sent_at = now() WHERE event_id = ? AND channel = 'EMAIL' AND status = 'PENDING'""", eventId); }}Why it is written this way
- No transaction spans the HTTP call. Holding a database transaction open across a network call ties up a connection and still cannot roll back an email.
- The idempotency key is derived from stored data (
eventIdand the channel), so every attempt sends the same key. A key generated per attempt would make each retry a new email. - If the call throws, the row stays
PENDINGand the listener's error handler retries the record. The retry repeats the call with the same key. - The header name and whether repeats are recognised depend on the provider: check its API documentation. Without provider support the email can be sent twice in the window between the call and step 3.
- The
RestClientbean is built elsewhere with the provider's base URL and credentials.
Configuration
The pattern is application code, but several Kafka and Spring settings decide how often duplicates arrive and how long they can keep arriving. Kafka defaults are from the 4.1 configuration reference, Spring defaults from the 3.3 reference.
enable.auto.commitDefault true (Kafka client). Spring for Apache Kafka sets it to false unless you configure it.
- What it does
- Whether the client commits offsets periodically in the background, regardless of whether your code has finished processing.
- When to change it
- Leave it off in Spring so the container commits after the listener returns. Background commits can advance past a record whose database transaction later rolls back, which loses the message. No dedup table repairs that.
spring.kafka.listener.ack-modeDefault BATCH
- What it does
- When the container commits offsets.
BATCHcommits after all records of a poll have been processed,RECORDafter each record. - When to change it
- With
BATCH, a crash mid-batch redelivers the records already processed in that batch. The pattern absorbs them.RECORDreduces duplicates at the cost of more commits.
max.poll.interval.msDefault 300000 (5 minutes)
- What it does
- Maximum time between polls before the consumer is considered failed and its partitions are reassigned.
- When to change it
- Exceeding it is the usual cause of two instances working on one record. Keep per-poll work well below it by lowering
max.poll.recordsbefore raising this.
max.poll.recordsDefault 500
- What it does
- Maximum records returned by one poll.
- When to change it
- Lower it when each record includes a slow call, so a batch always finishes inside the poll interval.
retention.ms (source, retry and dead letter topics)Default 604800000 (7 days)
- What it does
- How long records stay available, and therefore how far back a replay can deliver an old message again.
- When to change it
- The dedup retention must be longer than the longest of these, plus the time a replay might be started after it.
idempotency.retention-daysDefault none (application setting, 30 in this lesson)
- What it does
- Age after which
processed_messagerows are deleted by the cleanup job. - When to change it
- Raise it when a topic's retention is raised. Do not lower it to save space without checking every topic that can be replayed into the consumer.
isolation.levelDefault read_uncommitted
- What it does
- Whether the consumer also receives records from Kafka transactions that were aborted.
- When to change it
- Set
read_committedwhen the upstream producer uses transactions. Otherwise the consumer processes events from aborted transactions, and no dedup key can tell that they never happened.
Hands-on exercise
Deliver the same event twice and keep the stock right
Produce a duplicate by crashing a consumer between its database commit and its offset commit, see the effect applied twice, then add the processed-message check and repeat the crash.
Steps · 0 of 5 done
Crash between processing and commit
Simulation · not a real clusterAt-least-once consumption with a crash after the work is done. Run it with the idempotent consumer off, then on, and compare the outcomes.
What you should see
Without the check, the record is redelivered after the restart and the effect is applied twice. With it, the second delivery finds its key already recorded, the work is skipped, and the offset is committed: one effect. With the crash before processing, nothing was recorded, so the redelivery is processed normally. That last case shows why the dedup record must commit together with the effect and not before it.
What can go wrong
Dedup record and effect in different transactions
- Symptom
- Stock is occasionally not reduced for a paid order, and the event's id is present in
processed_message. - Cause
- The dedup insert ran in its own transaction (auto-commit,
REQUIRES_NEW, or a different data source) and committed. The business update then failed or the process died. The redelivery was skipped as a duplicate. - Fix
- Run the insert and the business statements on the same connection inside one transaction, and test it by forcing an exception after the insert and checking that the row is gone.
Catching the duplicate-key error inside a PostgreSQL transaction
- Symptom
- Duplicates produce
current transaction is aborted, commands ignored until end of transaction block, and the record is retried until it reaches the dead letter topic. - Cause
- The code inserts, catches
DuplicateKeyExceptionand carries on in the same transaction. PostgreSQL has already marked that transaction as failed. - Fix
- Use
INSERT ... ON CONFLICT DO NOTHINGand check the row count, or let the exception end the transaction and catch it outside.
Topic, partition and offset as the key
- Symptom
- Redeliveries are skipped correctly, but after a dead letter replay several orders have stock reduced twice.
- Cause
- The replayed record is a new record with a new offset, and often a different topic. Its coordinates were never seen before.
- Fix
- Key on an event id assigned by the producer, or on a business key. Keep coordinates only as a fallback for sources that provide neither.
An event id generated when the message is sent
- Symptom
- The dedup table fills as expected, yet upstream republications are still applied twice.
- Cause
- The producer calls
UUID.randomUUID()in its publishing code. Each publication of the same business fact gets a fresh id. - Fix
- Create the id in the transaction that records the business fact and store it, for example in the outbox row, so that every publication carries the same value.
Dedup rows deleted before the last possible duplicate
- Symptom
- A replay of 10-day-old dead letters reduces stock again for orders that were fulfilled long ago.
- Cause
- The cleanup removes rows after 7 days, while the dead letter topic keeps records for 14.
- Fix
- Set the retention from the longest replay window, and guard effects that must never repeat with a permanent unique business key.
Marking the message before an external call
- Symptom
- Some customers never receive their confirmation email, and the delivery row exists.
- Cause
- The consumer recorded the message as processed, then crashed before the provider call. The redelivery saw the record and skipped the send.
- Fix
- Use states: claim as
PENDING, call with a stable idempotency key, then markSENT. A redelivery that findsPENDINGrepeats the call.
A dedup table that is never cleaned
- Symptom
- Consumer throughput falls over months and lag appears at peak. The slowest statement is the insert into
processed_message. - Cause
- The table has grown to billions of rows and its primary key index no longer fits in memory, so each insert reads from disk.
- Fix
- Add the scheduled, batched delete with an index on the timestamp, or partition the table by time and drop old partitions.
How to troubleshoot
Suspect that an effect was applied twice for one order
Check
SELECT order_id, sku, count(*) FROM stock_reservation GROUP BY order_id, sku HAVING count(*) > 1; and SELECT * FROM processed_message WHERE message_id = '<event id>';
- How to read it
- With the unique key in place the first query returns nothing. One
processed_messagerow and a doubled effect means the effect ran outside the dedup transaction. No row at all means the key was cleaned up or never written.
Want to know how often duplicates arrive
Check
The application metric `inventory.duplicate.messages` (a counter incremented when the dedup insert affects 0 rows)
- How to read it
- A low steady rate after deployments and rebalances is normal. A rate that follows the incoming rate means something upstream publishes everything twice, or the group is rebalancing continuously.
Duplicates arrive in bursts
Check
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group inventory-service --state
- How to read it
- A state that keeps changing away from
Stablemeans repeated rebalances, each of which redelivers uncommitted records. Look in the application log for a commit failure that mentionsmax.poll.interval.ms.
Two records with the same event id on the topic
Check
kafka-console-consumer.sh --bootstrap-server broker:9092 --topic payments.events --from-beginning --property print.key=true --property print.headers=true --property print.offset=true
- How to read it
- The same
idheader at two offsets is an upstream republication, which the pattern absorbs. Different ids for what is clearly one business action means the producer creates ids at send time. Recent 4.x releases deprecate--propertyhere in favour of--formatter-property.
The dedup table is large or the cleanup is behind
Check
SELECT count(*), min(processed_at) FROM processed_message WHERE consumer = 'inventory-service';
- How to read it
- The oldest row should be about as old as the configured retention. Much older means the cleanup job is not running or cannot keep up with the insert rate.
Key takeaways
- At-least-once delivery plus an idempotent consumer is how effects happen once. Kafka does not deduplicate for consumers.
- The idempotency key must identify the business fact and be created once at the source. Offsets identify a record, not a fact.
- Insert the key first, under a unique constraint, in the same transaction as the effect. The constraint also settles the race between two instances.
- On PostgreSQL use
ON CONFLICT DO NOTHING. Elsewhere, catch the violation outside the transaction. - Keep dedup rows longer than any topic that can be replayed into the consumer, and delete them in batches.
- For an external call, use claim, call with a stable idempotency key, complete. Marking first loses the effect, calling first repeats it.
- Prefer a permanent unique business key for anything that must never happen twice.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
An inventory consumer deduplicates on (topic, partition, offset) stored in the same transaction as the stock update. It has run cleanly for months. After a bug, 1,200 records land in the dead letter topic. The fix is deployed, the dead letters are republished to the main topic, and 85 orders end up with stock reduced twice.
Why did the dedup not catch those 85?
Scenario 2 of 3
A listener on PostgreSQL does this inside one @Transactional method: insert the event id into processed_event; if DuplicateKeyException is thrown, catch it and return; otherwise update the balance. In production every duplicate ends with an error about an aborted transaction, is retried ten times and goes to the dead letter topic.
What is the smallest correct change?
Scenario 3 of 3
A notification consumer inserts (event_id, 'EMAIL') into a dedup table and then calls the email provider, all in one database transaction. The provider call takes 300 ms on average. During a provider slowdown the connection pool is exhausted, and afterwards 40 customers report two confirmation emails.
Which redesign addresses both problems?
References
References: Apache Kafka 4.1: design, message delivery semantics ↗ · Apache Kafka 4.1: consumer configuration reference (enable.auto.commit, isolation.level, max.poll.interval.ms, max.poll.records) ↗ · Apache Kafka 4.1: producer configuration reference (enable.idempotence and what it covers) ↗ · Spring for Apache Kafka 3.3 reference: message listener containers (AckMode, committing offsets) ↗ · PostgreSQL documentation: INSERT, the ON CONFLICT clause ↗ · PostgreSQL documentation: index uniqueness checks (a conflicting insert waits for the other transaction) ↗ · Stripe API reference: idempotent requests (an example of provider-side idempotency keys) ↗ · Chris Richardson, microservices.io: Idempotent consumer pattern ↗ · Report a problem