Path 5 · lesson 5 of 5
Effectively-once processing for payments and orders
Delivery stays at-least-once. Design the payment and order flow so each business effect happens once: stable keys, unique constraints, an outbox and safe retries.
What you will learn
- Design an event so that every copy of it carries the same identity, from the API call to the last consumer
- Make a database write reject its own repeat with a unique constraint or a processed-events row in the same transaction
- Call a payment provider so that a retry after a timeout cannot charge the card twice
- Trace what happens when a record was processed but its offset was not committed, and show that the redelivery is harmless
- Separate retryable failures, business outcomes and conflicting duplicates, and route each one correctly
- Implement an idempotent payment consumer in Spring Boot with JdbcTemplate and explain every transaction boundary in it
Before you start
- Duplicates and idempotency
- Kafka transactions and your database
- Offset management
- Unique constraints and transactions in a relational database
The concept
Customers do not care how many times a record was delivered. They care that the card was charged once, the order was marked paid once and one confirmation arrived. Effectively-once processing means exactly that: delivery is at-least-once, the code may run several times for the same event, and the business effect still happens once because every step is built to recognise its own repeat.
The design has four parts, one for each place a repeat can enter or leave the service:
| Where the repeat appears | Mechanism | Who enforces it |
|---|---|---|
| The same business fact is published twice | An identity created once with the fact (event id, order id) and carried in every copy | The producing service |
| The same record is processed twice | A unique constraint or a processed-events row, in the same database transaction as the business write | Your database |
| The external call is made twice | An idempotency key that is stored before the call and sent with every attempt | The provider |
| The outgoing event is produced twice | An outbox row with a deterministic event id, then deduplication downstream | Your database, then the next consumer |
Each row covers a different hop. Leave one out and the duplicate gets through at that hop, however careful the other three are.
Why it exists
The previous lessons established that the repeats cannot be configured away. A consumer that commits after processing will, after a crash, a rebalance or a failed commit, read some records again. An outbox relay sends a row again if it dies before marking it published. A user double-clicks, a mobile client retries a timed-out request. Each of these is correct behaviour by a component that prefers repeating to losing.
Before services were built this way, the usual defences were hope and a nightly reconciliation report. In payments, a duplicate is not a cosmetic defect: a second capture takes real money, a second OrderPaid ships a second parcel, and undoing either means refunds, support time and sometimes a regulator. The cost of the extra table and the extra constraint is small next to that.
What Kafka gives you
- At-least-once delivery when you commit after processing
- No duplicate appends from producer retries, within a producer session
- Atomic output records and offsets for topic-to-topic processing
- Order within a partition, as appended
What you must build
- An identity for every business fact, created once
- A database that refuses the second copy of an effect
- External calls that are safe to repeat
- A decision for records that can never succeed or that contradict an earlier one
Idempotency keys
Simulation, not a real clusterThe animated Build lesson shows a retried payment request with and without an idempotency key. Simulated.
Step 1 · The pattern
One tap, one charge
A shopper taps Pay. The app calls the payments API, the API charges the card, and a success response comes back. One order, one charge.
- payment request
- success
How it works inside
1. Identity is created once, at the source
A consumer can only recognise a repeat if both copies carry the same identity. So the identity must be created when the business fact is created, not when a message is sent. The checkout API accepts an Idempotency-Key from the client and stores it under a unique constraint, so a retried request returns the existing order. The order service creates the eventId of OrderPlaced in the same transaction as the order and stores it in the outbox row. However many times the relay publishes that row, the id is the same. An id generated inside the send path would be different on every attempt and useless.
2. The dedupe record commits with the business write
For a consumer whose whole effect is in its own database, insert the event id into a processed_events table in the same database transaction as the business write. The primary key makes the second insert fail or affect zero rows; you skip the work and let the offset move on. Both writes commit together or not at all, so there is no state in which the work is done but not recorded, or recorded but not done. A dedupe check in a separate transaction, in a cache, or in memory reintroduces exactly that gap.
Often a business key does the same job with no extra table: UNIQUE (order_id) on payments means one payment per order, whichever event asked for it. That is stronger than an event-id check, because it also catches two different events that ask for the same thing.
3. An external call needs a state machine
A card charge cannot join a database transaction. If you mark the event as processed first and then crash before the call, the redelivery is skipped and the customer is never charged. If you call first and crash before recording it, the redelivery calls again. The way out is to split the work into steps that each tolerate repetition:
- Claim. In one transaction, insert the payment as
PENDINGwith a newpayment_id, relying onUNIQUE (order_id). If the row already exists, read it instead. - Charge. Call the provider with
payment_idas the idempotency key. The key was stored in step 1, so every attempt for this order sends the same one, and the provider answers a repeat with the original result instead of charging again. - Complete. In one transaction, move the row from
PENDINGtoCAPTUREDorDECLINEDand insert the outgoing event into the outbox. The update is conditional on the row still beingPENDING.
A redelivery at any point resumes instead of restarting. If the row is already final, there is nothing to do. If it is still PENDING, the charge is repeated with the same key and the completion runs. Many provider APIs accept such a key; Stripe's API, for example, takes an Idempotency-Key header, compares the parameters of a repeated request with the original, and may prune keys once they are at least 24 hours old. Check your provider's rules, because a retry that arrives after the key was forgotten is a new charge.
4. Processed, but the offset was not committed
An offset commit records the position of the next record to read for a group and partition. It is a separate write to Kafka, made after your database transaction. If the process dies between the two, the group resumes from the last committed position and the record arrives again, possibly on another instance. Under this design the second pass finds a final payment row and returns without side effects, and the offset is committed this time. The redelivery costs one indexed lookup.
Processed, crashed, redelivered
Simulation · not a real clusterAt-least-once with a crash between processing and the offset commit. Run it with the idempotent consumer on and compare with it off.
5. Records that should not simply be retried or skipped
| Situation | Example | What to do |
|---|---|---|
| Transient failure | Provider timeout, database connection lost | Throw. The error handler redelivers with backoff, and the same idempotency key makes the repeat safe. |
| Business outcome | Card declined | Not an error. Record DECLINED, publish the result, commit. Retrying would only repeat the decline. |
| Exact duplicate | Same order, same amount, payment already final | Skip quietly and let the offset advance. Count it in a metric. |
| Conflicting duplicate | Same order id, different amount or currency | Do not skip and do not process. Fail without retries and send the record to the dead-letter topic for a person to look at. |
| Poison record | Payload cannot be deserialised or fails validation | No retry can fix it. Dead-letter it so the partition is not blocked behind it. |
| Uncertain result | The charge request timed out | Never invent a new key. Retry with the stored key until the provider gives a definite answer. |
Architecture
Drawing diagram…
Flowchart of a payment consumer. It looks up the payment row for the order and either inserts a pending row, charges the provider using the payment id as idempotency key, skips an exact duplicate or dead-letters a conflicting one. A successful charge leads to one database transaction that updates the status and inserts an outbox row.
flowchart TD
IN["orders.order-placed, key orderId"] --> L["OrderPlacedListener"]
L --> C{"Payment row for this order"}
C -->|"none"| INS["Insert PENDING with new payment_id"]
C -->|"PENDING"| CH["Charge provider, key is payment_id"]
C -->|"final, same amount"| SKIP["Skip, exact duplicate"]
C -->|"different amount"| DLT["Dead-letter topic, no retry"]
INS --> CH
CH -->|"approved or declined"| TX["One DB transaction"]
CH -->|"timeout"| RETRY["Throw, redeliver with backoff"]
RETRY --> L
TX --> UPD["Update status where PENDING"]
TX --> OBX["Insert outbox row, event id is payment_id"]
OBX --> REL["Relay publishes payments.payment-result"]
SKIP --> ACK["Offset advances"]
UPD --> ACK- The decision node reads state from the database, not from memory, so it gives the same answer on any instance after any restart.
- The provider call sits between two short database transactions. No database connection or lock is held while waiting on the network.
- The status update and the outbox insert share a transaction, so a result event exists exactly when the payment reached a final state.
- The offset advances only on paths that end in a committed, final state or in a dead-lettered record. A timeout loops back and holds the offset where it is.
Message flow
Drawing diagram…
Sequence diagram. The payment service receives OrderPlaced at offset 42, inserts a pending payment, charges the provider with an idempotency key, and commits the captured status with an outbox row. It crashes before committing the offset. The record is delivered again, the insert conflicts, the row is read as captured, nothing else happens and offset 43 is committed.
sequenceDiagram participant K as Kafka participant P as Payment Service participant D as Payments DB participant C as Card Provider K->>P: OrderPlaced order 7001 at offset 42 P->>D: INSERT payment PENDING, unique order_id P->>C: charge with Idempotency-Key pay-91 C-->>P: approved, ref ch_5 P->>D: UPDATE to CAPTURED and INSERT outbox, one commit Note over P: crash before the offset commit K->>P: OrderPlaced order 7001 at offset 42 again P->>D: INSERT payment, conflict, 0 rows P->>D: SELECT payment for order 7001 D-->>P: CAPTURED Note over P: exact duplicate, no charge, no new event P->>K: commit offset 43
- First pass: three durable steps in order. Claim in the database, charge at the provider, complete in the database together with the outbox row.
- The crash loses only the offset commit. Everything that matters to the customer is already durable.
- Second pass: the unique constraint turns the insert into a no-op and the read shows a final state. The service returns normally, which is what lets the offset finally move to 43.
- Had the crash come after the charge and before the completing transaction, the second pass would find
PENDING, repeat the charge with key pay-91, receive the original approval and complete.
In a real system
A typical shop checkout, the same system as the example project. Topics are keyed by orderId so that one order's events stay in order within a partition.
| Hop | What can repeat | What makes the effect single |
|---|---|---|
Browser to Order Service, POST /orders | Double click, client retry after a timeout | Idempotency-Key header stored under a unique constraint on orders |
Order Service to orders.order-placed (12 partitions) | Relay re-sends an outbox row | eventId created with the order and stored in the outbox row |
orders.order-placed to Payment Service, group payment-service | Redelivery after a crash, rebalance or failed commit | UNIQUE (order_id) on payments and the status check |
| Payment Service to card provider | Retry after a timeout, redelivery while PENDING | payment_id sent as the idempotency key on every attempt |
Payment Service to payments.payment-result | Relay re-sends | Outbox row whose event id is the payment_id |
payments.payment-result to Order Service, group order-service | Redelivery | processed_events row in the same transaction as the order status update |
| Order Service to Notification Service | Redelivery | Unique (order_id, template) on sent notifications, as far as the mail provider allows |
The last row is honest about a limit: an email provider without idempotency keys can still send twice if the service crashes after the call and before recording it. Decide per effect how much that matters. A duplicate confirmation email is tolerable; a duplicate capture is not, which is why the payment hop gets the full state machine.
Java and Spring Boot
The payment service's consumer with JdbcTemplate and explicit transaction boundaries, the order service's simpler consumer with a processed-events table, the SQL for both and the error handling. The provider's HTTP API is invented for the example: map your provider's real responses.
CREATE TABLE payments ( payment_id uuid PRIMARY KEY, -- also the provider idempotency key order_id uuid NOT NULL, amount_minor bigint NOT NULL, currency varchar(3) NOT NULL, status varchar(16) NOT NULL CHECK (status IN ('PENDING', 'CAPTURED', 'DECLINED')), provider_ref varchar(64), source_event_id uuid NOT NULL, -- the OrderPlaced that created the row created_at timestamptz NOT NULL DEFAULT now(), updated_at timestamptz NOT NULL DEFAULT now(), CONSTRAINT uq_payments_order UNIQUE (order_id));CREATE TABLE outbox ( id bigserial PRIMARY KEY, event_id uuid NOT NULL UNIQUE, topic varchar(128) NOT NULL, record_key varchar(128) NOT NULL, payload text NOT NULL, created_at timestamptz NOT NULL DEFAULT now(), published_at timestamptz);-- In the order service's database: generic dedupe for consumers whose-- whole effect is a local write.CREATE TABLE processed_events ( consumer varchar(64) NOT NULL, event_id uuid NOT NULL, processed_at timestamptz NOT NULL DEFAULT now(), PRIMARY KEY (consumer, event_id));Why it is written this way
uq_payments_orderis the guarantee. Application checks are an optimisation; the constraint is what holds when two instances race.payment_idis generated once, stored before the provider is called, and reused as the idempotency key. A key generated per attempt would defeat the provider's deduplication.outbox.event_idis unique and set to thepayment_id, so even a bug that completes a payment twice cannot enqueue two result events.processed_eventsis keyed by consumer and event id so several consumers can share the table. Delete old rows only after the longest period in which a record could still be redelivered or replayed, which is at least the topic's retention if you ever reset offsets.
package com.example.payments;import java.time.Instant;import java.util.UUID;/** Consumed from orders.order-placed. eventId is created once by the order service. */record OrderPlaced(UUID eventId, UUID orderId, String customerId, long amountMinor, String currency, Instant occurredAt) {}/** Published to payments.payment-result through the outbox. eventId equals paymentId. */record PaymentResult(UUID eventId, UUID paymentId, UUID orderId, boolean approved, String providerRef, Instant occurredAt) {}enum PaymentStatus { PENDING, CAPTURED, DECLINED }record Payment(UUID paymentId, UUID orderId, long amountMinor, String currency, PaymentStatus status) {}record ChargeResult(boolean approved, String providerRef) {}/** Implementations must send idempotencyKey to the provider on every attempt. */interface PaymentProvider { ChargeResult charge(String idempotencyKey, long amountMinor, String currency, String customerId);}/** Same order, different money: a person has to look at this. Never retried. */class ConflictingDuplicateException extends RuntimeException { ConflictingDuplicateException(String message) { super(message); }}Why it is written this way
- Amounts are integers in minor units (cents). Floating point has no place in a payment record.
- A declined card is a
ChargeResult, not an exception. Exceptions are reserved for cases where the outcome is unknown or the record is wrong. ConflictingDuplicateExceptiongets its own type so the error handler can route it to the dead-letter topic without retries.
package com.example.payments;import java.time.Instant;import java.util.UUID;import com.fasterxml.jackson.core.JsonProcessingException;import com.fasterxml.jackson.databind.ObjectMapper;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.stereotype.Service;import org.springframework.transaction.PlatformTransactionManager;import org.springframework.transaction.support.TransactionTemplate;@Servicepublic class PaymentProcessor { private static final Logger log = LoggerFactory.getLogger(PaymentProcessor.class); private final JdbcTemplate jdbc; private final TransactionTemplate tx; private final PaymentProvider provider; private final ObjectMapper json; public PaymentProcessor(JdbcTemplate jdbc, PlatformTransactionManager transactionManager, PaymentProvider provider, ObjectMapper json) { this.jdbc = jdbc; this.tx = new TransactionTemplate(transactionManager); this.provider = provider; this.json = json; } /** Safe to call any number of times for the same event, on any instance. */ public void handle(OrderPlaced event) { // Step 1, own transaction: claim the order or find the existing claim. Payment payment = tx.execute(status -> claim(event)); if (payment.amountMinor() != event.amountMinor() || !payment.currency().equals(event.currency())) { throw new ConflictingDuplicateException("Order " + event.orderId() + " already has payment " + payment.paymentId() + " for a different amount or currency"); } if (payment.status() != PaymentStatus.PENDING) { log.info("Duplicate OrderPlaced {} for order {}: payment {} is already {}", event.eventId(), event.orderId(), payment.paymentId(), payment.status()); return; } // Step 2, no transaction open: the stored payment id is the idempotency key. ChargeResult result = provider.charge(payment.paymentId().toString(), payment.amountMinor(), payment.currency(), event.customerId()); // Step 3, own transaction: final state and outgoing event together. tx.executeWithoutResult(status -> complete(payment, result)); } private Payment claim(OrderPlaced event) { jdbc.update(""" INSERT INTO payments (payment_id, order_id, amount_minor, currency, status, source_event_id) VALUES (?, ?, ?, ?, 'PENDING', ?) ON CONFLICT (order_id) DO NOTHING """, UUID.randomUUID(), event.orderId(), event.amountMinor(), event.currency(), event.eventId()); return jdbc.queryForObject(""" SELECT payment_id, order_id, amount_minor, currency, status FROM payments WHERE order_id = ? """, (rs, n) -> new Payment( rs.getObject("payment_id", UUID.class), rs.getObject("order_id", UUID.class), rs.getLong("amount_minor"), rs.getString("currency"), PaymentStatus.valueOf(rs.getString("status"))), event.orderId()); } private void complete(Payment payment, ChargeResult result) { int updated = jdbc.update(""" UPDATE payments SET status = ?, provider_ref = ?, updated_at = now() WHERE payment_id = ? AND status = 'PENDING' """, result.approved() ? "CAPTURED" : "DECLINED", result.providerRef(), payment.paymentId()); if (updated == 0) { // Another delivery of the same order completed first. Its transaction wrote the event. return; } PaymentResult out = new PaymentResult(payment.paymentId(), payment.paymentId(), payment.orderId(), result.approved(), result.providerRef(), Instant.now()); jdbc.update( "INSERT INTO outbox (event_id, topic, record_key, payload) VALUES (?, ?, ?, ?)", out.eventId(), "payments.payment-result", payment.orderId().toString(), toJson(out)); } private String toJson(PaymentResult out) { try { return json.writeValueAsString(out); } catch (JsonProcessingException e) { throw new IllegalStateException("Cannot serialise " + out, e); } }}Why it is written this way
TransactionTemplatemakes the three boundaries visible and avoids a classic trap:@Transactionalon a private method, or on a method called from the same class, is not applied by Spring's proxy.claimuses insert-then-read instead of read-then-insert. Two instances racing on the same order both attempt the insert, one wins, and both then read the same row.- The provider call happens with no database transaction open, so a slow provider cannot exhaust the connection pool or hold row locks.
completeis conditional onstatus = 'PENDING'. The row count tells this delivery whether it was the one that finished the payment, and only the one that did writes the outbox row.- The outgoing event id is the payment id, so it is deterministic: any copy of the result, from any attempt, has the same identity for the order service to deduplicate on.
- If the provider call throws, nothing is marked final. The exception leaves the listener, the offset stays put and the record is retried with the same key.
- This service has only the database transaction manager. If you also set a
transaction-id-prefix, aKafkaTransactionManagerexists too and you must name the one you want.
package com.example.payments;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderPlacedListener { private final PaymentProcessor processor; public OrderPlacedListener(PaymentProcessor processor) { this.processor = processor; } @KafkaListener(topics = "orders.order-placed", groupId = "payment-service") public void onOrderPlaced(OrderPlaced event) { // Returning normally lets the container commit the offset. // Throwing hands the record to the error handler and keeps the offset where it is. processor.handle(event); }}Why it is written this way
- The listener is deliberately thin. Whether a record is acknowledged is decided by one thing only: does
handlereturn or throw. - No manual acknowledgement is needed. The container commits offsets after the listener returns; the design does not depend on exactly when, because a late or lost commit only causes a harmless redelivery.
package com.example.payments;import java.util.Objects;import org.springframework.beans.factory.annotation.Value;import org.springframework.stereotype.Component;import org.springframework.web.client.HttpClientErrorException;import org.springframework.web.client.RestClient;@Componentpublic class HttpPaymentProvider implements PaymentProvider { private record ChargeRequest(long amount, String currency, String customer) {} private record ChargeResponse(String id, String status) {} private final RestClient http; public HttpPaymentProvider(RestClient.Builder builder, @Value("${payments.provider.base-url}") String baseUrl) { this.http = builder.baseUrl(baseUrl).build(); } @Override public ChargeResult charge(String idempotencyKey, long amountMinor, String currency, String customerId) { try { ChargeResponse response = http.post() .uri("/v1/charges") .header("Idempotency-Key", idempotencyKey) .body(new ChargeRequest(amountMinor, currency, customerId)) .retrieve() .body(ChargeResponse.class); return new ChargeResult(true, Objects.requireNonNull(response, "empty provider response").id()); } catch (HttpClientErrorException.UnprocessableEntity declined) { // In this invented API, 422 means the card was declined: a definite business outcome. return new ChargeResult(false, null); } // Timeouts, connection errors and 5xx propagate: the outcome is unknown, so the record is retried. }}Why it is written this way
- The key is a parameter, not something this class invents. It comes from the stored payment row, which is what keeps it identical across retries, restarts and instances.
- Only a definite decline is turned into a result. Anything uncertain is thrown so the same request is sent again; the provider then returns the original outcome instead of charging a second time.
- Configure connect and read timeouts on the client. A call that hangs longer than
max.poll.interval.msgets the consumer removed from the group, and the record goes to another instance while this one is still waiting. - Learn how long your provider remembers a key and what it does when the same key arrives with different parameters. Retries must finish inside that period.
package com.example.payments;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;import org.springframework.kafka.listener.DefaultErrorHandler;import org.springframework.kafka.support.ExponentialBackOffWithMaxRetries;@Configurationpublic class KafkaErrorConfig { @Bean public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) { // Publishes the failed record to orders.order-placed-dlt, same partition number. DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template); // Waits 1, 2, 4, 8, 10, 10 seconds between attempts, then gives up. ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(6); backOff.setInitialInterval(1_000L); backOff.setMultiplier(2.0); backOff.setMaxInterval(10_000L); DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff); handler.addNotRetryableExceptions(ConflictingDuplicateException.class); return handler; }}Why it is written this way
- Spring Boot attaches a
CommonErrorHandlerbean to the default listener container factory, so declaring it is enough. - Without this bean the default is ten delivery attempts with no delay, after which the record is logged and skipped. For a payment, a logged-and-skipped record is a lost charge.
- The retries block the partition for about 35 seconds in total, well inside the default
max.poll.interval.msof 5 minutes. Longer waits belong in retry topics. - A conflicting duplicate goes straight to the dead-letter topic. Retrying cannot change the fact that two events disagree about the amount.
- The dead-letter topic must exist with at least as many partitions as the source topic, and somebody must be alerted when it receives a record.
spring: kafka: bootstrap-servers: localhost:9092 consumer: group-id: payment-service auto-offset-reset: earliest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: "[spring.json.value.default.type]": com.example.payments.OrderPlaced "[spring.json.trusted.packages]": com.example.payments producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer listener: ack-mode: recordpayments: provider: base-url: https://provider.example.testWhy it is written this way
ack-mode: recordcommits after each record instead of after each polled batch. It shortens the stretch that is redelivered after a crash; it does not remove redelivery, and correctness does not rely on it.auto-offset-reset: earliestmakes a brand-new group start from the oldest retained record instead of silently skipping what was published before it first ran.- The producer settings here serve the dead-letter publisher. An outbox relay that sends pre-serialised JSON strings needs its own template with a
StringSerializer. - If the order service's serializer adds type headers naming its own classes, map or disable them as described in the serialization lesson. A payload that cannot be deserialised is a poison record and should be dead-lettered by an
ErrorHandlingDeserializersetup.
package com.example.orders;import java.time.Instant;import java.util.UUID;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;record PaymentResult(UUID eventId, UUID paymentId, UUID orderId, boolean approved, String providerRef, Instant occurredAt) {}@Componentpublic class PaymentResultListener { private final JdbcTemplate jdbc; public PaymentResultListener(JdbcTemplate jdbc) { this.jdbc = jdbc; } @KafkaListener(topics = "payments.payment-result", groupId = "order-service") @Transactional public void onPaymentResult(PaymentResult event) { int firstTime = jdbc.update(""" INSERT INTO processed_events (consumer, event_id) VALUES ('order-service', ?) ON CONFLICT DO NOTHING """, event.eventId()); if (firstTime == 0) { return; // seen before: the earlier transaction already updated the order } jdbc.update( "UPDATE orders SET status = ? WHERE order_id = ? AND status = 'PLACED'", event.approved() ? "PAID" : "PAYMENT_FAILED", event.orderId()); }}Why it is written this way
- The simple case: the whole effect is one local write, so a single database transaction holds the dedupe row and the business update.
- The dedupe insert comes first and uses the primary key. If the update then fails, the transaction rolls back and the dedupe row disappears with it, so the redelivery is processed properly.
@Transactionalon the public listener method is bound to the database transaction manager. Kafka is not part of it; the offset is committed afterwards by the container.- The update is also conditional on the current status, a second line of defence that keeps a late or out-of-order result from overwriting a later state.
Configuration
No setting below makes processing effectively-once. They decide how often repeats happen and how failures are routed. Kafka defaults are from the 4.1 configuration reference, Spring behaviour from the Spring for Apache Kafka 3.3 reference.
enable.auto.commit (consumer)Default true (Spring for Apache Kafka sets it to false unless you set it explicitly)
- What it does
- Whether the client commits offsets on a timer regardless of what your code has finished.
- When to change it
- Keep it off so the listener container commits only after the listener returns. A timed commit can pass a record whose processing then fails, which loses the payment instead of repeating it.
spring.kafka.listener.ack-modeDefault BATCH (the container's default AckMode)
- What it does
- When the container commits:
BATCHafter all records of a poll are processed,RECORDafter each one,MANUALwhen your code says so. - When to change it
RECORDreduces how many records are redelivered after a crash at the cost of more commit requests. Choose by volume. None of the modes removes the need for idempotent processing.
max.poll.interval.ms (consumer)Default 300000 (5 minutes)
- What it does
- The longest allowed gap between two calls to
pollbefore the consumer is considered failed and its partitions are reassigned. - When to change it
- Processing of one polled batch, including provider calls and blocking retries, must fit inside it. Otherwise the record is handed to another instance while the first is still working on it.
max.poll.records (consumer)Default 500
- What it does
- The maximum number of records one
pollreturns. - When to change it
- Lower it when each record involves a remote call. 500 charges at 1 second each do not fit in 5 minutes.
enable.idempotence (producer)Default true
- What it does
- The broker discards duplicate appends caused by this producer's own retries, per partition, within one producer session.
- When to change it
- Leave it on. Do not count it as part of this design: a relay or an application that sends the same event twice produces two records, and nothing a consumer does is covered.
isolation.level (consumer)Default read_uncommitted
- What it does
- Whether the consumer also receives records from Kafka transactions that are open or were aborted.
- When to change it
- Set
read_committedif any upstream producer writes the topic transactionally. It is unrelated to deduplication but stops you acting on events that were rolled back.
Hands-on exercise
Charge once, deliver twice
Reproduce the processed-but-not-committed redelivery, count the charges with and without an idempotent consumer, then do the same against the lab's simulated payment system.
Steps · 0 of 6 done
At-least-once with a crash before the commit
Simulation · not a real clusterThe record is processed, the process dies, the offset was never committed. Toggle the idempotent consumer and compare the number of charges.
What you should see
Without the idempotent consumer the card is charged twice and the offset ends one past the record. With it, the record is still delivered twice and the offset ends in the same place, but there is one charge. The delivery count did not change; only the effect did.
What can go wrong
Dedupe check outside the business transaction
- Symptom
- Rare duplicate payments, or rare payments that are skipped entirely, both under load or during restarts.
- Cause
- The event id is recorded in a cache, in another transaction or before the work. A crash between the two leaves the work done but unrecorded, or recorded but not done.
- Fix
- Insert the dedupe row or rely on the unique business key inside the same database transaction as the write it protects.
A new idempotency key on every attempt
- Symptom
- The provider's dashboard shows two captures for one order, a few seconds apart, after a timeout.
- Cause
- The key is generated in the HTTP client or per delivery, so the provider sees two unrelated requests.
- Fix
- Generate the key once, store it with the pending payment before the first call, and send the stored value on every attempt.
Treating producer idempotence as deduplication
- Symptom
- Consumers receive the same
eventIdtwice althoughenable.idempotence=trueon the producer. - Cause
- The outbox relay or the application called
sendtwice for the same event. To the broker these are two distinct records; idempotence only covers the client's internal retries of one send. - Fix
- Keep producer idempotence on, and deduplicate in the consumer on the event id or business key.
Dedupe on topic, partition and offset
- Symptom
- Duplicates slip through after the relay republished, after a replay from another cluster, or after a topic was recreated.
- Cause
- The offset identifies a position in one log, not a business fact. The same fact published twice has two offsets.
- Fix
- Deduplicate on an id that was created with the business fact. Reserve offset-based tracking for the store-offsets-with-the-data design.
Silently skipping a conflicting duplicate
- Symptom
- An order was changed from 40.00 to 55.00, but the customer was charged 40.00 and nobody noticed for weeks.
- Cause
- The consumer saw an existing payment for the order and skipped the second event without comparing its content.
- Fix
- Compare the fields that matter. On a mismatch, fail without retries, dead-letter the record and alert.
Dedupe rows deleted too early
- Symptom
- After an offset reset to replay a week of data, old payments are processed again.
- Cause
processed_eventsrows are purged after a day, while records stay in the topic for seven.- Fix
- Keep dedupe rows at least as long as a record can be redelivered or replayed, or rely on a permanent business key such as
UNIQUE (order_id).
How to troubleshoot
Suspected double charges
Check
SELECT order_id, count(*) FROM payments GROUP BY order_id HAVING count(*) > 1;
- How to read it
- Any row means the unique constraint is missing or was dropped. If the table is clean but the provider shows two captures, the idempotency key differed between attempts: compare the keys in the provider's request log.
Payments stuck in PENDING
Check
SELECT payment_id, order_id, created_at FROM payments WHERE status = 'PENDING' AND created_at < now() - interval '10 minutes';
- How to read it
- The claim committed and completion never did. Either the record is still being retried (look at lag) or it was dead-lettered after the provider kept failing. Replaying it is safe, because the key is stored.
The same records keep being processed
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- A
CURRENT-OFFSETthat does not advance while the log shows the same event ids means processing finishes but commits fail, often after a rebalance caused by exceedingmax.poll.interval.ms. Duplicates are being absorbed by the design, but throughput is zero.
Records are missing downstream
Check
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-placed-dlt --from-beginning
- How to read it
- Records here were given up on. A conflicting duplicate needs a human decision. A provider outage that outlasted the retries can be replayed once the provider is back.
How often duplicates actually arrive
Check
Application log line: Duplicate OrderPlaced ... payment ... is already CAPTURED
- How to read it
- A steady trickle around deploys and rebalances is normal. A sudden flood points at a relay republishing a backlog or a group whose commits are failing.
Key takeaways
- Effectively-once is a property of your design: at-least-once delivery plus effects that recognise their own repeat. Kafka does not provide it for databases or external calls.
- Create the identity once, with the business fact, and carry it in every copy: API idempotency key, event id, payment id.
- The dedupe record or unique constraint must commit in the same database transaction as the write it protects.
- An external call needs a stored key and a state machine: claim, call with the key, complete. Each step is safe to repeat.
- A record that was processed but not committed will come back. The correct response is to find the finished state and return normally.
- Exact duplicates are skipped, business outcomes are recorded, uncertain results are retried with the same key, and conflicting or unreadable records go to a dead-letter topic.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A payment consumer inserts the event id into processed_events and commits. Then it calls the card provider, then updates payments in a second transaction. During a deploy, 4 orders end up with a processed_events row, no charge at the provider and status PENDING. Kafka shows no lag for the group.
What is wrong with the design?
Scenario 2 of 3
A charge request times out after 5 seconds. The listener catches the timeout, generates a fresh UUID as idempotency key and tries again; the second attempt succeeds. Next morning finance finds 11 customers charged twice, all from a 3-minute window in which the provider was slow.
Which change prevents this?
Scenario 3 of 3
payments has UNIQUE (order_id). An OrderPlaced for order 9001 arrives with amount 4000 and is captured. Ten minutes later another OrderPlaced for order 9001 arrives with a different event id and amount 5500. The consumer logs it as a duplicate and skips it.
How should the consumer treat the second event?
References
References: Apache Kafka documentation: Design, message delivery semantics ↗ · KafkaProducer Javadoc (4.1): what the idempotent producer does and does not cover ↗ · Spring for Apache Kafka 3.3 reference: Exactly Once Semantics ↗ · Spring for Apache Kafka 3.3 reference: Message listener containers (committing offsets, AckMode) ↗ · Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler, dead-letter publishing) ↗ · Spring for Apache Kafka 3.3 reference: Examples of Kafka transactions with other transaction managers ↗ · Apache Kafka 4.1 consumer configuration reference ↗ · Stripe API reference: Idempotent requests ↗ · Report a problem