Skip to content

Path 5 · lesson 5 of 5

Advanced25 minApache Kafka 4.x (documentation for 4.1) · Java 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 (Boot 3.5 manages kafka-clients 3.9) · PostgreSQL syntax in the SQL

Effectively-once processing for payments and orders

Delivery stays at-least-once. Design the payment and order flow so each business effect happens once: stable keys, unique constraints, an outbox and safe retries.

What you will learn

  • Design an event so that every copy of it carries the same identity, from the API call to the last consumer
  • Make a database write reject its own repeat with a unique constraint or a processed-events row in the same transaction
  • Call a payment provider so that a retry after a timeout cannot charge the card twice
  • Trace what happens when a record was processed but its offset was not committed, and show that the redelivery is harmless
  • Separate retryable failures, business outcomes and conflicting duplicates, and route each one correctly
  • Implement an idempotent payment consumer in Spring Boot with JdbcTemplate and explain every transaction boundary in it

Before you start

The concept

Customers do not care how many times a record was delivered. They care that the card was charged once, the order was marked paid once and one confirmation arrived. Effectively-once processing means exactly that: delivery is at-least-once, the code may run several times for the same event, and the business effect still happens once because every step is built to recognise its own repeat.

The design has four parts, one for each place a repeat can enter or leave the service:

Where the repeat appearsMechanismWho enforces it
The same business fact is published twiceAn identity created once with the fact (event id, order id) and carried in every copyThe producing service
The same record is processed twiceA unique constraint or a processed-events row, in the same database transaction as the business writeYour database
The external call is made twiceAn idempotency key that is stored before the call and sent with every attemptThe provider
The outgoing event is produced twiceAn outbox row with a deterministic event id, then deduplication downstreamYour database, then the next consumer

Each row covers a different hop. Leave one out and the duplicate gets through at that hop, however careful the other three are.

Why it exists

The previous lessons established that the repeats cannot be configured away. A consumer that commits after processing will, after a crash, a rebalance or a failed commit, read some records again. An outbox relay sends a row again if it dies before marking it published. A user double-clicks, a mobile client retries a timed-out request. Each of these is correct behaviour by a component that prefers repeating to losing.

Before services were built this way, the usual defences were hope and a nightly reconciliation report. In payments, a duplicate is not a cosmetic defect: a second capture takes real money, a second OrderPaid ships a second parcel, and undoing either means refunds, support time and sometimes a regulator. The cost of the extra table and the extra constraint is small next to that.

What Kafka gives you

  • At-least-once delivery when you commit after processing
  • No duplicate appends from producer retries, within a producer session
  • Atomic output records and offsets for topic-to-topic processing
  • Order within a partition, as appended

What you must build

  • An identity for every business fact, created once
  • A database that refuses the second copy of an effect
  • External calls that are safe to repeat
  • A decision for records that can never succeed or that contradict an earlier one

Idempotency keys

Simulation, not a real cluster

The animated Build lesson shows a retried payment request with and without an idempotency key. Simulated.

Step 1 · The pattern

One tap, one charge

A shopper taps Pay. The app calls the payments API, the API charges the card, and a success response comes back. One order, one charge.

  • payment request
  • success
Open the full animated lesson

How it works inside

1. Identity is created once, at the source

A consumer can only recognise a repeat if both copies carry the same identity. So the identity must be created when the business fact is created, not when a message is sent. The checkout API accepts an Idempotency-Key from the client and stores it under a unique constraint, so a retried request returns the existing order. The order service creates the eventId of OrderPlaced in the same transaction as the order and stores it in the outbox row. However many times the relay publishes that row, the id is the same. An id generated inside the send path would be different on every attempt and useless.

2. The dedupe record commits with the business write

For a consumer whose whole effect is in its own database, insert the event id into a processed_events table in the same database transaction as the business write. The primary key makes the second insert fail or affect zero rows; you skip the work and let the offset move on. Both writes commit together or not at all, so there is no state in which the work is done but not recorded, or recorded but not done. A dedupe check in a separate transaction, in a cache, or in memory reintroduces exactly that gap.

Often a business key does the same job with no extra table: UNIQUE (order_id) on payments means one payment per order, whichever event asked for it. That is stronger than an event-id check, because it also catches two different events that ask for the same thing.

3. An external call needs a state machine

A card charge cannot join a database transaction. If you mark the event as processed first and then crash before the call, the redelivery is skipped and the customer is never charged. If you call first and crash before recording it, the redelivery calls again. The way out is to split the work into steps that each tolerate repetition:

  1. Claim. In one transaction, insert the payment as PENDING with a new payment_id, relying on UNIQUE (order_id). If the row already exists, read it instead.
  2. Charge. Call the provider with payment_id as the idempotency key. The key was stored in step 1, so every attempt for this order sends the same one, and the provider answers a repeat with the original result instead of charging again.
  3. Complete. In one transaction, move the row from PENDING to CAPTURED or DECLINED and insert the outgoing event into the outbox. The update is conditional on the row still being PENDING.

A redelivery at any point resumes instead of restarting. If the row is already final, there is nothing to do. If it is still PENDING, the charge is repeated with the same key and the completion runs. Many provider APIs accept such a key; Stripe's API, for example, takes an Idempotency-Key header, compares the parameters of a repeated request with the original, and may prune keys once they are at least 24 hours old. Check your provider's rules, because a retry that arrives after the key was forgotten is a new charge.

4. Processed, but the offset was not committed

An offset commit records the position of the next record to read for a group and partition. It is a separate write to Kafka, made after your database transaction. If the process dies between the two, the group resumes from the last committed position and the record arrives again, possibly on another instance. Under this design the second pass finds a final payment row and returns without side effects, and the offset is committed this time. The redelivery costs one indexed lookup.

Processed, crashed, redelivered

Simulation · not a real cluster

At-least-once with a crash between processing and the offset commit. Run it with the idempotent consumer on and compare with it off.

5. Records that should not simply be retried or skipped

SituationExampleWhat to do
Transient failureProvider timeout, database connection lostThrow. The error handler redelivers with backoff, and the same idempotency key makes the repeat safe.
Business outcomeCard declinedNot an error. Record DECLINED, publish the result, commit. Retrying would only repeat the decline.
Exact duplicateSame order, same amount, payment already finalSkip quietly and let the offset advance. Count it in a metric.
Conflicting duplicateSame order id, different amount or currencyDo not skip and do not process. Fail without retries and send the record to the dead-letter topic for a person to look at.
Poison recordPayload cannot be deserialised or fails validationNo retry can fix it. Dead-letter it so the partition is not blocked behind it.
Uncertain resultThe charge request timed outNever invent a new key. Retry with the stored key until the provider gives a definite answer.
Idempotency handles exact repeats. The other rows need an explicit decision.

Architecture

Drawing diagram…

Flowchart of a payment consumer. It looks up the payment row for the order and either inserts a pending row, charges the provider using the payment id as idempotency key, skips an exact duplicate or dead-letters a conflicting one. A successful charge leads to one database transaction that updates the status and inserts an outbox row.

flowchart TD
  IN["orders.order-placed, key orderId"] --> L["OrderPlacedListener"]
  L --> C{"Payment row for this order"}
  C -->|"none"| INS["Insert PENDING with new payment_id"]
  C -->|"PENDING"| CH["Charge provider, key is payment_id"]
  C -->|"final, same amount"| SKIP["Skip, exact duplicate"]
  C -->|"different amount"| DLT["Dead-letter topic, no retry"]
  INS --> CH
  CH -->|"approved or declined"| TX["One DB transaction"]
  CH -->|"timeout"| RETRY["Throw, redeliver with backoff"]
  RETRY --> L
  TX --> UPD["Update status where PENDING"]
  TX --> OBX["Insert outbox row, event id is payment_id"]
  OBX --> REL["Relay publishes payments.payment-result"]
  SKIP --> ACK["Offset advances"]
  UPD --> ACK
The payment consumer as a state machine: every path is safe to walk twice.
  • The decision node reads state from the database, not from memory, so it gives the same answer on any instance after any restart.
  • The provider call sits between two short database transactions. No database connection or lock is held while waiting on the network.
  • The status update and the outbox insert share a transaction, so a result event exists exactly when the payment reached a final state.
  • The offset advances only on paths that end in a committed, final state or in a dead-lettered record. A timeout loops back and holds the offset where it is.

Message flow

Drawing diagram…

Sequence diagram. The payment service receives OrderPlaced at offset 42, inserts a pending payment, charges the provider with an idempotency key, and commits the captured status with an outbox row. It crashes before committing the offset. The record is delivered again, the insert conflicts, the row is read as captured, nothing else happens and offset 43 is committed.

sequenceDiagram
  participant K as Kafka
  participant P as Payment Service
  participant D as Payments DB
  participant C as Card Provider
  K->>P: OrderPlaced order 7001 at offset 42
  P->>D: INSERT payment PENDING, unique order_id
  P->>C: charge with Idempotency-Key pay-91
  C-->>P: approved, ref ch_5
  P->>D: UPDATE to CAPTURED and INSERT outbox, one commit
  Note over P: crash before the offset commit
  K->>P: OrderPlaced order 7001 at offset 42 again
  P->>D: INSERT payment, conflict, 0 rows
  P->>D: SELECT payment for order 7001
  D-->>P: CAPTURED
  Note over P: exact duplicate, no charge, no new event
  P->>K: commit offset 43
A record that was processed but not committed is redelivered and changes nothing.
  1. First pass: three durable steps in order. Claim in the database, charge at the provider, complete in the database together with the outbox row.
  2. The crash loses only the offset commit. Everything that matters to the customer is already durable.
  3. Second pass: the unique constraint turns the insert into a no-op and the read shows a final state. The service returns normally, which is what lets the offset finally move to 43.
  4. Had the crash come after the charge and before the completing transaction, the second pass would find PENDING, repeat the charge with key pay-91, receive the original approval and complete.

In a real system

A typical shop checkout, the same system as the example project. Topics are keyed by orderId so that one order's events stay in order within a partition.

HopWhat can repeatWhat makes the effect single
Browser to Order Service, POST /ordersDouble click, client retry after a timeoutIdempotency-Key header stored under a unique constraint on orders
Order Service to orders.order-placed (12 partitions)Relay re-sends an outbox roweventId created with the order and stored in the outbox row
orders.order-placed to Payment Service, group payment-serviceRedelivery after a crash, rebalance or failed commitUNIQUE (order_id) on payments and the status check
Payment Service to card providerRetry after a timeout, redelivery while PENDINGpayment_id sent as the idempotency key on every attempt
Payment Service to payments.payment-resultRelay re-sendsOutbox row whose event id is the payment_id
payments.payment-result to Order Service, group order-serviceRedeliveryprocessed_events row in the same transaction as the order status update
Order Service to Notification ServiceRedeliveryUnique (order_id, template) on sent notifications, as far as the mail provider allows
One order, seven hops, and a named mechanism at each.

The last row is honest about a limit: an email provider without idempotency keys can still send twice if the service crashes after the call and before recording it. Decide per effect how much that matters. A duplicate confirmation email is tolerable; a duplicate capture is not, which is why the payment hop gets the full state machine.

Java and Spring Boot

The payment service's consumer with JdbcTemplate and explicit transaction boundaries, the order service's simpler consumer with a processed-events table, the SQL for both and the error handling. The provider's HTTP API is invented for the example: map your provider's real responses.

payment-service/src/main/resources/schema.sqlSQL
CREATE TABLE payments (    payment_id      uuid PRIMARY KEY,           -- also the provider idempotency key    order_id        uuid         NOT NULL,    amount_minor    bigint       NOT NULL,    currency        varchar(3)   NOT NULL,    status          varchar(16)  NOT NULL                    CHECK (status IN ('PENDING', 'CAPTURED', 'DECLINED')),    provider_ref    varchar(64),    source_event_id uuid         NOT NULL,      -- the OrderPlaced that created the row    created_at      timestamptz  NOT NULL DEFAULT now(),    updated_at      timestamptz  NOT NULL DEFAULT now(),    CONSTRAINT uq_payments_order UNIQUE (order_id));CREATE TABLE outbox (    id           bigserial PRIMARY KEY,    event_id     uuid         NOT NULL UNIQUE,    topic        varchar(128) NOT NULL,    record_key   varchar(128) NOT NULL,    payload      text         NOT NULL,    created_at   timestamptz  NOT NULL DEFAULT now(),    published_at timestamptz);-- In the order service's database: generic dedupe for consumers whose-- whole effect is a local write.CREATE TABLE processed_events (    consumer     varchar(64) NOT NULL,    event_id     uuid        NOT NULL,    processed_at timestamptz NOT NULL DEFAULT now(),    PRIMARY KEY (consumer, event_id));

Why it is written this way

  • uq_payments_order is the guarantee. Application checks are an optimisation; the constraint is what holds when two instances race.
  • payment_id is generated once, stored before the provider is called, and reused as the idempotency key. A key generated per attempt would defeat the provider's deduplication.
  • outbox.event_id is unique and set to the payment_id, so even a bug that completes a payment twice cannot enqueue two result events.
  • processed_events is keyed by consumer and event id so several consumers can share the table. Delete old rows only after the longest period in which a record could still be redelivered or replayed, which is at least the topic's retention if you ever reset offsets.

Configuration

No setting below makes processing effectively-once. They decide how often repeats happen and how failures are routed. Kafka defaults are from the 4.1 configuration reference, Spring behaviour from the Spring for Apache Kafka 3.3 reference.

enable.auto.commit (consumer)Default true (Spring for Apache Kafka sets it to false unless you set it explicitly)
What it does
Whether the client commits offsets on a timer regardless of what your code has finished.
When to change it
Keep it off so the listener container commits only after the listener returns. A timed commit can pass a record whose processing then fails, which loses the payment instead of repeating it.
spring.kafka.listener.ack-modeDefault BATCH (the container's default AckMode)
What it does
When the container commits: BATCH after all records of a poll are processed, RECORD after each one, MANUAL when your code says so.
When to change it
RECORD reduces how many records are redelivered after a crash at the cost of more commit requests. Choose by volume. None of the modes removes the need for idempotent processing.
max.poll.interval.ms (consumer)Default 300000 (5 minutes)
What it does
The longest allowed gap between two calls to poll before the consumer is considered failed and its partitions are reassigned.
When to change it
Processing of one polled batch, including provider calls and blocking retries, must fit inside it. Otherwise the record is handed to another instance while the first is still working on it.
max.poll.records (consumer)Default 500
What it does
The maximum number of records one poll returns.
When to change it
Lower it when each record involves a remote call. 500 charges at 1 second each do not fit in 5 minutes.
enable.idempotence (producer)Default true
What it does
The broker discards duplicate appends caused by this producer's own retries, per partition, within one producer session.
When to change it
Leave it on. Do not count it as part of this design: a relay or an application that sends the same event twice produces two records, and nothing a consumer does is covered.
isolation.level (consumer)Default read_uncommitted
What it does
Whether the consumer also receives records from Kafka transactions that are open or were aborted.
When to change it
Set read_committed if any upstream producer writes the topic transactionally. It is unrelated to deduplication but stops you acting on events that were rolled back.

Hands-on exercise

Charge once, deliver twice

Reproduce the processed-but-not-committed redelivery, count the charges with and without an idempotent consumer, then do the same against the lab's simulated payment system.

Steps · 0 of 6 done

At-least-once with a crash before the commit

Simulation · not a real cluster

The record is processed, the process dies, the offset was never committed. Toggle the idempotent consumer and compare the number of charges.

Take it further in the lab: Duplicate message processing

What can go wrong

Dedupe check outside the business transaction
Symptom
Rare duplicate payments, or rare payments that are skipped entirely, both under load or during restarts.
Cause
The event id is recorded in a cache, in another transaction or before the work. A crash between the two leaves the work done but unrecorded, or recorded but not done.
Fix
Insert the dedupe row or rely on the unique business key inside the same database transaction as the write it protects.
A new idempotency key on every attempt
Symptom
The provider's dashboard shows two captures for one order, a few seconds apart, after a timeout.
Cause
The key is generated in the HTTP client or per delivery, so the provider sees two unrelated requests.
Fix
Generate the key once, store it with the pending payment before the first call, and send the stored value on every attempt.
Treating producer idempotence as deduplication
Symptom
Consumers receive the same eventId twice although enable.idempotence=true on the producer.
Cause
The outbox relay or the application called send twice for the same event. To the broker these are two distinct records; idempotence only covers the client's internal retries of one send.
Fix
Keep producer idempotence on, and deduplicate in the consumer on the event id or business key.
Dedupe on topic, partition and offset
Symptom
Duplicates slip through after the relay republished, after a replay from another cluster, or after a topic was recreated.
Cause
The offset identifies a position in one log, not a business fact. The same fact published twice has two offsets.
Fix
Deduplicate on an id that was created with the business fact. Reserve offset-based tracking for the store-offsets-with-the-data design.
Silently skipping a conflicting duplicate
Symptom
An order was changed from 40.00 to 55.00, but the customer was charged 40.00 and nobody noticed for weeks.
Cause
The consumer saw an existing payment for the order and skipped the second event without comparing its content.
Fix
Compare the fields that matter. On a mismatch, fail without retries, dead-letter the record and alert.
Dedupe rows deleted too early
Symptom
After an offset reset to replay a week of data, old payments are processed again.
Cause
processed_events rows are purged after a day, while records stay in the topic for seven.
Fix
Keep dedupe rows at least as long as a record can be redelivered or replayed, or rely on a permanent business key such as UNIQUE (order_id).

How to troubleshoot

Suspected double charges

Check

SELECT order_id, count(*) FROM payments GROUP BY order_id HAVING count(*) > 1;
How to read it
Any row means the unique constraint is missing or was dropped. If the table is clean but the provider shows two captures, the idempotency key differed between attempts: compare the keys in the provider's request log.
Payments stuck in PENDING

Check

SELECT payment_id, order_id, created_at FROM payments WHERE status = 'PENDING' AND created_at < now() - interval '10 minutes';
How to read it
The claim committed and completion never did. Either the record is still being retried (look at lag) or it was dead-lettered after the provider kept failing. Replaying it is safe, because the key is stored.
The same records keep being processed

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
A CURRENT-OFFSET that does not advance while the log shows the same event ids means processing finishes but commits fail, often after a rebalance caused by exceeding max.poll.interval.ms. Duplicates are being absorbed by the design, but throughput is zero.
Records are missing downstream

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-placed-dlt --from-beginning
How to read it
Records here were given up on. A conflicting duplicate needs a human decision. A provider outage that outlasted the retries can be replayed once the provider is back.
How often duplicates actually arrive

Check

Application log line: Duplicate OrderPlaced ... payment ... is already CAPTURED
How to read it
A steady trickle around deploys and rebalances is normal. A sudden flood points at a relay republishing a backlog or a group whose commits are failing.

Key takeaways

  • Effectively-once is a property of your design: at-least-once delivery plus effects that recognise their own repeat. Kafka does not provide it for databases or external calls.
  • Create the identity once, with the business fact, and carry it in every copy: API idempotency key, event id, payment id.
  • The dedupe record or unique constraint must commit in the same database transaction as the write it protects.
  • An external call needs a stored key and a state machine: claim, call with the key, complete. Each step is safe to repeat.
  • A record that was processed but not committed will come back. The correct response is to find the finished state and return normally.
  • Exact duplicates are skipped, business outcomes are recorded, uncertain results are retried with the same key, and conflicting or unreadable records go to a dead-letter topic.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A payment consumer inserts the event id into processed_events and commits. Then it calls the card provider, then updates payments in a second transaction. During a deploy, 4 orders end up with a processed_events row, no charge at the provider and status PENDING. Kafka shows no lag for the group.

What is wrong with the design?

Scenario 2 of 3

A charge request times out after 5 seconds. The listener catches the timeout, generates a fresh UUID as idempotency key and tries again; the second attempt succeeds. Next morning finance finds 11 customers charged twice, all from a 3-minute window in which the provider was slow.

Which change prevents this?

Scenario 3 of 3

payments has UNIQUE (order_id). An OrderPlaced for order 9001 arrives with amount 4000 and is captured. Ten minutes later another OrderPlaced for order 9001 arrives with a different event id and amount 5500. The consumer logs it as a duplicate and skips it.

How should the consumer treat the second event?

References

References: Apache Kafka documentation: Design, message delivery semantics ↗ · KafkaProducer Javadoc (4.1): what the idempotent producer does and does not cover ↗ · Spring for Apache Kafka 3.3 reference: Exactly Once Semantics ↗ · Spring for Apache Kafka 3.3 reference: Message listener containers (committing offsets, AckMode) ↗ · Spring for Apache Kafka 3.3 reference: Handling exceptions (DefaultErrorHandler, dead-letter publishing) ↗ · Spring for Apache Kafka 3.3 reference: Examples of Kafka transactions with other transaction managers ↗ · Apache Kafka 4.1 consumer configuration reference ↗ · Stripe API reference: Idempotent requests ↗ · Report a problem