Skip to content

Path 5 · lesson 1 of 5

Intermediate22 minApache Kafka 4.x (defaults checked against the 4.4 documentation) · Java 21 · Spring Boot 3.x with Spring for Apache Kafka 3.3

At-most-once, at-least-once and exactly-once

The three delivery guarantees, defined by what a crash between processing and committing does, and which part of the pipeline each one covers.

What you will learn

  • Work out which guarantee a consumer has from the order of its process and commit steps
  • Say for any guarantee which hop it covers: producer to broker, broker to consumer, or the whole business flow
  • Predict the outcome of a crash at each point of a payment or order flow: nothing, a lost record, or a repeated one
  • Choose producer and consumer settings that give at-least-once, and recognise settings that silently give at-most-once
  • Explain why at-least-once delivery with idempotent handling is the normal design for money and orders

Before you start

The concept

A delivery guarantee is a statement about failures. When nothing fails, every design delivers each record once. The guarantees only differ in what happens when a process dies or a network call is lost between two steps that cannot be done at the same instant.

For a consumer those two steps are process (do the work: charge the card, reserve the stock) and commit (record in Kafka that the group has moved past this record). A committed offset is the position of the next record the group will read from a partition. It is a bookmark for the whole partition, not an acknowledgement of one record.

GuaranteeOrder of stepsCrash between the two stepsWhat the business sees
At-most-onceCommit, then processThe bookmark has already moved. The restarted consumer starts after the record.The record is never processed. A payment is silently skipped.
At-least-onceProcess, then commitThe bookmark has not moved. The restarted consumer reads the record again.The record is processed twice unless the handler can tell it has seen it.
Exactly-onceProcess and commit as one atomic stepEither both happened or neither did.One effect. Only possible when the result and the position are stored by the same system.
The names describe how many times a record can be processed when a failure lands between the steps.

A pipeline has more than one hop, and each hop has its own guarantee. Mixing them up is the usual source of wrong claims such as "we enabled idempotence, so we have exactly-once".

HopThe two steps that can be split by a failureWhat decides the guarantee
Producer to brokerThe broker appends the record, and the producer learns that it didacks, retries / delivery.timeout.ms, enable.idempotence
Broker to consumerThe consumer processes the record, and the group's offset is committedWhether the commit happens before or after processing
End to endA business action happens, and every later stage has taken effect exactly onceThe weakest hop, plus whether each handler is idempotent or transactional

Why it exists

The three guarantees exist because of one unavoidable fact: when a sender does not get a reply, it cannot tell whether the request was lost or the reply was lost. A producer that times out waiting for an acknowledgement does not know if the record is in the log. A consumer group that sees a member disappear does not know if that member finished its last record.

There are only two things to do in that situation. Do not retry, and accept that the work may never have happened: at-most-once. Retry, and accept that the work may now happen twice: at-least-once. Everything called exactly-once is at-least-once delivery with a mechanism that makes the repeat harmless.

A lost record

  • Nothing downstream knows it existed, so nothing can detect it
  • An order is paid for and never shipped, or shipped and never charged
  • Found days later by reconciliation, or by a customer
  • Recovering means replaying from a point in time and hoping the handlers tolerate it

A repeated record

  • The handler holds the record, with its identifier, when it arrives the second time
  • It can check whether that identifier was already applied and skip it
  • The check is local, cheap and under your control
  • Handlers that tolerate repeats also make deliberate replays safe

That asymmetry is why at-least-once plus idempotent handling is the default design for payments and orders. A duplicate can be recognised at the moment it arrives. A loss cannot. At-most-once still has a place, but only for data where one missing record does not matter: a metrics sample, a cache warm-up hint, a "user is typing" signal.

How it works inside

Producer to broker

The producer sends a batch and waits for the partition leader's response. With acks=all (the default) the leader replies once every replica that is currently in sync has the batch. If the response is lost, the producer retries until delivery.timeout.ms expires. Without further protection the retry can append the same batch a second time, which is at-least-once at this hop.

With enable.idempotence=true (the default since Kafka 3.0) the broker gives the producer an id, and the producer numbers the batches it sends to each partition. A retried batch carries the sequence number the leader already has, so the leader acknowledges it without appending it again. This hop becomes exactly-once for that producer's retries within one producer session. It says nothing about the consumer, and nothing about your application calling send twice for the same business event.

  • acks=0, or retries disabled: a failed send is simply gone. At-most-once.
  • acks=all with retries and no idempotence: no loss, but possible duplicates and reordering. At-least-once.
  • acks=all with idempotence (the 4.x defaults): no loss and no duplicates from retries, as long as the topic's replication factor and min.insync.replicas leave room for a broker failure.

Broker to consumer

A consumer keeps an in-memory position per partition and advances it as poll() returns records. Separately, the group has a committed offset stored in the internal __consumer_offsets topic. After a crash or a rebalance, whichever consumer owns the partition next starts from the committed offset. Everything between the committed offset and the point the old consumer had reached is read again. Everything before it is never read again, whether or not it was processed.

  • Commit as soon as records arrive, then process them: a crash during processing loses the rest of that batch. At-most-once.
  • Process every record from a poll, then commit: a crash before the commit repeats the batch. At-least-once.
  • enable.auto.commit=true commits on a timer from inside poll(). It is at-least-once only if every record from the previous poll was fully processed before the next poll() call. Hand records to another thread and it turns into at-most-once without any error.

End to end

End-to-end exactly-once needs the result of processing and the consumer's position to change together. Kafka can do that when the result is itself a write to Kafka: a transaction puts the output records and the offset commit in one atomic unit (see Kafka transactions). When the result is a row in a database, a call to a payment provider or an email, Kafka cannot include it. Then you make the handler idempotent, so that the repeat that at-least-once will eventually deliver changes nothing (see Duplicates and idempotency).

Move the crash, watch the outcome

Simulation · not a real cluster

A payment consumer crashes after it has charged the card and before the offset is committed. Step through it, then change the mode and the crash point.

Architecture

Drawing diagram…

Flowchart of an order service producing to a partition leader that replicates to followers, and a payment consumer that first writes to a payments database and then commits the next offset to the group offsets topic. The pipeline is grouped into the producer to broker hop, the broker to consumer hop, and the database outside Kafka.

flowchart LR
  OS["Order Service"] -->|"send with acks=all"| L["Partition leader"]
  L -->|"replicate"| F["In-sync followers"]
  L -->|"ack, may be lost"| OS
  L -->|"poll"| PS["Payment consumer"]
  PS -->|"1 process"| DB["Payments database"]
  PS -->|"2 commit next offset"| CO["Group offsets topic"]
  subgraph H1["Hop 1 producer to broker"]
    OS
    L
    F
  end
  subgraph H2["Hop 2 broker to consumer"]
    PS
    CO
  end
  subgraph H3["Outside Kafka"]
    DB
  end
Three places a failure can split two steps: the produce acknowledgement, process versus commit, and the write to a system Kafka does not control.
  • Hop 1 fails when the acknowledgement is lost. The producer retries. Idempotence stops the retry from being appended a second time.
  • Hop 2 fails when the consumer stops between step 1 and step 2. Which step comes first decides whether the record is lost or repeated.
  • The database is outside both hops. No Kafka setting reaches it, so the handler has to cope with a repeated record on its own.
  • The end-to-end guarantee is the weakest of the hops. Exactly-once on hop 1 does not repair an at-most-once consumer on hop 2.

Message flow

Drawing diagram…

Sequence diagram. Consumer A polls offset 41 for order 7001, inserts a payment into the database and crashes before committing the offset. The group coordinator reassigns the partition to consumer B, which fetches committed offset 41, reads the same record again, inserts a second payment unless the handler is idempotent, and commits offset 42.

sequenceDiagram
  participant K as Kafka partition
  participant C1 as Payment consumer A
  participant DB as Payments database
  participant G as Group coordinator
  participant C2 as Payment consumer B
  C1->>K: poll
  K-->>C1: offset 41 order 7001
  C1->>DB: insert payment for order 7001
  DB-->>C1: committed
  Note over C1: crash before the offset commit
  G->>C2: partition reassigned
  C2->>G: fetch committed offset
  G-->>C2: 41
  C2->>K: poll from 41
  K-->>C2: offset 41 order 7001 again
  C2->>DB: insert payment for order 7001
  Note over C2,DB: second charge unless the handler is idempotent
  C2->>G: commit offset 42
At-least-once: the payment is written, the consumer dies before committing, and the next owner of the partition reads offset 41 again.
  1. The committed offset is still 41 because consumer A never reached the commit. Kafka has no record that order 7001 was handled.
  2. Consumer B does exactly what it should: it starts from the committed offset. From Kafka's side nothing went wrong.
  3. Swap the two steps in consumer A and the same crash produces the opposite failure: offset 42 is committed, the payment is never written, and no consumer reads offset 41 again.
  4. The only place the repeat can be stopped is the insert. The database can refuse a second payment for order 7001. Kafka cannot.

In a real system

Take a typical online shop. The Order Service publishes an OrderPlaced event to the topic orders.placed, keyed by order id, with 12 partitions and replication factor 3. The Payment Service reads it in consumer group payment-service, charges the customer and writes a row to its payments table. A separate Analytics consumer in group order-analytics counts orders per minute for a dashboard.

StageGuarantee to aim forWhy
Order Service to orders.placedNo loss, no duplicate from retriesProducer defaults: acks=all, idempotence on. Topic with replication factor 3 and min.insync.replicas=2.
orders.placed to Payment ServiceAt-least-once, idempotent handlerA skipped payment ships goods for free. A repeated one is caught by a unique constraint on the order id.
orders.placed to AnalyticsAt-least-once, or at-most-once if simplerA dashboard that is off by one order for a minute after a crash costs nothing.
Payment Service to the card providerAt-least-once with an idempotency keyThe provider is outside Kafka. The key lets it recognise the same charge request sent twice.

Both consumer groups read the same partitions and each has its own committed offsets, so each can make its own choice. The guarantee belongs to a group and its handler, not to the topic.

Offsets & at-least-once

Simulation, not a real cluster

The animated Build lesson shows the same process-then-commit window on a partition, one record at a time.

Step 1 · The pattern

Read, process, remember your place

The consumer reads a message, does the work (here, sending an email) and periodically commits its offset: a bookmark saying 'I have handled everything up to here'.

  • message
  • processed
Open the full animated lesson

Java and Spring Boot

A plain Kafka consumer makes the ordering of the two steps explicit, so it is the clearest way to see the guarantee. The Spring listener below it gets the same at-least-once behaviour from the container.

src/main/java/com/example/payments/PaymentPollLoop.javaJava
package com.example.payments;import java.time.Duration;import java.util.List;import java.util.Properties;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.KafkaConsumer;import org.apache.kafka.common.errors.WakeupException;import org.apache.kafka.common.serialization.StringDeserializer;public final class PaymentPollLoop implements Runnable {    private final KafkaConsumer<String, String> consumer;    private final PaymentHandler handler;    public PaymentPollLoop(String bootstrapServers, PaymentHandler handler) {        Properties props = new Properties();        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);        props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-service");        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());        this.consumer = new KafkaConsumer<>(props);        this.handler = handler;    }    @Override    public void run() {        try {            consumer.subscribe(List.of("orders.placed"));            while (true) {                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));                // AT-LEAST-ONCE: process first ...                for (ConsumerRecord<String, String> record : records) {                    handler.handle(record.key(), record.value());                }                // ... then commit. A crash on any line above leaves the                // committed offset where it was, so the batch is read again.                if (!records.isEmpty()) {                    consumer.commitSync();                }                // AT-MOST-ONCE would be the same two blocks in the other order:                // commitSync() first, then the for loop. A crash inside the loop                // would then skip every record that had not been handled yet.            }        } catch (WakeupException shutdown) {            // thrown by poll() after another thread calls consumer.wakeup()        } finally {            consumer.close();        }    }    public void stop() {        consumer.wakeup();    }    /** Must tolerate being called twice with the same order. */    public interface PaymentHandler {        void handle(String orderId, String orderPlacedJson);    }}

Why it is written this way

  • enable.auto.commit=false puts the commit under your control. The Kafka default is true, which commits on a timer from inside poll().
  • commitSync() with no arguments commits the positions reached by the last poll(), which is the offset of the next record to read for each assigned partition.
  • The handler runs before the commit, so an exception or a crash repeats the batch instead of skipping it. The interface comment states the consequence: it must be safe to call twice.
  • auto.offset.reset=earliest only applies when the group has no committed offset at all. With latest (the Kafka default) a brand-new group starts at the end and never sees older records, which looks like loss but is a configuration choice.

Configuration

These settings decide the guarantee on each hop. Defaults are those documented for Apache Kafka 4.4 clients and for Spring for Apache Kafka 3.3.

acksDefault all
What it does
How many acknowledgements the leader needs before a produce request counts as done. all waits for every replica that is in sync at that moment.
When to change it
Keep all for orders and payments. 0 or 1 trade possible loss for latency, and any value other than all is incompatible with idempotence.
enable.idempotenceDefault true
What it does
Makes the producer's own retries safe: the broker drops a batch it has already appended from this producer for this partition.
When to change it
Leave it on. It is silently disabled if you set a conflicting value (acks other than all, retries=0, or max.in.flight.requests.per.connection above 5) without setting it explicitly, so set it explicitly.
delivery.timeout.msDefault 120000 (2 minutes)
What it does
Upper bound on the time from send() returning to success or failure being reported, including retries.
When to change it
This, not retries, is the knob for how long the producer keeps trying. When it expires the send fails and the application has to decide what to do with the event.
enable.auto.commitDefault true (Spring for Apache Kafka sets false unless you configure it)
What it does
Commits the consumer's position in the background on a timer, from inside poll().
When to change it
Set false and commit after processing whenever a lost record matters. Auto-commit is only safe if all work for a poll is finished before the next poll.
auto.commit.interval.msDefault 5000 (5 seconds)
What it does
How often offsets are auto-committed when enable.auto.commit is true.
When to change it
Lowering it shrinks the replay window after a crash but does not change the guarantee. It has no effect with manual or container-managed commits.
spring.kafka.listener.ack-modeDefault batch
What it does
When the Spring listener container commits: record after each record, batch after all records of a poll, manual / manual-immediate when the listener calls Acknowledgment.acknowledge().
When to change it
Use record when repeats are expensive, batch for throughput. All of them are at-least-once as long as the commit follows the work.
isolation.levelDefault read_uncommitted
What it does
Whether the consumer returns records from transactions that are still open or were aborted.
When to change it
Set read_committed on any consumer that reads a topic written by a transactional producer. Without it, exactly-once on the producing side does not reach the reader.

Hands-on exercise

Find the guarantee by moving the crash

Use the delivery simulator to see that the guarantee is decided by where the commit sits relative to the work, and that an idempotent handler is what turns a repeat into a single charge.

Steps · 0 of 5 done

Delivery simulator

Simulation · not a real cluster

Choose the commit order and where the consumer crashes, then step through what the restarted consumer reads.

Take it further in the lab: Consumer crash before offset commit

What can go wrong

Auto-commit with work handed to another thread
Symptom
After a deploy or a crash a handful of orders have no payment. There are no errors in the logs and consumer lag is zero.
Cause
enable.auto.commit=true committed the offsets of records that had been passed to an executor and were still queued when the process stopped. The commit ran ahead of the work, which is at-most-once.
Fix
Disable auto-commit and commit only after the work is done. In Spring, do the work on the listener thread, or use manual acknowledgement and acknowledge when the task completes.
Reading "idempotence is on" as "no duplicates anywhere"
Symptom
A customer is charged twice after a consumer restart even though the producer has enable.idempotence=true.
Cause
Producer idempotence covers the producer-to-broker hop. The duplicate came from the consumer hop: the record was processed, the commit did not happen, and it was delivered again.
Fix
Make the payment handler idempotent with a unique business key, as in Duplicates and idempotency.
Catching the exception and carrying on
Symptom
Records that failed to process never show up again and there is no dead letter record for them.
Cause
The handler logged the exception and returned normally, so the offset was committed past a record that was never processed. That is at-most-once for every failure.
Fix
Let the exception reach the error handler so the record is retried and then routed to a dead letter topic, or publish it there yourself before returning.
Durability assumed from acks=all alone
Symptom
Acknowledged order events are missing after a broker failure.
Cause
acks=all means all replicas in sync at that moment. With min.insync.replicas=1 (the default) that set can shrink to the leader alone, and a record acknowledged by one broker is lost with it.
Fix
Use replication factor 3 with min.insync.replicas=2 on topics that carry money or orders, so an acknowledgement always means at least two copies.

How to troubleshoot

Records were processed twice after a restart or a rebalance

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
Compare CURRENT-OFFSET with what your service had actually processed. A committed offset behind the processed position is the at-least-once window, and its size is how many records were repeated. Shrink it with more frequent commits, and make the handler idempotent because it can never reach zero.
Records are missing downstream but present in the topic

Check

Compare the committed offset from the same --describe output with the offset of the missing record, and look for enable.auto.commit=true in the ConsumerConfig values logged at startup
How to read it
A committed offset beyond a record that was never processed means the commit ran before the work: auto-commit with asynchronous processing, a swallowed exception, or commit-then-process code.
A new consumer group never sees records that were produced before it started

Check

auto.offset.reset in the consumer configuration, and whether the group had any committed offset: kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-analytics
How to read it
With no committed offset the consumer starts where auto.offset.reset says. latest skips everything already in the log. That is a starting position, not a delivery failure.
The same record appears twice in the topic itself

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.placed --partition 3 --offset 1200 --max-messages 20 --formatter-property print.offset=true --formatter-property print.key=true
How to read it
(On clients before Kafka 4.2 the option is --property.) Two offsets with the same payload mean the duplicate was created before the consumer: a producer without idempotence retried, or the application sent the event twice. Fix the producing side, and still deduplicate in the consumer.

Key takeaways

  • A guarantee describes what a failure between two steps does. Commit then process loses records. Process then commit repeats them.
  • A committed offset is the next position to read for a group and partition. It does not acknowledge individual records.
  • Each hop has its own guarantee: producer to broker (acks, retries, idempotence), broker to consumer (commit order), and the business effect in systems outside Kafka.
  • Producer idempotence fixes duplicates from the producer's retries only. It does nothing for what the consumer does with a record.
  • Exactly-once in Kafka is exactly-once effect on Kafka state. For a database or an external API it has to come from idempotent handling.
  • A duplicate can be detected when it arrives and a loss cannot, so the normal design is at-least-once delivery with idempotent handlers.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A Payment Service listener hands each OrderPlaced record to a thread pool and returns at once. enable.auto.commit is true with the 5 second interval. During a rolling restart, 37 orders end up with no payment row. The topic still contains all 37 events and the group shows no lag.

What should change first?

Scenario 2 of 3

A team reports: "Our producer has enable.idempotence=true and acks=all, so payments are exactly-once." Their consumer processes a record by inserting into payments and then the container commits the offset. Last week a pod was killed and one order got two payment rows.

Which statement about their setup is correct?

Scenario 3 of 3

You own two consumers of orders.placed. One updates a live "orders per minute" tile on an internal dashboard. The other reserves stock in the inventory database. Both currently process and then commit. Someone proposes switching both to commit-then-process "to stop duplicates for good".

What is the right response?

References

References: Apache Kafka documentation: Design, Message Delivery Semantics ↗ · Apache Kafka documentation: Producer configs (acks, enable.idempotence, delivery.timeout.ms) ↗ · Apache Kafka documentation: Consumer configs (enable.auto.commit, auto.commit.interval.ms, isolation.level) ↗ · KafkaProducer Javadoc: idempotent and transactional producer ↗ · Spring for Apache Kafka reference: Message listener containers, committing offsets ↗ · Report a problem