Skip to content

Kafka / streaming · 4 of 9

Step 1 · The pattern

Read, process, remember your place

The consumer reads a message, does the work (here, sending an email) and periodically commits its offset: a bookmark saying 'I have handled everything up to here'.

  • message
  • processed

In short

Offsets & at-least-once: how it works and how it breaks

A Kafka consumer keeps its place with an offset: the position of the next message it should read. It saves, or commits, that offset back to Kafka from time to time. Processing a message and committing its offset are two separate steps, so a crash can always land between them. If you commit after processing, a crash means the last few messages are processed again on restart: at-least-once delivery. If you commit before processing, a crash means those messages are never processed: at-most-once. Losing messages is usually worse, so most systems choose at-least-once and then make duplicates harmless. Give every message an ID and record the IDs you have handled, in the same transaction as the work. Or make the work naturally repeatable, like setting a value instead of adding to it. Kafka's transactions give exactly-once only for work that stays inside Kafka.

In Spring Boot

Commit after processing in Spring Kafka

application.properties
# the listener container commits offsets, not the Kafka client
spring.kafka.consumer.enable-auto-commit=false
# commit after each record the listener finishes (default: batch)
spring.kafka.listener.ack-mode=record
# only matters when producers use transactions
spring.kafka.consumer.isolation-level=read-committed

Spring Kafka turns the client's auto-commit off and commits for you once the listener returns. With record mode that is after every record; the default, batch, commits after everything from one poll has been processed. read-committed hides records from transactions that were aborted.

Gotcha: Both modes commit after processing, so both are at-least-once: a crash before the commit replays the record, or with batch mode the whole poll. The listener has to be safe to run twice on the same record.

How this comes up in interviews

"A consumer sends an email for each message. After a crash some customers got the same email twice. Why, and how do you prevent it?"

Then they ask

  1. A teammate suggests committing the offset before processing. What failure does that swap in?
  2. Where exactly do you record the handled message IDs so that the check itself cannot be skipped by a crash?
  3. Kafka offers exactly-once transactions. Why do they not solve the duplicate email?

A strong answer covers

  • Processing a message and committing its offset are separate steps, so a crash can land between them.
  • Commit after processing gives at-least-once: the last few messages repeat on restart.
  • Commit before processing gives at-most-once: a crash means those messages are never processed.
  • Choose at-least-once and make duplicates harmless: record message IDs in the same transaction as the work.
  • Kafka's transactions give exactly-once only for work that stays inside Kafka.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.