Path 4 · lesson 3 of 5
Offset management and commits
What a committed offset is, where it is stored, when automatic and manual commits happen, what Spring's ack modes do, and how to reset a group's position.
What you will learn
- State exactly what an offset commit stores and what it does not acknowledge
- Say when auto-commit fires and identify code where it can skip unprocessed records
- Choose between
commitSyncandcommitAsyncand combine them safely - Pick a Spring
AckModefor a listener from its documented behaviour - Predict where a consumer starts for a new group, an expired group and a restarted one
- Rewind or skip a consumer group with
kafka-consumer-groups.sh --reset-offsets
Before you start
The concept
A consumer has two offsets for every partition it reads. The position is the offset of the next record poll will return. It lives in the consumer's memory and advances every time poll hands out records. The committed offset is the position that has been stored durably for the group. It is the only one that survives a crash, a restart or a rebalance: whoever reads the partition next for this group starts from the committed offset.
A commit is a small write that says: for group G, topic T, partition P, the next record to read is offset N. It is one number per partition. It is not a list of handled records and it is not a per-record acknowledgement. Committing offset 47 claims that everything below 47 is done, whether or not that is true.
When you commit relative to processing decides the delivery guarantee. Commit before processing and a crash in between loses the records: at-most-once. Commit after processing and a crash in between repeats them: at-least-once. There is no ordering of the two steps that gives exactly-once on its own, because processing and committing are two separate writes to two systems.
Crash between processing and commit
Simulation · not a real clusterRun the payment consumer with the crash placed after processing and before the commit. Then move the commit before processing and crash again.
Why it exists
A traditional queue tracks each message on the broker: delivered, acknowledged, redeliver after a timeout. That is flexible, but the broker carries state for every in-flight message of every consumer. Kafka's log makes a much cheaper scheme possible. Records in a partition are read in order, so the progress of a group through a partition can be summarised by a single number. The broker stores one small value per group and partition, however many records have gone by.
Keeping that number in Kafka, rather than in each consumer's local disk, is what makes groups work. When a partition moves to another member during a rebalance, or a pod is replaced by a new one, the new reader asks the coordinator for the committed offset and continues from there. And because progress is only a number, it can be moved on purpose: rewind it to replay a day of events after fixing a bug, or advance it to skip a backlog nobody needs.
The cost of the single-number design is the callout above. A commit cannot express that one record in the middle failed. Handling individual failures needs something else: retries, a dead letter topic, or an idempotent consumer. Those are the subject of the reliability path.
How it works inside
Where commits are stored
Commits go to the group's coordinator, which appends them to the internal topic __consumer_offsets. That topic is compacted: the record key identifies the group, topic and partition, the value holds the offset with some metadata, and compaction keeps the latest value for each key. The coordinator also keeps the offsets in memory so that it can answer offset lookups quickly, and it acknowledges a commit only after the replicas of the offsets topic have it. By default the topic has 50 partitions (offsets.topic.num.partitions) and replication factor 3 (offsets.topic.replication.factor).
Committed offsets do not live for ever. Once a group has no members left, its offsets are removed after offsets.retention.minutes, 7 days by default. A consumer that comes back after that finds no committed offset and behaves like a brand-new group.
Automatic commits
With enable.auto.commit=true, which is the Kafka client default, the consumer commits its positions about every auto.commit.interval.ms (5 seconds). The commit is not made by an independent timer that understands your processing. In the classic consumer the check runs inside poll: if the interval has elapsed, the consumer commits the positions it has reached, meaning everything earlier polls returned. It also commits when the consumer is closed and before partitions are given up in a rebalance.
This gives at-least-once delivery on one condition, which the KafkaConsumer Javadoc spells out: you must finish processing everything a poll returned before you call poll again or close the consumer. If the loop processes each batch completely on the polling thread, a commit made by the next poll only ever covers finished work. After a crash, the records processed since the last auto-commit are delivered again: at-least-once with a window of up to the commit interval plus one batch.
Manual commits: sync and async
commitSync | commitAsync | |
|---|---|---|
| Blocks the poll thread | Yes, until success, an unrecoverable error, or default.api.timeout.ms | No |
| On failure | Throws to the caller | Passes the error to the callback if one was given, otherwise discards it |
| Typical cost | One round trip to the coordinator per call | Negligible for the caller |
| Use it for | The last commit before giving up a partition or shutting down | Routine commits after each batch |
A common combination is commitAsync after each batch, because a single failed commit is harmless when the next one supersedes it, and commitSync in onPartitionsRevoked and on shutdown, where there is no next one. Both also accept an explicit map of partition to OffsetAndMetadata, which lets you commit less than the whole batch, for example after each partition's records.
Only members of the current group generation may commit. A consumer that exceeded max.poll.interval.ms and was removed from the group gets CommitFailedException from commitSync. By then its partitions belong to someone else, and the work since the previous commit will be done again by the new owner.
Spring ack modes
Spring's listener container does its own committing. Since version 2.3 it sets enable.auto.commit to false unless you set it explicitly, and commits according to ContainerProperties.AckMode. The definitions below are those of the Spring for Apache Kafka reference, for a container not using transactions.
| AckMode | When the container commits |
|---|---|
BATCH (default) | When all the records returned by the poll have been processed |
RECORD | When the listener returns after processing each record |
TIME | When all records from the poll have been processed, provided ackTime has passed since the last commit |
COUNT | When all records from the poll have been processed, provided ackCount records have been received since the last commit |
COUNT_TIME | Like TIME and COUNT, when either condition is true |
MANUAL | The listener calls Acknowledgment.acknowledge(). After that the same semantics as BATCH apply: the commit is made once the records from the poll have been processed |
MANUAL_IMMEDIATE | Immediately when the listener calls Acknowledgment.acknowledge() |
syncCommits is true) and commitAsync when it is set to false.BATCH suits most listeners: one commit per poll, and a crash repeats at most one batch. RECORD narrows the repeat to one record at the price of a commit round trip per record. The manual modes are for when the listener itself must decide, for instance when it acknowledges only after an asynchronous step has completed. In every mode a commit still means everything below this offset is done.
When there is no committed offset
auto.offset.reset is consulted in exactly two situations: the group has no committed offset for the partition, or the committed offset no longer exists on the broker because retention deleted that part of the log. In every other start, restart or rebalance it is ignored and the committed offset is used. Changing it on a group that already has commits has no effect.
| Value | Start position when no valid committed offset exists |
|---|---|
latest (default) | The end of the log. Records written before the consumer first got the partition are skipped |
earliest | The oldest record still retained. The whole retained history is processed |
none | No automatic choice: the consumer throws an exception and you decide |
by_duration:<duration> | The offset for the current time minus an ISO-8601 duration, for example by_duration:PT1H. Listed in the 4.x consumer configuration; older clients do not accept it |
Moving the position on purpose
Inside a consumer, seek(partition, offset), seekToBeginning and seekToEnd change the in-memory position, and the next poll reads from there. In Spring, a listener that implements ConsumerSeekAware receives a callback for the same purpose. From outside, kafka-consumer-groups.sh --reset-offsets rewrites the committed offsets of a group. The documentation tells you to make sure the consumer instances are inactive first. The tool only displays the result unless you add --execute, since --dry-run is its default.
Offsets & at-least-once
Simulation, not a real clusterThe animated Build lesson walks through commit timing and a crash, record by record.
Step 1 · The pattern
Read, process, remember your place
The consumer reads a message, does the work (here, sending an email) and periodically commits its offset: a bookmark saying 'I have handled everything up to here'.
- message
- processed
Architecture
Drawing diagram…
Flowchart of a partition log divided into four ranges: offsets 0 to 41 processed and committed, 42 to 46 processed but not committed, 47 to 49 returned by poll and in progress, and 50 to 57 not yet returned. The committed offset points at 42 and the position at 50. The consumer sends a commit to the group coordinator, which appends it to the compacted __consumer_offsets topic, from which the next owner of the partition looks up where to resume.
flowchart TB
subgraph LOG["orders.events partition 2"]
R1["Offsets 0 to 41: processed and committed"]
R2["Offsets 42 to 46: processed, not yet committed"]
R3["Offsets 47 to 49: returned by poll, in progress"]
R4["Offsets 50 to 57: not yet returned"]
R1 --> R2 --> R3 --> R4
end
CO["Committed offset 42"] -.-> R2
POS["Position 50"] -.-> R4
C["Consumer in group payment-service"]
GC["Group coordinator"]
OT["Internal topic __consumer_offsets, compacted"]
NEXT["Next owner after a crash or rebalance"]
C -->|"commit group, topic, partition 2, offset 47"| GC
GC -->|"append latest offset per key"| OT
OT -->|"committed offset lookup"| NEXT
NEXT -->|"resumes at the committed offset"| LOG- The committed offset (42) and the position (50) are different numbers at almost every moment. Lag reported by the tools is measured from the committed offset to the log end offset.
- If the consumer dies now, the next owner starts at 42. Offsets 42 to 46 are processed a second time and 47 to 49 possibly a second time. Nothing is lost.
- The pending commit carries 47, the offset after the last finished record, not 46.
__consumer_offsetsis an ordinary replicated topic with compaction. Losing a broker does not lose the group's progress.
Message flow
Drawing diagram…
Sequence diagram. Consumer A resumes partition 2 at committed offset 42, polls offsets 42 to 46 and stores payments for 42, 43 and 44, then crashes before committing. After the session timeout the coordinator assigns the partition to consumer B, which reads the committed offset 42, polls the same records, stores 42 to 44 a second time and 45 and 46 for the first time, and commits offset 47.
sequenceDiagram participant A as Consumer A participant D as Payments database participant G as Group coordinator participant B as Consumer B A->>G: fetch committed offset for partition 2 G-->>A: 42 A->>A: poll returns offsets 42 to 46 A->>D: store payments for 42, 43, 44 Note over A: process crashes before the commit Note over G: no heartbeat for session.timeout.ms G-->>B: rebalance, partition 2 assigned to B B->>G: fetch committed offset for partition 2 G-->>B: 42 B->>B: poll returns offsets 42 to 46 B->>D: store payments for 42, 43, 44 again B->>D: store payments for 45, 46 B->>G: commit offset 47 G-->>B: commit acknowledged
- Consumer B cannot know that A already handled 42 to 44. The only shared knowledge is the committed offset, and it still says 42.
- This is at-least-once working as designed. The duplicates must be made harmless by the processing itself, for example a unique constraint on the order id.
- Had A committed 47 before writing to the database and then crashed, B would have started at 47 and the five payments would never have been stored.
- The commit of 47 covers all five records with a single number.
In a real system
A typical payment service consumes orders.events (12 partitions, key orderId) in the group payment-service. For every OrderPlaced event it charges the customer through a provider and stores a payment row. Two things must hold: no order is skipped, and no customer is charged twice.
- No skipped orders comes from the commit policy: commit only after the payment row is stored. The service uses Spring's
MANUAL_IMMEDIATEmode and acknowledges at the end of the listener, or simply the defaultBATCH. - No double charges cannot come from the commit policy, since at-least-once will redeliver after a crash. The service stores payments under a unique key on
orderIdand sends that same id to the provider as an idempotency key, so a redelivered event is detected and skipped. - First deploy uses
auto-offset-reset: earliest, so orders placed before the service first started are not skipped. - Replay after a bug: a release mis-rounded amounts for two hours. After the fix, the team stops the service, resets the group to the time of the bad deploy with
--to-datetime, and starts it again. The idempotent write turns the replay into corrections rather than second charges.
What can go wrong with this bookkeeping at scale is shown in the Break card The offset that went backwards. Making the processing safe to repeat is the topic of Duplicates and idempotency.
Java and Spring Boot
A plain consumer that commits explicit offsets, asynchronously per batch and synchronously when it loses partitions or shuts down. Then the Spring equivalent with manual acknowledgement, and the CLI commands for a controlled replay.
package com.example.payments.consumer;import java.time.Duration;import java.util.Collection;import java.util.HashMap;import java.util.List;import java.util.Map;import java.util.Properties;import java.util.concurrent.atomic.AtomicBoolean;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.KafkaConsumer;import org.apache.kafka.clients.consumer.OffsetAndMetadata;import org.apache.kafka.common.TopicPartition;import org.apache.kafka.common.errors.WakeupException;import org.apache.kafka.common.serialization.StringDeserializer;import org.slf4j.Logger;import org.slf4j.LoggerFactory;public class CommittingOrderConsumer implements Runnable, ConsumerRebalanceListener { private static final Logger log = LoggerFactory.getLogger(CommittingOrderConsumer.class); private final AtomicBoolean closed = new AtomicBoolean(false); private final KafkaConsumer<String, String> consumer; /** Next offset to read for each partition this consumer has finished work on. */ private final Map<TopicPartition, OffsetAndMetadata> processed = new HashMap<>(); public CommittingOrderConsumer(String bootstrapServers) { Properties props = new Properties(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-service"); props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); this.consumer = new KafkaConsumer<>(props); } @Override public void run() { try { consumer.subscribe(List.of("orders.events"), this); while (!closed.get()) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1)); for (ConsumerRecord<String, String> record : records) { handle(record); processed.put(new TopicPartition(record.topic(), record.partition()), new OffsetAndMetadata(record.offset() + 1)); } if (!records.isEmpty()) { consumer.commitAsync(Map.copyOf(processed), (offsets, exception) -> { if (exception != null) { log.warn("async commit failed for {}", offsets, exception); } }); } } } catch (WakeupException e) { if (!closed.get()) { throw e; } } finally { try { consumer.commitSync(processed); } catch (RuntimeException e) { log.warn("final commit failed, the last batch will be read again", e); } finally { consumer.close(); } } } @Override public void onPartitionsRevoked(Collection<TopicPartition> partitions) { Map<TopicPartition, OffsetAndMetadata> toCommit = new HashMap<>(); for (TopicPartition partition : partitions) { OffsetAndMetadata offset = processed.remove(partition); if (offset != null) { toCommit.put(partition, offset); } } consumer.commitSync(toCommit); } @Override public void onPartitionsAssigned(Collection<TopicPartition> partitions) { // positions are initialised from the committed offsets by the consumer itself } @Override public void onPartitionsLost(Collection<TopicPartition> partitions) { // already owned by someone else: a commit would be rejected, so only forget them partitions.forEach(processed::remove); } public void shutdown() { closed.set(true); consumer.wakeup(); } private void handle(ConsumerRecord<String, String> record) { // store the payment idempotently, keyed by the order id in record.key() }}Why it is written this way
record.offset() + 1is the value committed: the next record to read, never the offset of the record just handled.- The offset is recorded only after
handlereturns, so a commit can never run ahead of the work. Ifhandlethrows, the loop ends and the record is read again by the next owner. commitAsynckeeps the loop fast. A failed async commit is only logged, because the next batch's commit carries a higher offset anyway.onPartitionsRevokedruns insidepollon the same thread, before the partitions go to another member. A synchronous commit there keeps the new owner from repeating finished work.onPartitionsLostis overridden because its default implementation callsonPartitionsRevoked, and committing for partitions that have already been reassigned fails.- The final
commitSyncinfinallycovers a clean shutdown.Map.copyOfgives the async commit its own snapshot, since the map keeps changing.
package com.example.payments.consumer;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.support.Acknowledgment;import org.springframework.stereotype.Component;@Componentpublic class PaymentOrderListener { private final PaymentStore payments; public PaymentOrderListener(PaymentStore payments) { this.payments = payments; } @KafkaListener(id = "payment-orders", topics = "orders.events", groupId = "payment-service") public void onOrderPlaced(OrderPlaced event, Acknowledgment ack) { payments.storeOnce(event); ack.acknowledge(); } /** Implemented elsewhere: inserts with a unique key on the order id and ignores a repeat. */ public interface PaymentStore { void storeOnce(OrderPlaced event); } public record OrderPlaced(String orderId, String customerId, long amountMinor, String currency) { }}Why it is written this way
- The
Acknowledgmentparameter is only supplied when the container uses a manual ack mode. With any other mode it is null. acknowledge()is called after the database write. IfstoreOncethrows, nothing is acknowledged and the container's error handler takes over.- With
MANUAL_IMMEDIATEthe commit happens at theacknowledge()call. WithMANUALit is queued and committed once the records from the poll have been processed. - For this simple listener the default
BATCHmode without theAcknowledgmentparameter would give the same guarantee with fewer commits. Manual modes earn their keep when the listener finishes work somewhere else. storeOnceis what protects against double charges. No ack mode can do that.
spring: kafka: bootstrap-servers: localhost:9092 consumer: group-id: payment-service auto-offset-reset: earliest enable-auto-commit: false key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: spring.json.trusted.packages: com.example.payments.consumer spring.json.value.default.type: com.example.payments.consumer.PaymentOrderListener$OrderPlaced listener: ack-mode: manual_immediateWhy it is written this way
enable-auto-commit: falseis already what Spring for Apache Kafka sets when the property is absent. It is written out so nobody turns it on by accident: the container's ack modes only apply when it is false.ack-modeselects theContainerProperties.AckModefor all listener containers created by Spring Boot's default factory.auto-offset-reset: earliestmatters only the first time this group reads a partition, or after its offsets have expired.
# 1. Stop every instance of the service, then confirm the group has no active members.bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --describe --group payment-service --state# 2. Look at the current committed offsets and lag per partition.bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --describe --group payment-service# 3. Preview the reset. Without --execute nothing is changed.bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --reset-offsets --group payment-service --topic orders.events \ --to-datetime 2026-11-14T09:00:00.000 --dry-run# 4. Apply it, then start the service again.bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \ --reset-offsets --group payment-service --topic orders.events \ --to-datetime 2026-11-14T09:00:00.000 --executeWhy it is written this way
- The documentation asks you to make sure the consumer instances are inactive before resetting. Stop the service first.
--dry-runis the default and prints the new offset for each partition. Read that output before running with--execute.- Other documented targets:
--to-earliest,--to-latest,--to-offset,--shift-bywith a positive or negative number,--by-durationand--from-file. - Rewinding makes the group process old records again. Only do it when the processing is idempotent or the repeat is intended.
Configuration
Offset-related settings with their documented defaults for Kafka 4.x, and the Spring settings that replace some of them. The client defaults are the same in the 3.8 and 3.9 clients used by Spring for Apache Kafka 3.3.
enable.auto.commitDefault true (Spring for Apache Kafka sets false unless you set it)
- What it does
- Whether the consumer commits its positions periodically by itself.
- When to change it
- Leave it false in Spring and let the container commit. In a plain consumer, turn it off whenever records are handed to other threads or a commit must follow a specific write.
auto.commit.interval.msDefault 5000 (5 seconds)
- What it does
- How often offsets are auto-committed when
enable.auto.commitis true. - When to change it
- A lower value shrinks the window of records repeated after a crash and adds commit traffic. It has no effect once auto-commit is off.
auto.offset.resetDefault latest
- What it does
- Where to start when the group has no committed offset for a partition, or the committed offset is no longer on the broker.
- When to change it
- Use
earliestfor services that must not miss records produced before their first start. Usenonewhen a silent jump in either direction would be dangerous and you prefer a failure.
spring.kafka.listener.ack-modeDefault BATCH (container default)
- What it does
- The container's
AckMode: when it commits offsets on behalf of the listener. - When to change it
- Keep
BATCHfor most listeners. UseRECORDwhen repeating a whole batch after a crash is too costly, and a manual mode when the listener must decide the moment.
ContainerProperties.syncCommitsDefault true
- What it does
- Whether the Spring container commits with
commitSync(true) orcommitAsync(false). - When to change it
- Leave it true unless commit latency is measurably limiting throughput. Async commits report failures only through the commit callback.
offsets.retention.minutes (broker)Default 10080 (7 days)
- What it does
- How long committed offsets are kept after a group has lost all its members.
- When to change it
- Raise it if consumers are routinely stopped for longer, such as weekly batch jobs or environments that are switched off. Otherwise they restart from
auto.offset.reset.
offsets.topic.replication.factor (broker)Default 3
- What it does
- Replication factor of
__consumer_offsets. The topic cannot be created until the cluster has that many brokers. - When to change it
- Keep 3 in production. A single-broker development cluster needs it set to 1 for consumer groups to work at all.
Hands-on exercise
Move the commit and crash the consumer
Use the delivery semantics simulation to see how the position of the commit relative to processing decides between lost and repeated payments, then confirm it in the crash lab.
Steps · 0 of 5 done
Where the commit sits
Simulation · not a real clusterChange the commit order and the crash point, and watch the committed offset against the stored payments.
What you should see
Commit after processing with a crash in between repeats the uncommitted records: payments appear twice and none are missing. Commit before processing with a crash after the commit loses the in-flight records: no duplicates, but a payment is missing. With the idempotent consumer on, the records are still delivered twice, yet each payment is stored once.
What can go wrong
Auto-commit with processing on other threads
- Symptom
- After a crash or a deploy, a handful of records were never processed, and there is no error and no lag.
- Cause
- The poll thread handed records to a pool and polled again. The auto-commit inside that poll covered records the workers had not finished.
- Fix
- Set
enable.auto.commit=falseand commit only offsets whose records are complete, from the polling thread. Pause the partitions while workers are busy, as the Javadoc suggests.
A new group that starts at the end
- Symptom
- A newly deployed service ignores everything already in the topic and only reacts to records produced after it started.
- Cause
- The group had no committed offsets and
auto.offset.resetwas left at its default,latest. - Fix
- Set
auto.offset.reset=earliestbefore the first start of a group that must see existing data. For a group that already started, stop it and reset its offsets with the CLI.
Offsets expired while the consumer was off
- Symptom
- A job that was stopped for two weeks either skips to the newest records or reprocesses the entire topic when it comes back.
- Cause
- The group was empty for longer than
offsets.retention.minutes(7 days), so its committed offsets were deleted andauto.offset.resetapplied. - Fix
- Raise the broker's offset retention to cover planned downtime, or record the offsets before a long stop and restore them with
--reset-offsets --from-file.
Treating the commit as a per-record acknowledgement
- Symptom
- One failed record disappears: later records were processed and committed, and the failed one is never seen again.
- Cause
- The code caught the exception, moved on, and a later commit carried an offset beyond the failed record.
- Fix
- Decide explicitly what happens to a failed record: stop and retry it in place, or publish it to a retry or dead letter topic before moving on. See the reliability path.
Resetting offsets while the group is running
- Symptom
- The reset command refuses to run and reports that offsets can only be reset when the group is inactive.
- Cause
- The group still has active members. They would keep committing their own positions over whatever the tool wrote, so the tool checks the group state first.
- Fix
- Stop all instances, confirm the group has no members with
--describe --state, run the reset with--execute, then start the instances.
How to troubleshoot
Need to know where a group is in each partition
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- CURRENT-OFFSET is the committed offset, LOG-END-OFFSET the end of the partition, LAG the difference. A CURRENT-OFFSET that does not move while the service logs processed records means commits are not happening.
A consumer started somewhere unexpected
Check
Consumer log at INFO: 'Found no committed offset for partition' followed by 'Resetting offset for partition ... to position'
- How to read it
- The group had no committed offset for that partition, so
auto.offset.resetchose the start. Check the group id for a typo or a change, and whether the group was empty past the offset retention.
CommitFailedException in the logs
Check
The lines before it: a poll timeout or a rebalance, and the consumer metric `time-between-poll-max`
- How to read it
- The consumer was no longer a member when it tried to commit, usually because the batch took longer than
max.poll.interval.ms. The records since the last successful commit are processed again by the new owner.
Suspect commits are slow or failing
Check
Consumer metrics `commit-latency-avg` and `commit-rate` (group consumer-coordinator-metrics)
- How to read it
- A commit rate near zero with steady consumption means nothing is being committed. High commit latency with
RECORDor per-record manual acks means the commit round trip is limiting throughput.
The same records are processed after every restart
Check
Compare CURRENT-OFFSET from --describe with the last offset the service logged as processed
- How to read it
- If the committed offset equals the last processed offset instead of that offset plus one, the code commits the record's own offset. If it is far behind, commits are too rare or are failing.
Key takeaways
- A commit stores one number per group and partition: the next offset to read. It does not acknowledge individual records.
- Commit before processing gives at-most-once. Commit after processing gives at-least-once. Duplicates are handled in the processing, not in the commit.
- Auto-commit runs from
polland is safe only when each batch is fully processed before the nextpollorclose. - Use
commitAsyncfor routine commits andcommitSyncwhen partitions are revoked and at shutdown. - Spring turns auto-commit off and commits by
AckMode.BATCHis the default, and the manual modes still commit a position, not a record. auto.offset.resetapplies only when there is no valid committed offset. Its default,latest, skips existing records for a new group.- Stop the group before resetting offsets, and read the dry run first.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A plain-Java consumer uses default settings. To go faster, its loop submits every record from poll to a 16-thread executor and immediately polls again. After a node was killed, 37 orders were found that no instance ever processed. The consumer group shows zero lag.
What should change first?
Scenario 2 of 3
A reporting service ran for a year, then was switched off for 16 days during a migration. The broker uses default settings and the consumer has auto.offset.reset=latest. The topic retains 30 days of data. The service is started again with the same group id.
Where does it start reading?
Scenario 3 of 3
A Spring listener uses the default ack mode. max.poll.records is 500 and each record triggers an email. A pod is killed after it has sent emails for 320 records of a batch. Customers complain about receiving the same email twice.
Which change reduces the repeats to at most one record per crash without adding new components?
References
References: KafkaConsumer Javadoc (Kafka 4.4): offsets and consumer position, automatic and manual offset control ↗ · Apache Kafka 4.4 documentation: Consumer Configs ↗ · Apache Kafka 4.4 documentation: Distribution (consumer offset tracking, __consumer_offsets) ↗ · Apache Kafka 4.4 documentation: Broker Configs (offsets.retention.minutes, offsets.topic.*) ↗ · Apache Kafka 4.4 documentation: Basic Kafka Operations (managing consumer groups, --reset-offsets) ↗ · Apache Kafka 4.4 documentation: Monitoring (consumer metrics) ↗ · Spring for Apache Kafka 3.3 reference: Message Listener Containers (Committing Offsets, AckMode) ↗ · Spring for Apache Kafka 3.3 reference: Seeking to a Specific Offset ↗ · Spring Boot 3.4 reference: common application properties (spring.kafka.*) ↗ · Report a problem