Path 6 · lesson 5 of 5
Out-of-order events, graceful shutdown and recovery
Why events arrive out of order and how a consumer stays correct anyway, how to stop a consumer without causing duplicates or stalls, and how to recover after an incident.
What you will learn
- List the ways records for one entity end up processed out of order, even though each partition is ordered
- Write a consumer that ignores events older than the state it holds, and design events so that this is enough
- Describe what a consumer must do to shut down cleanly and what Spring's listener container does on stop()
- Set the shutdown timeouts of the container, Spring Boot and Kubernetes so that each one fits inside the next
- Choose between restarting from committed offsets, resetting offsets, replaying a dead letter topic and rebuilding state, and say what each one requires
Before you start
The concept
This lesson joins three subjects that are usually taught apart, because in production they are one subject. A consumer that is stopped badly reprocesses records. Recovery after an incident reprocesses records on purpose. And reprocessing means records are handled in a different order from the one in which they were written. A consumer that copes with old and repeated events makes shutdown mistakes harmless and makes every recovery method available. A consumer that does not is fragile in all three situations.
1. Out-of-order events
Kafka's ordering guarantee is narrow: records in one partition are stored, and delivered to a consumer, in the order the broker appended them. Nothing is promised across partitions, across topics, or about the order in which your application finishes processing them. Events about one order can therefore be handled out of sequence in several ordinary ways.
| How it happens | Mechanism |
|---|---|
| Events for one entity in different partitions | The key is missing, differs between event types (order id on one, customer id on another), or the events are in different topics. Each partition is consumed independently. |
| Partition count changed | The default partitioner hashes the key over the number of partitions. After partitions are added, the same key maps to a new partition while older events for it still wait in the old one. |
| Producer retries without idempotence | With several requests in flight, a failed batch that is retried can be appended after a later batch. Idempotence, which current clients enable by default, prevents this; a producer that has it switched off does not. |
| Several producers for one entity | Two service instances, or two services, each send an event about the same order. The broker appends them in arrival order, which is not necessarily the order of the business changes. |
| Retry topics and dead letter replays | A failed record is processed later than records that came after it. Spring's reference says directly that non-blocking retries give up the ordering guarantee for the topic. |
| Offset resets and redelivery | After a reset, a crash or a rebalance, the consumer sees records it has already applied, followed again by newer ones: old state arrives after new state. |
2. Graceful shutdown
A consumer is stopped far more often by a deployment than by a failure. A clean stop has four steps, in this order: stop polling for new records, finish the records already in hand, commit the offsets of what was finished, and leave the group so the coordinator hands the partitions to another member at once. Skip the commit and the next owner repeats work. Skip the leave and the partitions sit idle until the session times out.
3. Recovery
After an incident the question is always the same: from which position should each consumer group continue, and what happens to the records in between? Kafka gives you four tools, because it keeps the data after it has been read: continue from the committed offset, move the committed offset, replay what was parked in a dead letter topic, or throw derived state away and rebuild it from a topic. Each has a precondition, and the preconditions must be in place before the incident.
Why it exists
Each of the three failures below happens to teams that had no incident on the broker at all.
- The stale overwrite. A customer cancels an order. A delayed
PAIDevent for the same order is processed afterwards, the consumer writes it over the cancellation, and the warehouse ships. No record was lost and none was duplicated; they were applied in the wrong order. - The noisy deployment. Every rolling restart produces a burst of duplicate emails and a minute of lag. The pods are killed while a listener is mid-record, the offsets of finished work are never committed, and the partitions wait for the session timeout before anyone else reads them.
- The recovery that cannot be run. A bug wrote wrong totals for three hours. The fix is deployed, and then the team finds that rewinding the group would send three hours of customer emails again, because the consumer is not idempotent. The data to recover from is still in Kafka, and it cannot be used.
The common cause is a consumer written for the happy path: it assumes each record arrives once, in order, and that the process stops only between records. Kafka's design gives up those assumptions deliberately, in exchange for durability and the ability to re-read. The engineering response is to make the consumer tolerant, not to try to make delivery perfect.
How it works inside
Handling out-of-order events
Start with the key. Choose a key that identifies the entity whose order matters, use it for every event type about that entity, and keep those events in one topic. That removes the first cause. It does not remove retries, replays or resets, so a second defence is needed.
Put a version in every event. The service that owns the entity increments a counter in the same database transaction that changes the entity, and publishes it in the event. The consumer stores the version next to the state it derives and applies an event only if its version is higher than the stored one. An older or equal version is ignored, and that counts as success: the offset moves on. This single comparison handles reordering and duplicates together, since a repeated event has an equal version. Use a counter from the owning service, not a wall-clock timestamp: clocks on different machines disagree, and two changes in the same millisecond cannot be told apart.
Event as a delta
- Example: stock reduced by 2.
- Must be applied exactly once, and in order unless the operation commutes.
- A skipped or repeated event corrupts the result for good.
- Needs a record of applied event ids, or a gap check on a sequence number.
- Out-of-order events must be buffered until the gap closes.
Event as state
- Example: stock is now 14, version 31.
- Applying it twice gives the same result.
- A skipped event is repaired by the next one.
- Needs only: ignore if not newer.
- Works with compaction and with replays of any length.
Buffering is the tool for the cases where deltas are unavoidable, or where one event depends on another that has not arrived (a payment for an order the consumer has not seen yet). The consumer keeps the early event, in a table or by republishing it for later, until its predecessor arrives or a deadline passes. This costs state, a timeout policy and an answer for what to do when the gap never closes. Design events as full state where you can, and buffer only where you must.
What a clean shutdown is, in the consumer API
KafkaConsumer is not thread-safe, with one exception: wakeup() may be called from another thread. It makes a blocked poll() throw WakeupException, or the next call if none is blocked. The documented pattern is a flag plus wakeup() from the shutdown hook, and close() in a finally block on the polling thread. close() waits up to 30 seconds by default (or the timeout you pass) to complete pending commits and leave the group, and wakeup() cannot interrupt it. If auto-commit is on, close() commits the current position; with manual commits you must commit before closing.
A rebalance takes partitions away in the same way a shutdown does, so it has the same rule. ConsumerRebalanceListener.onPartitionsRevoked is called when the consumer has to give partitions up, including during close(), and the Kafka documentation recommends committing offsets there. onPartitionsLost is different: it is called when the partitions may already belong to someone else, for instance after a session timeout, and a commit at that point can no longer succeed.
What Spring's listener container does on stop()
- Listener containers implement
SmartLifecycleand run in a late phase, so on application shutdown they are stopped before beans such as the data source are destroyed. stop()marks the container as not running and wakes the consumer if it is blocked inpoll().- The consumer thread finishes the records returned by the previous poll. With
stopImmediateset to true it stops after the current record instead. - It commits the offsets that are pending for the records it processed, unsubscribes and closes the consumer, which leaves the group.
- It publishes
ConsumerStoppingEventbefore andConsumerStoppedEventafter the close, thenContainerStoppedEventonce all consumers of the container have stopped. - The calling thread waits for this for at most
shutdownTimeout, 10 seconds by default. If the listener is still busy when the time is up,stop()returns anyway and the application context goes on closing around it.
Static membership changes the last step. A consumer with group.instance.id set is a static member. When it shuts down, its partitions are not reassigned immediately; the group waits for the session timeout, expecting the same instance to come back. That is the point of the feature: a restart that finishes inside session.timeout.ms (45 seconds by default with the classic protocol) causes no rebalance at all. The cost is that a pod which does not come back leaves its partitions unread for the full timeout.
Pausing is not stopping. pause() on a listener container keeps the consumer in the group: it continues to call poll(), so no rebalance happens, but it fetches no records. Use it to wait out a dependency outage. Use stop() when the process is going away.
Leave cleanly, then crash
Simulation · not a real clusterRemove one consumer and watch how fast its partitions move. Then crash one instead and compare the gap. Repeat both with static membership switched on.
Recovery strategies
| Strategy | What it does | What it requires |
|---|---|---|
| Restart from committed offsets | The default. The group continues from the last committed position of each partition. Records processed but not committed before the stop are delivered again. | An idempotent consumer. Committed offsets that still exist (see offsets.retention.minutes). |
| Reset offsets backwards, by time or position | Moves the group's committed offsets to an earlier point so a window is processed again, for example everything since the faulty deployment. | The group must be stopped. The records must still be within retention. The consumer must be idempotent and must ignore stale versions, and side effects such as emails must be suppressed or deduplicated. |
| Reset offsets forwards | Skips a backlog or a block of records that should not be processed, for example a test flood. | A deliberate decision that the skipped records are not needed, or another way to reconcile them. Skipped records are not parked anywhere. |
| Replay a dead letter topic | Republishes only the records that failed, after the cause is fixed. | A dead letter topic with enough retention, a replay tool, and a consumer that ignores events older than its state. |
| Rebuild state from a topic | Discards a derived store (cache, search index, read model) and recreates it by reading the topic from the beginning. | A topic that still holds what is needed: a compacted topic keyed by entity, or full retention. Events that carry state, not deltas. |
kafka-consumer-groups.sh --reset-offsets changes the committed offsets of one group. You choose a scope (--topic, optionally with partitions, or --all-topics) and one target: --to-datetime, --by-duration, --to-earliest, --to-latest, --to-offset, --shift-by, --to-current or --from-file. Without --execute it only prints the offsets it would set, so the normal procedure is plan first, read the plan, then execute. The Kafka documentation tells you to make sure the consumer instances are inactive first. A target outside the available range is adjusted to the nearest available offset.
Rebuilding from a compacted topic has its own limit. Compaction keeps at least the latest record for each key; it does not promise exactly one, so the reader must still apply the version rule. A deleted key is represented by a tombstone, a record with a null value, and tombstones are themselves removed after delete.retention.ms (1 day by default). A rebuild that takes longer than that to read the topic from offset 0 can miss a deletion and keep an entity that no longer exists.
Architecture
Drawing diagram…
Decision flowchart. If processing was only interrupted, restart from committed offsets. If wrong results were written for a time window, stop the group and reset offsets to the start of the window. If records were parked in a dead letter topic, fix the cause and replay it. If derived state is lost or corrupt, rebuild it from a compacted or retained topic. Each method lists a requirement, and all four lead to one shared requirement: the consumer ignores stale versions and repeated events.
flowchart TD
INC["Incident is over, the consumer must continue"] --> Q1{"Was processing only interrupted"}
Q1 -->|"yes"| R1["Restart from committed offsets"]
Q1 -->|"no"| Q2{"Were wrong results written for a time window"}
Q2 -->|"yes"| R2["Stop the group, reset offsets to the start of the window"]
Q2 -->|"no"| Q3{"Were records parked in a dead letter topic"}
Q3 -->|"yes"| R3["Fix the cause, replay the dead letter topic"]
Q3 -->|"no"| Q4{"Is derived state lost or corrupt"}
Q4 -->|"yes"| R4["Drop the state, rebuild from the compacted or retained topic"]
R1 --> N1["Needs committed offsets that have not expired"]
R2 --> N2["Needs records still within retention"]
R3 --> N3["Needs dead letter retention and a replay tool"]
R4 --> N4["Needs events that carry full state"]
N1 --> G["Consumer ignores stale versions and repeats"]
N2 --> G
N3 --> G
N4 --> G- The questions are ordered from cheapest to most expensive. Most incidents end at the first box: the group restarts and repeats a few records.
- Each method has a precondition that has to exist before the incident: unexpired committed offsets, topic retention long enough to cover the window, a dead letter topic with a replay tool, or events that carry full state.
- All four methods deliver records the consumer has seen before, or records older than its current state. That is why they converge on one requirement. Without it, each method repairs one problem and creates another.
- Methods can be combined: reset offsets to reprocess a window, then replay the dead letters that the bug produced during that window.
Message flow
Drawing diagram…
Sequence diagram. The kubelet sends SIGTERM to the Spring Boot application, which stops the listener container. The container wakes the consumer, lets the listener finish the record at offset 207, commits offset 208 for partition orders-3, closes the consumer and leaves the group. The coordinator rebalances and assigns orders-3 to consumer B, which fetches the committed offset 208 and continues there. The process exits before the grace period ends.
sequenceDiagram participant K as Kubelet participant A as Spring Boot app participant C as Listener container participant L as Listener participant G as Group coordinator participant B as Consumer B K->>A: SIGTERM, grace period starts A->>C: stop C->>C: wake consumer, no further polls C->>L: finish record at offset 207 L-->>C: done C->>G: commit offset 208 for orders-3 C->>G: close consumer, leave group C-->>A: stopped within shutdownTimeout G->>B: rebalance, assign orders-3 B->>G: fetch committed offset for orders-3 G-->>B: 208 B->>B: continue at offset 208 A-->>K: process exits before grace period ends
- The commit says 208, the next record to read, after offset 207 was fully processed. Consumer B repeats nothing and skips nothing.
- Leaving the group is what makes the handover immediate. If the process is killed before it leaves, the coordinator notices only when heartbeats have been missing for the session timeout, and partition orders-3 is not read during that time.
- If SIGKILL arrives while the listener is still on offset 207, there is no commit. Consumer B starts from the last committed offset, which may be several records back, and processes those records again.
- Every step after SIGTERM is bounded by a timeout. The diagram is only true when the listener's remaining work fits in the container's, Spring's and the kubelet's limits.
In a real system
A typical order pipeline: the order service publishes OrderStatusChanged to the topic order-status (12 partitions, key = order id, 7 days retention). Each event carries the order's full current state and a version that the order service increments in the same transaction as the change. The notification service (group notification-service, four pods) keeps a table order_status with one row per order and emails the customer when the status changes. A second, compacted topic product-prices (key = product id) feeds an in-memory price view in each pod.
Ordering. A payment retry delays PAID (version 4) for order 7781 until after CANCELLED (version 5) has been processed. The upsert updates only where the stored version is lower, so version 4 changes no row, no email is sent, and the offset advances.
Shutdown. A rolling update replaces the four pods one after another. Each pod receives SIGTERM, finishes its current record (immediate-stop is on, and a record takes about 80 ms), commits and leaves the group. The remaining pods take over its partitions within a second or two. The pod's grace period is 60 seconds, far more than it needs.
Recovery. A release at 09:15 contains a bug that stores the wrong status for refunds. It is rolled back at 12:05. The team scales the deployment to zero, runs the reset with --to-datetime 2025-03-04T09:10:00.000 first without and then with --execute, and scales back up. Three hours of events are processed again. Orders that have moved on to a newer version since then are left alone by the version rule, and the rows the bug wrote wrongly are corrected (see the note below for the one adjustment this needed). Email sending is switched off for the run, and the few customers who had been told a wrong refund status are contacted separately.
Java and Spring Boot
Four pieces: a consumer that ignores stale events, the shutdown settings from the container out to Kubernetes, a listener that rebuilds a view from a compacted topic, and the plain-client shutdown loop that shows what Spring is doing for you. Written against the Spring for Apache Kafka 3.3 and Kafka client documentation, without a compiler at hand.
package com.example.notifications;import java.time.Instant;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;@Componentpublic class OrderStatusListener { /** Full current state of the order, not a change to apply. */ public record OrderStatusChanged(String orderId, long version, String status, long totalMinor, Instant occurredAt) { } private final JdbcTemplate jdbc; private final EmailOutbox emails; public OrderStatusListener(JdbcTemplate jdbc, EmailOutbox emails) { this.jdbc = jdbc; this.emails = emails; } @KafkaListener(id = "order-status", topics = "order-status", groupId = "notification-service") @Transactional public void onStatus(OrderStatusChanged event) { // PostgreSQL: insert the first version, or update only when the incoming version is higher. int changed = jdbc.update(""" INSERT INTO order_status (order_id, version, status, total_minor) VALUES (?, ?, ?, ?) ON CONFLICT (order_id) DO UPDATE SET version = EXCLUDED.version, status = EXCLUDED.status, total_minor = EXCLUDED.total_minor WHERE order_status.version < EXCLUDED.version """, event.orderId(), event.version(), event.status(), event.totalMinor()); if (changed == 0) { // Older than, or the same as, what we already hold: a late, repeated or replayed event. // Returning normally is correct. The container commits and moves on. return; } // Same database transaction as the row above, sent later by a separate relay. emails.enqueue(event.orderId(), event.version(), event.status()); }}Why it is written this way
- The comparison is done by the database inside the write, not by a read followed by a write in Java. Two pods that briefly hold the same partition during a rebalance cannot both win.
- An ignored event returns normally. Throwing here would send every late event through the retry and dead letter path, although nothing is wrong with it.
- The email is recorded in the same database transaction as the state change and sent by a relay (the outbox pattern), keyed by order id and version. A redelivered event finds the version already applied and enqueues nothing, so reprocessing a window does not send mail again.
@Transactionalhere is a database transaction. It does not include the Kafka offset commit, which happens afterwards; a crash in between repeats the record, and the version check absorbs it.- The event is a state event. If it said only that the status changed from PAID to CANCELLED, a missed or reordered event could not be repaired by a later one.
spring: kafka: bootstrap-servers: localhost:9092 consumer: group-id: notification-service # Used only when the group has no committed offset (new group, or offsets expired). # Explicit, so that case replays instead of silently skipping. auto-offset-reset: earliest max-poll-records: 100 listener: ack-mode: record # On stop(), finish the current record only, not the rest of the poll. immediate-stop: true lifecycle: # Upper bound for each SmartLifecycle phase during shutdown. Default 30s. timeout-per-shutdown-phase: 40sWhy it is written this way
immediate-stop: truemakes the remaining work at shutdown one record instead of up tomax-poll-records. The records not yet handed to the listener are simply fetched again by the next owner; they were never committed.ack-mode: recordcommits after every record. With the defaultBATCH, a kill in the middle of a poll repeats every record processed since the previous poll.auto-offset-reset: earliestis a statement about the empty-offset case. The client default islatest. With an idempotent consumer, replaying is the safer of the two.- The phase timeout is set above the container's
shutdownTimeoutand below the pod's grace period, so the three limits nest.
package com.example.notifications.config;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.context.event.EventListener;import org.springframework.kafka.config.ContainerCustomizer;import org.springframework.kafka.event.ConsumerStoppedEvent;import org.springframework.kafka.listener.ConcurrentMessageListenerContainer;import org.springframework.kafka.support.converter.RecordMessageConverter;import org.springframework.kafka.support.converter.StringJsonMessageConverter;@Configurationpublic class KafkaListenerConfig { private static final Logger log = LoggerFactory.getLogger(KafkaListenerConfig.class); /** Values arrive as strings (Boot's default deserializer); JSON is mapped to the listener's parameter type. */ @Bean RecordMessageConverter jsonMessageConverter() { return new StringJsonMessageConverter(); } /** Spring Boot applies this customizer to containers created by its default listener factory. */ @Bean ContainerCustomizer<Object, Object, ConcurrentMessageListenerContainer<Object, Object>> shutdownTimeout() { // stop() waits this long for the consumer threads to finish, commit and close. Default 10 000 ms. return container -> container.getContainerProperties().setShutdownTimeout(30_000L); } /** One log line per consumer thread on the way out; its absence in a pod's last logs means a hard kill. */ @EventListener public void onConsumerStopped(ConsumerStoppedEvent event) { log.info("Kafka consumer stopped: {}", event); }}Why it is written this way
- The message converter lets one container factory serve both listeners in this lesson:
OrderStatusChangedis converted from JSON, while the price view below receives the rawConsumerRecord. - There is a Spring Boot property for
immediate-stopbutshutdownTimeoutis set onContainerProperties, here through aContainerCustomizerbean. - 30 seconds is the budget for one in-flight record including a slow database call, not a target.
stop()returns as soon as the consumers have closed. ConsumerStoppedEventis published after the consumer is closed. Logging it gives you evidence, per pod, that the shutdown completed.
apiVersion: apps/v1kind: Deploymentmetadata: name: notification-servicespec: replicas: 4 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 # one consumer leaves the group at a time maxSurge: 0 selector: matchLabels: app: notification-service template: metadata: labels: app: notification-service spec: # SIGTERM first, SIGKILL after this many seconds. Default 30. terminationGracePeriodSeconds: 60 containers: - name: app image: registry.example.com/notification-service:1.8.2 # exec form, so the JVM is PID 1 and receives SIGTERM itself command: ["java", "-jar", "/app/notification-service.jar"]Why it is written this way
- The grace period (60 s) is longer than Spring's phase timeout (40 s), which is longer than the container's shutdown timeout (30 s). Each layer finishes before the one outside it gives up.
- The JVM must receive the signal. A shell wrapper that starts Java as a child process without
execswallows SIGTERM, and the consumer is killed hard every time, whatever the settings say. - Replacing one pod at a time keeps each rebalance small. With a cooperative assignor, only the departing pod's partitions move.
#!/usr/bin/env bashset -euo pipefailBOOTSTRAP=localhost:9092GROUP=notification-service# 1. Where is the group now? Keep this output: it is your way back.kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --describe --group "$GROUP"# 2. Stop every member (scale the deployment to 0), then confirm the group is empty.kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --describe --group "$GROUP" --state# 3. Plan only: without --execute the tool prints the offsets it would set.kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --group "$GROUP" \ --topic order-status --reset-offsets --to-datetime 2025-03-04T09:10:00.000# 4. Read the plan, then apply it.kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --group "$GROUP" \ --topic order-status --reset-offsets --to-datetime 2025-03-04T09:10:00.000 --execute# Other targets, same shape:# --by-duration PT3H three hours before now# --shift-by -500 500 records back on every partition in scope# --to-offset 81200 an exact offset (give the partition: --topic order-status:3)# --to-earliest / --to-latest start or end of what the broker still holdsWhy it is written this way
- Stop the group first. The Kafka documentation tells you to make sure the consumer instances are inactive, and a running member would overwrite the new offsets with its next commit in any case.
- The describe output from step 1 is the undo: it lists the offset of every partition before the change, and
--from-fileor--to-offsetcan put them back. --to-datetimefinds, for each partition, the first record whose timestamp is at or after the given time. That is the record timestamp, set by the producer by default, so allow a margin before the incident start.- A reset rewinds one group. Other groups reading the same topic are not affected, which is what makes it a surgical tool.
package com.example.notifications;import java.util.Map;import java.util.concurrent.ConcurrentHashMap;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.common.TopicPartition;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.listener.ConsumerSeekAware;import org.springframework.stereotype.Component;/** In-memory view of the compacted topic product-prices, rebuilt from offset 0 at every start. */@Componentpublic class ProductPriceView implements ConsumerSeekAware { private final Map<String, String> priceByProduct = new ConcurrentHashMap<>(); @Override public void onPartitionsAssigned(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) { // Ignore the committed position: this state lives only in memory, so read everything again. callback.seekToBeginning(assignments.keySet()); } // A group of its own per instance: every pod needs every partition of this topic. @KafkaListener(id = "price-view", topics = "product-prices", groupId = "price-view-${random.uuid}") public void onPrice(ConsumerRecord<String, String> record) { if (record.value() == null) { priceByProduct.remove(record.key()); // tombstone: the product was deleted } else { priceByProduct.put(record.key(), record.value()); } } public String priceOf(String productId) { return priceByProduct.get(productId); }}Why it is written this way
ConsumerSeekAware.onPartitionsAssignedis called whenever partitions are assigned, and the callback passed to it is the one to use there. Seeking to the beginning turns the topic into the recovery source for this state.- The same interface offers
seekToTimestamp, which is the in-application equivalent of a reset by time, for cases where the operator cannot stop the group and run the command-line tool. - The group id contains a random value, so each pod is alone in its group and is assigned all partitions. Such groups are abandoned at every restart and expire on the broker after
offsets.retention.minutes. - This works because
product-pricesis compacted and each record carries the full price. Applying records in offset order, later ones overwrite earlier ones, so reading a key twice is harmless. - The view is incomplete until the listener has caught up with the end of the topic. A real service gates its readiness on that.
package com.example.plain;import java.time.Duration;import java.util.Collection;import java.util.HashMap;import java.util.List;import java.util.Map;import java.util.concurrent.atomic.AtomicBoolean;import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.KafkaConsumer;import org.apache.kafka.clients.consumer.OffsetAndMetadata;import org.apache.kafka.common.TopicPartition;import org.apache.kafka.common.errors.WakeupException;/** The shutdown sequence without a framework. enable.auto.commit must be false. */public final class OrderConsumerLoop implements Runnable { private final AtomicBoolean closed = new AtomicBoolean(false); private final KafkaConsumer<String, String> consumer; private final Map<TopicPartition, OffsetAndMetadata> processed = new HashMap<>(); public OrderConsumerLoop(KafkaConsumer<String, String> consumer) { this.consumer = consumer; } @Override public void run() { try { consumer.subscribe(List.of("orders"), new ConsumerRebalanceListener() { @Override public void onPartitionsRevoked(Collection<TopicPartition> partitions) { commitProcessed(); // still the owner: hand over an exact position } @Override public void onPartitionsAssigned(Collection<TopicPartition> partitions) { } @Override public void onPartitionsLost(Collection<TopicPartition> partitions) { processed.keySet().removeAll(partitions); // no longer the owner: a commit would fail } }); while (!closed.get()) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1)); for (ConsumerRecord<String, String> record : records) { handle(record); processed.put(new TopicPartition(record.topic(), record.partition()), new OffsetAndMetadata(record.offset() + 1)); if (closed.get()) { break; // finish the record in hand, leave the rest uncommitted } } commitProcessed(); } } catch (WakeupException e) { if (!closed.get()) { throw e; } } finally { try { commitProcessed(); } finally { consumer.close(Duration.ofSeconds(10)); // completes commits, leaves the group } } } /** Called from the JVM shutdown hook, on another thread. */ public void shutdown() { closed.set(true); consumer.wakeup(); // the only KafkaConsumer method that is safe to call from another thread } private void commitProcessed() { if (!processed.isEmpty()) { consumer.commitSync(processed); processed.clear(); } } private void handle(ConsumerRecord<String, String> record) { // idempotent business processing }}Why it is written this way
- The committed value is
record.offset() + 1: the position of the next record to read, for each partition, and only for records that were fully handled. wakeup()interrupts a blockedpoll()with aWakeupException. The flag tells the loop that the exception is expected.- The commit in
finallyruns on the polling thread beforeclose(). With auto-commit disabled,close()does not commit for you. - The shutdown hook must wait for this thread to finish (for example
thread.join()with a limit) after callingshutdown(). Otherwise the JVM exits while the commit is still in progress. - This is what Spring's listener container implements for you, together with error handling. Read it to understand the container; use the container in a Spring application.
Configuration
The settings behind the three topics of this lesson, with the defaults documented for Kafka 4.1 and Spring for Apache Kafka 3.3 / Spring Boot 3.5.
ContainerProperties.shutdownTimeoutDefault 10000 (ms)
- What it does
- Longest time stop() blocks while the consumer threads finish, commit and close, before the container stopped event is published.
- When to change it
- Set it above the worst-case time of the work that remains at stop: one record with immediate stop, otherwise the rest of a poll.
spring.kafka.listener.immediate-stop (ContainerProperties.stopImmediate)Default false
- What it does
- When false, a stopping container still processes all records from the previous poll. When true, it stops after the current record.
- When to change it
- Turn it on when a poll can hold more work than your shutdown budget. The unprocessed records are not committed and are read by the next owner.
spring.lifecycle.timeout-per-shutdown-phaseDefault 30s
- What it does
- Spring Boot's limit for stopping all SmartLifecycle beans of one phase, which includes the listener containers.
- When to change it
- Keep it above shutdownTimeout and below the pod's termination grace period.
terminationGracePeriodSeconds (Kubernetes pod spec)Default 30
- What it does
- Time between SIGTERM and SIGKILL when a pod is deleted. A preStop hook, if any, uses the same budget.
- When to change it
- Raise it above the Spring timeouts. If it is shorter, the settings inside the JVM are irrelevant: the process is killed first.
max.poll.recordsDefault 500
- What it does
- Maximum number of records one poll() returns. It bounds the work in hand when a stop or a rebalance begins.
- When to change it
- Lower it when records are slow to process, so that both shutdown and max.poll.interval.ms have room.
session.timeout.msDefault 45000 (45 seconds)
- What it does
- With the classic group protocol, how long the coordinator waits without heartbeats before removing a member and rebalancing. With group.protocol=consumer the broker setting group.consumer.session.timeout.ms applies instead.
- When to change it
- This is how long partitions stay unread after a hard kill, and after any stop of a static member. Lower it for faster failover, at the price of more false alarms.
group.instance.idDefault null
- What it does
- Makes the consumer a static member. Its partitions are kept for it across a restart instead of being reassigned at once.
- When to change it
- Use it with stable identities (for example StatefulSet pod names) when restarts finish within the session timeout. Leave it unset if pods are replaced by new identities.
auto.offset.resetDefault latest
- What it does
- Where to start when the group has no committed offset for a partition, or the committed offset no longer exists: earliest, latest, by_duration (newer clients) or none, which throws.
- When to change it
- Choose it deliberately per consumer. With latest, a group whose offsets expired skips everything produced while it was away.
offsets.retention.minutes (broker)Default 10080
- What it does
- How long committed offsets are kept after a group has lost all its members, 7 days by default.
- When to change it
- Raise it if consumers can legitimately be stopped for longer, such as batch jobs or disabled features.
retention.ms (topic)Default 604800000 (7 days)
- What it does
- How long records are kept under the delete policy. It is the furthest back any offset reset or rebuild can reach.
- When to change it
- Set it from your recovery needs: at least the longest period after which you would still want to reprocess.
delete.retention.ms (topic)Default 86400000 (1 day)
- What it does
- How long tombstones stay in a compacted topic. A reader starting at offset 0 must finish within this time to be sure of seeing every deletion.
- When to change it
- Raise it if a full rebuild from the compacted topic can take longer than a day.
Hands-on exercise
Kill a consumer between processing and commit
Crash a consumer at the worst moment, see exactly which records are repeated, and find out what the consumer needs in order to make that harmless. Then compare it with a clean stop.
Steps · 0 of 7 done
Crash after processing, before the commit
Simulation · not a real clusterThe consumer has charged the payment but has not committed. Crash it there, restart, and watch what is delivered again. Then switch on the idempotent consumer.
What you should see
With the crash between processing and commit, the record is delivered again after the restart and the payment is taken twice. The committed offset still points at that record, because a commit names the next record to read and none was made. With the idempotent consumer the record is still delivered twice, but the second delivery changes nothing. With the crash after the commit there is no redelivery, which is the state a graceful shutdown always reaches.
What can go wrong
Last write wins on arrival order
- Symptom
- Now and then an entity shows an old state: a cancelled order reappears as paid, or a stock level jumps back. It follows retries, replays or deployments.
- Cause
- The consumer applies each event as the newest truth. A late or redelivered event overwrites newer state.
- Fix
- Carry a version from the owning service in each event, store it with the derived state and update only when the incoming version is higher. Make the comparison part of the write statement.
Killed before the commit on every deployment
- Symptom
- Each rollout produces a burst of duplicate side effects, and pod logs end without a consumer-stopped line. Pods take the full grace period to disappear.
- Cause
- The remaining records of a poll take longer than the shutdown timeouts, or the JVM never receives SIGTERM because a shell wrapper is PID 1. The process is killed with uncommitted work.
- Fix
- Start the JVM in exec form, enable immediate stop or lower max.poll.records, and nest the three timeouts: container below Spring phase below pod grace period.
Partitions idle after a pod is removed
- Symptom
- After scaling down, some partitions show growing lag for about 45 seconds although other consumers are healthy.
- Cause
- The consumer did not leave the group: it was killed hard, or it is a static member, whose partitions are held until the session timeout.
- Fix
- Make the shutdown clean so the consumer leaves. For static members this wait is by design; shorten session.timeout.ms or drop static membership if pods do not return under the same identity.
A group that was stopped too long skips its backlog
- Symptom
- A consumer that was disabled for two weeks is switched back on. It shows zero lag immediately and has processed nothing from those two weeks.
- Cause
- Its committed offsets expired after offsets.retention.minutes. With no committed offset, auto.offset.reset=latest placed it at the end of each partition.
- Fix
- Set auto.offset.reset on purpose (earliest for idempotent consumers, none if a human should decide), and raise offsets.retention.minutes if long stops are normal. Also check that the topic's retention covers the stop.
Offset reset repeats side effects
- Symptom
- After rewinding a group by three hours to repair data, customers receive the emails of those three hours again.
- Cause
- The database writes were idempotent, but the email, HTTP call or downstream event was triggered for every record processed.
- Fix
- Tie each side effect to a state change that actually happened (an outbox row keyed by entity and version), or run the reprocessing with side effects switched off.
Reset attempted while the group is running
- Symptom
- The reset command prints an error saying assignments can only be reset if the group is inactive, and the offsets are unchanged.
- Cause
- Group members are still connected. The tool refuses to move offsets under a live consumer, which would overwrite them with its next commit anyway.
- Fix
- Stop all members, confirm with --describe --state that the group has none, run the reset with --execute, and only then start the consumers.
How to troubleshoot
You need to know where a group is and whether it still has members before any recovery.
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group notification-service --state
- How to read it
- The state and member count tell you whether a reset is possible: it must show no members. Run --describe without --state for CURRENT-OFFSET, LOG-END-OFFSET and LAG for each partition, and save that output before changing anything.
You want to see what a reset would do before doing it.
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group notification-service --topic order-status --reset-offsets --to-datetime 2025-03-04T09:10:00.000
- How to read it
- Without --execute the command prints the new offset for each partition and changes nothing. Compare it with the saved current offsets: the difference is the number of records that will be processed again.
Duplicates appear after every deployment.
Check
The last log lines of a terminated pod: look for the container's Consumer stopped message or your ConsumerStoppedEvent log line
- How to read it
- If the line is there, the stop was clean and the duplicates have another cause. If the log ends in the middle of processing, the pod was killed: compare the time from SIGTERM to the end of the log with terminationGracePeriodSeconds.
A rewind does not reach as far back as requested.
Check
kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic order-status --time -2
- How to read it
- This prints the earliest offset still available for each partition. If it is higher than the offset you wanted, retention has already deleted those records and the reset was adjusted to the earliest available one.
Events for one entity are suspected to be spread over several partitions.
Check
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic order-status --from-beginning --property print.key=true --property print.partition=true --property print.offset=true
- How to read it
- Filter the output for one order id. More than one partition for the same key means the partition count changed or producers use different keys or partitioners. Kafka 4.2 and later accept --formatter-property in place of --property.
Stale events are being ignored and you want to know how often.
Check
A counter in the listener for writes that changed zero rows, tagged by topic
- How to read it
- A low steady rate is normal redelivery. A spike after a deployment points to unclean shutdowns; a spike after a replay is expected; a constant high rate suggests a keying problem or several producers racing.
Key takeaways
- Kafka orders records within one partition as the broker appended them. Different keys, a changed partition count, retries, replays and resets all break per-entity order.
- Give every event a version from the owning service and apply it only if it is newer than what you hold. That one rule covers late, repeated and replayed events.
- Prefer events that carry full state over deltas. State events can be skipped, repeated and rebuilt from; deltas cannot.
- A clean stop is: stop polling, finish the work in hand, commit, leave the group. Spring's container does this on stop(), within shutdownTimeout.
- The container's shutdown timeout, Spring Boot's phase timeout and the pod's termination grace period must nest, or the process is killed with uncommitted work.
- Recovery choices are restart, reset offsets, replay the dead letter topic or rebuild state. Each depends on retention you set earlier, and all depend on an idempotent, version-checking consumer.
- Committed offsets expire 7 days after a group becomes empty, and then auto.offset.reset, default latest, decides where it restarts.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A listener takes about 150 ms per record. max.poll.records is 500, ack mode is BATCH, and all shutdown settings are defaults: container shutdownTimeout 10 s, Spring phase timeout 30 s, pod grace period 30 s. After each rolling update the team sees a few hundred records processed twice.
Which change removes most of these duplicates at the lowest cost?
Scenario 2 of 3
An inventory consumer receives StockAdjusted events with a delta, for example minus 2, keyed by SKU. After a consumer bug, 1,200 events sit in the dead letter topic. Since then about 30,000 newer adjustments for the same SKUs have been applied. The fix is deployed and a colleague wants to replay the dead letter topic tonight.
What should you conclude before the replay runs?
Scenario 3 of 3
A reporting consumer group was switched off 9 days ago during a cost review. The topic has retention.ms of 14 days, the broker uses the default offsets.retention.minutes, and the consumer does not set auto.offset.reset. It is idempotent. You are asked to turn it back on so the reports are complete.
What do you do before starting it?
References
References: Apache Kafka 4.1 documentation: Basic Kafka Operations (managing consumer groups, --reset-offsets) ↗ · Apache Kafka 4.0 Javadoc: KafkaConsumer (wakeup, close, multi-threaded processing) ↗ · Apache Kafka 4.1 Javadoc: ConsumerRebalanceListener (commit on revoke, onPartitionsLost) ↗ · Apache Kafka 4.1 documentation: Consumer configs (session.timeout.ms, auto.offset.reset, group.instance.id) ↗ · Apache Kafka 4.1 documentation: Broker configs (offsets.retention.minutes) ↗ · Apache Kafka 4.1 documentation: Topic configs (retention.ms, delete.retention.ms) ↗ · Spring for Apache Kafka 3.3 reference: Listener Container Properties (shutdownTimeout, stopImmediate) ↗ · Spring for Apache Kafka 3.3 reference: Seeking to a Specific Offset (ConsumerSeekAware) ↗ · Spring for Apache Kafka 3.3 reference: Pausing and Resuming Listener Containers ↗ · Spring for Apache Kafka 3.3 reference: Application Events (ConsumerStoppedEvent) ↗ · Spring Boot reference: Common Application Properties (spring.kafka.listener.immediate-stop, spring.lifecycle.timeout-per-shutdown-phase) ↗ · Kubernetes documentation: Pod Lifecycle, termination of Pods ↗ · Report a problem