Skip to content

Path 6 · lesson 5 of 5

Advanced35 minApache Kafka 4.x brokers · Spring Boot 3.5 with Spring for Apache Kafka 3.3 (kafka-clients 3.9, classic group protocol by default) · Java 21. Config defaults read from the Kafka 4.1 configuration reference.

Out-of-order events, graceful shutdown and recovery

Why events arrive out of order and how a consumer stays correct anyway, how to stop a consumer without causing duplicates or stalls, and how to recover after an incident.

What you will learn

  • List the ways records for one entity end up processed out of order, even though each partition is ordered
  • Write a consumer that ignores events older than the state it holds, and design events so that this is enough
  • Describe what a consumer must do to shut down cleanly and what Spring's listener container does on stop()
  • Set the shutdown timeouts of the container, Spring Boot and Kubernetes so that each one fits inside the next
  • Choose between restarting from committed offsets, resetting offsets, replaying a dead letter topic and rebuilding state, and say what each one requires

Before you start

The concept

This lesson joins three subjects that are usually taught apart, because in production they are one subject. A consumer that is stopped badly reprocesses records. Recovery after an incident reprocesses records on purpose. And reprocessing means records are handled in a different order from the one in which they were written. A consumer that copes with old and repeated events makes shutdown mistakes harmless and makes every recovery method available. A consumer that does not is fragile in all three situations.

1. Out-of-order events

Kafka's ordering guarantee is narrow: records in one partition are stored, and delivered to a consumer, in the order the broker appended them. Nothing is promised across partitions, across topics, or about the order in which your application finishes processing them. Events about one order can therefore be handled out of sequence in several ordinary ways.

How it happensMechanism
Events for one entity in different partitionsThe key is missing, differs between event types (order id on one, customer id on another), or the events are in different topics. Each partition is consumed independently.
Partition count changedThe default partitioner hashes the key over the number of partitions. After partitions are added, the same key maps to a new partition while older events for it still wait in the old one.
Producer retries without idempotenceWith several requests in flight, a failed batch that is retried can be appended after a later batch. Idempotence, which current clients enable by default, prevents this; a producer that has it switched off does not.
Several producers for one entityTwo service instances, or two services, each send an event about the same order. The broker appends them in arrival order, which is not necessarily the order of the business changes.
Retry topics and dead letter replaysA failed record is processed later than records that came after it. Spring's reference says directly that non-blocking retries give up the ordering guarantee for the topic.
Offset resets and redeliveryAfter a reset, a crash or a rebalance, the consumer sees records it has already applied, followed again by newer ones: old state arrives after new state.

2. Graceful shutdown

A consumer is stopped far more often by a deployment than by a failure. A clean stop has four steps, in this order: stop polling for new records, finish the records already in hand, commit the offsets of what was finished, and leave the group so the coordinator hands the partitions to another member at once. Skip the commit and the next owner repeats work. Skip the leave and the partitions sit idle until the session times out.

3. Recovery

After an incident the question is always the same: from which position should each consumer group continue, and what happens to the records in between? Kafka gives you four tools, because it keeps the data after it has been read: continue from the committed offset, move the committed offset, replay what was parked in a dead letter topic, or throw derived state away and rebuild it from a topic. Each has a precondition, and the preconditions must be in place before the incident.

Why it exists

Each of the three failures below happens to teams that had no incident on the broker at all.

  • The stale overwrite. A customer cancels an order. A delayed PAID event for the same order is processed afterwards, the consumer writes it over the cancellation, and the warehouse ships. No record was lost and none was duplicated; they were applied in the wrong order.
  • The noisy deployment. Every rolling restart produces a burst of duplicate emails and a minute of lag. The pods are killed while a listener is mid-record, the offsets of finished work are never committed, and the partitions wait for the session timeout before anyone else reads them.
  • The recovery that cannot be run. A bug wrote wrong totals for three hours. The fix is deployed, and then the team finds that rewinding the group would send three hours of customer emails again, because the consumer is not idempotent. The data to recover from is still in Kafka, and it cannot be used.

The common cause is a consumer written for the happy path: it assumes each record arrives once, in order, and that the process stops only between records. Kafka's design gives up those assumptions deliberately, in exchange for durability and the ability to re-read. The engineering response is to make the consumer tolerant, not to try to make delivery perfect.

How it works inside

Handling out-of-order events

Start with the key. Choose a key that identifies the entity whose order matters, use it for every event type about that entity, and keep those events in one topic. That removes the first cause. It does not remove retries, replays or resets, so a second defence is needed.

Put a version in every event. The service that owns the entity increments a counter in the same database transaction that changes the entity, and publishes it in the event. The consumer stores the version next to the state it derives and applies an event only if its version is higher than the stored one. An older or equal version is ignored, and that counts as success: the offset moves on. This single comparison handles reordering and duplicates together, since a repeated event has an equal version. Use a counter from the owning service, not a wall-clock timestamp: clocks on different machines disagree, and two changes in the same millisecond cannot be told apart.

Event as a delta

  • Example: stock reduced by 2.
  • Must be applied exactly once, and in order unless the operation commutes.
  • A skipped or repeated event corrupts the result for good.
  • Needs a record of applied event ids, or a gap check on a sequence number.
  • Out-of-order events must be buffered until the gap closes.

Event as state

  • Example: stock is now 14, version 31.
  • Applying it twice gives the same result.
  • A skipped event is repaired by the next one.
  • Needs only: ignore if not newer.
  • Works with compaction and with replays of any length.

Buffering is the tool for the cases where deltas are unavoidable, or where one event depends on another that has not arrived (a payment for an order the consumer has not seen yet). The consumer keeps the early event, in a table or by republishing it for later, until its predecessor arrives or a deadline passes. This costs state, a timeout policy and an answer for what to do when the gap never closes. Design events as full state where you can, and buffer only where you must.

What a clean shutdown is, in the consumer API

KafkaConsumer is not thread-safe, with one exception: wakeup() may be called from another thread. It makes a blocked poll() throw WakeupException, or the next call if none is blocked. The documented pattern is a flag plus wakeup() from the shutdown hook, and close() in a finally block on the polling thread. close() waits up to 30 seconds by default (or the timeout you pass) to complete pending commits and leave the group, and wakeup() cannot interrupt it. If auto-commit is on, close() commits the current position; with manual commits you must commit before closing.

A rebalance takes partitions away in the same way a shutdown does, so it has the same rule. ConsumerRebalanceListener.onPartitionsRevoked is called when the consumer has to give partitions up, including during close(), and the Kafka documentation recommends committing offsets there. onPartitionsLost is different: it is called when the partitions may already belong to someone else, for instance after a session timeout, and a commit at that point can no longer succeed.

What Spring's listener container does on stop()

  1. Listener containers implement SmartLifecycle and run in a late phase, so on application shutdown they are stopped before beans such as the data source are destroyed.
  2. stop() marks the container as not running and wakes the consumer if it is blocked in poll().
  3. The consumer thread finishes the records returned by the previous poll. With stopImmediate set to true it stops after the current record instead.
  4. It commits the offsets that are pending for the records it processed, unsubscribes and closes the consumer, which leaves the group.
  5. It publishes ConsumerStoppingEvent before and ConsumerStoppedEvent after the close, then ContainerStoppedEvent once all consumers of the container have stopped.
  6. The calling thread waits for this for at most shutdownTimeout, 10 seconds by default. If the listener is still busy when the time is up, stop() returns anyway and the application context goes on closing around it.

Static membership changes the last step. A consumer with group.instance.id set is a static member. When it shuts down, its partitions are not reassigned immediately; the group waits for the session timeout, expecting the same instance to come back. That is the point of the feature: a restart that finishes inside session.timeout.ms (45 seconds by default with the classic protocol) causes no rebalance at all. The cost is that a pod which does not come back leaves its partitions unread for the full timeout.

Pausing is not stopping. pause() on a listener container keeps the consumer in the group: it continues to call poll(), so no rebalance happens, but it fetches no records. Use it to wait out a dependency outage. Use stop() when the process is going away.

Leave cleanly, then crash

Simulation · not a real cluster

Remove one consumer and watch how fast its partitions move. Then crash one instead and compare the gap. Repeat both with static membership switched on.

Recovery strategies

StrategyWhat it doesWhat it requires
Restart from committed offsetsThe default. The group continues from the last committed position of each partition. Records processed but not committed before the stop are delivered again.An idempotent consumer. Committed offsets that still exist (see offsets.retention.minutes).
Reset offsets backwards, by time or positionMoves the group's committed offsets to an earlier point so a window is processed again, for example everything since the faulty deployment.The group must be stopped. The records must still be within retention. The consumer must be idempotent and must ignore stale versions, and side effects such as emails must be suppressed or deduplicated.
Reset offsets forwardsSkips a backlog or a block of records that should not be processed, for example a test flood.A deliberate decision that the skipped records are not needed, or another way to reconcile them. Skipped records are not parked anywhere.
Replay a dead letter topicRepublishes only the records that failed, after the cause is fixed.A dead letter topic with enough retention, a replay tool, and a consumer that ignores events older than its state.
Rebuild state from a topicDiscards a derived store (cache, search index, read model) and recreates it by reading the topic from the beginning.A topic that still holds what is needed: a compacted topic keyed by entity, or full retention. Events that carry state, not deltas.

kafka-consumer-groups.sh --reset-offsets changes the committed offsets of one group. You choose a scope (--topic, optionally with partitions, or --all-topics) and one target: --to-datetime, --by-duration, --to-earliest, --to-latest, --to-offset, --shift-by, --to-current or --from-file. Without --execute it only prints the offsets it would set, so the normal procedure is plan first, read the plan, then execute. The Kafka documentation tells you to make sure the consumer instances are inactive first. A target outside the available range is adjusted to the nearest available offset.

Rebuilding from a compacted topic has its own limit. Compaction keeps at least the latest record for each key; it does not promise exactly one, so the reader must still apply the version rule. A deleted key is represented by a tombstone, a record with a null value, and tombstones are themselves removed after delete.retention.ms (1 day by default). A rebuild that takes longer than that to read the topic from offset 0 can miss a deletion and keep an entity that no longer exists.

Architecture

Drawing diagram…

Decision flowchart. If processing was only interrupted, restart from committed offsets. If wrong results were written for a time window, stop the group and reset offsets to the start of the window. If records were parked in a dead letter topic, fix the cause and replay it. If derived state is lost or corrupt, rebuild it from a compacted or retained topic. Each method lists a requirement, and all four lead to one shared requirement: the consumer ignores stale versions and repeated events.

flowchart TD
  INC["Incident is over, the consumer must continue"] --> Q1{"Was processing only interrupted"}
  Q1 -->|"yes"| R1["Restart from committed offsets"]
  Q1 -->|"no"| Q2{"Were wrong results written for a time window"}
  Q2 -->|"yes"| R2["Stop the group, reset offsets to the start of the window"]
  Q2 -->|"no"| Q3{"Were records parked in a dead letter topic"}
  Q3 -->|"yes"| R3["Fix the cause, replay the dead letter topic"]
  Q3 -->|"no"| Q4{"Is derived state lost or corrupt"}
  Q4 -->|"yes"| R4["Drop the state, rebuild from the compacted or retained topic"]
  R1 --> N1["Needs committed offsets that have not expired"]
  R2 --> N2["Needs records still within retention"]
  R3 --> N3["Needs dead letter retention and a replay tool"]
  R4 --> N4["Needs events that carry full state"]
  N1 --> G["Consumer ignores stale versions and repeats"]
  N2 --> G
  N3 --> G
  N4 --> G
Choosing a recovery method: each branch has its own precondition, and all of them rest on the same consumer property.
  • The questions are ordered from cheapest to most expensive. Most incidents end at the first box: the group restarts and repeats a few records.
  • Each method has a precondition that has to exist before the incident: unexpired committed offsets, topic retention long enough to cover the window, a dead letter topic with a replay tool, or events that carry full state.
  • All four methods deliver records the consumer has seen before, or records older than its current state. That is why they converge on one requirement. Without it, each method repairs one problem and creates another.
  • Methods can be combined: reset offsets to reprocess a window, then replay the dead letters that the bug produced during that window.

Message flow

Drawing diagram…

Sequence diagram. The kubelet sends SIGTERM to the Spring Boot application, which stops the listener container. The container wakes the consumer, lets the listener finish the record at offset 207, commits offset 208 for partition orders-3, closes the consumer and leaves the group. The coordinator rebalances and assigns orders-3 to consumer B, which fetches the committed offset 208 and continues there. The process exits before the grace period ends.

sequenceDiagram
  participant K as Kubelet
  participant A as Spring Boot app
  participant C as Listener container
  participant L as Listener
  participant G as Group coordinator
  participant B as Consumer B
  K->>A: SIGTERM, grace period starts
  A->>C: stop
  C->>C: wake consumer, no further polls
  C->>L: finish record at offset 207
  L-->>C: done
  C->>G: commit offset 208 for orders-3
  C->>G: close consumer, leave group
  C-->>A: stopped within shutdownTimeout
  G->>B: rebalance, assign orders-3
  B->>G: fetch committed offset for orders-3
  G-->>B: 208
  B->>B: continue at offset 208
  A-->>K: process exits before grace period ends
A clean stop during a rolling update: finish, commit, leave, and the next owner starts exactly where this one stopped.
  1. The commit says 208, the next record to read, after offset 207 was fully processed. Consumer B repeats nothing and skips nothing.
  2. Leaving the group is what makes the handover immediate. If the process is killed before it leaves, the coordinator notices only when heartbeats have been missing for the session timeout, and partition orders-3 is not read during that time.
  3. If SIGKILL arrives while the listener is still on offset 207, there is no commit. Consumer B starts from the last committed offset, which may be several records back, and processes those records again.
  4. Every step after SIGTERM is bounded by a timeout. The diagram is only true when the listener's remaining work fits in the container's, Spring's and the kubelet's limits.

In a real system

A typical order pipeline: the order service publishes OrderStatusChanged to the topic order-status (12 partitions, key = order id, 7 days retention). Each event carries the order's full current state and a version that the order service increments in the same transaction as the change. The notification service (group notification-service, four pods) keeps a table order_status with one row per order and emails the customer when the status changes. A second, compacted topic product-prices (key = product id) feeds an in-memory price view in each pod.

Ordering. A payment retry delays PAID (version 4) for order 7781 until after CANCELLED (version 5) has been processed. The upsert updates only where the stored version is lower, so version 4 changes no row, no email is sent, and the offset advances.

Shutdown. A rolling update replaces the four pods one after another. Each pod receives SIGTERM, finishes its current record (immediate-stop is on, and a record takes about 80 ms), commits and leaves the group. The remaining pods take over its partitions within a second or two. The pod's grace period is 60 seconds, far more than it needs.

Recovery. A release at 09:15 contains a bug that stores the wrong status for refunds. It is rolled back at 12:05. The team scales the deployment to zero, runs the reset with --to-datetime 2025-03-04T09:10:00.000 first without and then with --execute, and scales back up. Three hours of events are processed again. Orders that have moved on to a newer version since then are left alone by the version rule, and the rows the bug wrote wrongly are corrected (see the note below for the one adjustment this needed). Email sending is switched off for the run, and the few customers who had been told a wrong refund status are contacted separately.

Java and Spring Boot

Four pieces: a consumer that ignores stale events, the shutdown settings from the container out to Kubernetes, a listener that rebuilds a view from a compacted topic, and the plain-client shutdown loop that shows what Spring is doing for you. Written against the Spring for Apache Kafka 3.3 and Kafka client documentation, without a compiler at hand.

src/main/java/com/example/notifications/OrderStatusListener.javaJava
package com.example.notifications;import java.time.Instant;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;@Componentpublic class OrderStatusListener {    /** Full current state of the order, not a change to apply. */    public record OrderStatusChanged(String orderId, long version, String status, long totalMinor,            Instant occurredAt) {    }    private final JdbcTemplate jdbc;    private final EmailOutbox emails;    public OrderStatusListener(JdbcTemplate jdbc, EmailOutbox emails) {        this.jdbc = jdbc;        this.emails = emails;    }    @KafkaListener(id = "order-status", topics = "order-status", groupId = "notification-service")    @Transactional    public void onStatus(OrderStatusChanged event) {        // PostgreSQL: insert the first version, or update only when the incoming version is higher.        int changed = jdbc.update("""                INSERT INTO order_status (order_id, version, status, total_minor)                VALUES (?, ?, ?, ?)                ON CONFLICT (order_id) DO UPDATE                   SET version = EXCLUDED.version,                       status = EXCLUDED.status,                       total_minor = EXCLUDED.total_minor                 WHERE order_status.version < EXCLUDED.version                """, event.orderId(), event.version(), event.status(), event.totalMinor());        if (changed == 0) {            // Older than, or the same as, what we already hold: a late, repeated or replayed event.            // Returning normally is correct. The container commits and moves on.            return;        }        // Same database transaction as the row above, sent later by a separate relay.        emails.enqueue(event.orderId(), event.version(), event.status());    }}

Why it is written this way

  • The comparison is done by the database inside the write, not by a read followed by a write in Java. Two pods that briefly hold the same partition during a rebalance cannot both win.
  • An ignored event returns normally. Throwing here would send every late event through the retry and dead letter path, although nothing is wrong with it.
  • The email is recorded in the same database transaction as the state change and sent by a relay (the outbox pattern), keyed by order id and version. A redelivered event finds the version already applied and enqueues nothing, so reprocessing a window does not send mail again.
  • @Transactional here is a database transaction. It does not include the Kafka offset commit, which happens afterwards; a crash in between repeats the record, and the version check absorbs it.
  • The event is a state event. If it said only that the status changed from PAID to CANCELLED, a missed or reordered event could not be repaired by a later one.

Configuration

The settings behind the three topics of this lesson, with the defaults documented for Kafka 4.1 and Spring for Apache Kafka 3.3 / Spring Boot 3.5.

ContainerProperties.shutdownTimeoutDefault 10000 (ms)
What it does
Longest time stop() blocks while the consumer threads finish, commit and close, before the container stopped event is published.
When to change it
Set it above the worst-case time of the work that remains at stop: one record with immediate stop, otherwise the rest of a poll.
spring.kafka.listener.immediate-stop (ContainerProperties.stopImmediate)Default false
What it does
When false, a stopping container still processes all records from the previous poll. When true, it stops after the current record.
When to change it
Turn it on when a poll can hold more work than your shutdown budget. The unprocessed records are not committed and are read by the next owner.
spring.lifecycle.timeout-per-shutdown-phaseDefault 30s
What it does
Spring Boot's limit for stopping all SmartLifecycle beans of one phase, which includes the listener containers.
When to change it
Keep it above shutdownTimeout and below the pod's termination grace period.
terminationGracePeriodSeconds (Kubernetes pod spec)Default 30
What it does
Time between SIGTERM and SIGKILL when a pod is deleted. A preStop hook, if any, uses the same budget.
When to change it
Raise it above the Spring timeouts. If it is shorter, the settings inside the JVM are irrelevant: the process is killed first.
max.poll.recordsDefault 500
What it does
Maximum number of records one poll() returns. It bounds the work in hand when a stop or a rebalance begins.
When to change it
Lower it when records are slow to process, so that both shutdown and max.poll.interval.ms have room.
session.timeout.msDefault 45000 (45 seconds)
What it does
With the classic group protocol, how long the coordinator waits without heartbeats before removing a member and rebalancing. With group.protocol=consumer the broker setting group.consumer.session.timeout.ms applies instead.
When to change it
This is how long partitions stay unread after a hard kill, and after any stop of a static member. Lower it for faster failover, at the price of more false alarms.
group.instance.idDefault null
What it does
Makes the consumer a static member. Its partitions are kept for it across a restart instead of being reassigned at once.
When to change it
Use it with stable identities (for example StatefulSet pod names) when restarts finish within the session timeout. Leave it unset if pods are replaced by new identities.
auto.offset.resetDefault latest
What it does
Where to start when the group has no committed offset for a partition, or the committed offset no longer exists: earliest, latest, by_duration (newer clients) or none, which throws.
When to change it
Choose it deliberately per consumer. With latest, a group whose offsets expired skips everything produced while it was away.
offsets.retention.minutes (broker)Default 10080
What it does
How long committed offsets are kept after a group has lost all its members, 7 days by default.
When to change it
Raise it if consumers can legitimately be stopped for longer, such as batch jobs or disabled features.
retention.ms (topic)Default 604800000 (7 days)
What it does
How long records are kept under the delete policy. It is the furthest back any offset reset or rebuild can reach.
When to change it
Set it from your recovery needs: at least the longest period after which you would still want to reprocess.
delete.retention.ms (topic)Default 86400000 (1 day)
What it does
How long tombstones stay in a compacted topic. A reader starting at offset 0 must finish within this time to be sure of seeing every deletion.
When to change it
Raise it if a full rebuild from the compacted topic can take longer than a day.

Hands-on exercise

Kill a consumer between processing and commit

Crash a consumer at the worst moment, see exactly which records are repeated, and find out what the consumer needs in order to make that harmless. Then compare it with a clean stop.

Steps · 0 of 7 done

Crash after processing, before the commit

Simulation · not a real cluster

The consumer has charged the payment but has not committed. Crash it there, restart, and watch what is delivered again. Then switch on the idempotent consumer.

Take it further in the lab: Consumer crash before offset commit

What can go wrong

Last write wins on arrival order
Symptom
Now and then an entity shows an old state: a cancelled order reappears as paid, or a stock level jumps back. It follows retries, replays or deployments.
Cause
The consumer applies each event as the newest truth. A late or redelivered event overwrites newer state.
Fix
Carry a version from the owning service in each event, store it with the derived state and update only when the incoming version is higher. Make the comparison part of the write statement.
Killed before the commit on every deployment
Symptom
Each rollout produces a burst of duplicate side effects, and pod logs end without a consumer-stopped line. Pods take the full grace period to disappear.
Cause
The remaining records of a poll take longer than the shutdown timeouts, or the JVM never receives SIGTERM because a shell wrapper is PID 1. The process is killed with uncommitted work.
Fix
Start the JVM in exec form, enable immediate stop or lower max.poll.records, and nest the three timeouts: container below Spring phase below pod grace period.
Partitions idle after a pod is removed
Symptom
After scaling down, some partitions show growing lag for about 45 seconds although other consumers are healthy.
Cause
The consumer did not leave the group: it was killed hard, or it is a static member, whose partitions are held until the session timeout.
Fix
Make the shutdown clean so the consumer leaves. For static members this wait is by design; shorten session.timeout.ms or drop static membership if pods do not return under the same identity.
A group that was stopped too long skips its backlog
Symptom
A consumer that was disabled for two weeks is switched back on. It shows zero lag immediately and has processed nothing from those two weeks.
Cause
Its committed offsets expired after offsets.retention.minutes. With no committed offset, auto.offset.reset=latest placed it at the end of each partition.
Fix
Set auto.offset.reset on purpose (earliest for idempotent consumers, none if a human should decide), and raise offsets.retention.minutes if long stops are normal. Also check that the topic's retention covers the stop.
Offset reset repeats side effects
Symptom
After rewinding a group by three hours to repair data, customers receive the emails of those three hours again.
Cause
The database writes were idempotent, but the email, HTTP call or downstream event was triggered for every record processed.
Fix
Tie each side effect to a state change that actually happened (an outbox row keyed by entity and version), or run the reprocessing with side effects switched off.
Reset attempted while the group is running
Symptom
The reset command prints an error saying assignments can only be reset if the group is inactive, and the offsets are unchanged.
Cause
Group members are still connected. The tool refuses to move offsets under a live consumer, which would overwrite them with its next commit anyway.
Fix
Stop all members, confirm with --describe --state that the group has none, run the reset with --execute, and only then start the consumers.

How to troubleshoot

You need to know where a group is and whether it still has members before any recovery.

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group notification-service --state
How to read it
The state and member count tell you whether a reset is possible: it must show no members. Run --describe without --state for CURRENT-OFFSET, LOG-END-OFFSET and LAG for each partition, and save that output before changing anything.
You want to see what a reset would do before doing it.

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group notification-service --topic order-status --reset-offsets --to-datetime 2025-03-04T09:10:00.000
How to read it
Without --execute the command prints the new offset for each partition and changes nothing. Compare it with the saved current offsets: the difference is the number of records that will be processed again.
Duplicates appear after every deployment.

Check

The last log lines of a terminated pod: look for the container's Consumer stopped message or your ConsumerStoppedEvent log line
How to read it
If the line is there, the stop was clean and the duplicates have another cause. If the log ends in the middle of processing, the pod was killed: compare the time from SIGTERM to the end of the log with terminationGracePeriodSeconds.
A rewind does not reach as far back as requested.

Check

kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic order-status --time -2
How to read it
This prints the earliest offset still available for each partition. If it is higher than the offset you wanted, retention has already deleted those records and the reset was adjusted to the earliest available one.
Events for one entity are suspected to be spread over several partitions.

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic order-status --from-beginning --property print.key=true --property print.partition=true --property print.offset=true
How to read it
Filter the output for one order id. More than one partition for the same key means the partition count changed or producers use different keys or partitioners. Kafka 4.2 and later accept --formatter-property in place of --property.
Stale events are being ignored and you want to know how often.

Check

A counter in the listener for writes that changed zero rows, tagged by topic
How to read it
A low steady rate is normal redelivery. A spike after a deployment points to unclean shutdowns; a spike after a replay is expected; a constant high rate suggests a keying problem or several producers racing.

Key takeaways

  • Kafka orders records within one partition as the broker appended them. Different keys, a changed partition count, retries, replays and resets all break per-entity order.
  • Give every event a version from the owning service and apply it only if it is newer than what you hold. That one rule covers late, repeated and replayed events.
  • Prefer events that carry full state over deltas. State events can be skipped, repeated and rebuilt from; deltas cannot.
  • A clean stop is: stop polling, finish the work in hand, commit, leave the group. Spring's container does this on stop(), within shutdownTimeout.
  • The container's shutdown timeout, Spring Boot's phase timeout and the pod's termination grace period must nest, or the process is killed with uncommitted work.
  • Recovery choices are restart, reset offsets, replay the dead letter topic or rebuild state. Each depends on retention you set earlier, and all depend on an idempotent, version-checking consumer.
  • Committed offsets expire 7 days after a group becomes empty, and then auto.offset.reset, default latest, decides where it restarts.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A listener takes about 150 ms per record. max.poll.records is 500, ack mode is BATCH, and all shutdown settings are defaults: container shutdownTimeout 10 s, Spring phase timeout 30 s, pod grace period 30 s. After each rolling update the team sees a few hundred records processed twice.

Which change removes most of these duplicates at the lowest cost?

Scenario 2 of 3

An inventory consumer receives StockAdjusted events with a delta, for example minus 2, keyed by SKU. After a consumer bug, 1,200 events sit in the dead letter topic. Since then about 30,000 newer adjustments for the same SKUs have been applied. The fix is deployed and a colleague wants to replay the dead letter topic tonight.

What should you conclude before the replay runs?

Scenario 3 of 3

A reporting consumer group was switched off 9 days ago during a cost review. The topic has retention.ms of 14 days, the broker uses the default offsets.retention.minutes, and the consumer does not set auto.offset.reset. It is idempotent. You are asked to turn it back on so the reports are complete.

What do you do before starting it?

References

References: Apache Kafka 4.1 documentation: Basic Kafka Operations (managing consumer groups, --reset-offsets) ↗ · Apache Kafka 4.0 Javadoc: KafkaConsumer (wakeup, close, multi-threaded processing) ↗ · Apache Kafka 4.1 Javadoc: ConsumerRebalanceListener (commit on revoke, onPartitionsLost) ↗ · Apache Kafka 4.1 documentation: Consumer configs (session.timeout.ms, auto.offset.reset, group.instance.id) ↗ · Apache Kafka 4.1 documentation: Broker configs (offsets.retention.minutes) ↗ · Apache Kafka 4.1 documentation: Topic configs (retention.ms, delete.retention.ms) ↗ · Spring for Apache Kafka 3.3 reference: Listener Container Properties (shutdownTimeout, stopImmediate) ↗ · Spring for Apache Kafka 3.3 reference: Seeking to a Specific Offset (ConsumerSeekAware) ↗ · Spring for Apache Kafka 3.3 reference: Pausing and Resuming Listener Containers ↗ · Spring for Apache Kafka 3.3 reference: Application Events (ConsumerStoppedEvent) ↗ · Spring Boot reference: Common Application Properties (spring.kafka.listener.immediate-stop, spring.lifecycle.timeout-per-shutdown-phase) ↗ · Kubernetes documentation: Pod Lifecycle, termination of Pods ↗ · Report a problem