Skip to content

Path 6 · lesson 1 of 5

Intermediate25 minApache Kafka 4.x (defaults checked against the 4.4 documentation) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 (kafka-clients 3.9)

What can fail, and what each failure looks like

A map of producer, broker, network, consumer and dependency failures: the symptom your application sees, what Kafka does by itself, and what your code still has to do.

What you will learn

  • Place a failure in one of five areas: producer, broker, network, consumer or downstream dependency
  • Recognise each failure from what the application sees: the exception, the log line or the metric
  • Say what the Kafka client or the cluster does about each failure without any help from your code
  • Classify a failure as transient or permanent and choose between waiting, retrying and setting the record aside
  • Identify the two partial failures that produce duplicates: processed but not committed, and sent but not acknowledged

Before you start

The concept

A Kafka pipeline has five places where something can stop working: the producer, the brokers, the network between any two of them, the consumer, and whatever the consumer calls next (a database, another service). Each failure reaches your application in a different shape. Some arrive as an exception, some as a rebalance, and some only as a lag graph that starts to climb.

Kafka's clients and brokers already handle a large part of this. A producer retries a failed request, the controller elects a new partition leader, the group coordinator hands a dead consumer's partitions to another member. What they cannot do is decide what a failure means for your business operation. That part is your code, and it starts with one question: is this failure transient or permanent?

Transient: the same attempt can succeed later

  • A partition leader moved to another broker
  • The ISR is briefly smaller than min.insync.replicas
  • A request timed out or a connection dropped
  • The database or a downstream service is restarting
  • Right response: wait and retry, keep the record, keep the order

Permanent: the same attempt will fail every time

  • The payload cannot be deserialised
  • The record is larger than the broker accepts
  • The client is not authorised for the topic
  • The data breaks a business rule, or the handler has a bug
  • Right response: do not retry, set the record aside and move on, or stop and fix

Getting the classification wrong is costly in both directions. Retrying a permanent failure blocks a partition forever (the next lesson, Poison messages and error handling). Giving up on a transient failure throws away good records because a database was down for two minutes.

FailureSymptom in your applicationWhat Kafka doesWhat your code must do
Producer cannot get metadata, or its buffer is fullsend() blocks the calling thread for up to max.block.ms (1 minute), then fails with TimeoutException (BufferExhaustedException when the buffer is the cause)Nothing more: the record never entered the producer's bufferTreat it as not sent. Do not block request threads on it without a limit. Slow the source down or fail the request
Leader change during a sendUsually nothing. Latency rises for a few seconds and record-retry-rate goes above zeroThe broker answers NotLeaderOrFollowerException, the producer refreshes metadata and resends to the new leaderNothing, as long as delivery.timeout.ms is longer than an election takes
Not enough in-sync replicasSends with acks=all fail with NotEnoughReplicasException after retrying for delivery.timeout.ms (2 minutes)The leader rejects the write while the ISR is smaller than min.insync.replicas. The producer keeps retryingKeep the event (outbox or fail the request). Do not lower acks to make the error go away
Sent but not acknowledgedThe send future fails with TimeoutException after delivery.timeout.msRetries until the deadline. With idempotence the broker drops a resend it has already appendedTreat the outcome as unknown: the record may be in the log. Publish again later and make consumers idempotent
Broker failureA short pause on the partitions it led. Consumers and producers log a metadata refreshThe controller elects a new leader from the ISR. Clients reconnect by themselvesNothing for acknowledged acks=all writes. Records acknowledged with acks=1 that were not yet replicated are gone
Consumer crashLag grows on its partitions until the session timeout (45 seconds) passes, then another member continuesThe coordinator removes the member and reassigns its partitions. The new owner starts from the last committed offsetHandle the records after the last commit a second time without doing harm
Slow processingRebalances with no deploy, CommitFailedException, the same records processed by two instancesA consumer that does not call poll() within max.poll.interval.ms (5 minutes) leaves the groupMake one poll's worth of work fit: lower max.poll.records, speed the handler up, or raise the interval
Downstream dependency downEvery record fails with the same exception, on every partition, starting at the same minuteNothing. Kafka does not know your database existsRetry with back-off or pause the listener. Do not commit past the records and do not dead-letter the whole stream
The map. Timeouts are the client defaults in Kafka 4.x.

Why it exists

In a single process, an operation either returns or throws, and after an exception you know it did not happen. Between processes there is a third outcome: you do not know. The request left, no answer came back. The other side may have done the work, may be doing it now, or may never have seen the request. Every reliability feature in Kafka exists to shrink that third outcome or to make it safe.

  • Replication and the ISR exist so that one broker dying does not lose acknowledged records or stop the partition.
  • Producer retries with a delivery deadline exist so that a leader change or a dropped connection is invisible to the application.
  • Producer idempotence exists so that those retries cannot append the same batch twice.
  • Committed offsets and group membership exist so that when a consumer dies, another one continues from a known position.

Each of these moves the problem to a narrower place; none removes it. Replication cannot help when every replica of a partition is down. Idempotence covers the producer's own retries inside one producer session and nothing else. A committed offset is a bookmark for the partition, so the records between the last commit and a crash are delivered again. A map of failures is useful because it tells you which of these mechanisms is already working for you in a given situation and where its cover ends.

Without the map, teams tend to respond to symptoms. A NotEnoughReplicasException appears, someone sets acks=1, the error goes away, and so does the durability guarantee. A rebalance storm appears, someone raises every timeout, and a dead consumer now holds its partitions for ten minutes. The fix that removes the symptom is often the one that removes the protection.

How it works inside

Producer: three clocks

A send passes through three time limits, and which one expires tells you where it got stuck.

  1. max.block.ms (default 1 minute) bounds the time send() itself may block: waiting for topic metadata, or for space in the buffer (buffer.memory, 32 MB by default). If records are produced faster than the brokers accept them, the buffer fills and callers start to block. A failure here means the record was never queued.
  2. request.timeout.ms (default 30 seconds) bounds one request to a broker. When it expires the producer resends, if the delivery deadline allows.
  3. delivery.timeout.ms (default 2 minutes) bounds everything after send() returned: time in the buffer, every attempt, and every wait between attempts. retries defaults to 2147483647, so in practice this deadline, not a retry count, ends the retrying.

Between attempts the client waits retry.backoff.ms (100 ms), growing exponentially up to retry.backoff.max.ms (1 second) with 20% jitter. Errors that extend RetriableException, such as NotLeaderOrFollowerException, NotEnoughReplicasException and request timeouts, are retried inside the client. Errors that cannot improve, such as a record that is too large, a serialisation failure or a missing permission, fail the send immediately.

With enable.idempotence=true (the default, which requires acks=all), every batch carries a producer id and a sequence number. If a retry reaches a leader that already appended the batch, the leader acknowledges it without appending again. This also holds across a leader change, because the sequence numbers are part of the replicated log. It does not hold across a producer restart or an application-level resend: those are new sends with new sequence numbers.

Broker: leader change and ISR shrink

A broker stays alive in the cluster's eyes by keeping a session with the KRaft controller. When that session ends, the controller picks a new leader for each partition the broker led, choosing from the in-sync replicas, and publishes the change in the cluster metadata. Clients that still talk to the old leader are told NotLeaderOrFollowerException, refresh their metadata and carry on. An acknowledged acks=all record is on every in-sync replica, so the new leader has it.

A follower is in sync by time: if it has not caught up to the leader's log end for replica.lag.time.max.ms (30 seconds), the leader removes it from the ISR. The partition keeps working with a smaller ISR. Only when the ISR is smaller than min.insync.replicas does the leader reject acks=all writes with NotEnoughReplicasException. Consumers can still read everything below the high watermark. This is why a replication factor of 3 with min.insync.replicas=2 is the usual setting: one broker can be down with no effect on producers, and the second failure stops writes instead of accepting records that exist on a single disk.

If every in-sync replica is gone, the partition has no leader. unclean.leader.election.enable is false by default, so the partition stays offline until a replica that is known to hold all committed data returns. Producers time out and consumers of that partition receive nothing. Turning unclean election on trades this outage for silent data loss.

Network: the same failure seen from two sides

A network interruption has no exception of its own. Each side sees silence and applies its own timeout. A producer cut off from a leader sees request timeouts. A follower cut off from its leader stops fetching and falls out of the ISR after 30 seconds. A consumer cut off from the group coordinator cannot heartbeat, and after the session timeout the coordinator gives its partitions away, while that consumer may still be running and processing. When the network returns, its commit is rejected because it no longer owns the partitions, and the records it processed in the meantime are processed again by the new owner.

Consumer: two liveness checks

The group has two separate ways to decide that a member is gone. Heartbeats come from a background thread and prove the process is alive: missing them for session.timeout.ms (45 seconds with the classic protocol; with the KIP-848 consumer protocol the broker setting group.consumer.session.timeout.ms, also 45 seconds) removes the member. max.poll.interval.ms (5 minutes) proves the application is making progress: if the listener thread does not come back to poll() in time, the client itself leaves the group, even though heartbeats were fine. A crash trips the first check. A slow handler, a long blocking retry or a stuck database call trips the second.

Either way the result is a rebalance, and the new owner of each partition starts at the last committed offset. With an eager assignor every member gives up all its partitions during the rebalance; with the cooperative assignor or the new protocol only the partitions that move are paused. A rebalance is a recovery mechanism, but each one costs a pause and, for uncommitted work, a repeat.

Partial failures: the two gaps

GapWhat happenedResultDefence
Processed but not committedThe handler finished (the row is written, the card is charged) and the consumer died or lost its partitions before the offset commitThe next owner reads the same record againAn idempotent handler: Duplicates and idempotency
Sent but not acknowledgedThe leader appended the batch and the acknowledgement was lost or lateA client retry is deduplicated by idempotence. An application resend after a timeout is a real duplicateA stable event id in the record, and consumers that deduplicate on it
Database committed, publish failedThe service wrote its own database and then could not publish the eventState changed and nobody was toldWrite the event in the same database transaction: Transactional outbox

Architecture

Drawing diagram…

A flowchart of an order event moving from the Order Service through the producer buffer to the partition leader, its followers, the Payment Service consumer and the payments database, with a labelled failure attached to each hop.

flowchart LR
  APP["Order Service"] --> BUF["Producer buffer"]
  BUF --> LEAD["Partition leader"]
  LEAD --> FOL["Follower replicas"]
  CTRL["KRaft controller"] --> LEAD
  LEAD --> CONS["Payment Service consumer"]
  CONS --> DB["Payments database"]
  CONS --> COORD["Group coordinator"]
  F1["Buffer full or no metadata"] -.-> BUF
  F2["Request timeout or lost ack"] -.-> LEAD
  F3["Follower lags, ISR shrinks"] -.-> FOL
  F4["Broker dies, leader election"] -.-> CTRL
  F5["Crash or slow poll, rebalance"] -.-> COORD
  F6["Database down"] -.-> DB
One order event from producer to database, with the six points where it can get stuck.
  • Read the solid arrows as the path of one record and the dotted arrows as the failures of this lesson. Each failure attaches to one hop, and the hop tells you who notices first: the caller of send(), the producer's sender thread, the leader, the controller, the coordinator or your listener.
  • The first three failures are on the write path and end as a failed send or as extra latency. The last three are on the read path and end as lag, a rebalance or a failing listener. A single network fault can light up several of them at once.
  • Nothing in the diagram watches the last hop. Kafka has no knowledge of the database, so a failure there is visible only through your own exceptions and the lag they cause.

Message flow

Drawing diagram…

A sequence diagram in which a producer sends a batch to broker 1, broker 1 replicates it to broker 2 and dies, the controller makes broker 2 the leader, and the producer's retry is acknowledged by broker 2 without a second append.

sequenceDiagram
  participant P as Producer
  participant B1 as Broker 1 old leader
  participant C as Controller
  participant B2 as Broker 2 follower
  P->>B1: Produce batch seq 7 with acks=all
  B1->>B2: Replicate batch seq 7
  Note over B1: Broker 1 dies before it answers
  P--xB1: No response within request.timeout.ms
  C->>B2: Broker 2 is the new leader
  P->>B2: Metadata request
  B2-->>P: Leader is broker 2
  P->>B2: Retry batch seq 7
  Note over B2: Sequence 7 is already in the log
  B2-->>P: Ack with the original offset
  Note over P: Application sees one successful send
A leader dies after replicating a batch and before acknowledging it. The retry is deduplicated by sequence number.
  1. The producer cannot tell whether broker 1 appended the batch. It only knows that no answer came. This is the sent but not acknowledged gap, and the client's answer to it is to send the same batch again.
  2. Broker 2 already holds the batch because it was an in-sync follower. The producer id and sequence number were replicated with the records, so broker 2 recognises the retry and acknowledges it without writing a second copy.
  3. Take idempotence away and the same sequence of events appends the batch twice. Take acks=all away and broker 1 may have acknowledged a batch that broker 2 never received.
  4. All of this must fit inside delivery.timeout.ms. If the election takes longer than the deadline, the application gets a TimeoutException and is back to not knowing.

In a real system

Take a typical order flow, the one used in the example project. The Order Service publishes OrderPlaced to the topic orders.placed: 6 partitions, replication factor 3, min.insync.replicas=2, keyed by order id. The Payment Service reads it in the group payment-service with three instances, charges the card through a payment provider and writes the result to its own database.

During a rolling restart of the brokers, each broker is down for about a minute. Producers see a burst of retries and a latency spike at each step, and no errors: an election fits easily inside the two-minute delivery deadline. One broker then fails to come back, and a second is restarted anyway. Partitions that had both of those brokers as replicas now have an ISR of one. The Order Service starts receiving NotEnoughReplicasException for those partitions only, so roughly a third of orders fail to publish while the rest work. The symptom looks random until you group the failures by partition.

The same week, the payments database fails over for 90 seconds. Every Payment Service instance throws on every record. If the error handler retries ten times with no delay and then moves on, all records that arrived in those 90 seconds are skipped or dead-lettered within seconds, although nothing was wrong with any of them. The correct behaviour is the opposite of what a poison message needs: keep the records, wait, and try again.

What the team sawAreaClassRight response
Latency spike, record-retry-rate above zero, no errorsBroker: leader changeTransientNone. The client handled it
NotEnoughReplicasException on two of six partitionsBroker: ISR below minimumTransientRestore the broker. Keep events in the outbox meanwhile
All records fail for 90 seconds with a connection errorDependencyTransientBack off and retry in place, or pause the listener
One record fails with a parse error on every deliveryRecordPermanentSend it to a dead letter topic and continue

Java and Spring Boot

Two pieces of code decide what a failure means: the producer's completion callback and the consumer's exception types. Both make the transient or permanent decision explicit, so the framework can act on it.

order-service/src/main/java/com/example/orders/OrderEventPublisher.javaJava
package com.example.orders;import org.apache.kafka.common.errors.RetriableException;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Component;@Componentpublic class OrderEventPublisher {    private static final Logger log = LoggerFactory.getLogger(OrderEventPublisher.class);    private static final String TOPIC = "orders.placed";    private final KafkaTemplate<String, OrderPlaced> kafkaTemplate;    private final PendingEventStore pendingEvents;    public OrderEventPublisher(KafkaTemplate<String, OrderPlaced> kafkaTemplate,                               PendingEventStore pendingEvents) {        this.kafkaTemplate = kafkaTemplate;        this.pendingEvents = pendingEvents;    }    public void publish(OrderPlaced event) {        kafkaTemplate.send(TOPIC, event.orderId(), event)                .whenComplete((result, failure) -> {                    if (failure == null) {                        pendingEvents.markPublished(event.eventId());                        return;                    }                    Throwable cause = failure.getCause() != null ? failure.getCause() : failure;                    if (cause instanceof RetriableException) {                        // Includes TimeoutException: the client gave up waiting.                        // The record may or may not be in the log, so keep it pending.                        log.warn("Outcome unknown for event {}: {}", event.eventId(), cause.toString());                        pendingEvents.keepForRepublish(event.eventId());                    } else {                        // Too large, not authorised, cannot serialise: a resend cannot help.                        log.error("Event {} can never be published", event.eventId(), cause);                        pendingEvents.markRejected(event.eventId(), cause.getMessage());                    }                });    }}

Why it is written this way

  • KafkaTemplate.send returns a CompletableFuture. The failure arrives there, on the producer's sender thread, long after the calling method returned. Code that ignores the future never learns that a send failed.
  • The client has already retried every RetriableException until delivery.timeout.ms. Seeing one here means the deadline passed, so the outcome is unknown, not negative.
  • keepForRepublish sends the same event again later with the same eventId. That resend is a new record as far as the broker is concerned, which is why consumers must deduplicate on the event id.
  • PendingEventStore stands for a table of events written with the order. The full version of that idea is the transactional outbox.

Configuration

These settings are the timers behind every symptom in this lesson. Defaults are from the Kafka 4.4 configuration reference; the ones marked as changed differ on older clients.

delivery.timeout.msDefault 120000 (2 minutes)
What it does
Producer. Upper bound on the time to report success or failure after send() returns, covering buffering, every attempt and every back-off.
When to change it
Keep it longer than a leader election and a broker restart take in your cluster. It must be at least request.timeout.ms + linger.ms.
request.timeout.msDefault 30000 (30 seconds)
What it does
Producer and consumer. How long the client waits for the response to one request before resending or failing it.
When to change it
The producer documentation advises keeping it larger than the broker's replica.lag.time.max.ms, to avoid needless retries.
max.block.msDefault 60000 (1 minute)
What it does
Producer. How long send() may block waiting for metadata or buffer space.
When to change it
Lower it when send() is called from request threads, so a broker outage does not become a thread-pool outage.
retriesDefault 2147483647
What it does
Producer. Number of resends for a transient error. Bounded in practice by delivery.timeout.ms.
When to change it
Leave it. The documentation recommends controlling retry behaviour with delivery.timeout.ms instead.
retry.backoff.ms / retry.backoff.max.msDefault 100 / 1000 (1 second)
What it does
Client. First wait before retrying a failed request, growing exponentially to the maximum, with 20% jitter.
When to change it
Rarely changed. This is the client's own request retry and is separate from any retry of your listener.
session.timeout.msDefault 45000 (45 seconds)
What it does
Consumer, classic group protocol. Time without heartbeats after which the coordinator removes the member. With group.protocol=consumer the broker's group.consumer.session.timeout.ms (also 45 seconds) applies instead.
When to change it
Lower it to detect a crashed consumer sooner, at the price of rebalances during long garbage-collection pauses or short network blips.
max.poll.interval.msDefault 300000 (5 minutes)
What it does
Consumer. Longest allowed gap between two poll() calls before the consumer leaves the group.
When to change it
Size it for the slowest realistic batch: max.poll.records times the worst processing time, plus any blocking retry delay.
replica.lag.time.max.msDefault 30000 (30 seconds)
What it does
Broker. A follower that has not caught up to the leader's log end for this long is removed from the ISR.
When to change it
Seldom changed. It sets how long a slow follower can delay acks=all writes before it is dropped.
min.insync.replicasDefault 1
What it does
Broker or topic. Smallest ISR for which the leader accepts an acks=all write.
When to change it
Set it to 2 with replication factor 3. The default of 1 lets a write be acknowledged while it exists on one broker only.
unclean.leader.election.enableDefault false
What it does
Broker or topic. Whether a replica outside the ISR may become leader when no in-sync replica is available.
When to change it
Leave it off for anything you cannot afford to lose. Turning it on restores availability by discarding records.

Hands-on exercise

Break a broker and read the cluster's reaction

Stop brokers in a three-broker cluster and connect what the cluster does (leader election, ISR shrink) to what a producer with acks=all would see at each step.

Steps · 0 of 5 done

Stop brokers and watch the ISR

Simulation · not a real cluster

Three brokers, one topic with replication factor 3 and min.insync.replicas 2. Stop brokers one at a time and read the leader and ISR of each partition.

Take it further in the lab: Broker failure and leader election

What can go wrong

Lowering acks to silence NotEnoughReplicasException
Symptom
After a broker incident someone sets acks=1. The exceptions stop. Weeks later a broker dies and a few acknowledged orders are missing from the topic.
Cause
The exception was the protection working: the leader refused to acknowledge a write that only it held. acks=1 removes the refusal, not the risk.
Fix
Keep acks=all with min.insync.replicas=2. Fix the under-replicated partitions, and hold events in an outbox while writes are rejected.
Reading a send timeout as a clean failure
Symptom
Duplicate orders downstream after a network incident, although the producer is idempotent.
Cause
The application caught TimeoutException, assumed the record was not written and published the event again. The first copy had been appended. Idempotence covers the client's own retries only.
Fix
Treat a timeout as an unknown outcome. Resend with the same event id and deduplicate in the consumer.
Treating a dependency outage like a bad record
Symptom
A database failover of one minute puts thousands of healthy records into the dead letter topic.
Cause
The error handler gives every exception the same few fast retries and then recovers the record. During an outage every record exhausts its retries within milliseconds.
Fix
Throw distinct exception types. Retry dependency failures with a long, capped back-off or pause the listener; dead-letter only failures that belong to the record.
Raising max.poll.interval.ms until the rebalances stop
Symptom
Rebalances stop, but when a handler hangs on a stuck call its partitions sit unprocessed for the whole interval.
Cause
max.poll.interval.ms is the only check that the application is making progress. A very large value disables it.
Fix
First make the work fit: lower max.poll.records and put timeouts on downstream calls. Then set the interval with a reasoned margin.
Blocking request threads in send()
Symptom
During a broker outage the web service stops answering every endpoint, including ones that do not publish.
Cause
send() blocks for up to max.block.ms (1 minute by default) when metadata is unavailable or the buffer is full. Enough blocked callers exhaust the request thread pool.
Fix
Lower max.block.ms, or write the event to an outbox table in the request and publish from a separate thread.

How to troubleshoot

Sends fail on some partitions only

Check

kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --under-min-isr-partitions
How to read it
Lists partitions whose ISR is smaller than min.insync.replicas. If the failing partitions are in the list, the producer is being refused correctly. Find the brokers missing from the Isr column.
Latency spikes on the producer with no errors

Check

Producer metrics record-retry-rate and record-error-rate (kafka.producer:type=producer-metrics)
How to read it
Retries above zero with an error rate of zero means transient broker errors that the client absorbed, typically leader changes. An error rate above zero means sends are reaching the application as failures.
send() is slow or throws TimeoutException before anything is sent

Check

Producer metrics bufferpool-wait-ratio and buffer-available-bytes
How to read it
A wait ratio above zero and little available buffer means callers are blocking on a full buffer: the brokers accept records more slowly than the application produces them.
Brokers report trouble but you do not know which kind

Check

Broker metrics UnderReplicatedPartitions, UnderMinIsrPartitionCount and IsrShrinksPerSec (kafka.server:type=ReplicaManager)
How to read it
Under-replicated above zero means a follower is missing or slow and producers are still fine. Under-min-ISR above zero means acks=all producers are being rejected. A shrink rate without a broker restart points at a slow follower or a network problem between brokers.
Lag grows on a few partitions and nobody seems to own them

Check

kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group payment-service
How to read it
Each row shows the committed offset, the log end offset, the lag and the member that owns the partition. Rows without a consumer id have no owner right now: a member died and the session timeout has not passed, or a rebalance is in progress.
Rebalances with no deploy and no crash

Check

Consumer metrics time-between-poll-max and last-poll-seconds-ago, and CommitFailedException in the consumer log
How to read it
A maximum time between polls close to max.poll.interval.ms means the listener is too slow for the interval and the client is leaving the group by itself. Heartbeat problems would not show here.

Key takeaways

  • Sort every failure into producer, broker, network, consumer or dependency first, then into transient or permanent. The response follows from those two answers.
  • Kafka handles leader changes, broker failures and dead consumers by itself. It does not handle your database, your business rules or the meaning of a timeout.
  • A producer timeout means the outcome is unknown. Idempotence removes duplicates from the client's own retries, not from an application resend.
  • NotEnoughReplicasException is the durability guarantee refusing a risky write. Fix the ISR, not the acks setting.
  • A consumer leaves the group for two different reasons: no heartbeats for the session timeout, or no poll() within max.poll.interval.ms.
  • The two partial failures, processed but not committed and sent but not acknowledged, both end in a duplicate. Idempotent consumers are the common defence.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

An Order Service produces to orders.placed (6 partitions, replication factor 3, min.insync.replicas=2, acks=all). Since 14:05 about a third of sends fail with NotEnoughReplicasException after two minutes each. The failures are all on partitions 1 and 4. Broker 2 is down for a disk replacement and broker 3 was restarted at 14:04 and is still recovering its logs.

What should the team do now?

Scenario 2 of 3

A Payment Service consumer group shows a rebalance every six to seven minutes. No pods restart. The logs contain CommitFailedException. Heartbeat metrics look normal. max.poll.records is 500, max.poll.interval.ms is the default, and since yesterday's release each record calls a fraud API that takes about 800 ms.

Which explanation fits these symptoms?

Scenario 3 of 3

A producer with default settings sends an OrderPlaced event. Two minutes later the future completes with TimeoutException. The service catches it, logs publish failed and immediately sends the same event again, which succeeds. The next day finance reports that this order was charged twice.

What is the most likely cause of the double charge?

References

References: Apache Kafka documentation: Producer configs (delivery.timeout.ms, request.timeout.ms, max.block.ms, retries, enable.idempotence) ↗ · Apache Kafka documentation: Consumer configs (session.timeout.ms, max.poll.interval.ms, group.protocol) ↗ · Apache Kafka documentation: Design, replication (ISR, unclean leader election, availability and durability guarantees) ↗ · Apache Kafka documentation: Broker configs (replica.lag.time.max.ms, min.insync.replicas, unclean.leader.election.enable) ↗ · Apache Kafka documentation: Monitoring (producer, consumer and replica manager metrics) ↗ · Apache Kafka Javadoc: KafkaConsumer (detecting consumer failures) ↗ · Apache Kafka Javadoc: BufferExhaustedException ↗ · Spring for Apache Kafka reference: Sending messages with KafkaTemplate ↗ · Report a problem