Path 6 · lesson 1 of 5
What can fail, and what each failure looks like
A map of producer, broker, network, consumer and dependency failures: the symptom your application sees, what Kafka does by itself, and what your code still has to do.
What you will learn
- Place a failure in one of five areas: producer, broker, network, consumer or downstream dependency
- Recognise each failure from what the application sees: the exception, the log line or the metric
- Say what the Kafka client or the cluster does about each failure without any help from your code
- Classify a failure as transient or permanent and choose between waiting, retrying and setting the record aside
- Identify the two partial failures that produce duplicates: processed but not committed, and sent but not acknowledged
Before you start
The concept
A Kafka pipeline has five places where something can stop working: the producer, the brokers, the network between any two of them, the consumer, and whatever the consumer calls next (a database, another service). Each failure reaches your application in a different shape. Some arrive as an exception, some as a rebalance, and some only as a lag graph that starts to climb.
Kafka's clients and brokers already handle a large part of this. A producer retries a failed request, the controller elects a new partition leader, the group coordinator hands a dead consumer's partitions to another member. What they cannot do is decide what a failure means for your business operation. That part is your code, and it starts with one question: is this failure transient or permanent?
Transient: the same attempt can succeed later
- A partition leader moved to another broker
- The ISR is briefly smaller than
min.insync.replicas - A request timed out or a connection dropped
- The database or a downstream service is restarting
- Right response: wait and retry, keep the record, keep the order
Permanent: the same attempt will fail every time
- The payload cannot be deserialised
- The record is larger than the broker accepts
- The client is not authorised for the topic
- The data breaks a business rule, or the handler has a bug
- Right response: do not retry, set the record aside and move on, or stop and fix
Getting the classification wrong is costly in both directions. Retrying a permanent failure blocks a partition forever (the next lesson, Poison messages and error handling). Giving up on a transient failure throws away good records because a database was down for two minutes.
| Failure | Symptom in your application | What Kafka does | What your code must do |
|---|---|---|---|
| Producer cannot get metadata, or its buffer is full | send() blocks the calling thread for up to max.block.ms (1 minute), then fails with TimeoutException (BufferExhaustedException when the buffer is the cause) | Nothing more: the record never entered the producer's buffer | Treat it as not sent. Do not block request threads on it without a limit. Slow the source down or fail the request |
| Leader change during a send | Usually nothing. Latency rises for a few seconds and record-retry-rate goes above zero | The broker answers NotLeaderOrFollowerException, the producer refreshes metadata and resends to the new leader | Nothing, as long as delivery.timeout.ms is longer than an election takes |
| Not enough in-sync replicas | Sends with acks=all fail with NotEnoughReplicasException after retrying for delivery.timeout.ms (2 minutes) | The leader rejects the write while the ISR is smaller than min.insync.replicas. The producer keeps retrying | Keep the event (outbox or fail the request). Do not lower acks to make the error go away |
| Sent but not acknowledged | The send future fails with TimeoutException after delivery.timeout.ms | Retries until the deadline. With idempotence the broker drops a resend it has already appended | Treat the outcome as unknown: the record may be in the log. Publish again later and make consumers idempotent |
| Broker failure | A short pause on the partitions it led. Consumers and producers log a metadata refresh | The controller elects a new leader from the ISR. Clients reconnect by themselves | Nothing for acknowledged acks=all writes. Records acknowledged with acks=1 that were not yet replicated are gone |
| Consumer crash | Lag grows on its partitions until the session timeout (45 seconds) passes, then another member continues | The coordinator removes the member and reassigns its partitions. The new owner starts from the last committed offset | Handle the records after the last commit a second time without doing harm |
| Slow processing | Rebalances with no deploy, CommitFailedException, the same records processed by two instances | A consumer that does not call poll() within max.poll.interval.ms (5 minutes) leaves the group | Make one poll's worth of work fit: lower max.poll.records, speed the handler up, or raise the interval |
| Downstream dependency down | Every record fails with the same exception, on every partition, starting at the same minute | Nothing. Kafka does not know your database exists | Retry with back-off or pause the listener. Do not commit past the records and do not dead-letter the whole stream |
Why it exists
In a single process, an operation either returns or throws, and after an exception you know it did not happen. Between processes there is a third outcome: you do not know. The request left, no answer came back. The other side may have done the work, may be doing it now, or may never have seen the request. Every reliability feature in Kafka exists to shrink that third outcome or to make it safe.
- Replication and the ISR exist so that one broker dying does not lose acknowledged records or stop the partition.
- Producer retries with a delivery deadline exist so that a leader change or a dropped connection is invisible to the application.
- Producer idempotence exists so that those retries cannot append the same batch twice.
- Committed offsets and group membership exist so that when a consumer dies, another one continues from a known position.
Each of these moves the problem to a narrower place; none removes it. Replication cannot help when every replica of a partition is down. Idempotence covers the producer's own retries inside one producer session and nothing else. A committed offset is a bookmark for the partition, so the records between the last commit and a crash are delivered again. A map of failures is useful because it tells you which of these mechanisms is already working for you in a given situation and where its cover ends.
Without the map, teams tend to respond to symptoms. A NotEnoughReplicasException appears, someone sets acks=1, the error goes away, and so does the durability guarantee. A rebalance storm appears, someone raises every timeout, and a dead consumer now holds its partitions for ten minutes. The fix that removes the symptom is often the one that removes the protection.
How it works inside
Producer: three clocks
A send passes through three time limits, and which one expires tells you where it got stuck.
max.block.ms(default 1 minute) bounds the timesend()itself may block: waiting for topic metadata, or for space in the buffer (buffer.memory, 32 MB by default). If records are produced faster than the brokers accept them, the buffer fills and callers start to block. A failure here means the record was never queued.request.timeout.ms(default 30 seconds) bounds one request to a broker. When it expires the producer resends, if the delivery deadline allows.delivery.timeout.ms(default 2 minutes) bounds everything aftersend()returned: time in the buffer, every attempt, and every wait between attempts.retriesdefaults to 2147483647, so in practice this deadline, not a retry count, ends the retrying.
Between attempts the client waits retry.backoff.ms (100 ms), growing exponentially up to retry.backoff.max.ms (1 second) with 20% jitter. Errors that extend RetriableException, such as NotLeaderOrFollowerException, NotEnoughReplicasException and request timeouts, are retried inside the client. Errors that cannot improve, such as a record that is too large, a serialisation failure or a missing permission, fail the send immediately.
With enable.idempotence=true (the default, which requires acks=all), every batch carries a producer id and a sequence number. If a retry reaches a leader that already appended the batch, the leader acknowledges it without appending again. This also holds across a leader change, because the sequence numbers are part of the replicated log. It does not hold across a producer restart or an application-level resend: those are new sends with new sequence numbers.
Broker: leader change and ISR shrink
A broker stays alive in the cluster's eyes by keeping a session with the KRaft controller. When that session ends, the controller picks a new leader for each partition the broker led, choosing from the in-sync replicas, and publishes the change in the cluster metadata. Clients that still talk to the old leader are told NotLeaderOrFollowerException, refresh their metadata and carry on. An acknowledged acks=all record is on every in-sync replica, so the new leader has it.
A follower is in sync by time: if it has not caught up to the leader's log end for replica.lag.time.max.ms (30 seconds), the leader removes it from the ISR. The partition keeps working with a smaller ISR. Only when the ISR is smaller than min.insync.replicas does the leader reject acks=all writes with NotEnoughReplicasException. Consumers can still read everything below the high watermark. This is why a replication factor of 3 with min.insync.replicas=2 is the usual setting: one broker can be down with no effect on producers, and the second failure stops writes instead of accepting records that exist on a single disk.
If every in-sync replica is gone, the partition has no leader. unclean.leader.election.enable is false by default, so the partition stays offline until a replica that is known to hold all committed data returns. Producers time out and consumers of that partition receive nothing. Turning unclean election on trades this outage for silent data loss.
Network: the same failure seen from two sides
A network interruption has no exception of its own. Each side sees silence and applies its own timeout. A producer cut off from a leader sees request timeouts. A follower cut off from its leader stops fetching and falls out of the ISR after 30 seconds. A consumer cut off from the group coordinator cannot heartbeat, and after the session timeout the coordinator gives its partitions away, while that consumer may still be running and processing. When the network returns, its commit is rejected because it no longer owns the partitions, and the records it processed in the meantime are processed again by the new owner.
Consumer: two liveness checks
The group has two separate ways to decide that a member is gone. Heartbeats come from a background thread and prove the process is alive: missing them for session.timeout.ms (45 seconds with the classic protocol; with the KIP-848 consumer protocol the broker setting group.consumer.session.timeout.ms, also 45 seconds) removes the member. max.poll.interval.ms (5 minutes) proves the application is making progress: if the listener thread does not come back to poll() in time, the client itself leaves the group, even though heartbeats were fine. A crash trips the first check. A slow handler, a long blocking retry or a stuck database call trips the second.
Either way the result is a rebalance, and the new owner of each partition starts at the last committed offset. With an eager assignor every member gives up all its partitions during the rebalance; with the cooperative assignor or the new protocol only the partitions that move are paused. A rebalance is a recovery mechanism, but each one costs a pause and, for uncommitted work, a repeat.
Partial failures: the two gaps
| Gap | What happened | Result | Defence |
|---|---|---|---|
| Processed but not committed | The handler finished (the row is written, the card is charged) and the consumer died or lost its partitions before the offset commit | The next owner reads the same record again | An idempotent handler: Duplicates and idempotency |
| Sent but not acknowledged | The leader appended the batch and the acknowledgement was lost or late | A client retry is deduplicated by idempotence. An application resend after a timeout is a real duplicate | A stable event id in the record, and consumers that deduplicate on it |
| Database committed, publish failed | The service wrote its own database and then could not publish the event | State changed and nobody was told | Write the event in the same database transaction: Transactional outbox |
Architecture
Drawing diagram…
A flowchart of an order event moving from the Order Service through the producer buffer to the partition leader, its followers, the Payment Service consumer and the payments database, with a labelled failure attached to each hop.
flowchart LR APP["Order Service"] --> BUF["Producer buffer"] BUF --> LEAD["Partition leader"] LEAD --> FOL["Follower replicas"] CTRL["KRaft controller"] --> LEAD LEAD --> CONS["Payment Service consumer"] CONS --> DB["Payments database"] CONS --> COORD["Group coordinator"] F1["Buffer full or no metadata"] -.-> BUF F2["Request timeout or lost ack"] -.-> LEAD F3["Follower lags, ISR shrinks"] -.-> FOL F4["Broker dies, leader election"] -.-> CTRL F5["Crash or slow poll, rebalance"] -.-> COORD F6["Database down"] -.-> DB
- Read the solid arrows as the path of one record and the dotted arrows as the failures of this lesson. Each failure attaches to one hop, and the hop tells you who notices first: the caller of
send(), the producer's sender thread, the leader, the controller, the coordinator or your listener. - The first three failures are on the write path and end as a failed send or as extra latency. The last three are on the read path and end as lag, a rebalance or a failing listener. A single network fault can light up several of them at once.
- Nothing in the diagram watches the last hop. Kafka has no knowledge of the database, so a failure there is visible only through your own exceptions and the lag they cause.
Message flow
Drawing diagram…
A sequence diagram in which a producer sends a batch to broker 1, broker 1 replicates it to broker 2 and dies, the controller makes broker 2 the leader, and the producer's retry is acknowledged by broker 2 without a second append.
sequenceDiagram participant P as Producer participant B1 as Broker 1 old leader participant C as Controller participant B2 as Broker 2 follower P->>B1: Produce batch seq 7 with acks=all B1->>B2: Replicate batch seq 7 Note over B1: Broker 1 dies before it answers P--xB1: No response within request.timeout.ms C->>B2: Broker 2 is the new leader P->>B2: Metadata request B2-->>P: Leader is broker 2 P->>B2: Retry batch seq 7 Note over B2: Sequence 7 is already in the log B2-->>P: Ack with the original offset Note over P: Application sees one successful send
- The producer cannot tell whether broker 1 appended the batch. It only knows that no answer came. This is the sent but not acknowledged gap, and the client's answer to it is to send the same batch again.
- Broker 2 already holds the batch because it was an in-sync follower. The producer id and sequence number were replicated with the records, so broker 2 recognises the retry and acknowledges it without writing a second copy.
- Take idempotence away and the same sequence of events appends the batch twice. Take
acks=allaway and broker 1 may have acknowledged a batch that broker 2 never received. - All of this must fit inside
delivery.timeout.ms. If the election takes longer than the deadline, the application gets aTimeoutExceptionand is back to not knowing.
In a real system
Take a typical order flow, the one used in the example project. The Order Service publishes OrderPlaced to the topic orders.placed: 6 partitions, replication factor 3, min.insync.replicas=2, keyed by order id. The Payment Service reads it in the group payment-service with three instances, charges the card through a payment provider and writes the result to its own database.
During a rolling restart of the brokers, each broker is down for about a minute. Producers see a burst of retries and a latency spike at each step, and no errors: an election fits easily inside the two-minute delivery deadline. One broker then fails to come back, and a second is restarted anyway. Partitions that had both of those brokers as replicas now have an ISR of one. The Order Service starts receiving NotEnoughReplicasException for those partitions only, so roughly a third of orders fail to publish while the rest work. The symptom looks random until you group the failures by partition.
The same week, the payments database fails over for 90 seconds. Every Payment Service instance throws on every record. If the error handler retries ten times with no delay and then moves on, all records that arrived in those 90 seconds are skipped or dead-lettered within seconds, although nothing was wrong with any of them. The correct behaviour is the opposite of what a poison message needs: keep the records, wait, and try again.
| What the team saw | Area | Class | Right response |
|---|---|---|---|
Latency spike, record-retry-rate above zero, no errors | Broker: leader change | Transient | None. The client handled it |
NotEnoughReplicasException on two of six partitions | Broker: ISR below minimum | Transient | Restore the broker. Keep events in the outbox meanwhile |
| All records fail for 90 seconds with a connection error | Dependency | Transient | Back off and retry in place, or pause the listener |
| One record fails with a parse error on every delivery | Record | Permanent | Send it to a dead letter topic and continue |
Java and Spring Boot
Two pieces of code decide what a failure means: the producer's completion callback and the consumer's exception types. Both make the transient or permanent decision explicit, so the framework can act on it.
package com.example.orders;import org.apache.kafka.common.errors.RetriableException;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Component;@Componentpublic class OrderEventPublisher { private static final Logger log = LoggerFactory.getLogger(OrderEventPublisher.class); private static final String TOPIC = "orders.placed"; private final KafkaTemplate<String, OrderPlaced> kafkaTemplate; private final PendingEventStore pendingEvents; public OrderEventPublisher(KafkaTemplate<String, OrderPlaced> kafkaTemplate, PendingEventStore pendingEvents) { this.kafkaTemplate = kafkaTemplate; this.pendingEvents = pendingEvents; } public void publish(OrderPlaced event) { kafkaTemplate.send(TOPIC, event.orderId(), event) .whenComplete((result, failure) -> { if (failure == null) { pendingEvents.markPublished(event.eventId()); return; } Throwable cause = failure.getCause() != null ? failure.getCause() : failure; if (cause instanceof RetriableException) { // Includes TimeoutException: the client gave up waiting. // The record may or may not be in the log, so keep it pending. log.warn("Outcome unknown for event {}: {}", event.eventId(), cause.toString()); pendingEvents.keepForRepublish(event.eventId()); } else { // Too large, not authorised, cannot serialise: a resend cannot help. log.error("Event {} can never be published", event.eventId(), cause); pendingEvents.markRejected(event.eventId(), cause.getMessage()); } }); }}Why it is written this way
KafkaTemplate.sendreturns aCompletableFuture. The failure arrives there, on the producer's sender thread, long after the calling method returned. Code that ignores the future never learns that a send failed.- The client has already retried every
RetriableExceptionuntildelivery.timeout.ms. Seeing one here means the deadline passed, so the outcome is unknown, not negative. keepForRepublishsends the same event again later with the sameeventId. That resend is a new record as far as the broker is concerned, which is why consumers must deduplicate on the event id.PendingEventStorestands for a table of events written with the order. The full version of that idea is the transactional outbox.
package com.example.orders;import java.time.Instant;public record OrderPlaced(String eventId, String orderId, String customerId, long amountMinor, String currency, Instant placedAt) {}Why it is written this way
eventIdis created once, when the order is placed, and never changes on a resend. It is what lets a consumer recognise a duplicate.
package com.example.payments;import org.springframework.dao.TransientDataAccessException;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class PaymentListener { private final PaymentService paymentService; public PaymentListener(PaymentService paymentService) { this.paymentService = paymentService; } @KafkaListener(topics = "orders.placed", groupId = "payment-service") public void onOrderPlaced(OrderPlaced event) { if (event.amountMinor() <= 0) { // Permanent: this record will be wrong on every delivery. throw new InvalidOrderException("Non-positive amount for order " + event.orderId()); } try { paymentService.charge(event); } catch (TransientDataAccessException ex) { // Transient: the database, not the record, is the problem. throw new DependencyUnavailableException("Payments database unavailable", ex); } }}Why it is written this way
- The listener does not decide how often to retry. It only says which kind of failure this is by the exception type it throws. The error handler, configured in the next two lessons, maps the types to retry or dead-letter.
InvalidOrderExceptionandDependencyUnavailableExceptionare plainRuntimeExceptionsubclasses of this application.paymentService.chargemust be safe to call twice for the sameeventId, because a crash after it returns and before the offset commit delivers the record again.
spring: kafka: bootstrap-servers: kafka-1:9092,kafka-2:9092,kafka-3:9092 producer: acks: all key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: enable.idempotence: true delivery.timeout.ms: 120000 request.timeout.ms: 30000 max.block.ms: 10000Why it is written this way
acks, idempotence and the two timeouts are the defaults, written out so that nobody changes one without seeing the others.max.block.msis lowered from 1 minute to 10 seconds: this producer is called from HTTP request threads, and a full buffer should fail the request quickly instead of holding every thread for a minute.
Configuration
These settings are the timers behind every symptom in this lesson. Defaults are from the Kafka 4.4 configuration reference; the ones marked as changed differ on older clients.
delivery.timeout.msDefault 120000 (2 minutes)
- What it does
- Producer. Upper bound on the time to report success or failure after
send()returns, covering buffering, every attempt and every back-off. - When to change it
- Keep it longer than a leader election and a broker restart take in your cluster. It must be at least
request.timeout.ms+linger.ms.
request.timeout.msDefault 30000 (30 seconds)
- What it does
- Producer and consumer. How long the client waits for the response to one request before resending or failing it.
- When to change it
- The producer documentation advises keeping it larger than the broker's
replica.lag.time.max.ms, to avoid needless retries.
max.block.msDefault 60000 (1 minute)
- What it does
- Producer. How long
send()may block waiting for metadata or buffer space. - When to change it
- Lower it when
send()is called from request threads, so a broker outage does not become a thread-pool outage.
retriesDefault 2147483647
- What it does
- Producer. Number of resends for a transient error. Bounded in practice by
delivery.timeout.ms. - When to change it
- Leave it. The documentation recommends controlling retry behaviour with
delivery.timeout.msinstead.
retry.backoff.ms / retry.backoff.max.msDefault 100 / 1000 (1 second)
- What it does
- Client. First wait before retrying a failed request, growing exponentially to the maximum, with 20% jitter.
- When to change it
- Rarely changed. This is the client's own request retry and is separate from any retry of your listener.
session.timeout.msDefault 45000 (45 seconds)
- What it does
- Consumer, classic group protocol. Time without heartbeats after which the coordinator removes the member. With
group.protocol=consumerthe broker'sgroup.consumer.session.timeout.ms(also 45 seconds) applies instead. - When to change it
- Lower it to detect a crashed consumer sooner, at the price of rebalances during long garbage-collection pauses or short network blips.
max.poll.interval.msDefault 300000 (5 minutes)
- What it does
- Consumer. Longest allowed gap between two
poll()calls before the consumer leaves the group. - When to change it
- Size it for the slowest realistic batch:
max.poll.recordstimes the worst processing time, plus any blocking retry delay.
replica.lag.time.max.msDefault 30000 (30 seconds)
- What it does
- Broker. A follower that has not caught up to the leader's log end for this long is removed from the ISR.
- When to change it
- Seldom changed. It sets how long a slow follower can delay
acks=allwrites before it is dropped.
min.insync.replicasDefault 1
- What it does
- Broker or topic. Smallest ISR for which the leader accepts an
acks=allwrite. - When to change it
- Set it to 2 with replication factor 3. The default of 1 lets a write be acknowledged while it exists on one broker only.
unclean.leader.election.enableDefault false
- What it does
- Broker or topic. Whether a replica outside the ISR may become leader when no in-sync replica is available.
- When to change it
- Leave it off for anything you cannot afford to lose. Turning it on restores availability by discarding records.
Hands-on exercise
Break a broker and read the cluster's reaction
Stop brokers in a three-broker cluster and connect what the cluster does (leader election, ISR shrink) to what a producer with acks=all would see at each step.
Steps · 0 of 5 done
Stop brokers and watch the ISR
Simulation · not a real clusterThree brokers, one topic with replication factor 3 and min.insync.replicas 2. Stop brokers one at a time and read the leader and ISR of each partition.
What you should see
After the first stop, every partition still has a leader and an ISR of two: producers see retries and no errors. After the second stop the ISR is one, below min.insync.replicas, so acks=all writes are rejected with NotEnoughReplicasException while already committed records stay readable. After the restarts the ISR returns to three.
What can go wrong
Lowering acks to silence NotEnoughReplicasException
- Symptom
- After a broker incident someone sets
acks=1. The exceptions stop. Weeks later a broker dies and a few acknowledged orders are missing from the topic. - Cause
- The exception was the protection working: the leader refused to acknowledge a write that only it held.
acks=1removes the refusal, not the risk. - Fix
- Keep
acks=allwithmin.insync.replicas=2. Fix the under-replicated partitions, and hold events in an outbox while writes are rejected.
Reading a send timeout as a clean failure
- Symptom
- Duplicate orders downstream after a network incident, although the producer is idempotent.
- Cause
- The application caught
TimeoutException, assumed the record was not written and published the event again. The first copy had been appended. Idempotence covers the client's own retries only. - Fix
- Treat a timeout as an unknown outcome. Resend with the same event id and deduplicate in the consumer.
Treating a dependency outage like a bad record
- Symptom
- A database failover of one minute puts thousands of healthy records into the dead letter topic.
- Cause
- The error handler gives every exception the same few fast retries and then recovers the record. During an outage every record exhausts its retries within milliseconds.
- Fix
- Throw distinct exception types. Retry dependency failures with a long, capped back-off or pause the listener; dead-letter only failures that belong to the record.
Raising max.poll.interval.ms until the rebalances stop
- Symptom
- Rebalances stop, but when a handler hangs on a stuck call its partitions sit unprocessed for the whole interval.
- Cause
max.poll.interval.msis the only check that the application is making progress. A very large value disables it.- Fix
- First make the work fit: lower
max.poll.recordsand put timeouts on downstream calls. Then set the interval with a reasoned margin.
Blocking request threads in send()
- Symptom
- During a broker outage the web service stops answering every endpoint, including ones that do not publish.
- Cause
send()blocks for up tomax.block.ms(1 minute by default) when metadata is unavailable or the buffer is full. Enough blocked callers exhaust the request thread pool.- Fix
- Lower
max.block.ms, or write the event to an outbox table in the request and publish from a separate thread.
How to troubleshoot
Sends fail on some partitions only
Check
kafka-topics.sh --bootstrap-server kafka-1:9092 --describe --under-min-isr-partitions
- How to read it
- Lists partitions whose ISR is smaller than
min.insync.replicas. If the failing partitions are in the list, the producer is being refused correctly. Find the brokers missing from theIsrcolumn.
Latency spikes on the producer with no errors
Check
Producer metrics record-retry-rate and record-error-rate (kafka.producer:type=producer-metrics)
- How to read it
- Retries above zero with an error rate of zero means transient broker errors that the client absorbed, typically leader changes. An error rate above zero means sends are reaching the application as failures.
send() is slow or throws TimeoutException before anything is sent
Check
Producer metrics bufferpool-wait-ratio and buffer-available-bytes
- How to read it
- A wait ratio above zero and little available buffer means callers are blocking on a full buffer: the brokers accept records more slowly than the application produces them.
Brokers report trouble but you do not know which kind
Check
Broker metrics UnderReplicatedPartitions, UnderMinIsrPartitionCount and IsrShrinksPerSec (kafka.server:type=ReplicaManager)
- How to read it
- Under-replicated above zero means a follower is missing or slow and producers are still fine. Under-min-ISR above zero means
acks=allproducers are being rejected. A shrink rate without a broker restart points at a slow follower or a network problem between brokers.
Lag grows on a few partitions and nobody seems to own them
Check
kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group payment-service
- How to read it
- Each row shows the committed offset, the log end offset, the lag and the member that owns the partition. Rows without a consumer id have no owner right now: a member died and the session timeout has not passed, or a rebalance is in progress.
Rebalances with no deploy and no crash
Check
Consumer metrics time-between-poll-max and last-poll-seconds-ago, and CommitFailedException in the consumer log
- How to read it
- A maximum time between polls close to
max.poll.interval.msmeans the listener is too slow for the interval and the client is leaving the group by itself. Heartbeat problems would not show here.
Key takeaways
- Sort every failure into producer, broker, network, consumer or dependency first, then into transient or permanent. The response follows from those two answers.
- Kafka handles leader changes, broker failures and dead consumers by itself. It does not handle your database, your business rules or the meaning of a timeout.
- A producer timeout means the outcome is unknown. Idempotence removes duplicates from the client's own retries, not from an application resend.
NotEnoughReplicasExceptionis the durability guarantee refusing a risky write. Fix the ISR, not theackssetting.- A consumer leaves the group for two different reasons: no heartbeats for the session timeout, or no
poll()withinmax.poll.interval.ms. - The two partial failures, processed but not committed and sent but not acknowledged, both end in a duplicate. Idempotent consumers are the common defence.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
An Order Service produces to orders.placed (6 partitions, replication factor 3, min.insync.replicas=2, acks=all). Since 14:05 about a third of sends fail with NotEnoughReplicasException after two minutes each. The failures are all on partitions 1 and 4. Broker 2 is down for a disk replacement and broker 3 was restarted at 14:04 and is still recovering its logs.
What should the team do now?
Scenario 2 of 3
A Payment Service consumer group shows a rebalance every six to seven minutes. No pods restart. The logs contain CommitFailedException. Heartbeat metrics look normal. max.poll.records is 500, max.poll.interval.ms is the default, and since yesterday's release each record calls a fraud API that takes about 800 ms.
Which explanation fits these symptoms?
Scenario 3 of 3
A producer with default settings sends an OrderPlaced event. Two minutes later the future completes with TimeoutException. The service catches it, logs publish failed and immediately sends the same event again, which succeeds. The next day finance reports that this order was charged twice.
What is the most likely cause of the double charge?
References
References: Apache Kafka documentation: Producer configs (delivery.timeout.ms, request.timeout.ms, max.block.ms, retries, enable.idempotence) ↗ · Apache Kafka documentation: Consumer configs (session.timeout.ms, max.poll.interval.ms, group.protocol) ↗ · Apache Kafka documentation: Design, replication (ISR, unclean leader election, availability and durability guarantees) ↗ · Apache Kafka documentation: Broker configs (replica.lag.time.max.ms, min.insync.replicas, unclean.leader.election.enable) ↗ · Apache Kafka documentation: Monitoring (producer, consumer and replica manager metrics) ↗ · Apache Kafka Javadoc: KafkaConsumer (detecting consumer failures) ↗ · Apache Kafka Javadoc: BufferExhaustedException ↗ · Spring for Apache Kafka reference: Sending messages with KafkaTemplate ↗ · Report a problem