Skip to content

Path 10 · lesson 1 of 6

Advanced26 minApache Kafka 4.x, KRaft only (metric names checked against the 4.4 Monitoring page) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 and kafka-clients 3.9

Monitoring Kafka: the signals that matter first

Which broker, controller and client metrics tell you a cluster is unhealthy, how they reach a dashboard, and which of them deserve to wake someone up.

What you will learn

  • Name the small set of broker and controller metrics that show whether data is safe and partitions are available
  • Tell under-replicated, at-min-ISR, under-min-ISR and offline partitions apart and say what each means for producers
  • Read the request latency breakdown to find where a slow produce request spends its time
  • Expose broker metrics through JMX and client metrics through Micrometer, and know which parts are not Apache Kafka
  • Decide which signals page a person and which belong on a dashboard
  • Work from an alert and tool output back to which brokers are down

Before you start

The concept

A Kafka broker publishes several hundred metrics, and each client publishes dozens more. Monitoring Kafka well does not mean graphing all of them. It means answering four questions, continuously and in this order: Is acknowledged data still safely replicated? Can clients read and write every partition? Is the cluster fast enough? Are the applications keeping up? A handful of metrics answer each question. The rest are for diagnosis after one of those few has moved.

QuestionSignalWhere it is reportedHealthy value
Is data safely replicated?UnderReplicatedPartitions, UnderMinIsrPartitionCount, AtMinIsrPartitionCountEach broker, kafka.server:type=ReplicaManager0
Is replication stable?IsrShrinksPerSec, IsrExpandsPerSecEach broker, kafka.server:type=ReplicaManager0, except while a broker is down or returning
Can every partition be used?OfflinePartitionsCountControllers, kafka.controller:type=KafkaController0
Is someone in charge of metadata?ActiveControllerCountControllers, kafka.controller:type=KafkaControllerExactly one controller reports 1
Is the cluster fast enough?TotalTimeMs per request type, RequestHandlerAvgIdlePercent, NetworkProcessorAvgIdlePercentEach broker, kafka.network and kafka.serverLatency within your own target, idle ratios above about 0.3
Are requests failing?FailedProduceRequestsPerSec, FailedFetchRequestsPerSec, ErrorsPerSecEach broker, kafka.server:type=BrokerTopicMetrics and kafka.network:type=RequestMetricsClose to 0
Is there room on disk?Free space on the volumes under log.dirs, OfflineLogDirectoryCountThe operating system, and kafka.log:type=LogManagerHeadroom for retention plus a failure, and 0
Are applications keeping up?Consumer lag per group and partition, producer record-error-rateGroup offsets on the brokers, and each clientLag bounded, error rate 0
The first screen of a Kafka dashboard. Metric names are the MBean names in the Apache Kafka 4.4 documentation.

The four replication states are the ones people mix up. They describe the size of a partition's in-sync replica set (ISR) against two numbers: the replication factor and min.insync.replicas.

StateConditionWhat producers with acks=all seeWhat it means
Fully replicatedISR size equals the replica countNormalEvery copy is current.
Under-replicatedISR is smaller than the replica countNormalA copy is missing or behind. Safe for now, with less margin.
At min ISRISR size equals min.insync.replicasNormalOne more failure on this partition stops writes.
Under min ISRISR is smaller than min.insync.replicasNotEnoughReplicasExceptionWrites with acks=all are refused. Consumers can still read up to the high watermark.
OfflineNo live leaderTimeouts and leader-not-available errorsNothing can be written or read.

Stop a broker and read the indicators

Simulation · not a real cluster

Note the Under-replicated and Offline partitions counters and the acks=all verdict of each partition. Stop one broker, then a second, and say which indicator moved each time and why.

Why it exists

Kafka is built to keep working through failures, and that is exactly why it needs monitoring. When a broker dies, leaders move and clients carry on. When a follower falls behind, it is dropped from the ISR and writes continue. Each of these is the system spending its safety margin on your behalf, silently. Without metrics the first visible symptom arrives only when the margin is gone: a second failure takes a partition below min.insync.replicas and producers start failing.

Generic host monitoring does not catch this. CPU, memory and a process-is-up check can all be green while a broker's disk is slow enough that its followers keep leaving the ISR, or while a consumer group has stopped making progress. The useful signals are inside Kafka's own view of itself: replica state, controller state, request timing and group offsets.

Cause-based alerts

  • Fire on a suspected reason: high CPU, a long GC pause, a full request queue
  • There are hundreds of possible causes, so the list is never complete
  • Many fire without any effect on users, and people learn to ignore them
  • Useful on a dashboard, next to the symptom they might explain

Symptom-based alerts

  • Fire on an effect someone depends on: writes refused, partitions unavailable, lag growing
  • A short list covers every cause, including ones nobody predicted
  • Each alert means something is wrong for a client now or will be after one more failure
  • Belong on the pager, each with a runbook entry

Two real incidents in this app show the gap. In The graph nobody watched a team watched CPU and disk latency closely while outbound network traffic, the resource that was actually running out, grew unobserved. In Same symptom, new broker one broker's disk reads stood out against its peers long before anything failed, which is what comparing brokers side by side is for.

How it works inside

Where the numbers come from

The broker and the clients use two different metric libraries. The server reports through Yammer Metrics, which gives MBean names such as kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions. The Java clients use Kafka's own metrics registry, which gives names such as kafka.producer:type=producer-metrics,client-id=… with attributes like record-error-rate. Both are exposed through JMX, and both accept pluggable reporters (metric.reporters, which defaults to the JMX reporter).

Remote JMX is disabled by default. It is switched on with the JMX_PORT environment variable for processes started by the Kafka scripts, and it has no authentication unless you configure it, so the documentation tells you to secure it before using it in production.

From JMX to a dashboard

Apache Kafka ships the metrics and the JMX endpoint, and nothing else. Getting them into Prometheus and Grafana uses software from other projects, and it helps to know which is which when something breaks:

  • Prometheus JMX exporter: a separate open-source project under the Prometheus organisation, not part of Apache Kafka. It usually runs as a Java agent inside the broker JVM, reads MBeans locally and serves them over HTTP in Prometheus format. Because it reads in-process, remote JMX can stay off.
  • Exporter rules: a YAML file that maps MBean patterns to Prometheus metric names. The Prometheus names are therefore decided by your rules file, not by Kafka. Two companies' dashboards rarely use the same names for the same MBean.
  • Host metrics: disk space, disk latency, network throughput and CPU come from the operating system through a host agent such as the Prometheus node exporter, again a separate project.
  • Client metrics: in a Spring Boot application, Micrometer binds the Kafka client metrics and the actuator serves them. That path does not use JMX at all.

Replication metrics in detail

A follower leaves the ISR when it has not caught up to the leader's log end for replica.lag.time.max.ms (30 seconds by default). The leader proposes the change, the controller records it, and IsrShrinksPerSec ticks on the leader. When the follower catches up again, IsrExpandsPerSec ticks. One shrink followed later by one expand is the signature of a broker restart. Repeated shrinks and expands with every broker running are the signature of a follower that cannot keep up: a slow disk, a saturated network link, long GC pauses, or a partition receiving more than the follower can fetch. That pattern is one of the earliest warnings Kafka gives.

kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica shows how far, in messages, the worst follower on a broker is behind. FailedIsrUpdatesPerSec should be 0: a non-zero value means leaders are proposing ISR changes that the controller is not accepting.

Controller metrics in KRaft

ActiveControllerCount is reported by each controller node and is 0 or 1. Summed over the quorum it must be exactly 1. A sum of 0 means there is no active controller: the brokers keep serving with their last known metadata, but no leader can be elected and no ISR can change until the quorum has a majority again. OfflinePartitionsCount, FencedBrokerCount and ActiveBrokerCount come from the controller too, because it is the controller that knows which brokers are registered and heartbeating. kafka.server:type=raft-metrics gives each node's view of the quorum (current-state, current-leader, current-epoch), and brokers report how far behind the metadata log they are as last-applied-record-lag-ms in kafka.server:type=broker-metadata-metrics.

Request latency, broken into stages

kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce is the time a produce request spends in the broker. The same MBean family splits it into stages, and the stage that grew tells you where to look:

Stage metricWhat the request is waiting forIf this is the part that grew
RequestQueueTimeMsA free request handler threadThe broker is overloaded: check RequestHandlerAvgIdlePercent.
LocalTimeMsThe leader's own work, including the append to its logSlow disk or page cache pressure on this broker.
RemoteTimeMsFollowers, for acks=all produce requestsA follower is slow. The leader is healthy and waiting.
ResponseQueueTimeMsA free network threadCheck NetworkProcessorAvgIdlePercent.
ResponseSendTimeMsWriting the response to the socketNetwork congestion or a slow client.
The same breakdown exists for FetchConsumer and FetchFollower requests.

Client-side metrics

Broker metrics cannot tell you what an application experienced. A producer may be retrying, blocking on a full buffer or timing out records while every broker graph looks normal. These come from the clients:

ClientMetricWhat it tells you
Producerrecord-error-rateRecords that failed for good, after retries. This is data the application did not manage to send.
Producerrecord-retry-rateRetries. A burst during a leader change is normal; a steady rate is not.
Producerrequest-latency-avg, request-latency-maxRound-trip time as the producer sees it, including the network.
Producerbuffer-available-bytes, bufferpool-wait-ratioWhether send() is close to blocking because the brokers accept data more slowly than the application produces it.
Consumerrecords-lag-maxThe largest lag, in records, over the partitions this consumer is assigned. It is based on the consumer's current position, not its committed offset.
Consumerfetch-rateFetch requests per second. Zero means the consumer is not fetching at all.
Consumertime-between-poll-max, last-poll-seconds-agoHow long processing keeps the consumer away from poll(), to compare with max.poll.interval.ms.
Consumerrebalance-rate-per-hour, failed-rebalance-totalGroup instability.
Producer metrics are in kafka.producer:type=producer-metrics; consumer metrics in consumer-fetch-manager-metrics, consumer-metrics and consumer-coordinator-metrics.

Consumer lag itself, how to measure it from the broker side with kafka-consumer-groups.sh and how to tell slow processing from a stuck partition, is the subject of the Consumer lag lesson. For monitoring, the point to carry over is that lag measured from committed offsets on the brokers keeps working when the consumer is dead, while records-lag-max disappears together with the consumer that reported it. Alert on the first.

Disk

Kafka reports the size of each partition (kafka.log:type=Log,name=Size,topic=…,partition=…) and whether a log directory has failed (OfflineLogDirectoryCount, LogDirectoryOffline), but free space on the volume is a host metric. Alert on it early and on the trend: a broker that runs out of space fails writes to that log directory, takes the directory offline, and shuts down if it has no working directory left. Retention frees space only when whole segments age out, so a disk filling at a steady rate does not rescue itself.

Architecture

Drawing diagram…

A flowchart. Controllers and brokers expose MBeans that a JMX exporter agent turns into Prometheus metrics. A host agent reports disk and network. Spring Boot applications expose producer and consumer metrics through Micrometer. A lag exporter reads consumer group offsets from the brokers. All four feed Prometheus, which evaluates a small set of alert rules that reach the on-call engineer and serves Grafana dashboards.

flowchart LR
  subgraph K["Kafka cluster"]
    C["Controllers, kafka.controller MBeans"]
    B["Brokers, kafka.server and kafka.network MBeans"]
    H["Host, disk and network"]
  end
  subgraph A["Spring Boot applications"]
    P["Producer metrics"]
    U["Consumer metrics"]
  end
  C --> J["JMX exporter agent, separate project"]
  B --> J
  H --> N["Host agent"]
  P --> M["Micrometer and actuator endpoint"]
  U --> M
  B --> L["Lag exporter, reads group offsets"]
  J --> S["Prometheus"]
  N --> S
  M --> S
  L --> S
  S --> R["Alert rules, symptoms only"]
  S --> G["Grafana dashboards, causes and trends"]
  R --> O["On-call engineer"]
Four sources feed one metrics store. Only a short list of symptom rules reaches the pager.
  • Broker and controller metrics leave through a JMX exporter running inside each JVM. That exporter and the host agent are separate projects from Apache Kafka, so their versions and configuration are yours to manage.
  • Application metrics take a different path: Micrometer reads the Kafka client's own metric registry and the actuator endpoint serves it. A broker dashboard that is entirely green says nothing about this side.
  • Consumer lag is drawn as its own source because it should be computed from the offsets stored on the brokers, by a small exporter or a scheduled kafka-consumer-groups.sh, so it survives the death of the consumer.
  • Everything is stored and graphed, but only the symptom rules page. The same store answers the follow-up question, which is always which cause explains this symptom.

Message flow

Drawing diagram…

A sequence diagram. Broker 3, a follower with a degrading disk, falls behind the leader on broker 1. After thirty seconds the leader proposes a smaller ISR to the controller, which records it. The leader's metrics show an ISR shrink and one under-replicated partition, and after ten minutes the metrics pipeline opens a ticket. Later broker 2 is stopped for a patch, the ISR shrinks to the leader alone, the under-min-ISR count becomes one, and the pipeline pages the engineer, who starts by describing the under-min-ISR partitions.

sequenceDiagram
  participant B3 as Broker 3 follower
  participant B1 as Broker 1 leader
  participant C as Active controller
  participant X as Metrics pipeline
  participant E as On-call engineer
  Note over B3: Disk degrades and fetches slow down
  B3->>B1: Fetch orders.events-0, falling behind
  B1->>B1: Follower behind for 30 seconds
  B1->>C: Propose ISR 1 and 2
  C-->>B1: ISR change recorded
  B1->>X: IsrShrinksPerSec ticks and UnderReplicatedPartitions is 1
  X->>X: Under-replicated for 10 minutes
  X->>E: Ticket, cluster running with reduced margin
  Note over B1: Broker 2 is stopped for a patch
  B1->>C: Propose ISR 1 only
  C-->>B1: ISR change recorded
  B1->>X: UnderMinIsrPartitionCount is 1
  X->>E: Page, acks=all writes are being refused
  E->>B1: Describe under-min-ISR partitions
A slow follower becomes a ticket; a second replica lost on the same partition becomes a page.
  1. The leader, not the slow follower, reports the problem. Broker 3 would describe itself as running normally.
  2. Under-replication held for ten minutes opens a ticket rather than a page: nothing is failing for clients yet, but the cluster has lost the margin it needs to survive maintenance.
  3. The second ISR shrink is what turns a quiet degradation into refused writes, and it came from planned work. The ticket existed to stop that patch from starting.
  4. The page names the symptom, writes refused, and the first diagnostic step is the tool that lists exactly the affected partitions.

In a real system

A typical incident at an online retailer. The order platform runs six brokers and three controllers. Topic orders.events has 24 partitions, replication factor 3 and min.insync.replicas=2, keyed by order id. The Order Service produces with acks=all. Alerting covers broker process liveness, CPU above 85% and disk space above 90%. Nothing alerts on replica state.

TimeWhat happenedWhat monitoring showedWhat it should have shown
Mon 09:10The data volume on broker 4 starts to degrade: write latency several times its peers.Nothing. CPU on broker 4 is lower than usual.LocalTimeMs for produce on broker 4 diverging from the other brokers.
Mon 09:40 onwardsFollowers on broker 4 fall behind, leave the ISR, catch up, rejoin, and repeat.Nothing.IsrShrinksPerSec and IsrExpandsPerSec non-zero with every broker up; UnderReplicatedPartitions flapping.
Tue 14:00A routine rolling restart for an OS patch begins. The script waits for each broker's port to open.Process-down alerts, auto-resolved, as expected.A pre-flight check refusing to start while any partition is under-replicated.
Tue 14:07Broker 2 is stopped. Eleven partitions with replicas on both 2 and 4 drop to one in-sync replica.Nothing for two minutes.UnderMinIsrPartitionCount at 11: page immediately.
Tue 14:09Checkout error rate climbs. The Order Service logs NotEnoughReplicasException and then TimeoutException once delivery.timeout.ms expires.An application error-rate alert, pointing at the Order Service.Producer record-error-rate by topic next to the broker view.
Tue 14:31An engineer runs kafka-topics.sh --describe --under-min-isr-partitions, sees broker 4 missing from every ISR, and restarts broker 2.Recovery.The same list, twenty minutes sooner.
Illustrative timeline of a typical incident. No single alert was wrong; the right ones did not exist.

The review produced four changes, and they are a reasonable starting set for any cluster: page on under-min-ISR and offline partitions; open a ticket when partitions stay under-replicated or the ISR keeps shrinking with all brokers running; put the per-broker request latency breakdown on the dashboard so one slow broker stands out against its peers; and make the restart script check for under-replicated partitions before it touches the next broker. The last of these is covered in Upgrades and scaling.

Java and Spring Boot

Four pieces: exporter rules for the broker signals, alert rules built on them, the Spring Boot configuration that publishes client metrics, and the commands that confirm what an alert says. The Prometheus metric names in the first two files are examples: yours are whatever your exporter rules produce.

jmx-exporter/kafka-broker-rules.ymlYAML
# Rules for the Prometheus JMX exporter (a separate project from Apache Kafka),# attached to each broker and controller JVM as a Java agent.# The names on the right are our own choice. Alert rules must use the same names.lowercaseOutputName: truerules:  # Replica state, reported by each broker for the partitions it leads  - pattern: kafka.server<type=ReplicaManager, name=(UnderReplicatedPartitions|UnderMinIsrPartitionCount|AtMinIsrPartitionCount|OfflineReplicaCount)><>Value    name: kafka_replicamanager_$1    type: GAUGE  - pattern: kafka.server<type=ReplicaManager, name=(IsrShrinksPerSec|IsrExpandsPerSec|FailedIsrUpdatesPerSec)><>Count    name: kafka_replicamanager_$1_total    type: COUNTER  # Controller view (only controller nodes have these MBeans in KRaft)  - pattern: kafka.controller<type=KafkaController, name=(ActiveControllerCount|OfflinePartitionsCount|FencedBrokerCount|ActiveBrokerCount)><>Value    name: kafka_controller_$1    type: GAUGE  # Request latency by stage, as a 99th percentile  - pattern: kafka.network<type=RequestMetrics, name=(TotalTimeMs|RequestQueueTimeMs|LocalTimeMs|RemoteTimeMs|ResponseQueueTimeMs|ResponseSendTimeMs), request=(Produce|FetchConsumer|FetchFollower)><>99thPercentile    name: kafka_request_$1_p99    labels:      request: $2    type: GAUGE  # Failed requests per topic  - pattern: kafka.server<type=BrokerTopicMetrics, name=(FailedProduceRequestsPerSec|FailedFetchRequestsPerSec), topic=(.+)><>Count    name: kafka_topic_$1_total    labels:      topic: $2    type: COUNTER  # Thread pool headroom and failed log directories  - pattern: kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate    name: kafka_request_handler_idle_ratio    type: GAUGE  - pattern: kafka.log<type=LogManager, name=OfflineLogDirectoryCount><>Value    name: kafka_log_offline_directory_count    type: GAUGE

Why it is written this way

  • An explicit allow-list keeps the scrape small. Exporting every MBean of a broker with thousands of partitions produces a very large number of series, most of them never looked at.
  • Gauges such as UnderReplicatedPartitions are read from the Value attribute; rates such as IsrShrinksPerSec are exported from Count as counters so Prometheus computes the rate itself over whatever window the rule needs.
  • The controller rules match only on nodes that run the controller role. On a broker-only node they match nothing, which is correct.
  • Attribute names such as 99thPercentile and OneMinuteRate come from the Yammer metric types. Browse a running broker with a JMX client to confirm what your version exposes before relying on a rule.

Configuration

Few settings control monitoring itself. These are the ones that decide what is exposed and how the replica-state metrics behave. Defaults are from the Apache Kafka 4.4 configuration reference unless noted.

metric.reportersDefault org.apache.kafka.common.metrics.JmxReporter
What it does
Classes that are notified of every metric. The default publishes metrics as JMX MBeans. It exists on brokers and on clients.
When to change it
Leave the JMX reporter in place. If you add a custom reporter, list the JMX reporter explicitly as well, or the MBeans disappear and every JMX-based dashboard goes blank.
JMX_PORT (environment variable)Default none (remote JMX disabled)
What it does
When set for a process started by the Kafka scripts, opens a remote JMX port. JMX authentication is off unless configured through KAFKA_JMX_OPTS.
When to change it
Prefer an in-process exporter agent, which needs no remote port. If you must open remote JMX, enable authentication and TLS and restrict it by network.
replica.lag.time.max.msDefault 30000 (30 seconds)
What it does
How long a follower may go without catching up to the leader's log end before the leader removes it from the ISR.
When to change it
This is the delay between a follower stalling and the under-replicated metric moving. Lowering it gives faster detection and more ISR churn from ordinary pauses. Most clusters keep the default.
min.insync.replicasDefault 1
What it does
The ISR size below which acks=all produce requests are refused. It defines what the under-min-ISR and at-min-ISR metrics count against.
When to change it
With the default of 1 the under-min-ISR metric can only fire when a partition is already offline, so it warns of nothing. Use 2 with replication factor 3 for data you care about.
metrics.recording.levelDefault INFO
What it does
The most detailed level of client and broker Kafka metrics that is recorded: INFO, DEBUG or TRACE.
When to change it
Keep INFO in production. Raise it on one instance for a limited time when a diagnosis needs the more detailed sensors.
metrics.sample.window.ms / metrics.num.samplesDefault 30000 (30 seconds) / 2
What it does
The window over which Kafka Metrics rates and averages are computed, and how many windows are kept.
When to change it
Explains why client -rate and -avg metrics are smoothed over up to a minute. Scraping more often than the window does not add detail.
management.endpoints.web.exposure.includeDefault health
What it does
Spring Boot actuator endpoints reachable over HTTP.
When to change it
Add prometheus so client metrics can be scraped, and protect the management port as you would any internal endpoint.

Hands-on exercise

Troubleshoot: from two alerts to two brokers

You are given what an on-call engineer would see during an incident. Use the cluster explorer to work out which brokers are down, in which order, and which topics can no longer accept acks=all writes.

Steps · 0 of 7 done

Incident reconstruction: orders and payments

Simulation · not a real cluster

Four brokers, two topics, replication factor 3, min.insync.replicas 2. Reproduce the alerts from the exercise by stopping the right brokers in the right order.

What can go wrong

Alerting on liveness and host metrics only
Symptom
Every alert is green while producers fail with NotEnoughReplicasException, or the first alert of the incident comes from the application's error rate.
Cause
Process-up, CPU and memory checks do not see replica state. A cluster can have every broker running and still have partitions with one in-sync replica.
Fix
Page on under-min-ISR partitions, offline partitions and a missing active controller. Open tickets on sustained under-replication and ISR flapping.
Treating under-replicated as either an emergency or as noise
Symptom
Either the on-call is paged at every broker restart, or the under-replicated panel has been non-zero for a week and nobody looks at it.
Cause
The metric is expected to be non-zero briefly during restarts, so an alert without a duration fires on healthy maintenance and gets muted.
Fix
Give it a duration longer than a normal restart and a lower severity than under-min-ISR. A value that never returns to 0 is a broken or overloaded replica to fix, not a baseline.
Measuring consumer lag only from the consumer
Symptom
A consumer crashes or is stuck in a rebalance; the lag graph goes flat or empty and no alert fires while the backlog grows.
Cause
records-lag-max is published by the consumer. A consumer that is not running or has no assignment reports nothing, and missing data rarely triggers a threshold alert.
Fix
Compute lag from committed offsets on the brokers and alert on that. Add an alert for the absence of the client metric if you rely on it.
Looking for the problem on the broker with the highest count
Symptom
Engineers investigate broker 2 because it shows the most under-replicated partitions, and find nothing wrong with it.
Cause
The metric is reported by leaders. After a failure the surviving brokers take over leadership and report the missing follower. The failed or slow broker reports few or none.
Fix
Use kafka-topics.sh --describe --under-replicated-partitions and find the broker id missing from the Isr column.
Exporting every MBean
Symptom
Scrapes of the brokers take many seconds or time out, Prometheus memory grows, and broker CPU rises during scrapes.
Cause
Per-partition and per-client MBeans multiply with partition and client counts. A catch-all exporter rule turns all of them into time series.
Fix
Export an explicit list. Keep per-partition metrics for the few topics where you need them.

How to troubleshoot

UnderReplicatedPartitions is above 0 and stays there

Check

`kafka-topics.sh --bootstrap-server … --describe --under-replicated-partitions`, then the controller's `ActiveBrokerCount` and `FencedBrokerCount`
How to read it
If one broker id is missing from every Isr, that broker is down, fenced or too slow to keep up. If the missing ids vary and all brokers are active, followers are being outpaced: look at disk and network on the followers and at MaxLag.
ISR shrinks and expands repeatedly while every broker is running

Check

`IsrShrinksPerSec` per broker; `Shrinking ISR from … to …` lines in the leaders' server logs; disk latency and GC pauses on the broker that keeps being removed
How to read it
A follower repeatedly exceeds replica.lag.time.max.ms. The broker that is removed each time is the slow one. A single shrink followed by a single expand is just a restart.
Producers fail with NotEnoughReplicasException

Check

`kafka-topics.sh --bootstrap-server … --describe --under-min-isr-partitions`; `UnderMinIsrPartitionCount`
How to read it
The listed partitions have fewer in-sync replicas than min.insync.replicas. The data is not lost and consumers can still read. Bring a replica back; do not lower min.insync.replicas to make the error go away without accepting what that gives up.
Produce latency is high on some topics

Check

`kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce` per broker, then `RequestQueueTimeMs`, `LocalTimeMs` and `RemoteTimeMs`
How to read it
High RemoteTimeMs: the leader is waiting for a slow follower. High LocalTimeMs on one broker: that broker's disk. High RequestQueueTimeMs with low RequestHandlerAvgIdlePercent: the broker has too little request-handling capacity for its load.
OfflinePartitionsCount is above 0

Check

`kafka-topics.sh --bootstrap-server … --describe --unavailable-partitions`
How to read it
Every in-sync replica of those partitions is on a broker that is not running. Which brokers appear in their Replicas and last Isr tells you what to bring back. See Broker failures and incident response before considering an unclean election.
No controller reports ActiveControllerCount of 1, or topic creation hangs

Check

`kafka-metadata-quorum.sh --bootstrap-controller controller1.example.com:9093 describe --status`; `current-state` in `kafka.server:type=raft-metrics` on each controller
How to read it
The quorum has no leader, usually because fewer than a majority of controllers are running or can reach each other. Existing leaders keep serving, but no failure can be handled until the quorum is back.
Application reports send failures while broker dashboards are normal

Check

Producer `record-error-rate`, `record-retry-rate`, `buffer-available-bytes` and `request-latency-max` for that application; `ErrorsPerSec` for `request=Produce` on the brokers by error code
How to read it
Errors with retries and high client-side latency point at the network path or one partition leader. A shrinking buffer with blocked sends means the application produces faster than the cluster acknowledges. Broker-side ErrorsPerSec names the error code the clients were sent.

Key takeaways

  • Monitor in this order: is data replicated, is every partition available, is the cluster fast enough, are applications keeping up.
  • Under-replicated means less margin; under min ISR means acks=all writes are refused; offline means nothing works. They deserve different severities.
  • Replica-state metrics are reported by leaders, so sum them across brokers and find the faulty broker from the ISR lists, not from the highest count.
  • Repeated ISR shrinks and expands with all brokers up are an early warning of a slow broker.
  • Split request latency into queue, local, remote and response time before guessing at a cause.
  • Page on symptoms clients feel; keep causes on dashboards. Measure consumer lag from the brokers so it survives a dead consumer.
  • Apache Kafka provides the metrics and JMX. The Prometheus JMX exporter, host agents and Micrometer are separate projects that carry them to a dashboard.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A 6-broker cluster with replication factor 3 and min.insync.replicas=2. All six brokers are running and registered. For two days, IsrShrinksPerSec and IsrExpandsPerSec have been non-zero in bursts, and the removed replica is always on broker 5. Under-replicated partitions go to about 30 and return to 0 several times an hour. No client has reported an error.

What should you do with this?

Scenario 2 of 3

Produce p99 TotalTimeMs on broker 2 has risen from 15 ms to 400 ms for topics with acks=all. On broker 2, RequestQueueTimeMs is 1 ms, LocalTimeMs is 4 ms, RemoteTimeMs is 390 ms, and RequestHandlerAvgIdlePercent is 0.8. Other brokers show normal produce latency for the partitions they lead.

Where is the most likely problem?

Scenario 3 of 3

A Payment Service consumer group is alerted on with the client metric records-lag-max above 10,000 for 5 minutes. On Saturday its pods crash-loop on a bad configuration for three hours. No lag alert fires. On Monday the team finds 2 million unprocessed records on payments.events.

What is the right change to the alerting?

References

References: Apache Kafka documentation: Monitoring (broker, KRaft, producer and consumer metrics) ↗ · Apache Kafka documentation: Broker configuration reference ↗ · Apache Kafka documentation: Basic Kafka operations (checking consumer position, topic tool) ↗ · Apache Kafka documentation: KRaft vs ZooKeeper (metrics removed with ZooKeeper mode) ↗ · Spring for Apache Kafka reference: Monitoring (Micrometer timers and native metrics) ↗ · Spring Boot reference: Actuator metrics (Kafka metrics, Prometheus endpoint) ↗ · Prometheus JMX exporter (separate project) ↗ · Report a problem