Path 10 · lesson 1 of 6
Monitoring Kafka: the signals that matter first
Which broker, controller and client metrics tell you a cluster is unhealthy, how they reach a dashboard, and which of them deserve to wake someone up.
What you will learn
- Name the small set of broker and controller metrics that show whether data is safe and partitions are available
- Tell under-replicated, at-min-ISR, under-min-ISR and offline partitions apart and say what each means for producers
- Read the request latency breakdown to find where a slow produce request spends its time
- Expose broker metrics through JMX and client metrics through Micrometer, and know which parts are not Apache Kafka
- Decide which signals page a person and which belong on a dashboard
- Work from an alert and tool output back to which brokers are down
Before you start
The concept
A Kafka broker publishes several hundred metrics, and each client publishes dozens more. Monitoring Kafka well does not mean graphing all of them. It means answering four questions, continuously and in this order: Is acknowledged data still safely replicated? Can clients read and write every partition? Is the cluster fast enough? Are the applications keeping up? A handful of metrics answer each question. The rest are for diagnosis after one of those few has moved.
| Question | Signal | Where it is reported | Healthy value |
|---|---|---|---|
| Is data safely replicated? | UnderReplicatedPartitions, UnderMinIsrPartitionCount, AtMinIsrPartitionCount | Each broker, kafka.server:type=ReplicaManager | 0 |
| Is replication stable? | IsrShrinksPerSec, IsrExpandsPerSec | Each broker, kafka.server:type=ReplicaManager | 0, except while a broker is down or returning |
| Can every partition be used? | OfflinePartitionsCount | Controllers, kafka.controller:type=KafkaController | 0 |
| Is someone in charge of metadata? | ActiveControllerCount | Controllers, kafka.controller:type=KafkaController | Exactly one controller reports 1 |
| Is the cluster fast enough? | TotalTimeMs per request type, RequestHandlerAvgIdlePercent, NetworkProcessorAvgIdlePercent | Each broker, kafka.network and kafka.server | Latency within your own target, idle ratios above about 0.3 |
| Are requests failing? | FailedProduceRequestsPerSec, FailedFetchRequestsPerSec, ErrorsPerSec | Each broker, kafka.server:type=BrokerTopicMetrics and kafka.network:type=RequestMetrics | Close to 0 |
| Is there room on disk? | Free space on the volumes under log.dirs, OfflineLogDirectoryCount | The operating system, and kafka.log:type=LogManager | Headroom for retention plus a failure, and 0 |
| Are applications keeping up? | Consumer lag per group and partition, producer record-error-rate | Group offsets on the brokers, and each client | Lag bounded, error rate 0 |
The four replication states are the ones people mix up. They describe the size of a partition's in-sync replica set (ISR) against two numbers: the replication factor and min.insync.replicas.
| State | Condition | What producers with acks=all see | What it means |
|---|---|---|---|
| Fully replicated | ISR size equals the replica count | Normal | Every copy is current. |
| Under-replicated | ISR is smaller than the replica count | Normal | A copy is missing or behind. Safe for now, with less margin. |
| At min ISR | ISR size equals min.insync.replicas | Normal | One more failure on this partition stops writes. |
| Under min ISR | ISR is smaller than min.insync.replicas | NotEnoughReplicasException | Writes with acks=all are refused. Consumers can still read up to the high watermark. |
| Offline | No live leader | Timeouts and leader-not-available errors | Nothing can be written or read. |
Stop a broker and read the indicators
Simulation · not a real clusterNote the Under-replicated and Offline partitions counters and the acks=all verdict of each partition. Stop one broker, then a second, and say which indicator moved each time and why.
Why it exists
Kafka is built to keep working through failures, and that is exactly why it needs monitoring. When a broker dies, leaders move and clients carry on. When a follower falls behind, it is dropped from the ISR and writes continue. Each of these is the system spending its safety margin on your behalf, silently. Without metrics the first visible symptom arrives only when the margin is gone: a second failure takes a partition below min.insync.replicas and producers start failing.
Generic host monitoring does not catch this. CPU, memory and a process-is-up check can all be green while a broker's disk is slow enough that its followers keep leaving the ISR, or while a consumer group has stopped making progress. The useful signals are inside Kafka's own view of itself: replica state, controller state, request timing and group offsets.
Cause-based alerts
- Fire on a suspected reason: high CPU, a long GC pause, a full request queue
- There are hundreds of possible causes, so the list is never complete
- Many fire without any effect on users, and people learn to ignore them
- Useful on a dashboard, next to the symptom they might explain
Symptom-based alerts
- Fire on an effect someone depends on: writes refused, partitions unavailable, lag growing
- A short list covers every cause, including ones nobody predicted
- Each alert means something is wrong for a client now or will be after one more failure
- Belong on the pager, each with a runbook entry
Two real incidents in this app show the gap. In The graph nobody watched a team watched CPU and disk latency closely while outbound network traffic, the resource that was actually running out, grew unobserved. In Same symptom, new broker one broker's disk reads stood out against its peers long before anything failed, which is what comparing brokers side by side is for.
How it works inside
Where the numbers come from
The broker and the clients use two different metric libraries. The server reports through Yammer Metrics, which gives MBean names such as kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions. The Java clients use Kafka's own metrics registry, which gives names such as kafka.producer:type=producer-metrics,client-id=… with attributes like record-error-rate. Both are exposed through JMX, and both accept pluggable reporters (metric.reporters, which defaults to the JMX reporter).
Remote JMX is disabled by default. It is switched on with the JMX_PORT environment variable for processes started by the Kafka scripts, and it has no authentication unless you configure it, so the documentation tells you to secure it before using it in production.
From JMX to a dashboard
Apache Kafka ships the metrics and the JMX endpoint, and nothing else. Getting them into Prometheus and Grafana uses software from other projects, and it helps to know which is which when something breaks:
- Prometheus JMX exporter: a separate open-source project under the Prometheus organisation, not part of Apache Kafka. It usually runs as a Java agent inside the broker JVM, reads MBeans locally and serves them over HTTP in Prometheus format. Because it reads in-process, remote JMX can stay off.
- Exporter rules: a YAML file that maps MBean patterns to Prometheus metric names. The Prometheus names are therefore decided by your rules file, not by Kafka. Two companies' dashboards rarely use the same names for the same MBean.
- Host metrics: disk space, disk latency, network throughput and CPU come from the operating system through a host agent such as the Prometheus node exporter, again a separate project.
- Client metrics: in a Spring Boot application, Micrometer binds the Kafka client metrics and the actuator serves them. That path does not use JMX at all.
Replication metrics in detail
A follower leaves the ISR when it has not caught up to the leader's log end for replica.lag.time.max.ms (30 seconds by default). The leader proposes the change, the controller records it, and IsrShrinksPerSec ticks on the leader. When the follower catches up again, IsrExpandsPerSec ticks. One shrink followed later by one expand is the signature of a broker restart. Repeated shrinks and expands with every broker running are the signature of a follower that cannot keep up: a slow disk, a saturated network link, long GC pauses, or a partition receiving more than the follower can fetch. That pattern is one of the earliest warnings Kafka gives.
kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica shows how far, in messages, the worst follower on a broker is behind. FailedIsrUpdatesPerSec should be 0: a non-zero value means leaders are proposing ISR changes that the controller is not accepting.
Controller metrics in KRaft
ActiveControllerCount is reported by each controller node and is 0 or 1. Summed over the quorum it must be exactly 1. A sum of 0 means there is no active controller: the brokers keep serving with their last known metadata, but no leader can be elected and no ISR can change until the quorum has a majority again. OfflinePartitionsCount, FencedBrokerCount and ActiveBrokerCount come from the controller too, because it is the controller that knows which brokers are registered and heartbeating. kafka.server:type=raft-metrics gives each node's view of the quorum (current-state, current-leader, current-epoch), and brokers report how far behind the metadata log they are as last-applied-record-lag-ms in kafka.server:type=broker-metadata-metrics.
Request latency, broken into stages
kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce is the time a produce request spends in the broker. The same MBean family splits it into stages, and the stage that grew tells you where to look:
| Stage metric | What the request is waiting for | If this is the part that grew |
|---|---|---|
RequestQueueTimeMs | A free request handler thread | The broker is overloaded: check RequestHandlerAvgIdlePercent. |
LocalTimeMs | The leader's own work, including the append to its log | Slow disk or page cache pressure on this broker. |
RemoteTimeMs | Followers, for acks=all produce requests | A follower is slow. The leader is healthy and waiting. |
ResponseQueueTimeMs | A free network thread | Check NetworkProcessorAvgIdlePercent. |
ResponseSendTimeMs | Writing the response to the socket | Network congestion or a slow client. |
FetchConsumer and FetchFollower requests.Client-side metrics
Broker metrics cannot tell you what an application experienced. A producer may be retrying, blocking on a full buffer or timing out records while every broker graph looks normal. These come from the clients:
| Client | Metric | What it tells you |
|---|---|---|
| Producer | record-error-rate | Records that failed for good, after retries. This is data the application did not manage to send. |
| Producer | record-retry-rate | Retries. A burst during a leader change is normal; a steady rate is not. |
| Producer | request-latency-avg, request-latency-max | Round-trip time as the producer sees it, including the network. |
| Producer | buffer-available-bytes, bufferpool-wait-ratio | Whether send() is close to blocking because the brokers accept data more slowly than the application produces it. |
| Consumer | records-lag-max | The largest lag, in records, over the partitions this consumer is assigned. It is based on the consumer's current position, not its committed offset. |
| Consumer | fetch-rate | Fetch requests per second. Zero means the consumer is not fetching at all. |
| Consumer | time-between-poll-max, last-poll-seconds-ago | How long processing keeps the consumer away from poll(), to compare with max.poll.interval.ms. |
| Consumer | rebalance-rate-per-hour, failed-rebalance-total | Group instability. |
kafka.producer:type=producer-metrics; consumer metrics in consumer-fetch-manager-metrics, consumer-metrics and consumer-coordinator-metrics.Consumer lag itself, how to measure it from the broker side with kafka-consumer-groups.sh and how to tell slow processing from a stuck partition, is the subject of the Consumer lag lesson. For monitoring, the point to carry over is that lag measured from committed offsets on the brokers keeps working when the consumer is dead, while records-lag-max disappears together with the consumer that reported it. Alert on the first.
Disk
Kafka reports the size of each partition (kafka.log:type=Log,name=Size,topic=…,partition=…) and whether a log directory has failed (OfflineLogDirectoryCount, LogDirectoryOffline), but free space on the volume is a host metric. Alert on it early and on the trend: a broker that runs out of space fails writes to that log directory, takes the directory offline, and shuts down if it has no working directory left. Retention frees space only when whole segments age out, so a disk filling at a steady rate does not rescue itself.
Architecture
Drawing diagram…
A flowchart. Controllers and brokers expose MBeans that a JMX exporter agent turns into Prometheus metrics. A host agent reports disk and network. Spring Boot applications expose producer and consumer metrics through Micrometer. A lag exporter reads consumer group offsets from the brokers. All four feed Prometheus, which evaluates a small set of alert rules that reach the on-call engineer and serves Grafana dashboards.
flowchart LR
subgraph K["Kafka cluster"]
C["Controllers, kafka.controller MBeans"]
B["Brokers, kafka.server and kafka.network MBeans"]
H["Host, disk and network"]
end
subgraph A["Spring Boot applications"]
P["Producer metrics"]
U["Consumer metrics"]
end
C --> J["JMX exporter agent, separate project"]
B --> J
H --> N["Host agent"]
P --> M["Micrometer and actuator endpoint"]
U --> M
B --> L["Lag exporter, reads group offsets"]
J --> S["Prometheus"]
N --> S
M --> S
L --> S
S --> R["Alert rules, symptoms only"]
S --> G["Grafana dashboards, causes and trends"]
R --> O["On-call engineer"]- Broker and controller metrics leave through a JMX exporter running inside each JVM. That exporter and the host agent are separate projects from Apache Kafka, so their versions and configuration are yours to manage.
- Application metrics take a different path: Micrometer reads the Kafka client's own metric registry and the actuator endpoint serves it. A broker dashboard that is entirely green says nothing about this side.
- Consumer lag is drawn as its own source because it should be computed from the offsets stored on the brokers, by a small exporter or a scheduled
kafka-consumer-groups.sh, so it survives the death of the consumer. - Everything is stored and graphed, but only the symptom rules page. The same store answers the follow-up question, which is always which cause explains this symptom.
Message flow
Drawing diagram…
A sequence diagram. Broker 3, a follower with a degrading disk, falls behind the leader on broker 1. After thirty seconds the leader proposes a smaller ISR to the controller, which records it. The leader's metrics show an ISR shrink and one under-replicated partition, and after ten minutes the metrics pipeline opens a ticket. Later broker 2 is stopped for a patch, the ISR shrinks to the leader alone, the under-min-ISR count becomes one, and the pipeline pages the engineer, who starts by describing the under-min-ISR partitions.
sequenceDiagram participant B3 as Broker 3 follower participant B1 as Broker 1 leader participant C as Active controller participant X as Metrics pipeline participant E as On-call engineer Note over B3: Disk degrades and fetches slow down B3->>B1: Fetch orders.events-0, falling behind B1->>B1: Follower behind for 30 seconds B1->>C: Propose ISR 1 and 2 C-->>B1: ISR change recorded B1->>X: IsrShrinksPerSec ticks and UnderReplicatedPartitions is 1 X->>X: Under-replicated for 10 minutes X->>E: Ticket, cluster running with reduced margin Note over B1: Broker 2 is stopped for a patch B1->>C: Propose ISR 1 only C-->>B1: ISR change recorded B1->>X: UnderMinIsrPartitionCount is 1 X->>E: Page, acks=all writes are being refused E->>B1: Describe under-min-ISR partitions
- The leader, not the slow follower, reports the problem. Broker 3 would describe itself as running normally.
- Under-replication held for ten minutes opens a ticket rather than a page: nothing is failing for clients yet, but the cluster has lost the margin it needs to survive maintenance.
- The second ISR shrink is what turns a quiet degradation into refused writes, and it came from planned work. The ticket existed to stop that patch from starting.
- The page names the symptom, writes refused, and the first diagnostic step is the tool that lists exactly the affected partitions.
In a real system
A typical incident at an online retailer. The order platform runs six brokers and three controllers. Topic orders.events has 24 partitions, replication factor 3 and min.insync.replicas=2, keyed by order id. The Order Service produces with acks=all. Alerting covers broker process liveness, CPU above 85% and disk space above 90%. Nothing alerts on replica state.
| Time | What happened | What monitoring showed | What it should have shown |
|---|---|---|---|
| Mon 09:10 | The data volume on broker 4 starts to degrade: write latency several times its peers. | Nothing. CPU on broker 4 is lower than usual. | LocalTimeMs for produce on broker 4 diverging from the other brokers. |
| Mon 09:40 onwards | Followers on broker 4 fall behind, leave the ISR, catch up, rejoin, and repeat. | Nothing. | IsrShrinksPerSec and IsrExpandsPerSec non-zero with every broker up; UnderReplicatedPartitions flapping. |
| Tue 14:00 | A routine rolling restart for an OS patch begins. The script waits for each broker's port to open. | Process-down alerts, auto-resolved, as expected. | A pre-flight check refusing to start while any partition is under-replicated. |
| Tue 14:07 | Broker 2 is stopped. Eleven partitions with replicas on both 2 and 4 drop to one in-sync replica. | Nothing for two minutes. | UnderMinIsrPartitionCount at 11: page immediately. |
| Tue 14:09 | Checkout error rate climbs. The Order Service logs NotEnoughReplicasException and then TimeoutException once delivery.timeout.ms expires. | An application error-rate alert, pointing at the Order Service. | Producer record-error-rate by topic next to the broker view. |
| Tue 14:31 | An engineer runs kafka-topics.sh --describe --under-min-isr-partitions, sees broker 4 missing from every ISR, and restarts broker 2. | Recovery. | The same list, twenty minutes sooner. |
The review produced four changes, and they are a reasonable starting set for any cluster: page on under-min-ISR and offline partitions; open a ticket when partitions stay under-replicated or the ISR keeps shrinking with all brokers running; put the per-broker request latency breakdown on the dashboard so one slow broker stands out against its peers; and make the restart script check for under-replicated partitions before it touches the next broker. The last of these is covered in Upgrades and scaling.
Java and Spring Boot
Four pieces: exporter rules for the broker signals, alert rules built on them, the Spring Boot configuration that publishes client metrics, and the commands that confirm what an alert says. The Prometheus metric names in the first two files are examples: yours are whatever your exporter rules produce.
# Rules for the Prometheus JMX exporter (a separate project from Apache Kafka),# attached to each broker and controller JVM as a Java agent.# The names on the right are our own choice. Alert rules must use the same names.lowercaseOutputName: truerules: # Replica state, reported by each broker for the partitions it leads - pattern: kafka.server<type=ReplicaManager, name=(UnderReplicatedPartitions|UnderMinIsrPartitionCount|AtMinIsrPartitionCount|OfflineReplicaCount)><>Value name: kafka_replicamanager_$1 type: GAUGE - pattern: kafka.server<type=ReplicaManager, name=(IsrShrinksPerSec|IsrExpandsPerSec|FailedIsrUpdatesPerSec)><>Count name: kafka_replicamanager_$1_total type: COUNTER # Controller view (only controller nodes have these MBeans in KRaft) - pattern: kafka.controller<type=KafkaController, name=(ActiveControllerCount|OfflinePartitionsCount|FencedBrokerCount|ActiveBrokerCount)><>Value name: kafka_controller_$1 type: GAUGE # Request latency by stage, as a 99th percentile - pattern: kafka.network<type=RequestMetrics, name=(TotalTimeMs|RequestQueueTimeMs|LocalTimeMs|RemoteTimeMs|ResponseQueueTimeMs|ResponseSendTimeMs), request=(Produce|FetchConsumer|FetchFollower)><>99thPercentile name: kafka_request_$1_p99 labels: request: $2 type: GAUGE # Failed requests per topic - pattern: kafka.server<type=BrokerTopicMetrics, name=(FailedProduceRequestsPerSec|FailedFetchRequestsPerSec), topic=(.+)><>Count name: kafka_topic_$1_total labels: topic: $2 type: COUNTER # Thread pool headroom and failed log directories - pattern: kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate name: kafka_request_handler_idle_ratio type: GAUGE - pattern: kafka.log<type=LogManager, name=OfflineLogDirectoryCount><>Value name: kafka_log_offline_directory_count type: GAUGEWhy it is written this way
- An explicit allow-list keeps the scrape small. Exporting every MBean of a broker with thousands of partitions produces a very large number of series, most of them never looked at.
- Gauges such as
UnderReplicatedPartitionsare read from theValueattribute; rates such asIsrShrinksPerSecare exported fromCountas counters so Prometheus computes the rate itself over whatever window the rule needs. - The controller rules match only on nodes that run the controller role. On a broker-only node they match nothing, which is correct.
- Attribute names such as
99thPercentileandOneMinuteRatecome from the Yammer metric types. Browse a running broker with a JMX client to confirm what your version exposes before relying on a rule.
groups: - name: kafka-page # a client is affected now, or will be after one more failure rules: - alert: KafkaUnderMinIsrPartitions expr: sum(kafka_replicamanager_underminisrpartitioncount) > 0 for: 1m labels: { severity: page } annotations: summary: "Producers using acks=all are being refused on some partitions" runbook: "kafka-topics.sh --describe --under-min-isr-partitions" - alert: KafkaOfflinePartitions expr: max(kafka_controller_offlinepartitionscount) > 0 for: 1m labels: { severity: page } annotations: summary: "Partitions have no leader and cannot be read or written" - alert: KafkaNoActiveController expr: sum(kafka_controller_activecontrollercount) != 1 for: 2m labels: { severity: page } annotations: summary: "The controller quorum does not have exactly one active controller" - alert: KafkaBrokerDiskFillingUp # node exporter metric: will this volume be full within 12 hours at the current rate? expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/var/lib/kafka"}[6h], 12 * 3600) < 0 for: 30m labels: { severity: page } - name: kafka-ticket # margin is reduced, nobody is affected yet rules: - alert: KafkaUnderReplicatedPartitions expr: sum(kafka_replicamanager_underreplicatedpartitions) > 0 for: 10m labels: { severity: ticket } annotations: summary: "Partitions under-replicated for 10 minutes: do not start maintenance" - alert: KafkaIsrFlapping expr: sum by (instance) (increase(kafka_replicamanager_isrshrinkspersec_total[30m])) > 5 for: 15m labels: { severity: ticket } annotations: summary: "ISR keeps shrinking on {{ $labels.instance }}: a follower cannot keep up" - alert: KafkaProducerRecordErrors # Micrometer name of the producer metric record-error-rate expr: sum by (application) (kafka_producer_record_error_rate) > 0 for: 5m labels: { severity: ticket }Why it is written this way
- Two groups, two meanings. A page says a client is affected or one failure away from it. A ticket says the cluster has less margin than it was designed with and planned work should wait.
for:matters as much as the expression. A broker restart makes partitions under-replicated for a few minutes by design; ten minutes of it is no longer a restart.- The under-min-ISR rule sums over brokers because each leader reports its own partitions. The active-controller rule sums over controllers and requires exactly 1, so it catches both zero and, in theory, more than one.
- The disk rule alerts on the trend. A fixed threshold such as 90% fires too late on a fast-filling disk and too often on a full but stable one.
- Thresholds and windows here are starting points for illustration. Set them from your own traffic and restart times.
spring: application: name: order-service kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS} # A stable client.id makes client metrics and broker quota metrics attributable client-id: order-service producer: acks: allmanagement: endpoints: web: exposure: include: health,prometheus metrics: tags: application: ${spring.application.name}Why it is written this way
- With
spring-boot-starter-actuatorandmicrometer-registry-prometheuson the classpath, Spring Boot registers Micrometer listeners on its auto-configured producer and consumer factories, so every Kafka client metric is published without extra code. - Micrometer names follow the client metric group and name:
record-error-rateinproducer-metricsbecomeskafka.producer.record.error.rate, and Prometheus shows it askafka_producer_record_error_rate.records-lag-maxbecomeskafka_consumer_fetch_manager_records_lag_max. - The
prometheusendpoint is not exposed over HTTP unless it is listed undermanagement.endpoints.web.exposure.include. - If you build your own
DefaultKafkaProducerFactoryorDefaultKafkaConsumerFactory, addMicrometerProducerListener/MicrometerConsumerListeneryourself, as the Spring for Apache Kafka reference shows. Spring Kafka also records listener and template timers namedspring.kafka.listenerandspring.kafka.template.
# Which partitions are under-replicated, and which broker is missing from their ISR?bin/kafka-topics.sh --bootstrap-server broker1.example.com:9092 \ --describe --under-replicated-partitions# Illustrative output (not captured from a real cluster):# Topic: orders.events Partition: 0 Leader: 2 Replicas: 1,2,3 Isr: 2,3# Topic: orders.events Partition: 2 Leader: 3 Replicas: 3,4,1 Isr: 3,4# Topic: orders.events Partition: 3 Leader: 4 Replicas: 4,1,2 Isr: 4,2# Topic: payments.events Partition: 1 Leader: 3 Replicas: 3,4,1 Isr: 3,4# Partitions on which acks=all is being refused right nowbin/kafka-topics.sh --bootstrap-server broker1.example.com:9092 \ --describe --under-min-isr-partitions# Partitions with no leaderbin/kafka-topics.sh --bootstrap-server broker1.example.com:9092 \ --describe --unavailable-partitions# Is there an active controller, and are the voters caught up?bin/kafka-metadata-quorum.sh --bootstrap-server broker1.example.com:9092 describe --status# Lag of one group, from the offsets stored on the brokersbin/kafka-consumer-groups.sh --bootstrap-server broker1.example.com:9092 \ --describe --group payment-service# Size of each replica on each broker's log directories (JSON output)bin/kafka-log-dirs.sh --bootstrap-server broker1.example.com:9092 \ --describe --broker-list 1,2,3,4Why it is written this way
- An empty result from the first three commands is the healthy answer. They print only the partitions in that state.
- Read the output by comparing
ReplicaswithIsr: the broker id that is in the first list and absent from the second, across many lines, is the one to look at. In the sample that is broker 1. - The sample output is illustrative. The exact columns differ slightly between versions (newer versions print additional fields).
Configuration
Few settings control monitoring itself. These are the ones that decide what is exposed and how the replica-state metrics behave. Defaults are from the Apache Kafka 4.4 configuration reference unless noted.
metric.reportersDefault org.apache.kafka.common.metrics.JmxReporter
- What it does
- Classes that are notified of every metric. The default publishes metrics as JMX MBeans. It exists on brokers and on clients.
- When to change it
- Leave the JMX reporter in place. If you add a custom reporter, list the JMX reporter explicitly as well, or the MBeans disappear and every JMX-based dashboard goes blank.
JMX_PORT (environment variable)Default none (remote JMX disabled)
- What it does
- When set for a process started by the Kafka scripts, opens a remote JMX port. JMX authentication is off unless configured through
KAFKA_JMX_OPTS. - When to change it
- Prefer an in-process exporter agent, which needs no remote port. If you must open remote JMX, enable authentication and TLS and restrict it by network.
replica.lag.time.max.msDefault 30000 (30 seconds)
- What it does
- How long a follower may go without catching up to the leader's log end before the leader removes it from the ISR.
- When to change it
- This is the delay between a follower stalling and the under-replicated metric moving. Lowering it gives faster detection and more ISR churn from ordinary pauses. Most clusters keep the default.
min.insync.replicasDefault 1
- What it does
- The ISR size below which
acks=allproduce requests are refused. It defines what the under-min-ISR and at-min-ISR metrics count against. - When to change it
- With the default of 1 the under-min-ISR metric can only fire when a partition is already offline, so it warns of nothing. Use 2 with replication factor 3 for data you care about.
metrics.recording.levelDefault INFO
- What it does
- The most detailed level of client and broker Kafka metrics that is recorded: INFO, DEBUG or TRACE.
- When to change it
- Keep INFO in production. Raise it on one instance for a limited time when a diagnosis needs the more detailed sensors.
metrics.sample.window.ms / metrics.num.samplesDefault 30000 (30 seconds) / 2
- What it does
- The window over which Kafka Metrics rates and averages are computed, and how many windows are kept.
- When to change it
- Explains why client
-rateand-avgmetrics are smoothed over up to a minute. Scraping more often than the window does not add detail.
management.endpoints.web.exposure.includeDefault health
- What it does
- Spring Boot actuator endpoints reachable over HTTP.
- When to change it
- Add
prometheusso client metrics can be scraped, and protect the management port as you would any internal endpoint.
Hands-on exercise
Troubleshoot: from two alerts to two brokers
You are given what an on-call engineer would see during an incident. Use the cluster explorer to work out which brokers are down, in which order, and which topics can no longer accept acks=all writes.
Steps · 0 of 7 done
Incident reconstruction: orders and payments
Simulation · not a real clusterFour brokers, two topics, replication factor 3, min.insync.replicas 2. Reproduce the alerts from the exercise by stopping the right brokers in the right order.
What you should see
Broker 1 went down first: it is the one id present in every Replicas list and absent from every Isr list, and it holds replicas of 4 of the 6 partitions, which is why the count is 4. Every partition keeps two in-sync replicas, so acks=all still works. Broker 4 went down second. That leaves orders.events-2, orders.events-3 and payments.events-1 with a single in-sync replica: under min ISR, and Produce is rejected on exactly those three with NotEnoughReplicasException. All 6 partitions are now under-replicated and none is offline, because each still has a live leader. Stopping broker 3 instead would have broken orders.events-0, which the evidence says still works. Restarting broker 4 clears the under-min-ISR state; the under-replicated count returns to 4 until broker 1 is back.
What can go wrong
Alerting on liveness and host metrics only
- Symptom
- Every alert is green while producers fail with
NotEnoughReplicasException, or the first alert of the incident comes from the application's error rate. - Cause
- Process-up, CPU and memory checks do not see replica state. A cluster can have every broker running and still have partitions with one in-sync replica.
- Fix
- Page on under-min-ISR partitions, offline partitions and a missing active controller. Open tickets on sustained under-replication and ISR flapping.
Treating under-replicated as either an emergency or as noise
- Symptom
- Either the on-call is paged at every broker restart, or the under-replicated panel has been non-zero for a week and nobody looks at it.
- Cause
- The metric is expected to be non-zero briefly during restarts, so an alert without a duration fires on healthy maintenance and gets muted.
- Fix
- Give it a duration longer than a normal restart and a lower severity than under-min-ISR. A value that never returns to 0 is a broken or overloaded replica to fix, not a baseline.
Measuring consumer lag only from the consumer
- Symptom
- A consumer crashes or is stuck in a rebalance; the lag graph goes flat or empty and no alert fires while the backlog grows.
- Cause
records-lag-maxis published by the consumer. A consumer that is not running or has no assignment reports nothing, and missing data rarely triggers a threshold alert.- Fix
- Compute lag from committed offsets on the brokers and alert on that. Add an alert for the absence of the client metric if you rely on it.
Looking for the problem on the broker with the highest count
- Symptom
- Engineers investigate broker 2 because it shows the most under-replicated partitions, and find nothing wrong with it.
- Cause
- The metric is reported by leaders. After a failure the surviving brokers take over leadership and report the missing follower. The failed or slow broker reports few or none.
- Fix
- Use
kafka-topics.sh --describe --under-replicated-partitionsand find the broker id missing from theIsrcolumn.
Exporting every MBean
- Symptom
- Scrapes of the brokers take many seconds or time out, Prometheus memory grows, and broker CPU rises during scrapes.
- Cause
- Per-partition and per-client MBeans multiply with partition and client counts. A catch-all exporter rule turns all of them into time series.
- Fix
- Export an explicit list. Keep per-partition metrics for the few topics where you need them.
How to troubleshoot
UnderReplicatedPartitions is above 0 and stays there
Check
`kafka-topics.sh --bootstrap-server … --describe --under-replicated-partitions`, then the controller's `ActiveBrokerCount` and `FencedBrokerCount`
- How to read it
- If one broker id is missing from every
Isr, that broker is down, fenced or too slow to keep up. If the missing ids vary and all brokers are active, followers are being outpaced: look at disk and network on the followers and atMaxLag.
ISR shrinks and expands repeatedly while every broker is running
Check
`IsrShrinksPerSec` per broker; `Shrinking ISR from … to …` lines in the leaders' server logs; disk latency and GC pauses on the broker that keeps being removed
- How to read it
- A follower repeatedly exceeds
replica.lag.time.max.ms. The broker that is removed each time is the slow one. A single shrink followed by a single expand is just a restart.
Producers fail with NotEnoughReplicasException
Check
`kafka-topics.sh --bootstrap-server … --describe --under-min-isr-partitions`; `UnderMinIsrPartitionCount`
- How to read it
- The listed partitions have fewer in-sync replicas than
min.insync.replicas. The data is not lost and consumers can still read. Bring a replica back; do not lowermin.insync.replicasto make the error go away without accepting what that gives up.
Produce latency is high on some topics
Check
`kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce` per broker, then `RequestQueueTimeMs`, `LocalTimeMs` and `RemoteTimeMs`
- How to read it
- High
RemoteTimeMs: the leader is waiting for a slow follower. HighLocalTimeMson one broker: that broker's disk. HighRequestQueueTimeMswith lowRequestHandlerAvgIdlePercent: the broker has too little request-handling capacity for its load.
OfflinePartitionsCount is above 0
Check
`kafka-topics.sh --bootstrap-server … --describe --unavailable-partitions`
- How to read it
- Every in-sync replica of those partitions is on a broker that is not running. Which brokers appear in their
Replicasand lastIsrtells you what to bring back. See Broker failures and incident response before considering an unclean election.
No controller reports ActiveControllerCount of 1, or topic creation hangs
Check
`kafka-metadata-quorum.sh --bootstrap-controller controller1.example.com:9093 describe --status`; `current-state` in `kafka.server:type=raft-metrics` on each controller
- How to read it
- The quorum has no leader, usually because fewer than a majority of controllers are running or can reach each other. Existing leaders keep serving, but no failure can be handled until the quorum is back.
Application reports send failures while broker dashboards are normal
Check
Producer `record-error-rate`, `record-retry-rate`, `buffer-available-bytes` and `request-latency-max` for that application; `ErrorsPerSec` for `request=Produce` on the brokers by error code
- How to read it
- Errors with retries and high client-side latency point at the network path or one partition leader. A shrinking buffer with blocked sends means the application produces faster than the cluster acknowledges. Broker-side
ErrorsPerSecnames the error code the clients were sent.
Key takeaways
- Monitor in this order: is data replicated, is every partition available, is the cluster fast enough, are applications keeping up.
- Under-replicated means less margin; under min ISR means
acks=allwrites are refused; offline means nothing works. They deserve different severities. - Replica-state metrics are reported by leaders, so sum them across brokers and find the faulty broker from the ISR lists, not from the highest count.
- Repeated ISR shrinks and expands with all brokers up are an early warning of a slow broker.
- Split request latency into queue, local, remote and response time before guessing at a cause.
- Page on symptoms clients feel; keep causes on dashboards. Measure consumer lag from the brokers so it survives a dead consumer.
- Apache Kafka provides the metrics and JMX. The Prometheus JMX exporter, host agents and Micrometer are separate projects that carry them to a dashboard.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A 6-broker cluster with replication factor 3 and min.insync.replicas=2. All six brokers are running and registered. For two days, IsrShrinksPerSec and IsrExpandsPerSec have been non-zero in bursts, and the removed replica is always on broker 5. Under-replicated partitions go to about 30 and return to 0 several times an hour. No client has reported an error.
What should you do with this?
Scenario 2 of 3
Produce p99 TotalTimeMs on broker 2 has risen from 15 ms to 400 ms for topics with acks=all. On broker 2, RequestQueueTimeMs is 1 ms, LocalTimeMs is 4 ms, RemoteTimeMs is 390 ms, and RequestHandlerAvgIdlePercent is 0.8. Other brokers show normal produce latency for the partitions they lead.
Where is the most likely problem?
Scenario 3 of 3
A Payment Service consumer group is alerted on with the client metric records-lag-max above 10,000 for 5 minutes. On Saturday its pods crash-loop on a bad configuration for three hours. No lag alert fires. On Monday the team finds 2 million unprocessed records on payments.events.
What is the right change to the alerting?
References
References: Apache Kafka documentation: Monitoring (broker, KRaft, producer and consumer metrics) ↗ · Apache Kafka documentation: Broker configuration reference ↗ · Apache Kafka documentation: Basic Kafka operations (checking consumer position, topic tool) ↗ · Apache Kafka documentation: KRaft vs ZooKeeper (metrics removed with ZooKeeper mode) ↗ · Spring for Apache Kafka reference: Monitoring (Micrometer timers and native metrics) ↗ · Spring Boot reference: Actuator metrics (Kafka metrics, Prometheus endpoint) ↗ · Prometheus JMX exporter (separate project) ↗ · Report a problem