Skip to content
All outages

Incident · Easy · Kafka / streaming

The graph nobody watched

  • SEV1
  • TRAFFIC PEAK
  • BROKERS FALL IN SEQUENCE
90s

The situation

At the busiest hour your Kafka brokers stop responding one after another, like dominoes. CPU and disk latency, the two graphs you watch closely, look fine on every broker. Producers write only about 20 Mbps. Over the past year, though, the number of systems reading the full stream has grown: a second production pipeline, plus beta and staging environments.

The evidence

Broker network out (Mbps)illustrative
  • read by consumers
  • written by producers
Logs and signalsReconstructed
02:10broker-1: unresponsive
02:11consumers reconnect to remaining brokers
02:14broker-3: unresponsive
02:15broker-2: unresponsive
-- cpu: normal · disk latency: normal
-- produce: ~20 Mbps · fetch: 250+ Mbps
-- consumer environments: prod ×2, beta, staging

Your call

What ran out?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem