Incident · Easy · Kafka / streaming
The graph nobody watched
- SEV1
- TRAFFIC PEAK
- BROKERS FALL IN SEQUENCE
90s
The situation
At the busiest hour your Kafka brokers stop responding one after another, like dominoes. CPU and disk latency, the two graphs you watch closely, look fine on every broker. Producers write only about 20 Mbps. Over the past year, though, the number of systems reading the full stream has grown: a second production pipeline, plus beta and staging environments.
The evidence
- read by consumers
- written by producers
Logs and signalsReconstructed
02:10broker-1: unresponsive02:11consumers reconnect to remaining brokers02:14broker-3: unresponsive02:15broker-2: unresponsive-- cpu: normal · disk latency: normal-- produce: ~20 Mbps · fetch: 250+ Mbps-- consumer environments: prod ×2, beta, staging
Your call
What ran out?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.