Incident · Medium · Kafka / streaming
Same symptom, new broker
- NEAR MISS
- ONE BROKER
- 8× READ IOPS
90s
The situation
One Kafka broker in your cluster is doing about eight times the disk reads of its peers. Traffic is evenly balanced. You do the obvious thing and replace it with a fresh machine. The replacement shows exactly the same behaviour. Broker restarts, which used to take a minute, now take over twenty.
The evidence
- broker 7
- others
Logs and signalsReconstructed
09:00broker-7: read IOPS 8× cluster average10:30broker-7 replaced with new instance + new disk11:05broker-7 (new): read IOPS 8× again11:06tiered storage state: ~6M entries for deleted segments11:07restart time: >20 min
Your call
What is the cause?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.