Incident · Medium · Kafka / streaming
A producer for every request
- SEV1
- 03:53 UTC
- BROKERS OUT OF HEAP
90s
The situation
One Kafka broker runs out of JVM heap and falls over. It looks like a bad machine, so you add brokers and remove it. Then a second broker runs out of memory, and a third. Message volume is normal. Doubling the heap and restarting brings the cluster back, and hours later it happens again, faster. A small analytics feature has been ramping up behind a flag all week.
The evidence
- heap
- messages in
Logs and signalsReconstructed
03:53broker-2: GC thrashing · heap exhausted04:20suspect hardware · cluster expanded, broker-2 removed04:55broker-5: out of memory05:10new producers registered: 84× the usual rate06:40heap doubled · rolling restart · stable14:40feature flag: api-key audit 25% → 75%16:38broker heap exhausted again-- messages in: normal
Your call
What is filling the brokers' memory?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.