Skip to content
All outages

Incident · Medium · Kafka / streaming

A producer for every request

  • SEV1
  • 03:53 UTC
  • BROKERS OUT OF HEAP
90s

The situation

One Kafka broker runs out of JVM heap and falls over. It looks like a bad machine, so you add brokers and remove it. Then a second broker runs out of memory, and a third. Message volume is normal. Doubling the heap and restarting brings the cluster back, and hours later it happens again, faster. A small analytics feature has been ramping up behind a flag all week.

The evidence

Broker heap used (%)illustrative
  • heap
  • messages in
Logs and signalsReconstructed
03:53broker-2: GC thrashing · heap exhausted
04:20suspect hardware · cluster expanded, broker-2 removed
04:55broker-5: out of memory
05:10new producers registered: 84× the usual rate
06:40heap doubled · rolling restart · stable
14:40feature flag: api-key audit 25% → 75%
16:38broker heap exhausted again
-- messages in: normal

Your call

What is filling the brokers' memory?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem