Incident · Medium · Kafka / streaming
One server too many
- SEV1
- ONE REGION
- STREAMING FRONT END
90s
The situation
You add a small amount of capacity to the front-end fleet of your streaming service. Over the next hours, every front-end server, old ones included, starts failing to build its map of which shard lives where. Sign-in, metrics and serverless products that sit on top of the stream degrade with it.
The evidence
- threads
- OS limit
Logs and signalsReconstructed
02:44capacity add started: new front-end servers joining03:47capacity add complete05:15front-end: cannot create thread (all servers)05:16shard map: failed to build-- each server keeps one thread per peer server
Your call
What is the underlying cause?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.