Skip to content
All outages

Editor's note: A streaming service like Kafka, not Kafka itself: the lesson carries over.

Incident · Medium · Kafka / streaming

One server too many

  • SEV1
  • ONE REGION
  • STREAMING FRONT END
90s

The situation

You add a small amount of capacity to the front-end fleet of your streaming service. Over the next hours, every front-end server, old ones included, starts failing to build its map of which shard lives where. Sign-in, metrics and serverless products that sit on top of the stream degrade with it.

The evidence

Threads per front-end serverillustrative
  • threads
  • OS limit
Logs and signalsReconstructed
02:44capacity add started: new front-end servers joining
03:47capacity add complete
05:15front-end: cannot create thread (all servers)
05:16shard map: failed to build
-- each server keeps one thread per peer server

Your call

What is the underlying cause?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem