Skip to content
All outages

Incident · Hard · Infra · Databases

Bigger boxes didn't help

  • SEV0
  • 73 HOURS
  • SERVICE DISCOVERY
90s

The situation

Your service-discovery cluster, which every other system uses to find everything else, slows to a crawl. Writes to its key-value store take seconds. The whole platform goes down with it. The servers look overloaded, so you replace them with far more powerful machines. Performance gets no better. Your telemetry depends on the same cluster, so you can barely see.

The evidence

Key-value write latencyillustrative
  • latency
Logs and signalsReconstructed
earliernew 'streaming' feature enabled on discovery cluster
day 0node count up 50% · KV write latency seconds
day 0telemetry: unavailable (needs discovery)
day 1hardware swapped for bigger machines → no improvement
day 2profile: threads blocked on one shared channel · storage file mostly free-list

Your call

Why didn't bigger hardware help?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem