Incident · Hard · Infra · Databases
Bigger boxes didn't help
- SEV0
- 73 HOURS
- SERVICE DISCOVERY
90s
The situation
Your service-discovery cluster, which every other system uses to find everything else, slows to a crawl. Writes to its key-value store take seconds. The whole platform goes down with it. The servers look overloaded, so you replace them with far more powerful machines. Performance gets no better. Your telemetry depends on the same cluster, so you can barely see.
The evidence
- latency
Logs and signalsReconstructed
earliernew 'streaming' feature enabled on discovery clusterday 0node count up 50% · KV write latency secondsday 0telemetry: unavailable (needs discovery)day 1hardware swapped for bigger machines → no improvementday 2profile: threads blocked on one shared channel · storage file mostly free-list
Your call
Why didn't bigger hardware help?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.