Incident · Easy · Databases · URL shortener
One typo
- SEV1
- 09:37 PST
- OBJECT STORAGE
90s
The situation
An engineer debugging a slow billing system runs a well-used command to take a few servers out of service. Minutes later, the region's object storage cannot serve any request. Half the web's images and downloads break. Your own status dashboard cannot turn red, because its icons live in the same storage.
The evidence
- servers
Logs and signalsReconstructed
09:37ops: remove-capacity --count (mistyped: far too many)09:37tool: accepted, no minimum check09:38index subsystem: below required capacity09:38placement subsystem: cannot run without index09:45full restart begun · never done at this size
Your call
Why did a small mistake become hours of outage?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.