Skip to content
All outages

Editor's note: Object storage's index behaves like a URL shortener's lookup table.

Incident · Easy · Databases · URL shortener

One typo

  • SEV1
  • 09:37 PST
  • OBJECT STORAGE
90s

The situation

An engineer debugging a slow billing system runs a well-used command to take a few servers out of service. Minutes later, the region's object storage cannot serve any request. Half the web's images and downloads break. Your own status dashboard cannot turn red, because its icons live in the same storage.

The evidence

Index servers in serviceillustrative
  • servers
Logs and signalsReconstructed
09:37ops: remove-capacity --count (mistyped: far too many)
09:37tool: accepted, no minimum check
09:38index subsystem: below required capacity
09:38placement subsystem: cannot run without index
09:45full restart begun · never done at this size

Your call

Why did a small mistake become hours of outage?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem