Incident · Medium · URL shortener · Databases
Fixed the column, lost the API
- SEV1
- 15:06–18:57 UTC
- CONTROL PLANE
90s
The situation
Logins and deploys start failing: a column that points at another table's ID turns out to be a narrower number type than the ID itself, and the ID has just outgrown it. On-call widens the column. The migration succeeds and new rows save again. Minutes later every query on that table is crawling, and the outage spreads from logins to the whole API. Customer apps that are already running are untouched.
The evidence
- failed writes
- query time
Logs and signalsReconstructed
15:05INSERT … ERROR: value out of range for referencing column15:57status: investigating authentication errors (HTTP 500)16:13root cause identified: column narrower than the key it referenceslaterALTER TABLE … ALTER COLUMN … TYPE … done; inserts OK16:50queries on the altered table: very slow; API unavailable17:26status: platform API unavailable; running apps unaffected
Your call
The overflow is fixed. Why is everything slower than before?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.