Skip to content
All outages

Incident · Medium · URL shortener · Databases

Fixed the column, lost the API

  • SEV1
  • 15:06–18:57 UTC
  • CONTROL PLANE
90s

The situation

Logins and deploys start failing: a column that points at another table's ID turns out to be a narrower number type than the ID itself, and the ID has just outgrown it. On-call widens the column. The migration succeeds and new rows save again. Minutes later every query on that table is crawling, and the outage spreads from logins to the whole API. Customer apps that are already running are untouched.

The evidence

API errors and query timeillustrative
  • failed writes
  • query time
Logs and signalsReconstructed
15:05INSERT … ERROR: value out of range for referencing column
15:57status: investigating authentication errors (HTTP 500)
16:13root cause identified: column narrower than the key it references
laterALTER TABLE … ALTER COLUMN … TYPE … done; inserts OK
16:50queries on the altered table: very slow; API unavailable
17:26status: platform API unavailable; running apps unaffected

Your call

The overflow is fixed. Why is everything slower than before?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem