Skip to content
All outages

Incident · Medium · Payments · Databases

Crash-looping everywhere at once

  • SEV1
  • EVERY REGION
  • SAME MINUTE
90s

The situation

Your API gateway's quota-check service starts crash-looping. It happens in every region, within the same minute. Your on-call channel fills with "roll it back!"

The evidence

Task restarts / minillustrative
  • europe
  • asia
  • us-central1
Logs and signalsReconstructed
29 Mayrelease v2.x deployed · no alerts since
todayno binary deploys
10:45quota policy written to Spanner
10:45policy replicated globally (seconds)
10:46SIGSEGV: nil pointer dereference
10:46policy.fields: [blank, blank]
11:30spanner us-central1: overloaded

Your call

Why is everything crashing, and why is us-central1 recovering so slowly?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem