Incident · Medium · Payments · Databases
Crash-looping everywhere at once
- SEV1
- EVERY REGION
- SAME MINUTE
90s
The situation
Your API gateway's quota-check service starts crash-looping. It happens in every region, within the same minute. Your on-call channel fills with "roll it back!"
The evidence
- europe
- asia
- us-central1
Logs and signalsReconstructed
29 Mayrelease v2.x deployed · no alerts sincetodayno binary deploys10:45quota policy written to Spanner10:45policy replicated globally (seconds)10:46SIGSEGV: nil pointer dereference10:46policy.fields: [blank, blank]11:30spanner us-central1: overloaded
Your call
Why is everything crashing, and why is us-central1 recovering so slowly?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.