Incident · Hard · DNS/HTTP
The snapshot nobody could trust
- SEV1
- 15:35 UTC
- GLOBAL EDGE
90s
The situation
Your global edge service starts crashing across regions. A routine customer configuration change went out a few minutes ago and passed every health gate. The control plane is running two software builds side by side. Someone says: restore the Last Known Good snapshot.
The evidence
- gate pass %
- crashes
Logs and signalsReconstructed
15:35customer config change accepted by build A, processed by build B15:36health gates: passed15:36config saved as Last Known Good15:41data plane: crash in async processing15:43protection system: all config changes blocked15:48#incident: "restore Last Known Good?"
Your call
Why would restoring the snapshot not help?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.