Skip to content
All outages

Incident · Hard · DNS/HTTP

The snapshot nobody could trust

  • SEV1
  • 15:35 UTC
  • GLOBAL EDGE
90s

The situation

Your global edge service starts crashing across regions. A routine customer configuration change went out a few minutes ago and passed every health gate. The control plane is running two software builds side by side. Someone says: restore the Last Known Good snapshot.

The evidence

Health gate vs data-plane crashesillustrative
  • gate pass %
  • crashes
Logs and signalsReconstructed
15:35customer config change accepted by build A, processed by build B
15:36health gates: passed
15:36config saved as Last Known Good
15:41data plane: crash in async processing
15:43protection system: all config changes blocked
15:48#incident: "restore Last Known Good?"

Your call

Why would restoring the snapshot not help?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last reviewed: Official postmortem