Incident · Hard · DNS/HTTP · Databases
The database that vanished
- SEV1
- ONE REGION
- 06:48 UTC
90s
The situation
Every service calling your key-value database in your busiest region starts failing to connect. The storage team swears the data is intact and the nodes are healthy.
The evidence
Logs and signalsReconstructed
$ dig kv-db.region-east.internal;; ANSWER SECTION:;; (empty)dns-automationplanner : generating plans okenactor-A : delayed — applying plan #1041enactor-B : applied plan #1067 okenactor-B : cleanup: deleting stale plans < #1060storage nodes: healthy
Your call
Why does the endpoint resolve to nothing?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.