Skip to content
All outages

Incident · Hard · DNS/HTTP · Databases

The database that vanished

  • SEV1
  • ONE REGION
  • 06:48 UTC
90s

The situation

Every service calling your key-value database in your busiest region starts failing to connect. The storage team swears the data is intact and the nodes are healthy.

The evidence

Logs and signalsReconstructed
$ dig kv-db.region-east.internal
;; ANSWER SECTION:
;; (empty)
dns-automation
planner : generating plans ok
enactor-A : delayed — applying plan #1041
enactor-B : applied plan #1067 ok
enactor-B : cleanup: deleting stale plans < #1060
storage nodes: healthy

Your call

Why does the endpoint resolve to nothing?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem