Skip to content
All outages

Incident · Medium · DNS/HTTP

The rollback made it worse

  • SEV2
  • <1% OF USERS
  • ~24 HOURS
90s

The situation

You are enabling DNSSEC on your main domain, on the third attempt. The final step publishes a DS record at the parent zone, cached for up to 24 hours. A small share of users report that the app will not resolve. The team reacts fast and rolls back by removing the signatures from the zone. Now far more users cannot resolve it.

The evidence

Resolution failuresillustrative
  • SERVFAIL
Logs and signalsReconstructed
t+0 DS record published at parent (TTL 24h)
t+30msome resolvers: wildcard records fail validation
t+1h rollback: zone signatures removed
t+1h validating resolvers: DS cached, zone unsigned → SERVFAIL

Your call

Why did the rollback make it worse?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem