Incident · Medium · DNS/HTTP
Time went backwards
- SEV2
- 00:00 UTC, 1 JAN
- ~0.2% OF QUERIES
90s
The situation
Exactly at midnight UTC on New Year's Day, your DNS servers start failing a small share of lookups. Only names that are CNAMEs to somewhere else are affected, the ones where you must ask an upstream resolver. Direct answers are fine. A leap second was added at midnight.
The evidence
Logs and signalsReconstructed
23:59:60leap second inserted00:00:00upstream lookup: rtt = now − start = negative00:00:00panic: random-number call given a non-positive bound00:00:01A/AAAA answers (no upstream): OK00:00:05upstream resolvers: reachable
Your call
Why are CNAME lookups failing?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.