Skip to content
All outages

Incident · Medium · DNS/HTTP

Time went backwards

  • SEV2
  • 00:00 UTC, 1 JAN
  • ~0.2% OF QUERIES
90s

The situation

Exactly at midnight UTC on New Year's Day, your DNS servers start failing a small share of lookups. Only names that are CNAMEs to somewhere else are affected, the ones where you must ask an upstream resolver. Direct answers are fine. A leap second was added at midnight.

The evidence

Logs and signalsReconstructed
23:59:60leap second inserted
00:00:00upstream lookup: rtt = now − start = negative
00:00:00panic: random-number call given a non-positive bound
00:00:01A/AAAA answers (no upstream): OK
00:00:05upstream resolvers: reachable

Your call

Why are CNAME lookups failing?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem