Incident · Medium · DNS/HTTP · Payments
Failing slowly, then all at once
- SEV1
- ~15 HOURS
- IN-STORE PAYMENTS
90s
The situation
At 13:54 sellers suddenly cannot take card payments, and services cannot reach each other. Two changes went out today: a firewall policy change at 11:04 and a DNS zone change at 13:56. Internal DNS servers are failing requests, and your own access, sign-in and deploy tools are failing too, which makes any fix slow.
The evidence
- failing %
Logs and signalsReconstructed
11:04firewall policy change deployed → on-node ruleset much largertodayinternal DNS: service upgrade rolled out13:54payments: failing · services cannot reach dependencies13:56DNS zone change applied14:02incident response begins: DNS investigation18:00DNS capacity increased → no improvement
Your call
What best explains the pattern?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.