Skip to content
All outages

Incident · Medium · DNS/HTTP · Payments

Failing slowly, then all at once

  • SEV1
  • ~15 HOURS
  • IN-STORE PAYMENTS
90s

The situation

At 13:54 sellers suddenly cannot take card payments, and services cannot reach each other. Two changes went out today: a firewall policy change at 11:04 and a DNS zone change at 13:56. Internal DNS servers are failing requests, and your own access, sign-in and deploy tools are failing too, which makes any fix slow.

The evidence

Services unable to reach their dependenciesillustrative
  • failing %
Logs and signalsReconstructed
11:04firewall policy change deployed → on-node ruleset much larger
todayinternal DNS: service upgrade rolled out
13:54payments: failing · services cannot reach dependencies
13:56DNS zone change applied
14:02incident response begins: DNS investigation
18:00DNS capacity increased → no improvement

Your call

What best explains the pattern?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem