Skip to content
All outages

Editor's note: Payments-style date maths, in a cloud platform.

Incident · Easy · Payments · Infra

Bad hardware at midnight

  • SEV1
  • 00:00 UTC, 29 FEB
  • GLOBAL
90s

The situation

At midnight UTC, new virtual machines stop starting anywhere in your cloud. Each time a VM fails to start three times, the platform marks that physical server as faulty and moves its VMs to another server, where they fail again. The count of 'bad hardware' climbs by the minute.

The evidence

Servers marked as faultyillustrative
  • faulty
Logs and signalsReconstructed
00:00vm agent: create transfer certificate, expiry = today + 1 year
00:00error: certificate creation failed: date out of range
00:25agent start timed out (3 of 3) → server marked faulty
00:26self-healing: moving VMs to another server
00:27hardware diagnostics: pass

Your call

What is the cause?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem