Incident · Easy · Payments · Infra
Bad hardware at midnight
- SEV1
- 00:00 UTC, 29 FEB
- GLOBAL
90s
The situation
At midnight UTC, new virtual machines stop starting anywhere in your cloud. Each time a VM fails to start three times, the platform marks that physical server as faulty and moves its VMs to another server, where they fail again. The count of 'bad hardware' climbs by the minute.
The evidence
- faulty
Logs and signalsReconstructed
00:00vm agent: create transfer certificate, expiry = today + 1 year00:00error: certificate creation failed: date out of range00:25agent start timed out (3 of 3) → server marked faulty00:26self-healing: moving VMs to another server00:27hardware diagnostics: pass
Your call
What is the cause?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.