Incident · Hard · Databases · Infra
Every box, same second
- SEV2
- 00:00 UTC, 1 JUL
- WHOLE FLEET
90s
The situation
At midnight UTC, CPU jumps to 100% on every database host and every JVM application server you own, in every data centre, in the same second. Query volume is flat. No deploy, no cron job, no traffic spike. The processes are busy doing nothing useful.
The evidence
- CPU %
- queries
Logs and signalsReconstructed
23:59:60kernel: Clock: inserting leap second00:00:00mysqld: CPU 100% · threads spinning on timed waits00:00:00java: CPU 100% · same pattern00:00:10slow query log: nothing unusual
Your call
What is the cause?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.