Skip to content
All outages

Editor's note: Many companies hit by the same leap second.

Incident · Hard · Databases · Infra

Every box, same second

  • SEV2
  • 00:00 UTC, 1 JUL
  • WHOLE FLEET
90s

The situation

At midnight UTC, CPU jumps to 100% on every database host and every JVM application server you own, in every data centre, in the same second. Query volume is flat. No deploy, no cron job, no traffic spike. The processes are busy doing nothing useful.

The evidence

CPU, all database hosts vs queriesillustrative
  • CPU %
  • queries
Logs and signalsReconstructed
23:59:60kernel: Clock: inserting leap second
00:00:00mysqld: CPU 100% · threads spinning on timed waits
00:00:00java: CPU 100% · same pattern
00:00:10slow query log: nothing unusual

Your call

What is the cause?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Bug report