Skip to content
Chaos Lab

Challenge 26 · intermediate · Alerting

The noisy alert

Two hours of a checkout service, compressed so one second is one minute. Its target is 99.9% of requests succeeding. Ten times it has a one-minute blip of 2–4% errors that clears on its own. At minute 40 a dependency fails and 30% of requests fail for 5 minutes. At minute 75 a bad cache node quietly fails 1.5% of requests for half an hour. The alert pages when the error rate passes 1%, and every false page makes on-call slower to trust the next one.

Over the two hours: at most one false page, someone responding to the outage within 4 minutes of it starting, and to the slow problem within 15.

ready00:00 / 02:00Success · 10 s–p99 · 10 s–False pages0Firingno

Goal · measured 00:00–02:00

0 changes · par 1
  • False pages ≤ 1not measured yet
  • Minutes until someone responds to the outage ≤ 4.0 minnot measured yet
  • Minutes until someone responds to the slow problem ≤ 15.0 minnot measured yet

This is a simulation. Its numbers come from a model of a typical system, not from measuring real servers. The same link and seed always give the same run. How the Chaos Lab works.