Skip to content
All concepts

Concept

Dead man's switch

A dead man's switch is an alert that fires all the time on purpose, sent to a system outside the monitoring stack. That outside system pages someone when the signal stops arriving. It covers the one failure ordinary alerts cannot report: the alerting pipeline itself being down.

  1. Alert that always fires
  2. Heartbeat to an outside service (succeeds)
  3. Monitoring breaks (fails)
  4. Heartbeat stops (fails)
  5. Outside service pages (succeeds)
Silence is the signal. A broken alerting pipeline cannot send an alert, but it can stop sending one.

Where you'll see it

Nothing on the site names this term yet.

Sources: kube-prometheus runbooks: the Watchdog alert ↗ · Report a problem

The definition is in our own words.