Skip to content

Practice

Work a production incident, diagnose real outages, then practise the parts of on-call nobody teaches.
Kubernetes · 10 rounds · about 3 minutesDiagnose the clusterPods are failing. Read the evidence, find the cause, pick the first fix.

Incident rooms

25 incidents · 24 from real outages

Work a production incident from the inside: inspect the evidence, stop the impact, then find the cause.

SEV-2Databases · Composite incident · Illustrative data · 15–20 minutesIncident #001 — Checkout degradationCheckout requests are timing out. Successful orders have dropped by 35%. Stop the impact first, then find the cause.API p955.2 secondsAPI error rate18%CPU74%DB pool active50 / 50Take the incident
  1. 002SEV-1Config and deploysEvery core at 100%Every edge machine is at 100% CPU. Visitors get 502 errors and traffic served is down 80%.
  2. 003SEV-1Config and deploysThe network that keeps blinkingThe proxy fleet is returning 5xx on and off, in a five-minute rhythm. Your status page is unreachable.
  3. 004SEV-1Config and deploysCrash-looping everywhere at onceAPI requests are failing in every region. The quota-check service is crash-looping everywhere.
  4. 005SEV-1Caching and stateThe cache that never refillsClients are failing to boot. The membership database is saturated and the cache in front of it is not refilling.
  5. 006SEV-1DatabasesForty-three secondsThe site is slow and some pages show out-of-date data. Database primaries have moved to the other coast.
  6. 007SEV-1DatabasesFive backups, zero restoresThe site is down. About 300 GB of the primary database's data directory has just been deleted.
  7. 008SEV-1Networking and DNSNobody can find anybodyProduct and outfit pages are returning errors. Services in the cluster cannot resolve hostnames.
  8. 009SEV-1Platform and OSBigger boxes didn't helpPlayers online have fallen to half of normal. The Consul cluster everything depends on is slow and unstable.
  9. 010SEV-1Capacity and scalingOne server too manyPutting and getting stream records is failing. Metrics, logins and events that depend on the stream service are failing with it.
  10. 011SEV-1Networking and DNSThe database that vanishedClients cannot connect to the regional database endpoint. Services that store their state in it, including instance launches, are failing.
  11. 012SEV-1Platform and OS06:00 UTC, everywhereThe web app and APIs are unreachable, monitors are not alerting, and tens of thousands of nodes are offline across five regions.
  12. 013SEV-1DatabasesThe database stopped writingNo new events can be stored. The database is refusing writes, and the event backlog is growing out of control.
  13. 014SEV-1Capacity and scalingOverloaded, yet idleThe app is unavailable. The web tier is saturated, and your dashboards and alerting are down.
  14. 015SEV-1Queues and asyncThe message that kills its readerThe app is crashing every few minutes. Customers lose the dashboard and Slack interactions each time.
  15. 016SEV-1Databases2,147,483,647The app is effectively read-only. Nearly every action fails because one very busy table can no longer accept writes.
  16. 017SEV-1DatabasesThe failover that never cameThe API and dashboard are completely down. The primary database has lost its disks, and automatic failover has not happened.
  17. 018SEV-2Time and limitsTime went backwardsAbout 0.2% of DNS queries are failing, all for CNAME records, and some HTTP requests fail with them.
  18. 019SEV-1Caching and stateCharged again. And again.Customers with auto-recharge are being charged repeatedly. Balances read as zero and cannot be updated.
  19. 020SEV-1Capacity and scalingSmaller to save moneyThe dashboard has been down since 17:21. Build agents are still running builds, but nobody can see or manage them.
  20. 021SEV-1APIs and dependenciesThe dashboard that hammered its own APIThe dashboard is mostly unavailable and API errors are rising again. Every API request needs an authorization check from one overloaded service.
  21. 022SEV-1APIs and dependenciesZero usage, zero quotaSign-ins and credential checks are failing for every user account, so every app that needs authentication is down.
  22. 023SEV-1APIs and dependenciesThe backlog that came back at onceBuilds are barely starting, about one a minute instead of many a second. The build queue keeps growing and the main database is unresponsive.
  23. 024SEV-1APIs and dependenciesThe dependency that never said noAbout a third of user connections have stopped delivering messages. The servers that hold those connections are running out of memory and crashing.
  24. 025SEV-1APIs and dependenciesFail open or fail closedMany products are failing at once: identity logins, new device registrations, challenge widgets and dashboard logins. Their shared key-value store cannot reach its central storage.
Real outagesReal outages, 90 seconds each. Read the signals, pick the root cause, then see who it happened to.0 of 80 diagnosed

On-call skills

Take the page, find the cause, write it up afterwards, read the graphs, and tell the story in an interview. Every mode reuses the real outages from Break.