Practice
Work a production incident, diagnose real outages, then practise the parts of on-call nobody teaches.
Incident rooms
25 incidents · 24 from real outages
Work a production incident from the inside: inspect the evidence, stop the impact, then find the cause.
- 002SEV-1Config and deploysEvery core at 100%Every edge machine is at 100% CPU. Visitors get 502 errors and traffic served is down 80%.
- 003SEV-1Config and deploysThe network that keeps blinkingThe proxy fleet is returning 5xx on and off, in a five-minute rhythm. Your status page is unreachable.
- 004SEV-1Config and deploysCrash-looping everywhere at onceAPI requests are failing in every region. The quota-check service is crash-looping everywhere.
- 005SEV-1Caching and stateThe cache that never refillsClients are failing to boot. The membership database is saturated and the cache in front of it is not refilling.
- 006SEV-1DatabasesForty-three secondsThe site is slow and some pages show out-of-date data. Database primaries have moved to the other coast.
- 007SEV-1DatabasesFive backups, zero restoresThe site is down. About 300 GB of the primary database's data directory has just been deleted.
- 008SEV-1Networking and DNSNobody can find anybodyProduct and outfit pages are returning errors. Services in the cluster cannot resolve hostnames.
- 009SEV-1Platform and OSBigger boxes didn't helpPlayers online have fallen to half of normal. The Consul cluster everything depends on is slow and unstable.
- 010SEV-1Capacity and scalingOne server too manyPutting and getting stream records is failing. Metrics, logins and events that depend on the stream service are failing with it.
- 011SEV-1Networking and DNSThe database that vanishedClients cannot connect to the regional database endpoint. Services that store their state in it, including instance launches, are failing.
- 012SEV-1Platform and OS06:00 UTC, everywhereThe web app and APIs are unreachable, monitors are not alerting, and tens of thousands of nodes are offline across five regions.
- 013SEV-1DatabasesThe database stopped writingNo new events can be stored. The database is refusing writes, and the event backlog is growing out of control.
- 014SEV-1Capacity and scalingOverloaded, yet idleThe app is unavailable. The web tier is saturated, and your dashboards and alerting are down.
- 015SEV-1Queues and asyncThe message that kills its readerThe app is crashing every few minutes. Customers lose the dashboard and Slack interactions each time.
- 016SEV-1Databases2,147,483,647The app is effectively read-only. Nearly every action fails because one very busy table can no longer accept writes.
- 017SEV-1DatabasesThe failover that never cameThe API and dashboard are completely down. The primary database has lost its disks, and automatic failover has not happened.
- 018SEV-2Time and limitsTime went backwardsAbout 0.2% of DNS queries are failing, all for CNAME records, and some HTTP requests fail with them.
- 019SEV-1Caching and stateCharged again. And again.Customers with auto-recharge are being charged repeatedly. Balances read as zero and cannot be updated.
- 020SEV-1Capacity and scalingSmaller to save moneyThe dashboard has been down since 17:21. Build agents are still running builds, but nobody can see or manage them.
- 021SEV-1APIs and dependenciesThe dashboard that hammered its own APIThe dashboard is mostly unavailable and API errors are rising again. Every API request needs an authorization check from one overloaded service.
- 022SEV-1APIs and dependenciesZero usage, zero quotaSign-ins and credential checks are failing for every user account, so every app that needs authentication is down.
- 023SEV-1APIs and dependenciesThe backlog that came back at onceBuilds are barely starting, about one a minute instead of many a second. The build queue keeps growing and the main database is unresponsive.
- 024SEV-1APIs and dependenciesThe dependency that never said noAbout a third of user connections have stopped delivering messages. The servers that hold those connections are running out of memory and crashing.
- 025SEV-1APIs and dependenciesFail open or fail closedMany products are failing at once: identity logins, new device registrations, challenge widgets and dashboard logins. Their shared key-value store cannot reach its central storage.
On-call skills
Take the page, find the cause, write it up afterwards, read the graphs, and tell the story in an interview. Every mode reuses the real outages from Break.
- Pager simulatorA page goes off. Five simulated minutes to acknowledge, read the signals and stop the bleeding.10 scenarios · not started
- InvestigateA vague report and a handful of leads. Spend three looks on the right evidence, then name the cause.12 scenarios · not started
- Write the postmortemOrder the timeline, separate trigger from root cause, and pick the action items that are not traps.10 scenarios · not started
- Read the graphOne panel, trace or PromQL query a day. Which service is the bottleneck? Saturation or errors?18 scenarios · not started
- Tell me about an outageA STAR story template and five interviewer follow-ups for every Break card you have finished.80 scenarios · not started
- Interview roundsRounds of design questions on one theme. Pick an approach, watch what it does, compare with a model answer.79 scenarios · not started