From public postmortems
Real outages
Diagnose outages that really happened, in 90 seconds each. Who it happened to is revealed after you answer.
- Outages
- 80 incidents
- Failure types
- 13 failure types
- Solved
- 0/80 solved
Browse by what broke
Pick a failure and see every outage it caused.
Start here: 5 classics
The outages every engineer ends up talking about. 0/5 solved.
Interview: 10 to know
One outage for each failure interviewers ask about. 0/10 solved.
Kafka on-call pack
Brokers, consumers, offsets and queues under pressure. 0/11 solved.
When the fix is the outage
The rollback, the restart or the repair made it worse. 0/5 solved.
New this week
Added in the last seven days. 0/26 solved.
All outages
80 outages
- Overload and stormsCloud & infraNew
Every restart freezes again
A shared connection gateway goes dark at peak, and every batch of servers you restart freezes again. What is really taking it down?
Difficulty: On-call90s to solveCompany hidden - Overload and stormsCloud & infraNew
Did it go through?
A national payment switch drops to a 50% success rate. Isolating sites barely helps, and one status call is far busier than usual.
Difficulty: On-call90s to solveCompany hidden - Overload and stormsDNS & networkNew
Everything played but the fight
A record live boxing stream fails for many viewers while films on the same service play normally. Where does the evidence point?
Difficulty: On-call90s to solveCompany hidden - Hidden single pointConsensusKafka & queuesNew
Five nodes, one building
One data hall loses cooling. The exchange spans several zones and should survive it, yet nearly all trading stops for hours.
Difficulty: On-call90s to solveCompany hidden - Overload and stormsCloud & infraNew
Four good minutes
A resize ends 17% of sessions. Chat recovers, voice stays down, and a full restart of the call-routing cluster helps for four minutes.
Difficulty: On-call90s to solveCompany hidden - Overload and stormsCloud & infraNew
Four times the previous peak
A registered-fan presale opens to four times the site's previous peak traffic. Carts are losing tickets at checkout, and more sales are still to open.
Difficulty: On-call90s to solveCompany hidden - Overload and stormsDatabasesNew
One sale, every store
One store's sale takes down every store on its database shard. More caching did not save it, and the next sale is in a week.
Difficulty: On-call90s to solveCompany hidden - Overload and stormsCloud & infraNew
Validation pending
At market open a broker's vendor order system backs up. The vendor works on the queue of pending orders, and the queue keeps growing.
Difficulty: On-call90s to solveCompany hidden - Failover failedDatabasesNew
#7, then #39
Customer-facing reference numbers suddenly skip ahead by the same amount for many accounts. No data is missing and nothing crashed.
Difficulty: On-call90s to solveCompany hidden - Limits and exhaustionKafka & queuesNew
A producer for every request
Kafka brokers run out of heap one after another. Message volume is normal, more heap only buys time, and a small feature is mid-rollout.
Difficulty: On-call90s to solveCompany hidden - Failover failedRedis & cachingNew
Failed over. Nobody followed.
A Redis primary vanishes and the cluster promotes its secondary within a minute. Ten minutes on, part of the API is still failing. Why?
Difficulty: On-call90s to solveCompany hidden - Limits and exhaustionCloud & infraNew
Five characters from public
Share links are meant to be private because nobody else knows them. A researcher opens millions of them without being sent a single one.
Difficulty: On-call90s to solveCompany hidden - The fix made it worseDatabasesNew
Fixed the column, lost the API
An ID outgrows a narrower referencing column. The emergency migration works, then the whole API goes down. What did the fix break?
Difficulty: On-call90s to solveCompany hidden - Limits and exhaustionRedis & cachingNew
Full every ten minutes
A Redis LRU cache stalls for a few seconds every several minutes. Traffic is steady, and memory drops by far more than any key could explain.
Difficulty: Expert90s to solveCompany hidden - Hidden single pointRedis & cachingNew
More nodes, more outages
One Redis cluster node is pinned at 100% CPU with steady traffic. Moving to a larger cluster makes the outages more frequent. What is going on?
Difficulty: On-call90s to solveCompany hidden - Split brainKafka & queuesConsensusNew
One cluster, two truths
One in ten job writes fails and clients disagree about who leads each partition. All brokers are up. Coordination hosts are mid-replacement.
Difficulty: Expert90s to solveCompany hidden - Limits and exhaustionCloud & infraNew
Overloaded, yet idle
Millions of clients reconnect after the holidays, latency explodes, yet web-tier CPU drops. What is really going on?
Difficulty: On-call90s to solveCompany hidden - Config pushDNS & networkNew
Permanently out of credit
Customers who ran out of mobile data keep landing on the carrier's error page, even later on a different connection entirely.
Difficulty: Starter90s to solveCompany hidden - Overload and stormsRedis & cachingDatabasesNew
Quiet Saturday, loud Monday
A cache TTL is cut from 12 hours to 2 on a quiet Saturday. On Monday the core database overloads, and the TTL is only half the story.
Difficulty: On-call90s to solveCompany hidden - Overload and stormsKafka & queuesNew
Reverted in five minutes
A blank config reached a log pipeline and was reverted within five minutes. Hours later, over half the logs are still being lost. Why?
Difficulty: On-call90s to solveCompany hidden - Limits and exhaustionDatabasesNew
Same customers, same sick server
A fixed group of customers see requests hang for seven hours overnight. Everyone else is fine and the health check stays green.
Difficulty: On-call90s to solveCompany hidden - Data loss and exposureRedis & cachingNew
Someone else's chat titles
Users see other people's chat titles. Every cache key holds the right data, and the cache cluster is healthy. So how do replies get crossed?
Difficulty: On-call90s to solveCompany hidden - Overload and stormsKafka & queuesNew
The graph nobody watched
Kafka brokers fail one after another at peak. CPU and disk look fine and writes are small, but many more systems now read the stream.
Difficulty: On-call90s to solveCompany hidden - Bad inputKafka & queuesNew
The message that kills its reader
An app that serves web traffic and reads a queue keeps dying minutes after every restart. No deploy, and the logs cut off before each crash.
Difficulty: On-call90s to solveCompany hidden