Chaos Lab
Whole products
Six believable products, end to end: break them and make them survive.
System 1 of 6 · 8 challenges
Online shop
An online shop: people browse product pages (cached in Redis, stored in Postgres) and check out through an external payments provider.
- 01Payments brownout starterThe payments provider is not down, just degraded: one call in five fails at random. Recover the failed calls without making the site slower.Goal: requests succeed ≥ 99.0% · p99 latency ≤ 800 ms
- 02Redis dies at peak starterAt the lunchtime peak the Redis cache goes down and every read falls through to Postgres. Keep checkouts working while the cache is gone.Goal: checkouts succeed ≥ 99.0% · checkout p99 latency ≤ 1,500 ms
- 03The retry storm intermediateThe payments provider starts failing some calls. Your retries were meant to help, but they push it over capacity and it collapses. Tune them so most checkouts still succeed.Goal: checkouts succeed ≥ 75.0% · product pages load ≥ 99.0%
- 04Slow dependency, no threads left starterThe payments provider slows to four seconds a call. Every checkout thread waits for it, and soon product pages stop loading too. Keep the rest of the site up.Goal: product pages load ≥ 99.0% · product page p99 latency ≤ 500 ms
- 06The TTL expiry wave starterA scheduled cache refresh is coming. Last time, every key expired at once and the database went down under the recomputes. Prepare before it hits.Goal: requests succeed ≥ 99.5%
- 07The double charge intermediateThe payments provider charges the card but one reply in five never arrives. Your retry charges the customer again. Stop the double charges without leaving customers charged for orders that failed.Goal: duplicate charges ≤ 1.0 per 1,000 checkouts · checkouts succeed ≥ 98.5% · charged, but the checkout failed ≤ 15.0 per 1,000 checkouts
- 09The viral product intermediateHalf of all product reads are suddenly for one product, whose page is expensive to rebuild and expires every five seconds. Stop every expiry from turning into a pile-up.Goal: product reads succeed ≥ 99.5% · read p99 latency ≤ 500 ms · checkouts succeed ≥ 99.5%
- 10Sold more than we had intermediateA flash sale of 1,000 units, and half of all checkouts want one. Stop the shop selling stock it does not have, without taking the rest of the shop down.Goal: units sold beyond stock ≤ 1 units · checkouts succeed ≥ 99.0% · product reads succeed ≥ 99.0%
System 2 of 6 · 2 challenges
Phone app
A phone app's backend: people open the app (edge → API → cache and database), every phone keeps a live connection to a realtime gateway, and a push sender can notify every phone at once.
- 08Push notification stampede intermediateMarketing sends a push to every phone, and every app prefetches its personal home screen at once. Keep the app opening fast for the people actually using it.Goal: app opens succeed ≥ 99.5% · app open p99 ≤ 500 ms
- 11Everyone reconnects at once intermediateA routine deploy restarts the realtime gateways, and 20,000 phones lose their connection in the same second. Get them back without taking the app down for everyone else.Goal: app opens succeed ≥ 99.5% · phones connected (average) ≥ 95.0%
System 3 of 6 · 2 challenges
Chat
A messaging service: people read channels (through a data service to a message store partitioned by channel) and send messages, which fan-out workers copy into every member's inbox.
- 12The busy channel intermediateSomething happens in a channel with a million members, and everyone opens it at once. Every read lands on one partition of the message store. Keep reads answering.Goal: channel reads answer ≥ 99.5% · read p99 ≤ 300 ms
- 13The announcement intermediateOne message to a channel of 400,000 members becomes thousands of fan-out jobs, and everyone else's messages wait behind it. Keep ordinary messages arriving within a second.Goal: messages delivered within 1 s ≥ 99.0% · delivery p99 ≤ 1.0 s
System 4 of 6 · 3 challenges
Ticketing
A ticketing site: visitors (and, during a drop, bots) come through an edge that can run a waiting room, to a fleet of servers that autoscaling can grow, which hold seats while buyers pay.
- 14The TV ad minute intermediateA TV advert tells the whole country to open your site now: ten times the usual visitors, for ten seconds. Make sure nobody sees an error, even if some of them wait.Goal: visitors who see an error ≤ 1.0% · visitors served per second ≥ 350 served/s
- 15The ticket drop intermediate2,000 seats go on sale. Bots grab holds they never pay for, and fans are told it is sold out while seats sit held. Get the seats to the fans.Goal: seats sold to fans ≥ 95.0% of seats · most seats held by bots at once ≤ 200 seats
- 16Launch day intermediateThe launch goes better than planned: visitors double, then triple, then reach five times normal, and stay there. Serve them quickly, not just eventually.Goal: visitors who see an error ≤ 1.0% · served within 5 s ≥ 98.0%
System 5 of 6 · 3 challenges
Video streaming
An on-demand video service: viewers' players fetch two seconds of video at a time from a CDN; misses go to an origin shield, then to origin servers that read video storage.
- 52The evening peak starterAt 9 pm half your viewers' home internet slows to 2 Mbps and their players stall every few seconds. A bigger buffer only delays it. Keep everyone watching.Goal: time spent buffering ≤ 1.0% · average picture quality ≥ 3.0 Mbps
- 53The premiere intermediateA new episode drops and 60,000 viewers press play at once. None of its segments is cached anywhere, and the origin melts. Twice the servers does not help.Goal: time spent buffering ≤ 1.0% · segments delivered ≥ 99.5%
- 54The CDN goes dark intermediateYour CDN provider fails worldwide. A backup CDN takes the viewers, but its cache is cold and the origin melts under the misses. Keep playback running.Goal: time spent buffering ≤ 1.0% · segments delivered ≥ 99.5%
System 6 of 6 · 2 challenges
Multi-region app
A web app in two regions: traffic steering (DNS) sends users to region A; region B stands by with its own servers and a read-only replica of the database.
- 55The region goes dark intermediateYour main region loses power. A full standby region is ready and waiting, yet users see errors for a minute. Make the failover fast enough to matter.Goal: requests succeed ≥ 80.0% · seconds mostly down ≤ 10 s
- 56The standby that wasn't intermediateFailover works perfectly: traffic is in region B within seconds. Then region B falls over, and every order is refused. Make the standby a real one.Goal: requests succeed ≥ 80.0% · writes succeed ≥ 75.0%
Security
Find the weakness from the attacker's side, then close it.
System 1 of 7 · 1 challenge
Multi-tenant API
Companies keep invoices in one shared database and read them by id. Every request is authenticated; the question is whether the API checks whose invoice it is.
System 2 of 7 · 1 challenge
Token validation
An API accepts JSON Web Tokens and must decide, for every request, whether the token is genuine, still valid and meant for this API.
System 3 of 7 · 1 challenge
Link previews
Users paste a URL and the server fetches it to show a preview. The server sits inside the cloud network, next to the metadata service and an admin API. A fictional, self-contained simulation.
System 4 of 7 · 1 challenge
Staff roles
An internal tool: staff manage customers, admins manage staff accounts. Everyone is signed in; the API decides what each person may do.
System 5 of 7 · 1 challenge
Login service
A login service behind a gateway with a rate limiter. Password checks are slow on purpose, so every login attempt costs real CPU.
System 6 of 7 · 1 challenge
Secrets handling
A payments service holding a database password, a provider API key and customers' session tokens. Logs, a debug endpoint and error pages can all give them away.
System 7 of 7 · 3 challenges
OTP sign-in
Sign in with a 6-digit code sent by SMS and valid for 5 minutes: an OTP service, a code store and SMS providers.
- 57A million codes is not enough intermediateSign-in codes have six digits: a million possibilities. A botnet guesses 40,000 a second from 5,000 machines, and the per-IP limit never notices. Stop the guessing without locking real users out.Goal: attacker guesses checked a second ≤ 100/s · real sign-ins succeeding ≥ 99.0%
- 58No codes during the sale intermediateMid-sale, the SMS provider stops answering and nobody can sign in. Adding a backup provider is not enough on its own. Keep sign-ins working.Goal: sign-ins succeeding ≥ 95.0%
- 59One code, two sign-ins intermediateUsers tap Verify twice, and the app retries on flaky networks. Some codes arrive in two requests at the same moment, and both sign in. Make every code work exactly once.Goal: codes accepted twice ≤ 0.01% · sign-ins succeeding ≥ 99.0%
Performance engineering
Find the real bottleneck, then make the system fast under load.
System 1 of 6 · 2 challenges
Worker pool
A thread-per-request API in front of a database: requests queue for a worker thread, and each thread waits for one of a fixed number of database connections.
- 17Thread pool saturation intermediateTraffic jumps to two and a half times normal. The API's threads are all busy, requests pile up, and the obvious fix is more threads. Find out why it isn't.Goal: answered before clients give up ≥ 800 answered/s · p99 of those answers ≤ 1.0 s · time up, not restarting after a crash ≥ 100.0%
- 22The N+1 query intermediateA harmless-looking release lists each order's line items. Traffic is normal, no query is slow, and the API falls over anyway.Goal: answered before clients give up ≥ 580 answered/s · p99 of those answers ≤ 300 ms
System 2 of 6 · 1 challenge
JVM heap
A Java service with a garbage-collected heap. Short-lived objects are cheap; objects that stay reachable are not.
System 3 of 6 · 1 challenge
Search fan-out
A search request asks 20 index shards in parallel and waits for all of them. Each shard is fast, almost always.
System 4 of 6 · 1 challenge
Pricing API
A CPU-bound API on 4-core instances. Every millisecond of CPU per request decides how many requests an instance can take.
System 5 of 6 · 1 challenge
Orders database
An order-history page backed by one PostgreSQL primary with 8 cores. Every query costs database CPU.
System 6 of 6 · 1 challenge
Checkout fleet
Identical instances that can each do 500 requests a second flat out. The question is how close to flat out you can run them.
Observability & debugging
Diagnose a broken system from its metrics, logs and traces.
System 1 of 6 · 2 challenges
Microservices
A request enters at the gateway and goes to orders, which calls inventory and pricing in parallel. Each service reports only its own view: its response time, logs and spans.
- 21Find the bottleneck intermediateTwo services deploy at once and half of all requests start timing out. Every service upstream looks slow. Use the evidence to find the one that is, and fix only that.Goal: requests answered ≥ 99.0% · p99 at the gateway ≤ 200 ms
- 29The missing trace intermediateDistributed tracing is on, yet almost no trace shows a whole request, and after one deploy none do. Make traces whole again without blowing the tracing budget.Goal: stored traces that are complete ≥ 99.0% · share of spans kept ≤ 15% of spans
System 2 of 6 · 1 challenge
Alerting
A service's error rate feeds an alert rule that pages an on-call engineer. Two hours compressed into two minutes: one simulated second is one minute.
System 3 of 6 · 1 challenge
Checkout errors
Checkout calls a payment provider. When the provider fails, what the code does with the exception decides whether anyone can see it.
System 4 of 6 · 1 challenge
Sign-in chain
Web calls orders, orders calls the internal auth service over mutual TLS. When something breaks deep in the chain, everything above it alerts.
System 5 of 6 · 1 challenge
Invoices API
An API listing invoices. Most customers have dozens; one new customer has 40,000.
System 6 of 6 · 1 challenge
Request logs
A request crosses a gateway and three services, each writing its own logs. Following one customer's failed request means finding its lines in all four.
Kubernetes & infrastructure
Pods, limits, probes and scaling: what the cluster does with your settings.
System 1 of 8 · 1 challenge
CPU limits
A service running as Kubernetes pods with a CPU limit, which Linux enforces as a CPU-time quota every 100 ms, shared by all the pod's threads.
System 2 of 8 · 1 challenge
Rollout and probes
A Deployment of four pods behind a Service, replaced by a new version. New pods boot, then warm up; probes decide when they get traffic and when they get restarted.
System 3 of 8 · 1 challenge
Memory limits
A Java service in a pod with a memory limit. The kernel counts all of the container's memory, the JVM heap and everything outside it.
System 4 of 8 · 1 challenge
CrashLoopBackOff
Three pods of a quotes service. A release reads a new setting from a ConfigMap when it starts.
System 5 of 8 · 1 challenge
Horizontal Pod Autoscaler
An API scaled by a Horizontal Pod Autoscaler on CPU. Each pod handles 120 requests a second at full CPU.
System 6 of 8 · 1 challenge
Rolling update
Four pods at 90% CPU each at peak. A rolling update replaces them; new pods take 20 s to become ready.
System 7 of 8 · 1 challenge
Shared node
An API pod and a nightly batch job share a 4-core node. The API needs 2.5 cores at peak.
System 8 of 8 · 1 challenge
Node upgrade
Three replicas of an API on a three-node cluster. Two replicas carry the load; the third is spare. Upgrading a node drains it first.
API & traffic engineering
Rate limits, gateways, timeouts and load balancing under real traffic.
System 1 of 7 · 1 challenge
API gateway
Real users and one abusive client call an API through a gateway that can rate-limit requests, keyed globally, by client IP address, or by API key.
System 2 of 7 · 1 challenge
Load balancer
A load balancer spreads requests over four identical instances. One goes bad without going down.
System 3 of 7 · 1 challenge
Call chain
A request crosses three services inside a 1-second budget: the app, the API, the order service with a small thread pool, and the stock service it calls with a timeout and retries.
System 4 of 7 · 1 challenge
Flash sale
A shop whose backend can serve 950 requests a second. 80% of traffic is people viewing the product page; 20% are buying.
System 5 of 7 · 1 challenge
API gateway
Every request passes an API gateway, which asks the auth service whether the token is valid before routing it.
System 6 of 7 · 1 challenge
Event ingestion
Devices send events to an ingest API, which queues them in memory; consumers write them to a database that can take 1,000 writes a second.
System 7 of 7 · 1 challenge
Mobile API
A mobile app's API. 40% of people are still on an older app version that reads the price as "price".
Kafka & event streaming
Send messages through a simulated Kafka cluster, then break it: kill a broker, crash a consumer mid-batch, flood a hot partition. Watch the lag and the logs, fix it, run the same fault again.
- Lab 1Producer publishing messagesPublish OrderPlaced events, lose an acknowledged order when a leader dies under acks=1, then make the producer durable and run the same failure again.Foundation20 min
- Lab 2Partition key and orderingOrder status events sent without a key are consumed out of order. Key them by order id, retest, then see what adding a partition does to a key.Foundation20 min
- Lab 3Consumer group assignmentScale a group from one to three consumers on six partitions, overshoot to eight, then find out what happens when two services share one group.id.Foundation20 min
- Lab 4Adding and removing consumersA rolling deploy pauses a whole consumer group, and a slow consumer is thrown out and its work redone. Fix both with three consumer settings.Intermediate25 min
- Lab 5Duplicate message processingA payment consumer charges a card, crashes before committing its offset, and charges again after the restart. Make the consumer idempotent and retest.Intermediate25 min
- Lab 6Consumer crash before offset commitKill a consumer between processing and commit, read the committed offset to predict the replay, size the replay window, then make the replay harmless.Intermediate25 min
- Lab 7Poison message and dead-letter queueOne record the consumer cannot read stops a partition. Try the silent skip, then add bounded retries, a dead letter topic, an alert and a replay.Intermediate25 min
- Lab 8Retry and backoff strategyA downstream API goes away for 8 seconds. Compare a retry loop, Spring's default, exponential back-off with a cap, and non-blocking retry topics.Intermediate30 min
- Lab 9Broker failure and leader electionKill brokers under a replicated partition and follow the ISR, the leader epoch and the high watermark through one, two and three failures.Advanced30 min
- Lab 10Consumer lag during high loadA flash sale makes orders arrive faster than the consumers can process them. Watch lag grow, try two fixes that do not work, then add real capacity and measure the drain time.Intermediate30 min
- Lab 11Hot partitionOne large merchant sends most of the traffic under a single key. One partition and one consumer are saturated while the rest idle. Find out why scaling does not help and what each fix costs.Advanced35 min
- Lab 12Transactional outboxSave an order and publish its event as two separate writes, lose one of them in a crash, then fix it with an outbox table and a relay, and absorb the duplicate the relay can produce.Advanced30 min
- Lab 13Database unavailable while consuming Kafka messagesThe consumer's database goes down for a while. Try three common reactions, see what each one costs, then pause the listener, keep the offsets, and resume without losing a record.Advanced35 min
- Lab 14Kafka transactionA service reads orders and writes to two topics. Crash it between the two sends, make the sends and the offset one Kafka transaction, fence a zombie, then find where the transaction stops.Advanced35 min
- Lab 15Schema evolutionDeploy a producer change that existing consumers cannot read and watch the topic fill with poison records. Then add a compatibility check, evolve the schema safely, and get the deploy order right.Advanced35 min
How it works
The lab runs a model of each system in your browser, ten steps per simulated second. Each step sends traffic through the current state of its components, then updates queues, connection pools, caches and error rates. Queues building, pools filling, retry storms and backlogs are not scripted: they come out of the model. The numbers are a model of a typical system, not measurements of real servers.
Every challenge is tested on each build: it must fail if you change nothing, pass with its intended fix, and no smaller set of changes may pass, so par is exact. The same seed always gives the same run, so a shared link replays exactly what you saw.