Skip to content

Chaos Lab

Break production systems in your browser: an online shop, a phone app, a chat service, a ticketing site, a video streaming service and a two-region app. Watch the dashboards, logs and traces react, then change their defences until they survive.

Whole products

Six believable products, end to end: break them and make them survive.

System 1 of 6 · 8 challenges

Online shop

An online shop: people browse product pages (cached in Redis, stored in Postgres) and check out through an external payments provider.

600 req/sno timeoutno retriespool 2/30
  1. 01Payments brownout starterThe payments provider is not down, just degraded: one call in five fails at random. Recover the failed calls without making the site slower.Goal: requests succeed ≥ 99.0% · p99 latency ≤ 800 ms
  2. 02Redis dies at peak starterAt the lunchtime peak the Redis cache goes down and every read falls through to Postgres. Keep checkouts working while the cache is gone.Goal: checkouts succeed ≥ 99.0% · checkout p99 latency ≤ 1,500 ms
  3. 03The retry storm intermediateThe payments provider starts failing some calls. Your retries were meant to help, but they push it over capacity and it collapses. Tune them so most checkouts still succeed.Goal: checkouts succeed ≥ 75.0% · product pages load ≥ 99.0%
  4. 04Slow dependency, no threads left starterThe payments provider slows to four seconds a call. Every checkout thread waits for it, and soon product pages stop loading too. Keep the rest of the site up.Goal: product pages load ≥ 99.0% · product page p99 latency ≤ 500 ms
  5. 06The TTL expiry wave starterA scheduled cache refresh is coming. Last time, every key expired at once and the database went down under the recomputes. Prepare before it hits.Goal: requests succeed ≥ 99.5%
  6. 07The double charge intermediateThe payments provider charges the card but one reply in five never arrives. Your retry charges the customer again. Stop the double charges without leaving customers charged for orders that failed.Goal: duplicate charges ≤ 1.0 per 1,000 checkouts · checkouts succeed ≥ 98.5% · charged, but the checkout failed ≤ 15.0 per 1,000 checkouts
  7. 09The viral product intermediateHalf of all product reads are suddenly for one product, whose page is expensive to rebuild and expires every five seconds. Stop every expiry from turning into a pile-up.Goal: product reads succeed ≥ 99.5% · read p99 latency ≤ 500 ms · checkouts succeed ≥ 99.5%
  8. 10Sold more than we had intermediateA flash sale of 1,000 units, and half of all checkouts want one. Stop the shop selling stock it does not have, without taking the rest of the shop down.Goal: units sold beyond stock ≤ 1 units · checkouts succeed ≥ 99.0% · product reads succeed ≥ 99.0%

System 2 of 6 · 2 challenges

Phone app

A phone app's backend: people open the app (edge → API → cache and database), every phone keeps a live connection to a realtime gateway, and a push sender can notify every phone at once.

600 req/sretry every 1 ssession checkpool 3/60
  1. 08Push notification stampede intermediateMarketing sends a push to every phone, and every app prefetches its personal home screen at once. Keep the app opening fast for the people actually using it.Goal: app opens succeed ≥ 99.5% · app open p99 ≤ 500 ms
  2. 11Everyone reconnects at once intermediateA routine deploy restarts the realtime gateways, and 20,000 phones lose their connection in the same second. Get them back without taking the app down for everyone else.Goal: app opens succeed ≥ 99.5% · phones connected (average) ≥ 95.0%

System 3 of 6 · 2 challenges

Chat

A messaging service: people read channels (through a data service to a message store partitioned by channel) and send messages, which fan-out workers copy into every member's inbox.

1k reads/severy message×1 workersquery per read
  1. 12The busy channel intermediateSomething happens in a channel with a million members, and everyone opens it at once. Every read lands on one partition of the message store. Keep reads answering.Goal: channel reads answer ≥ 99.5% · read p99 ≤ 300 ms
  2. 13The announcement intermediateOne message to a channel of 400,000 members becomes thousands of fan-out jobs, and everyone else's messages wait behind it. Keep ordinary messages arriving within a second.Goal: messages delivered within 1 s ≥ 99.0% · delivery p99 ≤ 1.0 s

System 4 of 6 · 3 challenges

Ticketing

A ticketing site: visitors (and, during a drop, bots) come through an edge that can run a waiting room, to a fleet of servers that autoscaling can grow, which hold seats while buyers pay.

300/sall in
  1. 14The TV ad minute intermediateA TV advert tells the whole country to open your site now: ten times the usual visitors, for ten seconds. Make sure nobody sees an error, even if some of them wait.Goal: visitors who see an error ≤ 1.0% · visitors served per second ≥ 350 served/s
  2. 15The ticket drop intermediate2,000 seats go on sale. Bots grab holds they never pay for, and fans are told it is sold out while seats sit held. Get the seats to the fans.Goal: seats sold to fans ≥ 95.0% of seats · most seats held by bots at once ≤ 200 seats
  3. 16Launch day intermediateThe launch goes better than planned: visitors double, then triple, then reach five times normal, and stay there. Serve them quickly, not just eventually.Goal: visitors who see an error ≤ 1.0% · served within 5 s ≥ 98.0%

System 5 of 6 · 3 challenges

Video streaming

An on-demand video service: viewers' players fetch two seconds of video at a time from a CDN; misses go to an origin shield, then to origin servers that read video storage.

10,000/smisses100/s
  1. 52The evening peak starterAt 9 pm half your viewers' home internet slows to 2 Mbps and their players stall every few seconds. A bigger buffer only delays it. Keep everyone watching.Goal: time spent buffering ≤ 1.0% · average picture quality ≥ 3.0 Mbps
  2. 53The premiere intermediateA new episode drops and 60,000 viewers press play at once. None of its segments is cached anywhere, and the origin melts. Twice the servers does not help.Goal: time spent buffering ≤ 1.0% · segments delivered ≥ 99.5%
  3. 54The CDN goes dark intermediateYour CDN provider fails worldwide. A backup CDN takes the viewers, but its cache is cold and the origin melts under the misses. Keep playback running.Goal: time spent buffering ≤ 1.0% · segments delivered ≥ 99.5%

System 6 of 6 · 2 challenges

Multi-region app

A web app in two regions: traffic steering (DNS) sends users to region A; region B stands by with its own servers and a read-only replica of the database.

100%replication
  1. 55The region goes dark intermediateYour main region loses power. A full standby region is ready and waiting, yet users see errors for a minute. Make the failover fast enough to matter.Goal: requests succeed ≥ 80.0% · seconds mostly down ≤ 10 s
  2. 56The standby that wasn't intermediateFailover works perfectly: traffic is in region B within seconds. Then region B falls over, and every order is refused. Make the standby a real one.Goal: requests succeed ≥ 80.0% · writes succeed ≥ 75.0%

Security

Find the weakness from the attacker's side, then close it.

System 1 of 7 · 1 challenge

Multi-tenant API

Companies keep invoices in one shared database and read them by id. Every request is authenticated; the question is whether the API checks whose invoice it is.

their own invoicesWHERE id = ?
  1. 20Broken access control starterA signed-in customer of another company starts counting through invoice ids, and the API hands them over. Stop the leak without breaking anyone's real work.Goal: other companies' invoices refused ≥ 100.0% · real users' requests answered ≥ 99.0%

System 2 of 7 · 1 challenge

Token validation

An API accepts JSON Web Tokens and must decide, for every request, whether the token is genuine, still valid and meant for this API.

leeway 0 s
  1. 23JWT misconfiguration intermediateThe API checks that every request carries a JWT, and decodes it. An attacker shows that decoding is not validating. Close every hole without locking out real users.Goal: attacker requests refused ≥ 100.0% · real users' requests accepted ≥ 99.5%

System 3 of 7 · 1 challenge

Link previews

Users paste a URL and the server fetches it to show a preview. The server sits inside the cloud network, next to the metadata service and an admin API. A fictional, self-contained simulation.

  1. 27SSRF: the preview that fetched the keys intermediateA link-preview feature fetches any URL a user pastes, from inside the cloud network. An attacker points it at the metadata service and an admin API. Close every route without breaking real previews.Goal: attacker requests refused ≥ 100.0% · real previews working ≥ 99.0%

System 4 of 7 · 1 challenge

Staff roles

An internal tool: staff manage customers, admins manage staff accounts. Everyone is signed in; the API decides what each person may do.

/admin/*
  1. 32Signed in is not allowed starterEvery request to the internal tool is authenticated. A support agent finds the admin API anyway and starts deleting staff accounts. Authentication said who they were; nothing asked what they may do.Goal: non-admin admin calls refused ≥ 100.0% · real admin calls working ≥ 99.0%

System 5 of 7 · 1 challenge

Login service

A login service behind a gateway with a rate limiter. Password checks are slow on purpose, so every login attempt costs real CPU.

50/s
  1. 33Rate limit bypass intermediateThe login page has a per-IP rate limit. A credential-stuffing bot sends 1,000 logins a second from one machine and is never limited: it writes a different X-Forwarded-For on every request.Goal: bot attempts let through ≤ 2.0% · real logins succeeding ≥ 99.0%

System 6 of 7 · 1 challenge

Secrets handling

A payments service holding a database password, a provider API key and customers' session tokens. Logs, a debug endpoint and error pages can all give them away.

redactedsecrets at startup
  1. 34Secrets exposure starterThe secrets live in a secret manager, so they must be safe. Then the logs, a debug endpoint and an error page each hand them out. Stop every leak without losing the logs you debug with.Goal: secret copies exposed ≤ 1 copies · debugging ability from logs ≥ 90%

System 7 of 7 · 3 challenges

OTP sign-in

Sign in with a 6-digit code sent by SMS and valid for 5 minutes: an OTP service, a code store and SMS providers.

100102/s
  1. 57A million codes is not enough intermediateSign-in codes have six digits: a million possibilities. A botnet guesses 40,000 a second from 5,000 machines, and the per-IP limit never notices. Stop the guessing without locking real users out.Goal: attacker guesses checked a second ≤ 100/s · real sign-ins succeeding ≥ 99.0%
  2. 58No codes during the sale intermediateMid-sale, the SMS provider stops answering and nobody can sign in. Adding a backup provider is not enough on its own. Keep sign-ins working.Goal: sign-ins succeeding ≥ 95.0%
  3. 59One code, two sign-ins intermediateUsers tap Verify twice, and the app retries on flaky networks. Some codes arrive in two requests at the same moment, and both sign in. Make every code work exactly once.Goal: codes accepted twice ≤ 0.01% · sign-ins succeeding ≥ 99.0%

Performance engineering

Find the real bottleneck, then make the system fast under load.

System 1 of 6 · 2 challenges

Worker pool

A thread-per-request API in front of a database: requests queue for a worker thread, and each thread waits for one of a fixed number of database connections.

600/swait 0 msconn wait 0 ms
  1. 17Thread pool saturation intermediateTraffic jumps to two and a half times normal. The API's threads are all busy, requests pile up, and the obvious fix is more threads. Find out why it isn't.Goal: answered before clients give up ≥ 800 answered/s · p99 of those answers ≤ 1.0 s · time up, not restarting after a crash ≥ 100.0%
  2. 22The N+1 query intermediateA harmless-looking release lists each order's line items. Traffic is normal, no query is slow, and the API falls over anyway.Goal: answered before clients give up ≥ 580 answered/s · p99 of those answers ≤ 300 ms

System 2 of 6 · 1 challenge

JVM heap

A Java service with a garbage-collected heap. Short-lived objects are cheap; objects that stay reachable are not.

500/scache: unbounded
  1. 28Memory leak intermediateA release adds a cache. Within a minute the heap is climbing, GC is working harder, and then the service dies with OutOfMemoryError, again and again. Make it stop for good.Goal: requests answered ≥ 99.5% · p99 latency ≤ 150 ms

System 3 of 6 · 1 challenge

Search fan-out

A search request asks 20 index shards in parallel and waits for all of them. Each shard is fast, almost always.

300/s×20
  1. 35The P99 problem intermediateEvery shard answers in about 8 ms, yet one search in six takes over 400 ms. Averages look perfect; the slowest users don't.Goal: p99 search latency ≤ 100 ms · searches answered ≥ 99.0%

System 4 of 6 · 1 challenge

Pricing API

A CPU-bound API on 4-core instances. Every millisecond of CPU per request decides how many requests an instance can take.

30% CPU
  1. 36CPU bottleneck starterAt the lunchtime peak the pricing API pins at 100% CPU and requests time out. Doubling the threads makes it worse. Find where the CPU goes.Goal: requests answered ≥ 99.5% · p99 latency ≤ 150 ms · instances ≤ 2 instances

System 5 of 6 · 1 challenge

Orders database

An order-history page backed by one PostgreSQL primary with 8 cores. Every query costs database CPU.

Index Scan
  1. 37Database bottleneck starterA small feature adds one query. Database CPU goes to 100% and every page slows down. Bigger hardware is the obvious answer; it isn't the cheap one.Goal: pages answered ≥ 99.5% · p99 latency ≤ 200 ms · database cost ≤ 1× cost

System 6 of 6 · 1 challenge

Checkout fleet

Identical instances that can each do 500 requests a second flat out. The question is how close to flat out you can run them.

300/s each
  1. 38Load vs capacity starterEach instance can do 500 requests a second, so 4 instances can do 2,000. The sale brings 1,800. Why is checkout so slow?Goal: checkouts answered ≥ 99.5% · p99 latency ≤ 150 ms · instances ≤ 5

Observability & debugging

Diagnose a broken system from its metrics, logs and traces.

System 1 of 6 · 2 challenges

Microservices

A request enters at the gateway and goes to orders, which calls inventory and pricing in parallel. Each service reports only its own view: its response time, logs and spans.

orders v117inventorypricing
  1. 21Find the bottleneck intermediateTwo services deploy at once and half of all requests start timing out. Every service upstream looks slow. Use the evidence to find the one that is, and fix only that.Goal: requests answered ≥ 99.0% · p99 at the gateway ≤ 200 ms
  2. 29The missing trace intermediateDistributed tracing is on, yet almost no trace shows a whole request, and after one deploy none do. Make traces whole again without blowing the tracing budget.Goal: stored traces that are complete ≥ 99.0% · share of spans kept ≤ 15% of spans

System 2 of 6 · 1 challenge

Alerting

A service's error rate feeds an alert rule that pages an on-call engineer. Two hours compressed into two minutes: one simulated second is one minute.

scrapeevaluate
  1. 26The noisy alert intermediateThe checkout alert pages on every blip, so by the time the real outage comes, nobody hurries. Write a rule that pages for problems, not noise.Goal: false pages ≤ 1 · minutes until someone responds to the outage ≤ 4.0 min · minutes until someone responds to the slow problem ≤ 15.0 min

System 3 of 6 · 1 challenge

Checkout errors

Checkout calls a payment provider. When the provider fails, what the code does with the exception decides whether anyone can see it.

  1. 39The error that isn't in the logs starterCustomers say checkout shows a blank receipt. The dashboards are green and the logs are empty. The failures are real; the code is hiding them.Goal: failures visible in logs ≥ 99.0% · log volume (gb/day) ≤ 20 GB/day

System 4 of 6 · 1 challenge

Sign-in chain

Web calls orders, orders calls the internal auth service over mutual TLS. When something breaks deep in the chain, everything above it alerts.

mTLS
  1. 40Root cause starterWeb is throwing 502s, orders is erroring, and this morning's deploy is the obvious suspect. Every alert points somewhere different. Find the one thing that's actually broken.Goal: requests succeeding ≥ 99.0%

System 5 of 6 · 1 challenge

Invoices API

An API listing invoices. Most customers have dozens; one new customer has 40,000.

≈50 rows
  1. 41The P99 mystery intermediatep50 is 30 ms, unchanged. p99 has jumped to 900 ms. CPU is fine and adding servers doesn't move it. Who are the slow requests?Goal: p99 latency ≤ 200 ms · requests answered ≥ 99.0%

System 6 of 6 · 1 challenge

Request logs

A request crosses a gateway and three services, each writing its own logs. Following one customer's failed request means finding its lines in all four.

req_7f3a91c2no id
  1. 42Correlation ID starterA customer's order failed at 10:42. Four services logged something in that second, 1,900 lines in all. Make one request's story findable in one search.Goal: a request's log lines findable by id ≥ 99.0%

Kubernetes & infrastructure

Pods, limits, probes and scaling: what the cluster does with your settings.

Warm up: diagnose the cluster10 rounds of real-world failures. Read the evidence, find the cause, pick the first fix.

System 1 of 8 · 1 challenge

CPU limits

A service running as Kubernetes pods with a CPU limit, which Linux enforces as a CPU-time quota every 100 ms, shared by all the pod's threads.

40/slimit 2 cores16 threadsTHROTTLED 60%
  1. 19CPU throttling intermediateThe pod uses 84% of its CPU limit on average, and its p99 is half a second. Then traffic rises. Find out where the time goes.Goal: requests answered ≥ 99.0% · p99 latency ≤ 200 ms

System 2 of 8 · 1 challenge

Rollout and probes

A Deployment of four pods behind a Service, replaced by a new version. New pods boot, then warm up; probes decide when they get traffic and when they get restarted.

400/s
  1. 25Readiness vs liveness intermediateA routine deploy turns into a restart storm. New pods get traffic before they can handle it, then get killed for being slow. Make the rollout invisible to users.Goal: requests answered ≥ 99.0% · p99 latency ≤ 300 ms

System 3 of 8 · 1 challenge

Memory limits

A Java service in a pod with a memory limit. The kernel counts all of the container's memory, the JVM heap and everything outside it.

300/sheap 1glimit 1 Gi
  1. 30OOMKilled starterA Java pod with -Xmx1g and a 1 GiB memory limit runs fine all day, then dies every evening with exit code 137. Make it survive the peak without asking for more memory.Goal: requests answered ≥ 99.5% · restarts ≤ 1 · memory limit (gib) ≤ 1 GiB

System 4 of 8 · 1 challenge

CrashLoopBackOff

Three pods of a quotes service. A release reads a new setting from a ConfigMap when it starts.

v41
  1. 43CrashLoopBackOff starterv42 ships and every pod restarts, waits, restarts, waits longer. More memory, a longer delay: nothing changes. Read why it exits.Goal: requests served ≥ 95.0%

System 5 of 8 · 1 challenge

Horizontal Pod Autoscaler

An API scaled by a Horizontal Pod Autoscaler on CPU. Each pod handles 120 requests a second at full CPU.

2 replicas
  1. 44Autoscaling too late intermediateThe autoscaler works: it adds pods. They arrive a minute after the traffic did. Make scaling keep up with a fast ramp.Goal: requests answered ≥ 99.0% · average pods ≤ 8.0

System 6 of 8 · 1 challenge

Rolling update

Four pods at 90% CPU each at peak. A rolling update replaces them; new pods take 20 s to become ready.

unavail 1surge 0
  1. 45Rolling deployment failure starterA routine rolling update at peak traffic takes one pod out of four, and the other three can't carry the load. Roll out without losing capacity.Goal: requests answered ≥ 99.0% · rollout complete ≥ 100%

System 7 of 8 · 1 challenge

Shared node

An API pod and a nightly batch job share a 4-core node. The API needs 2.5 cores at peak.

same node
  1. 46Resource starvation intermediateAt 2 a.m. the API slows to a crawl. Nothing changed in the API. Something else on the same node is eating its CPU.Goal: requests answered ≥ 99.0% · p99 latency ≤ 150 ms

System 8 of 8 · 1 challenge

Node upgrade

Three replicas of an API on a three-node cluster. Two replicas carry the load; the third is spare. Upgrading a node drains it first.

no PDB
  1. 47Pod disruption intermediateA routine node upgrade drains the node holding two of the API's three replicas. Both are evicted at once. Make maintenance safe.Goal: requests answered ≥ 99.0% · nodes upgraded ≥ 100%

API & traffic engineering

Rate limits, gateways, timeouts and load balancing under real traffic.

System 1 of 7 · 1 challenge

API gateway

Real users and one abusive client call an API through a gateway that can rate-limit requests, keyed globally, by client IP address, or by API key.

400/s400/s
  1. 18Rate limiter under attack starterA scraper floods the API with one key spread across 300 addresses. Real users start timing out. Pick what the rate limit should count.Goal: real users' requests answered ≥ 99.0% · real users' p99 ≤ 300 ms

System 2 of 7 · 1 challenge

Load balancer

A load balancer spreads requests over four identical instances. One goes bad without going down.

800/s
  1. 24The slow instance starterOne of four instances gets a failing disk. It doesn't crash, it just gets slow, and the load balancer keeps sending it work. Make the balancer notice.Goal: requests answered ≥ 99.5% · p99 latency ≤ 200 ms

System 3 of 7 · 1 challenge

Call chain

A request crosses three services inside a 1-second budget: the app, the API, the order service with a small thread pool, and the stock service it calls with a timeout and retries.

1 swait1500ms×3
  1. 31Timeout budget intermediateThe app waits 1 second for an order. Deep in the call chain, a service waits 1.5 seconds per try, and retries twice. When its dependency browns out, nothing gets through.Goal: orders answered within 1 s ≥ 85.0%

System 4 of 7 · 1 challenge

Flash sale

A shop whose backend can serve 950 requests a second. 80% of traffic is people viewing the product page; 20% are buying.

500/s
  1. 48Traffic spike intermediateThe sale opens and 5,000 people a second arrive at a shop built for 950. Doubling the servers isn't enough, and isn't affordable. Keep the shop up.Goal: shoppers served ≥ 99.0% · longest wait (s) ≤ 60 s · backend instances ≤ 1×

System 5 of 7 · 1 challenge

API gateway

Every request passes an API gateway, which asks the auth service whether the token is valid before routing it.

every request
  1. 49Gateway bottleneck starterLaunch day doubles traffic. The services are 20% busy, yet the gateway returns 503s. Every request waits on the same small service.Goal: requests answered ≥ 99.5% · p99 latency ≤ 150 ms · auth instances ≤ 1 instances

System 6 of 7 · 1 challenge

Event ingestion

Devices send events to an ingest API, which queues them in memory; consumers write them to a database that can take 1,000 writes a second.

800/s
  1. 50Backpressure intermediateDevices reconnect after an outage and send events faster than they can be written. The ingest API accepts them all, until it runs out of memory and loses them all.Goal: events lost ≤ 1 events · peak database load (%) ≤ 90%

System 7 of 7 · 1 challenge

Mobile API

A mobile app's API. 40% of people are still on an older app version that reads the price as "price".

GET /ordersGET /orders
  1. 51API versioning starterv2 renames one field. Every new app is fine; 40% of users are on an older app, and it crashes on every order screen.Goal: requests working ≥ 99.0% · old app working ≥ 99.0%

Kafka & event streaming

Send messages through a simulated Kafka cluster, then break it: kill a broker, crash a consumer mid-batch, flood a hot partition. Watch the lag and the logs, fix it, run the same fault again.

  1. Lab 1Producer publishing messagesPublish OrderPlaced events, lose an acknowledged order when a leader dies under acks=1, then make the producer durable and run the same failure again.Foundation20 min
  2. Lab 2Partition key and orderingOrder status events sent without a key are consumed out of order. Key them by order id, retest, then see what adding a partition does to a key.Foundation20 min
  3. Lab 3Consumer group assignmentScale a group from one to three consumers on six partitions, overshoot to eight, then find out what happens when two services share one group.id.Foundation20 min
  4. Lab 4Adding and removing consumersA rolling deploy pauses a whole consumer group, and a slow consumer is thrown out and its work redone. Fix both with three consumer settings.Intermediate25 min
  5. Lab 5Duplicate message processingA payment consumer charges a card, crashes before committing its offset, and charges again after the restart. Make the consumer idempotent and retest.Intermediate25 min
  6. Lab 6Consumer crash before offset commitKill a consumer between processing and commit, read the committed offset to predict the replay, size the replay window, then make the replay harmless.Intermediate25 min
  7. Lab 7Poison message and dead-letter queueOne record the consumer cannot read stops a partition. Try the silent skip, then add bounded retries, a dead letter topic, an alert and a replay.Intermediate25 min
  8. Lab 8Retry and backoff strategyA downstream API goes away for 8 seconds. Compare a retry loop, Spring's default, exponential back-off with a cap, and non-blocking retry topics.Intermediate30 min
  9. Lab 9Broker failure and leader electionKill brokers under a replicated partition and follow the ISR, the leader epoch and the high watermark through one, two and three failures.Advanced30 min
  10. Lab 10Consumer lag during high loadA flash sale makes orders arrive faster than the consumers can process them. Watch lag grow, try two fixes that do not work, then add real capacity and measure the drain time.Intermediate30 min
  11. Lab 11Hot partitionOne large merchant sends most of the traffic under a single key. One partition and one consumer are saturated while the rest idle. Find out why scaling does not help and what each fix costs.Advanced35 min
  12. Lab 12Transactional outboxSave an order and publish its event as two separate writes, lose one of them in a crash, then fix it with an outbox table and a relay, and absorb the duplicate the relay can produce.Advanced30 min
  13. Lab 13Database unavailable while consuming Kafka messagesThe consumer's database goes down for a while. Try three common reactions, see what each one costs, then pause the listener, keep the offsets, and resume without losing a record.Advanced35 min
  14. Lab 14Kafka transactionA service reads orders and writes to two topics. Crash it between the two sends, make the sends and the offset one Kafka transaction, fence a zombie, then find where the transaction stops.Advanced35 min
  15. Lab 15Schema evolutionDeploy a producer change that existing consumers cannot read and watch the topic fill with poison records. Then add a compatibility check, evolve the schema safely, and get the deploy order right.Advanced35 min

How it works

The lab runs a model of each system in your browser, ten steps per simulated second. Each step sends traffic through the current state of its components, then updates queues, connection pools, caches and error rates. Queues building, pools filling, retry storms and backlogs are not scripted: they come out of the model. The numbers are a model of a typical system, not measurements of real servers.

Every challenge is tested on each build: it must fail if you change nothing, pass with its intended fix, and no smaller set of changes may pass, so par is exact. The same seed always gives the same run, so a shared link replays exactly what you saw.