BuildBreak labs
Make it work, break it, fix it. Each lab is a guided run through one failure: build the working flow, inject the fault, read what the logs and metrics show, apply the fix and run the same fault again.
Simulation, not a real cluster.There is no Kafka broker behind this page. Labs are deterministic simulations: the logs and metrics you see are simulated, and the same actions always give the same result.
0 / 15 labs completed
- Lab 1Producer publishing messagesPublish OrderPlaced events, lose an acknowledged order when a leader dies under acks=1, then make the producer durable and run the same failure again.Foundation20 min
- Lab 2Partition key and orderingOrder status events sent without a key are consumed out of order. Key them by order id, retest, then see what adding a partition does to a key.Foundation20 min
- Lab 3Consumer group assignmentScale a group from one to three consumers on six partitions, overshoot to eight, then find out what happens when two services share one group.id.Foundation20 min
- Lab 4Adding and removing consumersA rolling deploy pauses a whole consumer group, and a slow consumer is thrown out and its work redone. Fix both with three consumer settings.Intermediate25 min
- Lab 5Duplicate message processingA payment consumer charges a card, crashes before committing its offset, and charges again after the restart. Make the consumer idempotent and retest.Intermediate25 min
- Lab 6Consumer crash before offset commitKill a consumer between processing and commit, read the committed offset to predict the replay, size the replay window, then make the replay harmless.Intermediate25 min
- Lab 7Poison message and dead-letter queueOne record the consumer cannot read stops a partition. Try the silent skip, then add bounded retries, a dead letter topic, an alert and a replay.Intermediate25 min
- Lab 8Retry and backoff strategyA downstream API goes away for 8 seconds. Compare a retry loop, Spring's default, exponential back-off with a cap, and non-blocking retry topics.Intermediate30 min
- Lab 9Broker failure and leader electionKill brokers under a replicated partition and follow the ISR, the leader epoch and the high watermark through one, two and three failures.Advanced30 min
- Lab 10Consumer lag during high loadA flash sale makes orders arrive faster than the consumers can process them. Watch lag grow, try two fixes that do not work, then add real capacity and measure the drain time.Intermediate30 min
- Lab 11Hot partitionOne large merchant sends most of the traffic under a single key. One partition and one consumer are saturated while the rest idle. Find out why scaling does not help and what each fix costs.Advanced35 min
- Lab 12Transactional outboxSave an order and publish its event as two separate writes, lose one of them in a crash, then fix it with an outbox table and a relay, and absorb the duplicate the relay can produce.Advanced30 min
- Lab 13Database unavailable while consuming Kafka messagesThe consumer's database goes down for a while. Try three common reactions, see what each one costs, then pause the listener, keep the offsets, and resume without losing a record.Advanced35 min
- Lab 14Kafka transactionA service reads orders and writes to two topics. Crash it between the two sends, make the sends and the offset one Kafka transaction, fence a zombie, then find where the transaction stops.Advanced35 min
- Lab 15Schema evolutionDeploy a producer change that existing consumers cannot read and watch the topic fill with poison records. Then add a compatibility check, evolve the schema safely, and get the deploy order right.Advanced35 min