Skip to content
Chaos Lab

Debug the code · lab 4 · intermediate · about 7 min

The blip that would not end

Product pages check stock with the inventory service, 1,200 times a second. At 16:20 inventory restarts a node as planned and answers 503 for about 10 seconds. The node comes back, but inventory keeps failing for the next 20 minutes, and so do product pages.

  • Your fix must
  • Inventory sees at most 1.5× normal load during a blip
  • At least 99.5% of stock checks succeed through a 10-second blip
  • Stock check p99 under 600 ms
Stock check p998.00 starget < 600 ms

Pager timeline

Logs and signalsReconstructed
16:20inventory: planned node restart
16:20inventory returns 503 for 10 s
16:21requests to inventory 4.8× normal
16:24node healthy again; inventory still failing
Requests to inventory, per second
TimeFrom usersRetriesTotal
16:191,20001,200
16:201,2004,6005,800
16:231,2004,5005,700

Step 1 · find it

Tap the line that causes it

Mark up to 3 lines, then check. It takes 2 lines.

InventoryClient.javaJava 21 · Spring Boot 3.2+

No line marked yet

The code is real Spring Boot; the evidence and every patch's numbers are simulated and illustrative. Nothing runs in your browser.