BuildBreak lab 8 of 15
Retry and backoff strategy
A downstream API goes away for 8 seconds. Compare a retry loop, Spring's default, exponential back-off with a cap, and non-blocking retry topics.
Read first: Retries, back-off and retry topics · Retry topics and dead letter topics as architecture · Error handling, retries and dead letters in Spring · Consumer rebalancing
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- order-events: 2 partitions, RF 3
- Producers
- order-service
- Consumer groups
- inventory-service (1)
- Guided run
- 7 stages: build → break → observe → fix → retest
1
Business scenario
The inventory service consumes order-events and calls the warehouse's stock API for every event: reserve stock when an order is placed, adjust it when the order changes, release it when the order is cancelled. The stock API is redeployed a few times a week and answers 503 for about 8 seconds each time. That is a normal, short, transient failure, and the events should simply be processed a little later. Instead, each redeploy produces a different incident depending on who last touched the retry code: a flood of calls against a service that is starting up, a consumer thrown out of its group, or reservations that silently never happen.
2
Starting architecture
Drawing diagram…
The order service producer writes order events to the order-events topic, which has 2 partitions. The inventory service consumer polls it and makes one call to the stock API per processing attempt; the stock API is down for 8 seconds at each redeploy. In the non-blocking variant the consumer publishes a failed record to one of the topics order-events-retry-1 to 7, reads it again when its delay has passed, and publishes it to order-events-dlt when the attempts are exhausted.
flowchart LR OS["Order service producer"] -->|"order events"| T["Topic order-events, 2 partitions"] T -->|"poll"| INV["Inventory service consumer"] INV -->|"one call per attempt"| API["Stock API, down 8 s per redeploy"] INV -.->|"non-blocking variant"| RT["Topics order-events-retry-1 to 7"] RT -.->|"when the delay has passed"| INV INV -.->|"attempts exhausted"| DLT["Topic order-events-dlt"]
3
Expected behaviour
- An 8 second outage of the stock API delays the affected events by seconds. None is lost and none is applied twice.
- While the API is down, the consumer sends it a handful of calls, not hundreds.
- The consumer stays in its group throughout: no rebalance is caused by retrying.
- Events of one order are applied in the order they were written, unless you have decided to give that up.
4 · Build → break → observe → fix → retest
Guided run
Reserve stock
Do this
- Turn on Order service publishing (3 events per second) and press play.
- Watch the stock API panel: one call per event while everything is healthy.
Look for
- "Stock updates applied" rises and lag stays near 0.
- The listener was written with a retry loop: on any exception it calls the API again at once, for as long as it takes. So far that code has never run.
- The simulated consumer fetches one record per
poll()(max.poll.recordsis 1 here; Kafka's default is 500), so that it polls between two records as a Spring listener container does between two deliveries.
Not done yet. The simulator checks this for you:consumed >= 15 && callsWhileDown == 0
Simulator
Simulation · not a real clusterControls for this stage
3 order events per second, keyed by order id.
stock API
simulatedThe service the consumer calls for every record. A retry is a call: a consumer that retries without waiting sends its whole capacity at a service that is trying to recover.
- Status
- up
- Calls received while down
- 0
- Most calls in one second while down
- 0
- Calls in total
- 0
Simulated metrics
computed by the simulation- Stock updates applied
- 0
- Lag on order-events
- 0records
- Retries
- 0
- Removed from the group (max.poll.interval.ms)
- 0
- Rebalances
- 0
- Events applied twice
- 0
- Skipped and lost
- 0
- Dead-lettered
- 0
- Processed out of order
- 0
Consumer group inventory-service
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 1
- inventory-1runningprocessed 0order-events-0order-events-1
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| order-events-0 | inventory-1 | none | 0 |
| order-events-1 | inventory-1 | none | 0 |
Partition logs
simulated records- order-events-0leader b1ISR 1,2,3HW 0end 0
empty
inventory-service: committed none, lag 0
- order-events-1leader b2ISR 2,3,1HW 0end 0
empty
inventory-service: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, order-events (2 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.