Skip to content

BuildBreak lab 8 of 15

Retry and backoff strategy

A downstream API goes away for 8 seconds. Compare a retry loop, Spring's default, exponential back-off with a cap, and non-blocking retry topics.

Intermediate30 min

Read first: Retries, back-off and retry topics · Retry topics and dead letter topics as architecture · Error handling, retries and dead letters in Spring · Consumer rebalancing

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
order-events: 2 partitions, RF 3
Producers
order-service
Consumer groups
inventory-service (1)
Guided run
7 stages: build → break → observe → fix → retest

1

Business scenario

The inventory service consumes order-events and calls the warehouse's stock API for every event: reserve stock when an order is placed, adjust it when the order changes, release it when the order is cancelled. The stock API is redeployed a few times a week and answers 503 for about 8 seconds each time. That is a normal, short, transient failure, and the events should simply be processed a little later. Instead, each redeploy produces a different incident depending on who last touched the retry code: a flood of calls against a service that is starting up, a consumer thrown out of its group, or reservations that silently never happen.

2

Starting architecture

Drawing diagram…

The order service producer writes order events to the order-events topic, which has 2 partitions. The inventory service consumer polls it and makes one call to the stock API per processing attempt; the stock API is down for 8 seconds at each redeploy. In the non-blocking variant the consumer publishes a failed record to one of the topics order-events-retry-1 to 7, reads it again when its delay has passed, and publishes it to order-events-dlt when the attempts are exhausted.

flowchart LR
  OS["Order service producer"] -->|"order events"| T["Topic order-events, 2 partitions"]
  T -->|"poll"| INV["Inventory service consumer"]
  INV -->|"one call per attempt"| API["Stock API, down 8 s per redeploy"]
  INV -.->|"non-blocking variant"| RT["Topics order-events-retry-1 to 7"]
  RT -.->|"when the delay has passed"| INV
  INV -.->|"attempts exhausted"| DLT["Topic order-events-dlt"]
Every processing attempt is one call to the stock API. The dotted paths exist only with retry topics: a failed record is parked in a retry topic and the main partitions carry on.

3

Expected behaviour

  • An 8 second outage of the stock API delays the affected events by seconds. None is lost and none is applied twice.
  • While the API is down, the consumer sends it a handful of calls, not hundreds.
  • The consumer stays in its group throughout: no rebalance is caused by retrying.
  • Events of one order are applied in the order they were written, unless you have decided to give that up.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 7

Reserve stock

Do this

  1. Turn on Order service publishing (3 events per second) and press play.
  2. Watch the stock API panel: one call per event while everything is healthy.

Look for

  • "Stock updates applied" rises and lag stays near 0.
  • The listener was written with a retry loop: on any exception it calls the API again at once, for as long as it takes. So far that code has never run.
  • The simulated consumer fetches one record per poll() (max.poll.records is 1 here; Kafka's default is 500), so that it polls between two records as a Spring listener container does between two deliveries.

Not done yet. The simulator checks this for you:consumed >= 15 && callsWhileDown == 0

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

3 order events per second, keyed by order id.

stock API

simulated

The service the consumer calls for every record. A retry is a call: a consumer that retries without waiting sends its whole capacity at a service that is trying to recover.

Status
up
Calls received while down
0
Most calls in one second while down
0
Calls in total
0

Simulated metrics

computed by the simulation
Stock updates applied
0
Lag on order-events
0records
Retries
0
Removed from the group (max.poll.interval.ms)
0
Rebalances
0
Events applied twice
0
Skipped and lost
0
Dead-lettered
0
Processed out of order
0

Consumer group inventory-service

stablegeneration 1lag 0

RangeAssignor (eager)commit after batchmax.poll.records 1

  • inventory-1runningprocessed 0
    order-events-0order-events-1
Partitions of group inventory-service: owner, committed offset and lag
PartitionOwnerCommittedLag
order-events-0 inventory-1none0
order-events-1 inventory-1none0

Partition logs

simulated records
  • order-events-0leader b1ISR 1,2,3HW 0end 0

    empty

    inventory-service: committed none, lag 0

  • order-events-1leader b2ISR 2,3,1HW 0end 0

    empty

    inventory-service: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, order-events (2 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.