BuildBreak lab 7 of 15
Poison message and dead-letter queue
One record the consumer cannot read stops a partition. Try the silent skip, then add bounded retries, a dead letter topic, an alert and a replay.
Read first: Poison messages and error handling · Dead letter topics: park, inspect, replay · Error handling, retries and dead letters in Spring · Serialisation in Spring Kafka: JSON, type headers, Avro
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- shipment-requests: 3 partitions, RF 3
- Producers
- order-service
- Consumer groups
- shipping-service (3)
- Guided run
- 7 stages: build → break → observe → fix → retest
1
Business scenario
The shipping service consumes shipment-requests (3 partitions, 3 instances) and books a parcel label with the carrier for each one. This afternoon a canary of the order service went out with a change to the event: weightGrams, an integer, became weight: "1.2kg". The canary was rolled back after one record. Nobody thought about that record again until the evening, when a third of today's parcels had no label. The consumers were all running, the group was stable, there were no restarts and no alerts.
2
Starting architecture
Drawing diagram…
The order service producer writes ShipmentRequested events to the shipment-requests topic, which has 3 partitions. The shipping service, 3 consumers in one group, polls the topic and books a label with the carrier API for each record. After the fix, records the shipping service gives up on are published to the topic shipment-requests-dlt, and any record arriving there raises an alert to the owning team.
flowchart LR OS["Order service producer"] -->|"ShipmentRequested"| T["Topic shipment-requests, 3 partitions"] T -->|"poll"| SS["Shipping service, 3 consumers in one group"] SS -->|"book a label"| CA["Carrier API"] SS -.->|"after the fix, records it gives up on"| DLT["Topic shipment-requests-dlt"] DLT -.->|"any record here"| AL["Alert to the owning team"]
3
Expected behaviour
- Every shipment request gets a label, in order within its partition.
- A record that can never be processed does not stop the records behind it for more than a few seconds.
- No record disappears: one that could not be processed is kept, with the reason, somebody is told, and it can be processed later.
4 · Build → break → observe → fix → retest
Guided run
Book labels
Do this
- Turn on Order service publishing (6 shipment requests per second) and press play.
- In the Consumer group panel, check that each of the three consumers owns one partition and that the lag of every partition stays near 0.
Look for
- "Labels booked" rises steadily and "Blocked partitions" is 0.
- The group has no error handling configured at all: the first failure will show what that means.
Not done yet. The simulator checks this for you:consumed >= 24 && blockedPartitions == 0
Simulator
Simulation · not a real clusterControls for this stage
6 ShipmentRequested events per second, keyed by order id.
Consumer group shipping-service
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 10
- shipper-1runningprocessed 0shipment-requests-0
- shipper-2runningprocessed 0shipment-requests-1
- shipper-3runningprocessed 0shipment-requests-2
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| shipment-requests-0 | shipper-1 | none | 0 |
| shipment-requests-1 | shipper-2 | none | 0 |
| shipment-requests-2 | shipper-3 | none | 0 |
Simulated metrics
computed by the simulation- Labels booked
- 0
- Blocked partitions
- 0
- Largest lag on one partition
- 0records
- Failed processing attempts
- 0
- Skipped and lost
- 0
- Retries
- 0
- Dead-lettered
- 0
- Alerts raised
- 0
- Dead letters replayed
- 0
Dead letter topic shipment-requests-dlt
simulatedEmpty. A record arrives here when the error handler gives up on it; the original partition then moves on.
- Waiting for a fix
- 0
- Replayed
- 0
- Alerts raised
- 0
Partition logs
simulated records- shipment-requests-0leader b1ISR 1,2,3HW 0end 0
empty
shipping-service: committed none, lag 0
- shipment-requests-1leader b2ISR 2,3,1HW 0end 0
empty
shipping-service: committed none, lag 0
- shipment-requests-2leader b3ISR 3,1,2HW 0end 0
empty
shipping-service: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, shipment-requests (3 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.