Skip to content

BuildBreak lab 7 of 15

Poison message and dead-letter queue

One record the consumer cannot read stops a partition. Try the silent skip, then add bounded retries, a dead letter topic, an alert and a replay.

Intermediate25 min

Read first: Poison messages and error handling · Dead letter topics: park, inspect, replay · Error handling, retries and dead letters in Spring · Serialisation in Spring Kafka: JSON, type headers, Avro

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
shipment-requests: 3 partitions, RF 3
Producers
order-service
Consumer groups
shipping-service (3)
Guided run
7 stages: build → break → observe → fix → retest

1

Business scenario

The shipping service consumes shipment-requests (3 partitions, 3 instances) and books a parcel label with the carrier for each one. This afternoon a canary of the order service went out with a change to the event: weightGrams, an integer, became weight: "1.2kg". The canary was rolled back after one record. Nobody thought about that record again until the evening, when a third of today's parcels had no label. The consumers were all running, the group was stable, there were no restarts and no alerts.

2

Starting architecture

Drawing diagram…

The order service producer writes ShipmentRequested events to the shipment-requests topic, which has 3 partitions. The shipping service, 3 consumers in one group, polls the topic and books a label with the carrier API for each record. After the fix, records the shipping service gives up on are published to the topic shipment-requests-dlt, and any record arriving there raises an alert to the owning team.

flowchart LR
  OS["Order service producer"] -->|"ShipmentRequested"| T["Topic shipment-requests, 3 partitions"]
  T -->|"poll"| SS["Shipping service, 3 consumers in one group"]
  SS -->|"book a label"| CA["Carrier API"]
  SS -.->|"after the fix, records it gives up on"| DLT["Topic shipment-requests-dlt"]
  DLT -.->|"any record here"| AL["Alert to the owning team"]
Solid arrows are the starting system. The dotted path is what the fix adds: a place for records that cannot be processed, and someone who is told about them.

3

Expected behaviour

  • Every shipment request gets a label, in order within its partition.
  • A record that can never be processed does not stop the records behind it for more than a few seconds.
  • No record disappears: one that could not be processed is kept, with the reason, somebody is told, and it can be processed later.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 7

Book labels

Do this

  1. Turn on Order service publishing (6 shipment requests per second) and press play.
  2. In the Consumer group panel, check that each of the three consumers owns one partition and that the lag of every partition stays near 0.

Look for

  • "Labels booked" rises steadily and "Blocked partitions" is 0.
  • The group has no error handling configured at all: the first failure will show what that means.

Not done yet. The simulator checks this for you:consumed >= 24 && blockedPartitions == 0

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

6 ShipmentRequested events per second, keyed by order id.

Consumer group shipping-service

stablegeneration 1lag 0

RangeAssignor (eager)commit after batchmax.poll.records 10

  • shipper-1runningprocessed 0
    shipment-requests-0
  • shipper-2runningprocessed 0
    shipment-requests-1
  • shipper-3runningprocessed 0
    shipment-requests-2
Partitions of group shipping-service: owner, committed offset and lag
PartitionOwnerCommittedLag
shipment-requests-0 shipper-1none0
shipment-requests-1 shipper-2none0
shipment-requests-2 shipper-3none0

Simulated metrics

computed by the simulation
Labels booked
0
Blocked partitions
0
Largest lag on one partition
0records
Failed processing attempts
0
Skipped and lost
0
Retries
0
Dead-lettered
0
Alerts raised
0
Dead letters replayed
0

Dead letter topic shipment-requests-dlt

simulated

Empty. A record arrives here when the error handler gives up on it; the original partition then moves on.

Waiting for a fix
0
Replayed
0
Alerts raised
0

Partition logs

simulated records
  • shipment-requests-0leader b1ISR 1,2,3HW 0end 0

    empty

    shipping-service: committed none, lag 0

  • shipment-requests-1leader b2ISR 2,3,1HW 0end 0

    empty

    shipping-service: committed none, lag 0

  • shipment-requests-2leader b3ISR 3,1,2HW 0end 0

    empty

    shipping-service: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, shipment-requests (3 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.