BuildBreak lab 15 of 15
Schema evolution
Deploy a producer change that existing consumers cannot read and watch the topic fill with poison records. Then add a compatibility check, evolve the schema safely, and get the deploy order right.
Read first: Schema registries and schema evolution · Serialisation in Spring Kafka: JSON, type headers, Avro · Poison messages and error handling · Dead letter topics: park, inspect, replay
Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.
- Brokers
- 3 (KRaft)
- Topics
- orders: 3 partitions, RF 3
- Producers
- order-service
- Consumer groups
- payment-service (1)
- Guided run
- 10 stages: build → break → observe → fix → retest
1
Business scenario
The order service publishes OrderPlaced to the orders topic and the payment service consumes it. Both were built against the same schema: orderId, amount, currency and storeCode. The order team tidies up the event and renames amount to totalMinor. Their tests pass, because their own code reads the new name. Twenty minutes after the deploy the payment service's lag alarm fires, and it has not taken a payment since.
2
Starting architecture
Drawing diagram…
The order service producer writes OrderPlaced records, each written with a schema version, to the orders topic. The payment service consumer polls the topic, deserializes each record and writes to a payments table. Dashed lines show the optional schema registry: the producer checks and registers its schema at build time, and the consumer fetches the writer schema from it. The registry is not part of the broker.
flowchart LR
OS["Order service producer"] -->|"OrderPlaced written with a schema version"| T["Topic orders"]
T -->|"poll and deserialize"| PS["Payment service consumer"]
PS --> DB[("payments table")]
OS -.->|"check and register at build time"| SR["Schema registry"]
PS -.->|"fetch writer schema"| SR3
Expected behaviour
- The producer can change the event without any running consumer failing to read a record.
- A change that existing consumers cannot read is rejected in the producer's build, before any record is written with it.
- A record that still cannot be read is set aside, so that it does not block its partition.
4 · Build → break → observe → fix → retest
Guided run
One schema on both sides
Do this
- Turn on Order service publishing (4 orders per second) and press play.
- Read the fields table in the registry panel: the writer and the reader have the same four fields.
Look for
- Payments are inserted and "Records that failed to deserialize" is 0.
- "Compatibility mode" is NONE: the subject accepts any schema. Nothing checks a change yet.
Not done yet. The simulator checks this for you:inserted("payments") >= 10 && deserializationFailures == 0
Simulator
Simulation · not a real clusterControls for this stage
4 orders per second.
Schema registry: subject orders-value
simulatedA schema registry is separate software (for example Confluent Schema Registry or Apicurio Registry), not part of Apache Kafka: brokers store bytes and check nothing. Deploy order in mode NONE: no order is safe: nothing is checked.
- Compatibility mode
- NONE
- Producer writes
- v1
- Consumers read (tolerant reader)
- v1
- Registered versions
- 1
Fields: what the producer writes and what the consumers expect
| field | writer v1 | reader v1 |
|---|---|---|
| orderId | string | string |
| amount | long | long |
| currency | string = "GBP" | string = "GBP" |
| storeCode | string | string |
Possible next changes, checked against v1
| change | BACKWARD | FORWARD | FULL |
|---|---|---|---|
| rename amount to totalMinor | REJECTED | REJECTED | REJECTED |
| change amount to a string | REJECTED | REJECTED | REJECTED |
| add optional couponCode | passes | passes | passes |
| add optional giftNote | passes | passes | passes |
| remove storeCode | passes | REJECTED | REJECTED |
| add required channel | REJECTED | passes | REJECTED |
Simulated metrics
computed by the simulation- Payments taken
- 0
- Records that failed to deserialize
- 0
- Blocked partitions
- 0
- Payment service lag
- 0records
- Schemas rejected before deploy
- 0
- Records read across schema versions
- 0
- Dead-lettered records
- 0
- Failures caused by an unknown field
- 0
Consumer group payment-service
stablegeneration 1lag 0RangeAssignor (eager)commit after batchmax.poll.records 10
- payment-1runningprocessed 0orders-0orders-1orders-2
| Partition | Owner | Committed | Lag |
|---|---|---|---|
| orders-0 | payment-1 | none | 0 |
| orders-1 | payment-1 | none | 0 |
| orders-2 | payment-1 | none | 0 |
Partition logs
simulated records- orders-0leader b1ISR 1,2,3HW 0end 0
empty
payment-service: committed none, lag 0
- orders-1leader b2ISR 2,3,1HW 0end 0
empty
payment-service: committed none, lag 0
- orders-2leader b3ISR 3,1,2HW 0end 0
empty
payment-service: committed none, lag 0
dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice
Simulated log
newest first · not real broker output- 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (3 partitions, RF 3). KRaft mode, no ZooKeeper
The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.