Skip to content

BuildBreak lab 15 of 15

Schema evolution

Deploy a producer change that existing consumers cannot read and watch the topic fill with poison records. Then add a compatibility check, evolve the schema safely, and get the deploy order right.

Advanced35 min

Read first: Schema registries and schema evolution · Serialisation in Spring Kafka: JSON, type headers, Avro · Poison messages and error handling · Dead letter topics: park, inspect, replay

Simulation.This lab is a deterministic simulation. No Kafka broker is running; logs and metrics are simulated.

Brokers
3 (KRaft)
Topics
orders: 3 partitions, RF 3
Producers
order-service
Consumer groups
payment-service (1)
Guided run
10 stages: build → break → observe → fix → retest

1

Business scenario

The order service publishes OrderPlaced to the orders topic and the payment service consumes it. Both were built against the same schema: orderId, amount, currency and storeCode. The order team tidies up the event and renames amount to totalMinor. Their tests pass, because their own code reads the new name. Twenty minutes after the deploy the payment service's lag alarm fires, and it has not taken a payment since.

2

Starting architecture

Drawing diagram…

The order service producer writes OrderPlaced records, each written with a schema version, to the orders topic. The payment service consumer polls the topic, deserializes each record and writes to a payments table. Dashed lines show the optional schema registry: the producer checks and registers its schema at build time, and the consumer fetches the writer schema from it. The registry is not part of the broker.

flowchart LR
  OS["Order service producer"] -->|"OrderPlaced written with a schema version"| T["Topic orders"]
  T -->|"poll and deserialize"| PS["Payment service consumer"]
  PS --> DB[("payments table")]
  OS -.->|"check and register at build time"| SR["Schema registry"]
  PS -.->|"fetch writer schema"| SR
The broker stores bytes. The schema is an agreement between the two services; a registry, which is separate software, can store and check it.

3

Expected behaviour

  • The producer can change the event without any running consumer failing to read a record.
  • A change that existing consumers cannot read is rejected in the producer's build, before any record is written with it.
  • A record that still cannot be read is set aside, so that it does not block its partition.

4 · Build → break → observe → fix → retest

Guided run

buildStage 1 of 10

One schema on both sides

Do this

  1. Turn on Order service publishing (4 orders per second) and press play.
  2. Read the fields table in the registry panel: the writer and the reader have the same four fields.

Look for

  • Payments are inserted and "Records that failed to deserialize" is 0.
  • "Compatibility mode" is NONE: the subject accepts any schema. Nothing checks a change yet.

Not done yet. The simulator checks this for you:inserted("payments") >= 10 && deserializationFailures == 0

Simulator

Simulation · not a real cluster
simulated time 0.0s
Speed

Controls for this stage

4 orders per second.

Schema registry: subject orders-value

simulated

A schema registry is separate software (for example Confluent Schema Registry or Apicurio Registry), not part of Apache Kafka: brokers store bytes and check nothing. Deploy order in mode NONE: no order is safe: nothing is checked.

Compatibility mode
NONE
Producer writes
v1
Consumers read (tolerant reader)
v1
Registered versions
1

Fields: what the producer writes and what the consumers expect

fieldwriter v1reader v1
orderIdstringstring
amountlonglong
currencystring = "GBP"string = "GBP"
storeCodestringstring

Possible next changes, checked against v1

changeBACKWARDFORWARDFULL
rename amount to totalMinorREJECTEDREJECTEDREJECTED
change amount to a stringREJECTEDREJECTEDREJECTED
add optional couponCodepassespassespasses
add optional giftNotepassespassespasses
remove storeCodepassesREJECTEDREJECTED
add required channelREJECTEDpassesREJECTED

Simulated metrics

computed by the simulation
Payments taken
0
Records that failed to deserialize
0
Blocked partitions
0
Payment service lag
0records
Schemas rejected before deploy
0
Records read across schema versions
0
Dead-lettered records
0
Failures caused by an unknown field
0

Consumer group payment-service

stablegeneration 1lag 0

RangeAssignor (eager)commit after batchmax.poll.records 10

  • payment-1runningprocessed 0
    orders-0orders-1orders-2
Partitions of group payment-service: owner, committed offset and lag
PartitionOwnerCommittedLag
orders-0 payment-1none0
orders-1 payment-1none0
orders-2 payment-1none0

Partition logs

simulated records
  • orders-0leader b1ISR 1,2,3HW 0end 0

    empty

    payment-service: committed none, lag 0

  • orders-1leader b2ISR 2,3,1HW 0end 0

    empty

    payment-service: committed none, lag 0

  • orders-2leader b3ISR 3,1,2HW 0end 0

    empty

    payment-service: committed none, lag 0

dashed = above the high watermark (not replicated, not readable)dim = committed by every groupviolet = same event written twice

Simulated log

newest first · not real broker output
  1. 0.0sINFO [simulation] Simulated cluster ready: 3 brokers, orders (3 partitions, RF 3). KRaft mode, no ZooKeeper

The root cause, the solution with its code, and what the simulation leaves out appear when you finish the last stage.

The lab is recorded as complete when you finish its last stage. You can also mark it yourself.