Path 10 · lesson 6 of 6
Schema registries and schema evolution
What a schema registry stores, what each compatibility mode really promises, which order to deploy in, and how to evolve events safely with or without a registry.
What you will learn
- Explain what a schema registry stores and how a record on the topic points to its schema
- State precisely what backward, forward and full compatibility guarantee, and what the transitive variants add
- Decide whether adding, removing, renaming or retyping an Avro field is safe under a given mode
- Choose the order in which to deploy producers and consumers for a schema change
- Evolve JSON events without a registry using tolerant readers and additive changes
- Carry out a breaking change with a new topic or dual publishing, and diagnose a compatibility failure in production
Before you start
The concept
A Kafka broker stores and delivers bytes. It does not know whether a record value is JSON, Avro or a photograph, and it never checks. The meaning of those bytes is an agreement between the producer and every consumer: the schema. In a system that lives for years the schema will change, the producer and the consumers are deployed at different times, and a topic keeps records written under old versions for as long as its retention says. Schema evolution is the discipline of changing the agreement without breaking anyone who is on a different version.
A registry is a small service with a REST API that keeps every version of every schema and refuses new versions that break the rules you set.
| Term | Meaning | Example |
|---|---|---|
| Subject | The name under which a schema's versions are kept. With the default naming strategy it is derived from the topic | orders.placed-value |
| Version | The position of a schema within its subject, counted from 1 | Version 2 of orders.placed-value |
| Schema id | A number that identifies one schema across the whole registry. This is what travels with each record | 43 |
| Compatibility mode | The rule a new version must satisfy against earlier versions of the subject before it is accepted | BACKWARD (Confluent's default) |
Apicurio uses other words for the same ideas: an artifact has versions, artifacts can be collected in groups, each version has a global id, and rules for validity and compatibility are set globally, per group or per artifact.
Compatibility is always a statement about who can read what. Avro makes this concrete. Data is written with the writer's schema and read with the reader's schema, the one the consuming application was built with, and the Avro specification defines how the two are resolved when they differ.
| Mode | Guarantee | Checked against |
|---|---|---|
BACKWARD | A consumer using the new schema can read data written with the previous schema | The last registered version |
FORWARD | A consumer using the previous schema can read data written with the new schema | The last registered version |
FULL | Both of the above | The last registered version |
BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, FULL_TRANSITIVE | The same guarantee, against every earlier version | All registered versions |
NONE | Nothing. Any schema is accepted | Not checked |
Why it exists
Without a shared, enforced schema, the contract lives in two code bases and in people's heads. The producer team renames a field in what looks like a tidy refactoring. Their tests pass, because their tests use their own classes. The first check against any consumer happens in production, at the moment a consumer tries to deserialise the first new record.
Kafka makes this failure worse than the same mistake in an HTTP API, for three reasons.
- The bad data is durable. A broken HTTP response is gone when the deployment is rolled back. A record written with a broken schema stays in the log until retention removes it, and every consumer has to get past it.
- One bad record blocks a partition. Under at-least-once delivery a record that cannot be deserialised is fetched again and again. After an incompatible deployment it is not one poison record: every record written since is one.
- Readers come later. A consumer group may be created next year and start from the earliest offset. It must be able to read records written under every schema version still in the topic.
A registry moves the check from run time to the moment a schema is registered, ideally in the build pipeline of the producer. An incompatible schema is rejected there, with a reason, before any record is written. It also makes compact binary formats practical: an Avro record does not carry field names, so a reader needs the exact writer's schema to decode it, and the registry is where it finds that schema.
A registry is not the only way to evolve safely. Plain JSON with disciplined producers and tolerant consumers works for many teams. The registry's contribution is enforcement: the rule is checked by a machine on every change, not remembered by people.
How it works inside
How a record points to its schema
A registry-aware serializer runs inside the producer, in place of a plain JSON or string serializer. Before it writes, it obtains the id of the record's schema from the registry: by looking it up, or by registering the schema if the client is allowed to. It caches the answer. Then it writes a short prefix in front of the encoded data. In Confluent's default wire format that prefix is 5 bytes.
| Bytes | Content |
|---|---|
| 0 | A format version byte, 0 |
| 1 to 4 | The schema id, as a 4-byte integer |
| 5 onward | The data in the schema's format, for example Avro binary |
The deserializer on the consumer reads the id, fetches that schema from the registry once and caches it, decodes the bytes with it as the writer's schema, and resolves the result against the reader's schema. The registry is therefore not in the path of every record, only of the first record with each new id. It also means these bytes are not plain Avro or plain JSON: a consumer without the matching deserializer sees an unreadable prefix. Recent Confluent versions can alternatively carry a schema identifier in a record header; the principle is the same.
What Avro does when the schemas differ
The Avro specification's resolution rules for records are short, and every compatibility table follows from them. Fields are matched by name. A field the writer has and the reader lacks is ignored. A field the reader has and the writer lacks takes the reader's default value, and if the reader declares no default, reading fails. A writer's int can be read as long, float or double, and long as float or double, but not the other way.
| Change to an Avro record | BACKWARD | FORWARD | FULL | Why |
|---|---|---|---|---|
| Add a field with a default | Yes | Yes | Yes | New readers fill the default for old data. Old readers ignore the extra field |
| Add a field without a default | No | Yes | No | A new reader has no value for the field in old data |
| Remove a field that had a default | Yes | Yes | Yes | New readers ignore it in old data. Old readers fall back to their default for new data |
| Remove a field that had no default | Yes | No | No | Old readers require the field and new data does not contain it |
| Rename a field | Only if both have defaults | Only if both have defaults | Only if both have defaults | To a reader it is a removal plus an addition, matched by name. Even when accepted, the old readers see the default, not the value. Treat it as breaking |
Widen a type, int to long | Yes | No | No | A long reader can read int data. An int reader cannot read long data |
Change to an unrelated type, long to string | No | No | No | No promotion exists in either direction |
The pattern in the table: a change is safe in both directions only when it is an optional field appearing or disappearing. That is why the practical rule for Avro is to give every field a default from the first version, and why FULL or FULL_TRANSITIVE permits little else. For Protobuf, the equivalent rules are to never reuse or change a field number and to add fields as optional; Confluent's documentation recommends BACKWARD_TRANSITIVE for Protobuf subjects.
Which side to deploy first
| Mode of the subject | Deploy order | Reason |
|---|---|---|
BACKWARD or BACKWARD_TRANSITIVE | Consumers first, then producers | Only new readers are guaranteed to understand both old and new data. Old readers may not understand new data |
FORWARD or FORWARD_TRANSITIVE | Producers first, then consumers | Old readers understand new data. New readers are not guaranteed to understand old data still in the topic |
FULL or FULL_TRANSITIVE | Any order | Both directions hold, so producers and consumers can be released independently |
NONE | No safe order is implied | Nothing was checked |
Non-transitive modes compare with the last version only. Versions 1 to 2 and 2 to 3 can each pass while version 3 cannot read version 1 data. Since a topic holds old records for its whole retention and a new consumer may start from the beginning, the transitive modes are the ones that match how Kafka is used. Confluent's default is BACKWARD, chosen so that a consumer on the latest schema can be rewound over older data. Teams that want independent deployments usually choose FULL_TRANSITIVE and accept that only optional fields can change.
Evolving JSON without a registry
JSON carries its field names in every record, so a reader can skip what it does not know. That gives you forward compatibility for free, provided both sides keep four rules.
- Consumers are tolerant readers. Ignore unknown fields, and do not fail on a missing optional field: use a default. Read only the fields you need.
- Producers make additive changes only. Add optional fields. Do not remove a field or make an optional field required while any consumer may depend on it.
- Never repurpose a field. Do not change its type, unit or meaning under the same name.
totalin pounds becomingtotalin pence parses perfectly and is wrong everywhere. Add a new field with a new name. - Carry a version field, such as
schemaVersion, so that a consumer can branch deliberately when it has to, and so that the records in a dead letter topic tell you what produced them.
What JSON without a registry does not give you is the check. Nothing stops a producer from breaking rule 2 or 3 except review and contract tests between the teams. How the Spring serializers map JSON to classes, type headers and trusted packages is covered in Serialization in Spring for Apache Kafka.
When the change really is breaking
Some changes cannot be made compatible: a field changes type, one event splits into two, a key changes. Do not force them through by switching the subject to NONE. Give the new contract a new place and migrate.
New topic
- Create
orders.placed.v2with the new schema under its own subject - The producer writes to both topics during the migration (dual publishing), ideally from one outbox so the two cannot diverge
- Each consumer moves to the new topic in its own time, at a defined offset or point in time
- When no group reads the old topic any more, stop writing to it and let retention empty it
Expand and contract in place
- Add the new field beside the old one as an optional field, and fill both
- Move each consumer to the new field
- Stop filling the old field, then remove it, which is allowed once it is optional
- Works only when the old and new shape can coexist in one record
Both are the same idea as an expand, migrate, contract database migration, shown in the animated lesson Expand, migrate, contract: never make a change that requires two independently deployed parties to switch at the same instant.
Architecture
Drawing diagram…
A flowchart in which a build pipeline registers a schema version and the registry accepts or rejects it by compatibility. At run time the producer's serializer looks up the schema id and writes it in front of the Avro data, and the consumer's deserializer fetches the schema by id and resolves it against its own reader schema, which either yields an object or a deserialization failure.
flowchart TD
CI["Producer build pipeline"] -- "register new version" --> REG["Schema registry, third party"]
REG --> CHK{"Compatible with earlier versions of the subject?"}
CHK -- "No" --> REJ["Rejected, build fails"]
CHK -- "Yes" --> ID["Version 2 stored, schema id 43"]
P["Order Service"] --> SER["Registry-aware serializer"]
SER -- "look up schema id, cached" --> REG
SER --> REC["Record value: format byte, schema id 43, Avro data"]
REC --> T["Topic orders.placed"]
T --> DES["Registry-aware deserializer"]
DES -- "fetch schema 43 once, cached" --> REG
DES --> RES{"Writer schema resolves against reader schema?"}
RES -- "Yes" --> APP["Payment Service listener"]
RES -- "No" --> ERR["Deserialization failure, poison record"]
ERR --> EH["ErrorHandlingDeserializer and error handler"]- There are two moments of truth. The first is registration, where an incompatible schema can be stopped before it produces anything. The second is resolution in the consumer, where it is too late: the record is already in the log.
- The registry is consulted once per schema id by each client, then cached. An outage of the registry does not stop clients that already hold the ids they need, but a newly started client, or the first record with a new id, cannot proceed without it.
- The failure branch on the right is the subject of the poison message lessons. Schema evolution is about making sure that branch is never taken for a whole deployment's worth of records.
Message flow
Drawing diagram…
A sequence diagram in which a build pipeline tests and registers version 2 of a schema, the consumer is deployed and reads an old record by fetching schema 42 and applying a default, then the producer is deployed, looks up schema id 43 and writes a new record, which the consumer reads by fetching schema 43.
sequenceDiagram participant CI as Build pipeline participant R as Schema registry participant C as Payment Service participant P as Order Service participant K as Topic orders.placed CI->>R: Test OrderPlaced v2 against subject orders.placed-value R-->>CI: Compatible under FULL_TRANSITIVE CI->>R: Register v2 R-->>CI: Version 2, schema id 43 Note over C: Deployed with v2 as reader schema C->>K: Poll K-->>C: Old record, prefix says schema id 42 C->>R: Get schema 42 R-->>C: OrderPlaced v1 Note over C: Missing salesChannel takes the default web Note over P: Deployed with v2 as writer schema P->>R: Look up id of v2 for the subject R-->>P: Schema id 43 P->>K: Record with prefix 43 and Avro data C->>K: Poll K-->>C: New record, prefix says schema id 43 C->>R: Get schema 43 R-->>C: OrderPlaced v2 Note over C: Reads salesChannel as written
- The schema is registered by the pipeline, before either service is deployed. The producer at run time only looks the id up. A production client that registers schemas by itself turns a deployment into an unreviewed contract change.
- The consumer reads two generations of records with one reader schema. For the old record, the field that did not exist yet takes the default declared in version 2. This is backward compatibility at work.
- Under
FULL_TRANSITIVEthe order of the two deployments could be swapped. A consumer still on version 1 would simply ignoresalesChannelin new records. Under plainBACKWARDthe order shown here is the required one.
In a real system
Incident: the registry said yes
A typical order platform. The Order Service publishes OrderPlaced as Avro to orders.placed, 12 partitions, keyed by order id. The subject orders.placed-value is on the registry default, BACKWARD. Three services consume it with generated Avro classes: Payment, Inventory and Notification. The Payment Service wraps its deserializer in ErrorHandlingDeserializer and has a dead letter topic. The other two do not.
The Order Service team cleans up the event. The field legacyStoreCode, a required string with no default that nobody on their team uses, is removed. The pipeline's compatibility check passes: under BACKWARD, removing a field is allowed, because a reader on the new schema just ignores it in old data. The schema becomes version 7. The Order Service is deployed at 11:02. No consumer has been rebuilt.
| Time | What happened | Why |
|---|---|---|
| 11:02 | The first version 7 records are written to all 12 partitions | The producer is the first and only service on the new schema |
| 11:02 | Inventory and Notification stop on every partition, logging the same deserialization error several times a second | Their reader schema is version 6, which requires legacyStoreCode with no default. The writer's data does not contain it. The exception is thrown inside poll(), the position does not move, and the same record is fetched again |
| 11:02 onward | The Payment Service keeps moving, and sends every new order to its dead letter topic. No order is charged | The wrapper turns each failure into a dead-lettered record. With every record failing, the dead letter topic becomes the main flow |
| 11:09 | Lag alerts fire for two groups. The dead letter rate alert for Payment fires | Seven minutes of orders are affected, about 1,500 |
| 11:15 | The Order Service is rolled back to the previous version | New records are version 6 again and readable. The version 7 records written since 11:02 are still in the log |
| 11:15 to 11:50 | Inventory and Notification remain stuck at the first version 7 record on each partition | Rolling back a producer does not remove what it wrote. Each partition is still blocked at its first unreadable record |
| 11:50 | All three consumers are rebuilt with version 7 as their reader schema and deployed. They read version 6 and version 7 records alike | This is the direction BACKWARD does guarantee. The fix was the deployment that should have come first |
| 12:20 | The Payment Service's dead letter topic is replayed through the upgraded consumer | The dead-lettered records hold the original bytes, so they are readable once the reader understands version 7 |
Nothing in the tooling failed. The registry answered the question it was configured to ask, and the answer was correct. The team asked the wrong question for the deployment order they used: they needed FORWARD compatibility (old readers, new data) and had checked BACKWARD.
- If producers and consumers are released independently, set the subject to
FULL_TRANSITIVE. The removal would have been rejected at registration, with the reason. - If the mode stays
BACKWARD, the release process must enforce consumers first. A registry cannot enforce deployment order. - Give every field a default from version 1. A field with a default can later be removed in either direction.
- A rollback of the producer is not a fix once records are written. The ways forward are to upgrade the readers or to skip or dead-letter the unreadable range.
- When every record fails, a dead letter topic is not a safety net. Alert on its rate, and consider stopping the container on a burst of deserialization failures so that nothing is passed over.
For a real case of one unreadable record taking a consumer down repeatedly, see the Break card The message that kills its reader.
Java and Spring Boot
Two versions of an Avro schema that are compatible in both directions, the registry calls that check and register a version, a Spring Boot configuration for registry-aware serializers, and a JSON tolerant reader with Jackson for teams without a registry. The registry, its REST API and the io.confluent serializers are third-party software, not part of Apache Kafka or Spring.
{ "type": "record", "name": "OrderPlaced", "namespace": "com.example.orders.events", "fields": [ { "name": "eventId", "type": "string" }, { "name": "orderId", "type": "string" }, { "name": "customerId", "type": "string" }, { "name": "totalMinor", "type": "long", "doc": "Order total in minor units, for example pence" }, { "name": "currency", "type": "string", "default": "GBP" }, { "name": "placedAt", "type": { "type": "long", "logicalType": "timestamp-millis" } } ]}Why it is written this way
eventId,orderId,customerId,totalMinorandplacedAthave no default: they are the core of the event and can never be removed compatibly. That is a deliberate statement about the contract.totalMinoris alongin minor units with the unit in its name anddoc. Money as a floating point number, or a baretotal, is the kind of field that later gets repurposed.currencyhas a default, so it could be removed later without breaking readers in either direction.
{ "type": "record", "name": "OrderPlaced", "namespace": "com.example.orders.events", "fields": [ { "name": "eventId", "type": "string" }, { "name": "orderId", "type": "string" }, { "name": "customerId", "type": "string" }, { "name": "totalMinor", "type": "long", "doc": "Order total in minor units, for example pence" }, { "name": "currency", "type": "string", "default": "GBP" }, { "name": "placedAt", "type": { "type": "long", "logicalType": "timestamp-millis" } }, { "name": "salesChannel", "type": "string", "default": "web" }, { "name": "couponCode", "type": ["null", "string"], "default": null } ]}Why it is written this way
- Both new fields have defaults. A version 2 reader fills them in for version 1 data (backward), and a version 1 reader ignores them in version 2 data (forward). The change passes
FULL_TRANSITIVE. couponCodeis a union withnullfirst and anulldefault: the Avro way to say optional. Listingnullfirst with anulldefault is the form every Avro version accepts.salesChannelis a string, not an Avro enum. Adding a symbol to an enum is a change old readers cannot handle unless the enum declares its own default symbol.- Nothing was renamed, retyped or removed. If
totalMinorhad to become a decimal string, that would be a new field or a new topic, not version 3 of this one.
#!/usr/bin/env bash# Confluent Schema Registry REST API (third party). Apicurio Registry offers a compatible API.set -euo pipefailREGISTRY="${SCHEMA_REGISTRY_URL}"SUBJECT="orders.placed-value"BODY=$(jq -n --rawfile schema schemas/order-placed-v2.avsc '{schemaType: "AVRO", schema: $schema}')CT="Content-Type: application/vnd.schemaregistry.v1+json"# One-off, per subject: the rule new versions must satisfy.curl -sf -X PUT -H "$CT" --data '{"compatibility": "FULL_TRANSITIVE"}' \ "$REGISTRY/config/$SUBJECT"# 1. Ask first. verbose=true returns the reasons when the answer is no.RESULT=$(curl -sf -X POST -H "$CT" --data "$BODY" \ "$REGISTRY/compatibility/subjects/$SUBJECT/versions/latest?verbose=true")echo "$RESULT"[ "$(echo "$RESULT" | jq -r '.is_compatible')" = "true" ] || { echo "Schema is not compatible"; exit 1; }# 2. Register. An incompatible schema would be refused here as well, with 409 Conflict.curl -sf -X POST -H "$CT" --data "$BODY" "$REGISTRY/subjects/$SUBJECT/versions"Why it is written this way
- The check runs in the producer's pipeline, before any deployment. A rejected schema fails a build, not production.
- The compatibility test uses the mode configured for the subject, falling back to the registry's global mode if the subject has none. Setting it explicitly per subject avoids surprises from the global default.
- With a transitive mode the registry compares against all earlier versions when registering. Registration, step 2, is the authoritative check.
- Registration returns the schema id. Registering the same schema again returns the same id, so the step can be repeated safely.
spring: kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS} properties: # Read by the io.confluent serializer and deserializer, not by Kafka itself. schema.registry.url: ${SCHEMA_REGISTRY_URL} producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer properties: auto.register.schemas: false consumer: group-id: payment-service enable-auto-commit: false key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: spring.deserializer.value.delegate.class: io.confluent.kafka.serializers.KafkaAvroDeserializer specific.avro.reader: trueWhy it is written this way
KafkaAvroSerializerandKafkaAvroDeserializercome from Confluent'skafka-avro-serializerlibrary (Apache 2.0), published in Confluent's own Maven repository, not Maven Central. Apicurio provides its own serializer classes with the same role.auto.register.schemas: falsestops a running producer from registering whatever schema its classes happen to have. With it, a schema that the pipeline did not register makes the send fail, which is the behaviour you want in production.specific.avro.reader: truemakes the deserializer return the generatedOrderPlacedclass and use its schema as the reader's schema. The default isfalse, which returns aGenericRecord.- The Avro deserializer is the delegate of
ErrorHandlingDeserializer, so a record that cannot be resolved reaches the error handler and does not fail insidepoll(). See the poison message lesson for the handler. - The default subject naming derives the subject from the topic, here
orders.placed-value. It assumes one event type per topic.
package com.example.payments.events;import java.time.Instant;import java.util.Objects;import com.fasterxml.jackson.annotation.JsonIgnoreProperties;/** * The Payment Service's own view of the OrderPlaced event, read from JSON. * A tolerant reader: unknown fields are ignored, optional fields get defaults. */@JsonIgnoreProperties(ignoreUnknown = true)public record OrderPlaced( Integer schemaVersion, String eventId, String orderId, Long totalMinor, String currency, Instant placedAt, String salesChannel) { public OrderPlaced { // Required by this consumer: fail clearly, with the field name, if the producer drops one. Objects.requireNonNull(eventId, "eventId"); Objects.requireNonNull(orderId, "orderId"); Objects.requireNonNull(totalMinor, "totalMinor"); // Optional: absent in records written before the field existed. if (schemaVersion == null) { schemaVersion = 1; } if (currency == null) { currency = "GBP"; } if (salesChannel == null) { salesChannel = "web"; } }}Why it is written this way
- The consumer declares only the fields it uses.
customerIdandcouponCodeexist in the event and are absent here on purpose: a field you do not read is a field whose removal cannot break you. @JsonIgnoreProperties(ignoreUnknown = true)makes the tolerance part of the class, independent of how theObjectMapperwas configured. Plain Jackson fails on unknown properties by default; the mapper inside Spring Kafka'sJsonDeserializerturns that off, and stating it on the type removes the dependence.- Wrapper types (
Long,Integer) are used so that a missing field isnulland can be told apart from zero. A missingtotalMinormust be an error, not an order for nothing. salesChannelis aString. A Java enum here would fail on the first value this build has never seen, which turns an additive change by the producer into a poison record.schemaVersionis not used for routine additions. It exists so the consumer can branch on purpose in the rare case where meaning differs between versions, and so dead-lettered records explain themselves.
Configuration
Settings of the registry and of registry-aware clients that decide what is checked and when. These belong to Confluent Schema Registry and its serializers, which are third-party; defaults are from Confluent's documentation. The last entry is Jackson's.
compatibility (registry, global or per subject)Default BACKWARD
- What it does
- The rule a new schema version must pass: BACKWARD, FORWARD, FULL, their _TRANSITIVE forms, or NONE. Set with
PUT /configorPUT /config/{subject}. - When to change it
- Use
FULL_TRANSITIVEwhen producers and consumers deploy independently. KeepBACKWARDonly with a consumers-first release rule. TreatNONEon a production subject as an incident waiting to happen.
schema.registry.urlDefault none
- What it does
- Where the serializer and deserializer find the registry. Required for registry-aware clients.
- When to change it
- Clients cache schemas by id, but a client that starts while the registry is down cannot serialise or deserialise. Run the registry with the availability you expect of your consumers.
auto.register.schemasDefault true
- What it does
- Whether the serializer registers the schema of the object it is asked to write when the registry does not have it.
- When to change it
- Convenient in development. Set to
falsein production and register schemas from the build pipeline, so a deployment cannot change a contract as a side effect.
value.subject.name.strategyDefault TopicNameStrategy
- What it does
- How the subject is derived. The default uses the topic name, which ties one schema lineage to one topic.
- When to change it
- Keep the default for one event type per topic.
RecordNameStrategyandTopicRecordNameStrategyexist for topics that carry several event types.
specific.avro.readerDefault false
- What it does
- Whether the Avro deserializer returns generated classes (and uses their schema as the reader's schema) or
GenericRecord. - When to change it
- Set to
truewhen the consumer uses generated classes. Remember that the reader's schema is then fixed at build time: reading newer data depends on forward compatibility.
DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES (Jackson)Default true in plain Jackson; disabled in the mapper Spring Kafka's JsonDeserializer creates
- What it does
- Whether a JSON field with no matching property makes deserialisation fail.
- When to change it
- A tolerant reader needs it off. Do not rely on which mapper is in use: annotate event types with
@JsonIgnoreProperties(ignoreUnknown = true).
Hands-on exercise
Troubleshooting: every new record fails after a producer deployment
Diagnose a compatibility failure from a consumer's symptoms and the registry's answers, decide what restores processing, and then reproduce the mechanism in the schema evolution lab. All output below is illustrative: written for this exercise in the format the tools produce.
Steps · 0 of 6 done
What you should see
The registry accepted version 7 because the subject is BACKWARD and removing a field is backward compatible: a reader on version 7 ignores the field in older data. The Inventory Service is an old reader of new data, which is the forward direction. Its reader schema, version 6, requires legacyStoreCode and has no default, and the data written with schema id 48 does not contain it, so Avro resolution fails. The rollback stopped new version 7 records, but those already written are in the log, and each partition is blocked at its first one because the failure happens inside poll() and the position never moves. To recover without loss: deploy the Inventory Service with version 7 as its reader schema, which reads version 6 and version 7 alike. To prevent it: use FULL_TRANSITIVE for subjects whose producers and consumers deploy independently (or enforce consumers-first under BACKWARD), give fields defaults, and wrap the deserializer in ErrorHandlingDeserializer so that a failure reaches the error handler. In the lab you should see the same shape: lag rising on every partition after the incompatible deployment, no movement of committed offsets, and normal processing through the same deployment once the change is compatible.
What can go wrong
Reading BACKWARD as 'safe for existing consumers'
- Symptom
- A schema change passes the registry's check, the producer is deployed, and running consumers fail on every new record.
- Cause
BACKWARDguarantees that new readers can read old data. Running consumers are old readers of new data, which onlyFORWARDorFULLcovers.- Fix
- Deploy consumers first under
BACKWARD, or move the subject toFULL_TRANSITIVEso that either order is safe.
Compatibility switched to NONE to get a change through
- Symptom
- A subject that used to reject changes now accepts anything. Some time later an unrelated change breaks consumers.
- Cause
- The mode was set to
NONEfor one awkward release and never restored. Every later change went unchecked. - Fix
- Do not disable the check for a breaking change. Use a new topic or expand and contract. Audit
GET /config/{subject}for subjects onNONE.
Producers that register schemas at run time
- Symptom
- A new schema version appears in the registry at the moment of a deployment, and nobody reviewed it.
- Cause
auto.register.schemasis on by default, so the serializer registers the schema of whatever class the new build contains.- Fix
- Set
auto.register.schemas=falsein production and register from the pipeline, where a rejection fails the build.
Fields without defaults
- Symptom
- A harmless-looking clean-up, removing an unused field or adding a new one, is rejected by the registry or breaks one direction.
- Cause
- In Avro a field can only appear or disappear compatibly in both directions if it has a default value.
- Fix
- Give every non-essential field a default from the first version. Keep the set of fields without defaults small and permanent.
Repurposing a field
- Symptom
- No errors anywhere. Amounts are off by a factor of 100, or a status is interpreted wrongly, in some consumers only.
- Cause
- A field kept its name and type but changed unit or meaning. Every schema check passes because the schema did not change.
- Fix
- Never change the meaning of an existing field. Add a new field with a name that states the unit, migrate consumers, then retire the old one.
Rolling back the producer and waiting
- Symptom
- After a rollback, lag keeps growing and consumers stay on the same offsets.
- Cause
- The incompatible records are durable. The consumers are blocked at the first one in each partition.
- Fix
- Upgrade the readers so they can read the records, or deliberately skip or dead-letter the affected range. A rollback only stops the range from growing.
A strict JSON consumer
- Symptom
- A consumer fails with an unrecognised property error right after a producer added an optional field.
- Cause
- The consumer's mapper fails on unknown properties, or maps a field to an enum that lacks the new value.
- Fix
- Make consumers tolerant readers: ignore unknown properties, map open-ended values to strings, default missing optional fields.
How to troubleshoot
Lag grows on every partition right after a producer deployment
Check
kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 --describe --group inventory-service
- How to read it
- Run it twice. CURRENT-OFFSET frozen on all partitions while LOG-END-OFFSET rises, with members assigned, means the consumers are stuck, not slow. All partitions at once, starting at a deployment time, points at the data format and not at one bad record.
RecordDeserializationException or SerializationException in the consumer log
Check
The innermost cause, for example `AvroTypeException: ... missing required field legacyStoreCode`, and the schema id in the message
- How to read it
- A missing required field means the reader's schema needs a field without a default that the writer's schema lacks: a forward compatibility break. A type mismatch means a field was retyped. The schema id tells you which version wrote the record.
You need to know which schema a stuck record was written with
Check
kafka-console-consumer.sh --bootstrap-server kafka-1:9092 --topic orders.placed --partition 3 --offset 88210 --max-messages 1 | xxd | head -n 1
- How to read it
- In Confluent's default wire format the first byte is
00and the next four are the schema id in hexadecimal:00 00 00 00 30is id 48. Thencurl -s $SCHEMA_REGISTRY_URL/schemas/ids/48shows the writer's schema. A value that starts with7b(an opening brace) is plain JSON written to an Avro topic.
You need to see what changed between two versions
Check
curl -s $SCHEMA_REGISTRY_URL/subjects/orders.placed-value/versions/6 and the same for version 7
- How to read it
- Compare the
schemafields. Look for removed fields that had no default, added fields without a default, renamed fields and changed types, and match what you find to the table of safe changes.
The registry accepted a change that broke consumers
Check
curl -s "$SCHEMA_REGISTRY_URL/config/orders.placed-value?defaultToGlobal=true"
- How to read it
BACKWARDwith a producer-first deployment explains it: the check covered the other direction.NONEmeans nothing was checked. A non-transitive mode explains a break against a version older than the last.
The registry rejects a schema and you need the reason
Check
curl -s -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' --data @body.json "$SCHEMA_REGISTRY_URL/compatibility/subjects/orders.placed-value/versions/latest?verbose=true"
- How to read it
- The response has
is_compatibleand, withverbose=true, amessagesarray naming the field and the rule that failed. Fix the schema, usually by adding a default, and do not change the mode.
A JSON consumer fails after the producer added a field
Check
The exception class in the consumer log: Jackson's `UnrecognizedPropertyException` or `InvalidFormatException`
- How to read it
UnrecognizedPropertyExceptionmeans the consumer is strict about unknown fields: make it tolerant.InvalidFormatExceptionon an existing field means the producer changed a type or sent an enum value the consumer does not know: that is a breaking change on the producer's side, or an enum that should have been a string.
Key takeaways
- Kafka stores bytes. The schema is a contract between teams, and a schema registry, which is third-party software and not part of Apache Kafka, is where that contract is stored and checked.
- Each record carries a schema id, not the schema. Readers fetch the writer's schema by id and resolve it against their own.
BACKWARDmeans new readers can read old data.FORWARDmeans old readers can read new data.FULLis both, and the transitive forms check every earlier version.- The mode dictates the deployment order: consumers first under backward, producers first under forward, any order under full.
- In Avro, only optional fields, those with defaults, can come and go safely in both directions. Renames and type changes are breaking.
- Without a registry: tolerant readers, additive changes, never repurpose a field, carry a version.
- A breaking change gets a new topic or an expand and contract migration. Rolling back a producer does not remove the records it wrote.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
The subject payments.captured-value is on BACKWARD. The producing team wants to remove terminalId, a string field with no default, and the registry's compatibility test returns is_compatible: true. Four consumer services read the topic with generated Avro classes built from the current version. The producing team plans to release this afternoon; the consumer teams release weekly.
What has to happen for this change to go out safely?
Scenario 2 of 3
orders.placed carries totalMinor as a long. A new pricing engine needs fractional minor units, so the team wants total as a decimal encoded in a string. The subject is on FULL_TRANSITIVE and rejects every attempt to change the type. Six consumer groups read the topic, two of them owned by other departments with their own release schedules. Retention is 14 days.
How should the change be made?
Scenario 3 of 3
A team uses JSON without a registry. The Order Service adds an optional field giftMessage to OrderPlaced. The Spring-based Payment Service is unaffected. A reporting job written with a plain Jackson ObjectMapper and its default settings stops at partition 5, offset 20417, with UnrecognizedPropertyException: Unrecognized field "giftMessage", and its lag grows.
Which change addresses the cause?
References
References: Apache Avro 1.12 specification: schema resolution ↗ · Spring for Apache Kafka reference: Serialization, deserialization and message conversion ↗ · Confluent documentation: Schema evolution and compatibility ↗ · Confluent documentation: Formats, serializers and deserializers (wire format, subject naming) ↗ · Confluent documentation: Schema Registry API reference ↗ · Confluent Schema Registry repository: licence section of the README ↗ · Apicurio Registry documentation: Introduction to Apicurio Registry ↗ · Apicurio Registry repository ↗ · Report a problem