Path 9 · lesson 3 of 6
Serialisation in Spring Kafka: JSON, type headers, Avro
How Java objects become record bytes and back: JsonSerializer and JsonDeserializer, type headers and trusted packages, ErrorHandlingDeserializer, and Avro with a schema registry.
What you will learn
- Configure JsonSerializer and JsonDeserializer for a topic with one event type and for a topic with several
- Explain what the type headers contain, why they couple services, and remove that coupling with a default type or a type mapping
- Wrap deserializers in ErrorHandlingDeserializer and describe what happens to a record that cannot be parsed, with and without it
- Design event records that tolerate fields being added by the producer
- Describe what a schema registry stores, how a record refers to its schema, and what a compatibility check prevents
- State which parts of an Avro setup are Apache Kafka, which are Spring and which are separately licensed products
Before you start
The concept
A Kafka record is a key and a value, each an array of bytes, plus headers. The broker never interprets them. Turning an object into bytes is the job of a Serializer inside the producer; turning bytes back is the job of a Deserializer inside the consumer. Both are configured per client, separately for key and value. Kafka itself ships serializers for simple types (String, Long, byte[], UUID and so on).
Spring for Apache Kafka adds JsonSerializer and JsonDeserializer, built on Jackson, and ErrorHandlingDeserializer, a wrapper that turns a deserialisation failure into something the listener container can handle. For schema-based binary formats such as Avro you plug in serializers from a schema registry product; those are not part of Kafka or Spring.
Because producer and consumer are different programs, deployed at different times, the bytes on the topic are a contract. A database schema has one owner and one migration tool. An event format has one writer and many readers that the writer may not know, and old records stay in the log and are read again long after the code that wrote them is gone. Most of this lesson is about keeping that contract independent of any one service's Java classes.
| Format | On the wire | Reader needs | Typical use |
|---|---|---|---|
| JSON, no type information | UTF-8 JSON text | To be told the target class (default type) | One event type per topic. The simplest contract between services |
| JSON with type headers | JSON text plus a header naming the type | A class with that name, or a mapping from the name to its own class | Several event types on one topic |
| Avro with a schema registry | A schema id followed by compact binary data | Access to the registry to fetch the writer's schema | Many teams, long-lived topics, enforced compatibility |
Why it exists
The first version of most Spring Kafka systems uses JsonSerializer with its defaults on one side and JsonDeserializer on the other, and it works on the first try, as long as both sides are in the same code base. The trouble appears when a second service consumes the topic.
- By default
JsonSerializeradds a header__TypeId__containing the fully qualified Java class name of the value, for examplecom.example.orders.events.OrderPlaced. - By default
JsonDeserializeruses that header to decide which class to create, in preference to anything configured locally. - The consuming service has no class of that name, and even if it did, the package is not in the deserializer's trusted packages. Deserialisation fails for every record.
The quick fixes are worse than the problem: copying the producer's class into the same package in every consumer, publishing a shared library of event classes that every service must upgrade together, or setting trusted packages to *. Each of them makes a Java package name part of the wire contract. Renaming a package in the order service then breaks the payment service.
The second problem is what a single unreadable record does. Deserialisation happens inside poll, before any of your code or Spring's error handling runs. Without protection, one malformed record stops a partition for good. ErrorHandlingDeserializer exists for exactly this case.
How it works inside
What JsonSerializer writes
JsonSerializer writes the object as JSON with a Jackson ObjectMapper. Unless spring.json.add.type.headers is false, it also adds type headers: __TypeId__ for the value's class, and further headers for the element and key types of collections and maps. When the serializer is configured through properties, the mapper is Spring Kafka's own (JacksonUtils.enhancedObjectMapper()), not the ObjectMapper bean Spring Boot configures for web requests. Settings such as spring.jackson.* therefore do not apply to your events. To use a customised mapper, build the serializer in code and pass it to the producer factory.
Java records work as event types without annotations: Jackson writes the components as fields and reads them through the canonical constructor. Records are a good fit because an event is an immutable fact.
How JsonDeserializer chooses the target type
- If
spring.json.use.type.headersis true (the default) and the record has a type header, the header decides. Its value is first looked up inspring.json.type.mapping; if it is not a mapped token it is treated as a class name, which must be in a trusted package. - Otherwise the configured default type is used:
spring.json.value.default.type(orspring.json.key.default.typefor keys). - If there is neither a usable header nor a default type, deserialisation fails with a message that no type information is available.
Two consequences are easy to miss. A default type does not override a header: if the producer still sends headers, a consumer that only sets a default type keeps failing until it also sets spring.json.use.type.headers=false. And the deserializer's mapper does not fail on unknown JSON properties (Spring Kafka's mapper disables FAIL_ON_UNKNOWN_PROPERTIES), so a consumer record with fewer components than the producer's simply ignores the extra fields. That behaviour is what lets a producer add fields without breaking readers.
Three ways to agree on the type
| Approach | Producer | Consumer | Coupling |
|---|---|---|---|
| Class name in headers (the default) | Nothing to set | The producer's package in spring.json.trusted.packages and a class with the same name | Services share Java class and package names |
| No type information | spring.json.add.type.headers=false | spring.json.value.default.type set to its own class, spring.json.use.type.headers=false | Field names only. Works for one type per topic (or per listener) |
| Type mapping | spring.json.type.mapping=order-placed:com.example.orders.events.OrderPlaced | spring.json.type.mapping=order-placed:com.example.payments.events.OrderPlaced | A logical name per event type, plus field names. Works for several types per topic |
With a type mapping the consumer's class is taken from its own configuration, so no trusted package entry is needed for it. A token the consumer has not mapped is treated as a class name and rejected. In practice this means that when a producer starts publishing a new event type on a shared topic, every consumer of that topic must already map or deliberately handle it; see the pitfalls.
ErrorHandlingDeserializer
A deserializer runs inside the Kafka consumer while poll builds its result. If it throws, poll throws, the consumer's position has not moved, and the next poll meets the same record. The listener container never sees a record it could hand to an error handler, so it can neither skip nor dead-letter it. The partition is blocked and the log fills with the same exception.
ErrorHandlingDeserializer wraps the real deserializer (the delegate). When the delegate throws, the wrapper catches the exception, returns a null value and stores a DeserializationException, containing the cause and the raw bytes, in a record header (springDeserializerExceptionValue, or springDeserializerExceptionKey for keys). poll succeeds. For a record listener the container detects the header, does not call your method, and passes the record to its error handler, which can log, skip or publish the original bytes to a dead letter topic.
- Wrap both key and value deserializers if either can fail.
StringDeserializerandByteArrayDeserializercannot fail on content, so they need no wrapper. - With a batch listener the container still calls your method; the failed record is in the list with a null value and the exception in its header. Your code must check.
- A dead letter record for a deserialisation failure has a
byte[]value. The template used for dead letters needs a serializer that passes raw bytes through; this is set up in Spring error handling. - A null value is also what a tombstone on a compacted topic looks like. The difference is the exception header.
Avro and a schema registry, in concept
JSON carries field names in every record and has no enforced schema: nothing stops a producer from renaming a field on Friday afternoon. Avro is a binary format defined by a schema (record name, fields, types, defaults). The bytes contain only values, in schema order, so they are compact and cannot be read without the schema that wrote them. A schema registry is a separate service that solves the question of where the reader gets that schema.
- The registry stores schemas under subjects. With the common default naming, the value schema of topic
orders.placedlives under the subjectorders.placed-value. Each change is a new version of the subject, and every distinct schema has a numeric schema id. - The producer's serializer looks up (or registers) the schema, receives its id and writes the record value as a small prefix containing the id, followed by the Avro binary data. In Confluent's format the prefix is one version byte (0) and the four-byte id.
- The consumer's deserializer reads the id, fetches that schema from the registry once and caches it, and decodes the data. It can then convert from the writer's schema to the reader's own schema using Avro's resolution rules.
- When a new schema version is registered, the registry runs a compatibility check against earlier versions and rejects a change that breaks the configured rule.
The compatibility rule decides what may change and who must deploy first. Confluent's registry defaults to BACKWARD: a reader using the new schema can read data written with the previous one. Under that rule you may remove a field or add one with a default, and you upgrade consumers before producers. The modes, and what each allows, are covered in schema registry and evolution; the lab schema evolution lets you break and fix a consumer with a schema change.
JSON with Spring's serializers
- No extra infrastructure; readable with any console consumer
- Contract is by convention: field names and tolerant readers
- A breaking change is found by a consumer at runtime
- Larger records: field names repeat in every value
- Good for a handful of services owned by a few teams
Avro with a registry
- A registry to run, secure and keep available for producers and consumers
- Contract is a versioned schema that tools can check
- A breaking change is rejected when the schema is registered
- Compact binary; needs tooling to read
- Good when many teams share topics that live for years
Architecture
Drawing diagram…
A flowchart. In the order service an OrderPlaced record goes through JsonSerializer with type headers off and becomes a record whose value is JSON bytes and whose key is the order id, written to the topic orders.placed. In the payment service the consumer's poll passes the bytes to ErrorHandlingDeserializer, which delegates to JsonDeserializer with a default type. If parsing succeeds, the payment service's own OrderPlaced record is passed to the KafkaListener method. If it fails, the wrapper returns a null value with an exception header containing the raw bytes, the listener container skips the listener and calls the error handler, which sends the original bytes to a dead letter topic.
flowchart LR
Ev["OrderPlaced record in the order service"] --> JS["JsonSerializer, type headers off"]
JS --> Rec["Record value is JSON bytes, key is the order id"]
Rec --> Topic["Topic orders.placed"]
Topic --> Poll["Consumer poll in the payment service"]
Poll --> EHD["ErrorHandlingDeserializer"]
EHD --> JD["JsonDeserializer with default type"]
JD --> Ok{"Parsed"}
Ok -->|yes| Obj["OrderPlaced record of the payment service"]
Obj --> Lis["KafkaListener method"]
Ok -->|no| Hdr["Null value plus exception header with the raw bytes"]
Hdr --> Cont["Listener container skips the listener"]
Cont --> EH["Error handler"]
EH --> DLT["Dead letter topic gets the original bytes"]- The two
OrderPlacedboxes are different classes in different packages. The only thing they share is the JSON between them. - Deserialisation happens inside
poll, on the consumer thread, before the container sees the record. That is why the failure path has to start inside the deserializer. - On failure nothing is lost: the raw bytes travel in the exception header and can be published unchanged to a dead letter topic for inspection and replay.
- The error handler and the dead letter topic are container configuration, covered in the next lesson. Without them the default handler logs and skips the record.
Message flow
Drawing diagram…
A sequence diagram. The listener container calls poll. The Kafka consumer fetches from offset 70 and receives records 70 and 71 as bytes. For record 70 the ErrorHandlingDeserializer delegates to the JsonDeserializer and gets an OrderPlaced object. For record 71, which contains truncated JSON, the delegate throws a parse exception and the wrapper returns a null value with the exception stored in a header. Poll returns two records. The container calls the payment listener for offset 70. For record 71 it finds the exception header and calls the error handler, which recovers by publishing to the dead letter topic. The container then commits offset 72.
sequenceDiagram participant B as Broker, orders.placed-3 participant C as KafkaConsumer in poll participant E as ErrorHandlingDeserializer participant J as JsonDeserializer participant K as Listener container participant L as PaymentListener participant H as Error handler K->>C: poll C->>B: fetch from offset 70 B-->>C: records 70 and 71 as bytes C->>E: deserialize value of 70 E->>J: delegate J-->>E: OrderPlaced C->>E: deserialize value of 71, truncated JSON J-->>E: delegate throws a parse exception E-->>C: null value, exception stored in a header C-->>K: two records K->>L: onOrderPlaced for offset 70 L-->>K: return K->>H: record 71 has a deserialisation exception header H-->>K: recovered, published to the dead letter topic K->>B: commit offset 72
- Both records are deserialised before the listener is called for the first one. Deserialisation is part of
poll. - The wrapper converts an exception into data.
pollreturns normally, so the consumer's position moves past record 71. - The listener is never called for record 71. The error handler receives it with the original bytes available.
- After recovery the commit covers offset 71 as well. Without the wrapper,
pollwould throw at record 71 every time and offset 72 would never be reached.
In a real system
In the example project the order service publishes OrderPlaced to orders.placed, keyed by order id. Three services read it: payment, inventory and notification. Each needs different fields: payment wants the amount and currency, inventory the order lines, notification the customer id.
- The topic carries one event type, so the producer sends plain JSON without type headers.
- Each consumer declares its own record with only the fields it uses and names it as the default type. Unknown fields are ignored, so the order service can add a field without coordinating a release.
- Every consumer wraps its value deserializer in
ErrorHandlingDeserializer. One malformed record costs one dead letter, not a stopped partition. - The contract is written down as an example JSON document and a list of rules in the order service's repository: fields are only added, never renamed or retyped; new fields are optional for readers.
A second topic, payments.events, carries two types, PaymentCompleted and PaymentFailed, keyed by order id so that the outcome events of one order stay in order. Here the consumer has to know which type each record is, so the payment service maps logical names (payment-completed, payment-failed) to its classes and the notification service maps the same names to its own.
A typical point at which such a system moves to Avro and a registry is when topics are shared beyond the teams that talk to each other daily: an analytics platform reads everything, or events are retained for years. Then nobody can know all readers, and an automated compatibility check at registration time replaces the written rules.
Java and Spring Boot
Both sides of the two topics from the example: one event type without type information, and two event types with a type mapping. An Avro variant follows, clearly marked as depending on a registry product. Written against the Spring for Apache Kafka 3.3 serialisation reference; not compiled in this environment.
package com.example.orders.events;import java.time.Instant;import java.util.List;/** Published to orders.placed. The record key is orderId. */public record OrderPlaced( String eventId, String orderId, String customerId, long amountMinor, String currency, List<Line> lines, Instant occurredAt) { public record Line(String sku, int quantity) { }}Why it is written this way
- A record is the event DTO: immutable, no framework annotations, serialised by Jackson as one JSON field per component.
- It is a class made for the wire, not a JPA entity or a domain object. Publishing entities leaks internal structure and makes every refactoring a contract change.
- Money is an integer amount in minor units plus a currency code. Timestamps are
Instant. Check once how your mapper writes dates (as text or as a number) and write that into the contract; consumers in other languages need to know. eventIdidentifies this event for deduplication and tracing;orderIdis the business key and the Kafka key.
spring: kafka: producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: "[spring.json.add.type.headers]": falseWhy it is written this way
spring.json.add.type.headers: falsekeeps the Java class name out of the record. The value is plain JSON that any consumer in any language can read.- Keys are strings. A key that is a plain string can be read by every tool, and hashing it is the same in every client.
- These are serializer properties passed through the producer's
propertiesmap. They only take effect when the serializer is created from configuration, not when you pass a serializer instance to the factory.
package com.example.payments.events;/** * The payment service's view of the orders.placed event: only the fields it uses. * Other fields in the JSON are ignored. */public record OrderPlaced( String eventId, String orderId, long amountMinor, String currency) { public OrderPlaced { if (orderId == null || orderId.isBlank()) { throw new IllegalArgumentException("orderId is required"); } if (currency == null || currency.length() != 3) { throw new IllegalArgumentException("currency must be a three-letter code"); } }}Why it is written this way
- A different class in a different package from the producer's, with a subset of the fields. This is the tolerant reader: the producer can add fields freely.
- Unknown JSON properties are ignored because Spring Kafka's default mapper disables
FAIL_ON_UNKNOWN_PROPERTIES. If you supply your ownObjectMapper, keep that setting. - The compact constructor rejects JSON that parses but is not a usable event. Jackson calls the constructor, so the exception surfaces as a deserialisation failure, which
ErrorHandlingDeserializerturns into a handled record. - A missing numeric field would become 0 without complaint. If absent and zero must be distinguished, use
Longand check for null.
spring: kafka: consumer: group-id: payment-service key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: "[spring.deserializer.value.delegate.class]": org.springframework.kafka.support.serializer.JsonDeserializer "[spring.json.value.default.type]": com.example.payments.events.OrderPlaced "[spring.json.use.type.headers]": falseWhy it is written this way
- The configured value deserializer is the wrapper. The real one is named by
spring.deserializer.value.delegate.class, and the wrapper creates and configures it with the same properties. spring.json.value.default.typetellsJsonDeserializerwhich class to create. The package of the default type is trusted automatically.spring.json.use.type.headers: falsemakes the consumer ignore type headers even if some producer still sends them. Without it, a header would win over the default type.- No
spring.json.trusted.packagesline is needed, and certainly not*. - The key uses
StringDeserializer, which cannot fail on content, so it is not wrapped. A JSON or Avro key would getspring.deserializer.key.delegate.classin the same way. - A listener on another topic with another type overrides the default type per listener:
@KafkaListener(properties = "spring.json.value.default.type=com.example.payments.events.RefundRequested").
spring: kafka: producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: "[spring.json.type.mapping]": >- payment-completed:com.example.payments.events.PaymentCompleted, payment-failed:com.example.payments.events.PaymentFailedWhy it is written this way
- Type headers stay on (the default), but with a mapping the header value is the token (
payment-completed), not the class name. - The tokens are part of the contract, like topic names. Choose names that describe the event, not the class.
- A class that is sent but not listed in the mapping falls back to its fully qualified class name in the header. Add every event type of the topic to the mapping.
- The YAML folded scalar joins the lines with spaces; the reference's own example has spaces after the commas, and class names are trimmed.
spring: kafka: consumer: group-id: notification-service key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: "[spring.deserializer.value.delegate.class]": org.springframework.kafka.support.serializer.JsonDeserializer "[spring.json.type.mapping]": >- payment-completed:com.example.notifications.events.PaymentCompleted, payment-failed:com.example.notifications.events.PaymentFailedWhy it is written this way
- The same tokens, mapped to the notification service's own classes. Neither service knows the other's package names.
- Type headers are used (the default), because here the header is what distinguishes the two event types.
- The reference notes that mapped classes on both sides must be compatible: the same JSON must fit both.
- A record with a token this service has not mapped fails to deserialise and goes to the error handler through
ErrorHandlingDeserializer. Decide in the error handling configuration whether unknown types are skipped quietly or dead-lettered.
package com.example.notifications.kafka;import org.springframework.kafka.annotation.KafkaHandler;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;import com.example.notifications.events.PaymentCompleted;import com.example.notifications.events.PaymentFailed;import com.example.notifications.service.NotificationService;@Component@KafkaListener(id = "notification-payments", groupId = "notification-service", topics = "payments.events")public class PaymentEventsListener { private final NotificationService notifications; public PaymentEventsListener(NotificationService notifications) { this.notifications = notifications; } @KafkaHandler public void onCompleted(PaymentCompleted event) { notifications.sendReceipt(event.orderId(), event.amountMinor(), event.currency()); } @KafkaHandler public void onFailed(PaymentFailed event) { notifications.sendPaymentFailed(event.orderId(), event.reason()); } @KafkaHandler(isDefault = true) public void onOther(Object other) { // A type that was mapped and deserialised but has no handler here. }}Why it is written this way
@KafkaListeneron the class with@KafkaHandlermethods dispatches by the type of the deserialised payload. This requires the deserializer to have produced the right class already, which is what the type mapping does.- The default handler catches payload types without a specific method. It does not catch unmapped tokens: those fail earlier, in the deserializer.
- Both handlers run on the same consumer threads and in partition order, so
PaymentFailedfollowed byPaymentCompletedfor the same order is seen in that order.
{ "type": "record", "name": "OrderPlaced", "namespace": "com.example.orders.avro", "doc": "Published to orders.placed when an order has been accepted. Key: orderId.", "fields": [ { "name": "eventId", "type": "string" }, { "name": "orderId", "type": "string" }, { "name": "customerId", "type": "string" }, { "name": "amountMinor", "type": "long" }, { "name": "currency", "type": "string" }, { "name": "occurredAt", "type": { "type": "long", "logicalType": "timestamp-millis" } }, { "name": "couponCode", "type": ["null", "string"], "default": null } ]}Why it is written this way
- The schema is the contract, kept in version control. A build plugin generates the Java class from it; the class is an output, not the source of truth.
couponCodeshows how a field is added compatibly: a union withnulland a default. A reader with this schema can decode older data that lacks the field, because the default fills the gap.- Adding a field without a default would be rejected under backward compatibility, because old data could not be read with the new schema.
- The
namespacebecomes the Java package of the generated class, but readers resolve by schema, not by Java class name.
spring: kafka: producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer properties: "[schema.registry.url]": http://localhost:8081 "[auto.register.schemas]": false consumer: key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: "[spring.deserializer.value.delegate.class]": io.confluent.kafka.serializers.KafkaAvroDeserializer "[schema.registry.url]": http://localhost:8081 "[specific.avro.reader]": trueWhy it is written this way
KafkaAvroSerializerandKafkaAvroDeserializerare Confluent classes from thekafka-avro-serializerartifact (Apache License 2.0 according to the project README). They are not on the classpath through Spring Boot; you add the dependency and choose its version yourself, matching your registry.- They need a running schema registry at
schema.registry.url. The registry is a separate service to deploy; Confluent's server is under the Confluent Community License, Apicurio Registry under the Apache License 2.0. auto.register.schemas: falsestops applications from registering schemas at runtime. Schemas are then registered by the build pipeline, where a failed compatibility check blocks a release instead of a running producer. The serializer's default is true.specific.avro.reader: truemakes the deserializer return the generated class. The default returns aGenericRecord.- The Spring side is unchanged:
KafkaTemplate,@KafkaListenerandErrorHandlingDeserializerwork the same with anySerializerandDeserializer. A record whose schema id is unknown, or whose bytes are not in this format at all, becomes a handled deserialisation failure. - The registry is now a runtime dependency. Serializers cache schemas, so a short registry outage mostly affects new producers, new consumers and new schema versions.
Configuration
The spring.json.* and spring.deserializer.* keys and their defaults are from the Spring for Apache Kafka 3.3 serialisation reference and constants. They are set under spring.kafka.producer.properties or spring.kafka.consumer.properties. The last three keys belong to Confluent's serializers.
spring.json.add.type.headers (producer)Default true
- What it does
- Whether
JsonSerializeradds headers naming the type of the value (__TypeId__) and of container contents. - When to change it
- Set false for topics with one event type read by other services. Keep true together with a type mapping for topics with several types.
spring.json.type.mapping (producer and consumer)Default empty
- What it does
- Comma-separated
token:classNamepairs. The producer writes the token instead of the class name; the consumer maps the token to its own class. - When to change it
- Use it whenever type headers are needed across services. Treat the tokens as part of the public contract.
spring.json.value.default.type / spring.json.key.default.type (consumer)Default none
- What it does
- The class
JsonDeserializercreates when no type header is present or headers are ignored. - When to change it
- Set it to the consumer's own class. Combine with
spring.json.use.type.headers=falseso a stray header cannot override it.
spring.json.use.type.headers (consumer)Default true
- What it does
- Whether type headers, when present, decide the target type in preference to the default type.
- When to change it
- Set false on consumers of single-type topics. Leave true when you rely on a type mapping.
spring.json.trusted.packages (consumer)Default java.util, java.lang
- What it does
- Package patterns from which classes named in type headers may be instantiated.
*trusts everything. - When to change it
- Needed only when class names come from headers. Prefer a default type or mapping so you never list foreign packages, and avoid
*.
spring.json.remove.type.headers (consumer)Default true
- What it does
- Whether
JsonDeserializerremoves the type headers from the record after using them. - When to change it
- Set false only if later code, such as forwarding to another topic, needs the original headers.
spring.deserializer.value.delegate.class / spring.deserializer.key.delegate.class (consumer)Default none
- What it does
- The real deserializer that
ErrorHandlingDeserializerwraps, as a class name. - When to change it
- Required whenever
ErrorHandlingDeserializeris the configured deserializer. Wrap every deserializer that parses content.
spring.deserializer.value.function (consumer)Default none
- What it does
- A
Function<FailedDeserializationInfo, T>class that supplies a substitute value when deserialisation fails. The listener is then called with that value. - When to change it
- Rarely needed. The default behaviour (null value, error handler invoked) is usually what you want.
schema.registry.url (Confluent serializers)Default none
- What it does
- Address of the schema registry the Avro, Protobuf or JSON Schema serializers talk to. Not an Apache Kafka or Spring setting.
- When to change it
- Required with those serializers. Producers and consumers both need network access and credentials for the registry.
auto.register.schemas (Confluent serializers)Default true
- What it does
- Whether the serializer registers the schema of the object it is sending if the registry does not have it.
- When to change it
- Convenient in development. Set false in production and register schemas from the pipeline, so applications need no write access to the registry.
specific.avro.reader (Confluent Avro deserializer)Default false
- What it does
- Whether the deserializer returns generated classes (
SpecificRecord) instead ofGenericRecord. - When to change it
- Set true when your listeners take generated Avro classes as parameters.
Hands-on exercise
Read the contract off the wire, then break and repair a consumer
Look at exactly what a producer puts on a topic with and without type headers, see what one unparseable record does to a consumer, and then try a schema change in the schema evolution lab.
Steps · 0 of 7 done
What you should see
The first record carries __TypeId__ with the producer's fully qualified class name; the second has no type header. A consumer with only a default type fails on the first record with a message that the class is not in the trusted packages, because the header wins, and reads both once headers are ignored. With plain JsonDeserializer the broken record makes the consumer log the same deserialisation error again and again without advancing; with ErrorHandlingDeserializer the error is logged for that record, the record is skipped by the default error handler, and consumption continues with the next one.
What can go wrong
Class names in headers coupling services
- Symptom
- A consumer in another service logs that the class
com.example.orders.events.OrderPlacedis not in the trusted packages, for every record. - Cause
- The producer's
JsonSerializeradds the Java class name as a type header by default, and the consumer'sJsonDeserializerfollows the header by default. - Fix
- For a single-type topic, stop sending headers (
spring.json.add.type.headers=false) and give each consumer a default type withspring.json.use.type.headers=false. For several types, usespring.json.type.mappingon both sides. Do not answer withtrusted.packages=*or a copied class.
A poison record without ErrorHandlingDeserializer
- Symptom
- One partition's lag grows without limit while the others are fine. The log repeats the same
SerializationExceptionfor the same partition and offset many times a second. - Cause
- The deserializer throws inside
poll. The position does not advance, the container cannot hand the record to an error handler, and the nextpollfails on the same record. - Fix
- Configure
ErrorHandlingDeserializerwith the real deserializer as delegate, for key and value. To get past the record already blocking the group, deploy that change; as a last resort move the group's offset past it withkafka-consumer-groups.sh --reset-offsetswhile the group is stopped. The Break card The message that kills its reader shows the pattern.
Default type set but headers still winning
- Symptom
- After setting
spring.json.value.default.type, the consumer still fails with the trusted packages error. - Cause
spring.json.use.type.headersdefaults to true, so a__TypeId__header sent by the producer takes precedence over the default type.- Fix
- Set
spring.json.use.type.headers=falseon the consumer, or switch off the headers at the producer. Check what is really on the topic withprint.headers=true.
A renamed or retyped field
- Symptom
- After a producer release, consumers do not fail, but amounts are 0 or customer ids are null, and wrong data is written downstream.
- Cause
- JSON has no schema. The consumer's mapper ignores the unknown new field name and fills the missing old one with null or zero.
- Fix
- Only add fields. To rename, add the new field, keep writing the old one until every consumer has moved, then remove it. Validate required fields in the consumer's record constructor. For enforcement, move to a schema format with registry compatibility checks.
A new event type on a shared topic
- Symptom
- After the payment service starts publishing
PaymentRefundedtopayments.events, the notification service sends every such record to its dead letter topic, or logs errors for them. - Cause
- The consumer's type mapping has no entry for the new token, so the deserializer treats it as an untrusted class name and fails.
- Fix
- Agree beforehand how consumers treat unknown types. Either add the mapping in consumers before the producer starts sending, or configure the error handler to skip this specific failure quietly. A new topic for a new kind of event avoids the question.
Batch listeners and failed deserialisation
- Symptom
- A batch listener throws
NullPointerExceptiononrecord.value()now and then. - Cause
- With
ErrorHandlingDeserializer, a failed record is still part of the batch, with a null value and the exception in a header. The container does not filter it for batch listeners. - Fix
- Check for null values and the deserialisation exception header in batch listeners, and report the failing index to the error handler as described in the reference.
Expecting Boot's ObjectMapper settings in events
- Symptom
- Dates in events are formatted differently from the REST API, or a custom Jackson module has no effect on Kafka messages.
- Cause
- A
JsonSerializercreated from properties uses Spring Kafka's own mapper, not theObjectMapperbean configured throughspring.jackson.*. - Fix
- Build
JsonSerializerandJsonDeserializerin code with the mapper you want and pass them to the producer and consumer factories. Note that configuration properties are then not applied to instances you have configured with setters.
How to troubleshoot
Need to know what a producer really writes: headers, key and value
Check
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.placed --from-beginning --max-messages 5 --property print.key=true --property print.headers=true
- How to read it
- A
__TypeId__header with a Java class name means type headers are on without a mapping. A short token means a mapping is in use. No header means the consumer must supply a default type. The value shows the real field names and date format.
Consumer logs The class '...' is not in the trusted packages
Check
The consumer's `spring.json.use.type.headers`, `spring.json.value.default.type` and `spring.json.type.mapping`, against the header seen on the wire
- How to read it
- The header names a class the consumer does not trust. Decide which of the three approaches the topic uses and configure the consumer for it; adding the producer's package to the trusted list only works if the consumer also has that class.
Consumer logs No type information in headers and no default type provided
Check
Whether the record has a type header, and whether the default type property reached the deserializer (the `ConsumerConfig values` block does not list spring.json keys; check the properties map in your YAML for indentation and bracket notation)
- How to read it
- The producer sends no headers and the consumer has no default type, often because the property sits under the wrong YAML parent or the dotted key was not written in brackets.
One partition stuck, the same deserialisation error repeating
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- A
CURRENT-OFFSETthat does not move on one partition whileLOG-END-OFFSETgrows, together with the repeating error, is a poison record withoutErrorHandlingDeserializer. The stuck offset is the record to inspect.
Need to look at one specific record that fails
Check
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.placed --partition 3 --offset 71 --max-messages 1 --property print.headers=true
- How to read it
- Shows the raw value and headers of that record. Truncated JSON, a different event type, or binary data on a JSON topic are the usual findings. Binary output beginning with a zero byte and four more bytes before the data suggests a registry-framed Avro record.
With Avro: the deserializer fails with a schema not found or unknown magic byte error
Check
The registry's REST API, for example curl http://localhost:8081/subjects and curl http://localhost:8081/subjects/orders.placed-value/versions
- How to read it
- A missing subject or id means producer and consumer use different registries or the schema was never registered. An unknown magic byte means the record was not written by a registry serializer at all, for example plain JSON sent to an Avro topic.
Key takeaways
- The bytes on a topic are a contract between programs released at different times. Keep Java class and package names out of it.
JsonSerializeradds the class name as a type header by default andJsonDeserializerfollows it by default. For one type per topic switch both off and use a default type; for several types use a type mapping.- Trusted packages protect against records choosing arbitrary classes. Designs with a default type or mapping never need
*. - Deserialisation happens inside
poll. WithoutErrorHandlingDeserializer, one unparseable record blocks its partition; with it, the record reaches the error handler with its raw bytes. - Events are records made for the wire. Add fields, never rename or retype them, and let consumers ignore what they do not use.
- A schema registry stores versioned schemas per subject, records carry a schema id, and compatibility checks reject breaking changes at registration time.
- Registries and their serializers are separate products, not part of Apache Kafka or Spring. Check their licences.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
The order service publishes to orders.placed with JsonSerializer defaults. A new shipping service, in package com.example.shipping, configures JsonDeserializer with spring.json.value.default.type=com.example.shipping.events.OrderPlaced and nothing else. Every record fails with: The class 'com.example.orders.events.OrderPlaced' is not in the trusted packages.
Which change fixes the shipping service without tying it to the order service's classes?
Scenario 2 of 3
A consumer uses JsonDeserializer directly (no wrapper), AckMode RECORD and a DefaultErrorHandler with a dead letter topic. At 02:10 an upstream bug writes one record with truncated JSON to partition 4 of orders.placed. By 02:30 lag on partition 4 is 18,000 and rising, other partitions are at zero, and nothing has arrived on the dead letter topic.
Why did the dead letter topic not receive the record, and what prevents a repeat?
Scenario 3 of 3
Twelve services owned by five teams read customers.updated, published as JSON without type headers. Twice this quarter a producer release renamed a field; consumers kept running and wrote nulls into their databases for hours before anyone noticed. The team proposes moving the topic to Avro with a schema registry using BACKWARD compatibility.
What does that change actually buy for this problem?
References
References: Spring for Apache Kafka 3.3 reference: Serialization, Deserialization, and Message Conversion (JsonSerializer, type mapping, ErrorHandlingDeserializer) ↗ · Spring for Apache Kafka 3.3 Javadoc: constant values (spring.json.* and spring.deserializer.* keys, header names) ↗ · Spring for Apache Kafka 3.3 reference: @KafkaListener on a Class (@KafkaHandler) ↗ · Spring Boot 3.5 reference: Apache Kafka Support, Additional Kafka Properties (JsonDeserializer properties in application.yml) ↗ · Apache Kafka 3.9: Consumer configs (key.deserializer, value.deserializer) ↗ · Apache Avro specification: schema declaration and schema resolution ↗ · Confluent documentation: Formats, serializers and deserializers (wire format, subject naming, auto.register.schemas) ↗ · Confluent documentation: Schema evolution and compatibility (BACKWARD is the default) ↗ · confluentinc/schema-registry README: licence of the server and of the client and serializer modules ↗ · Apicurio Registry: project site (Apache License 2.0, Confluent-compatible API) ↗ · Report a problem