Skip to content

Path 9 · lesson 6 of 6

Advanced27 minJava 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 · kafka-clients 3.9 (the versions Spring Boot 3.5 manages: spring-kafka 3.3.x, kafka-clients 3.9.x, Testcontainers 1.21.x) · tests run against an Apache Kafka 4.x broker image

Testing and observing Spring Kafka applications

Test listeners as plain classes and against a real broker with Testcontainers, then make them observable: Micrometer metrics, traces across topics and safe, searchable logs.

What you will learn

  • Split Kafka code so the business decision can be unit-tested without Spring or a broker
  • Choose between @EmbeddedKafka and Testcontainers, knowing which broker each one runs in this version
  • Write an integration test with a KafkaContainer, @ServiceConnection and Awaitility that does not depend on timing
  • Test the paths that hurt in production: a duplicate delivery, a bad record and a failing dependency
  • Find the Micrometer meters Spring Boot exposes for Kafka clients, templates and listeners, and enable observations so traces cross a topic
  • Put topic, partition, offset and key into every log line of a listener, and keep payloads and personal data out

Before you start

The concept

A Kafka listener is asynchronous in both directions. In a test, nothing returns when the work is done: a record goes in, and some time later a row, another record or a log line appears. In production it is the same problem with higher stakes: when a customer asks why an order was not charged, the only evidence is what the service measured and logged while it handled one record among millions. Testing and observability are two answers to the same question: what did this code do with this record?

For tests, the answer is layered. Each layer proves something the others cannot, and the cheap layers should carry most of the cases.

LayerWhat runsWhat it provesWhat it cannot prove
Plain unit testYour listener and processor classes, constructed with new. No Spring, no broker.Business decisions: valid or invalid, which exception type, which calls are made. Milliseconds per test.Anything about serialization, annotations, offsets, retries or configuration.
@EmbeddedKafkaThe Spring context and a broker started inside the test JVM from the Kafka libraries on the test classpath.Wiring: deserializers, the error handler, the dead-letter path, topic names. No Docker needed.Behaviour of the broker version you run in production. Under Spring Boot 3.5 the embedded broker is built from the 3.9 Kafka jars.
TestcontainersThe Spring context and a real broker in a container, at the image version you choose, plus real dependencies such as PostgreSQL.The service as deployed: client against a 4.x broker, transactions, the database, idempotency across both.Multi-broker failures, production load, network partitions. Those need a staging cluster or fault-injection tooling.

For observability, Spring Boot already produces most of the raw material. With Actuator on the classpath it binds the Kafka clients' own metrics to Micrometer, times every listener invocation and every template send, and, if you enable observations and add a tracing bridge, carries a trace across the topic in a record header. What it cannot do for you is decide what your log lines say.

Why it exists

Before containers were practical in a build, teams tested Kafka code in one of two unsatisfying ways. Either everything was mocked, and the first real record in the test environment failed to deserialize; or a shared test cluster was used, and tests failed because someone else's records were in the topic. The embedded broker fixed isolation by starting a broker per test class inside the JVM. Testcontainers fixed fidelity by starting the same image you deploy.

The observability story has the same shape. A service that only logs processing failed forces you to reproduce a problem to understand it, and an asynchronous flow through four services is hard to reproduce. Three things changed that, and each answers a different question:

SignalQuestion it answersExample for the payment service
MetricsIs the flow healthy right now, and since when not?Listener failures per minute went from 0 to 40 at 14:02; the consumer's maximum lag is rising.
TracesWhere did this one business operation spend its time, and where did it stop?The order's trace shows the order service's send, 3 seconds in the topic, then a payment listener span that ended in an error.
LogsWhat exactly happened to this record?orders.order-placed-3@1041 key=7c1e... dead-lettered: provider returned 422.

Each signal needs the others. A metric tells you to look; a trace or the record's coordinates tell you where; a log line tells you what. The coordinates of a Kafka record (topic, partition, offset) are the one identifier that every tool in the chain understands, which is why they belong in every log line a listener writes.

How it works inside

The embedded broker in this version

Spring for Apache Kafka 3.3 ships two embedded brokers in spring-kafka-test: EmbeddedKafkaZKBroker, which starts an embedded ZooKeeper, and EmbeddedKafkaKraftBroker, which runs KRaft in combined controller and broker mode. The reference states that the ZooKeeper one is still the default for @EmbeddedKafka in this line; set kraft = true on the annotation (available since 3.2) to get the KRaft one. Kafka 4.0 removed ZooKeeper mode, and the Spring for Apache Kafka 4.x reference lists only the KRaft broker, so tests that rely on the old default will need the attribute when you upgrade.

Since 3.0.10 the embedded broker publishes its address as spring.kafka.bootstrap-servers, so a @SpringBootTest connects to it without extra properties. It creates the topics named in the annotation, with 2 partitions each unless you set partitions.

Testcontainers and Spring Boot

  • The current class is org.testcontainers.kafka.KafkaContainer, for the apache/kafka and apache/kafka-native images. org.testcontainers.kafka.ConfluentKafkaContainer is for confluentinc/cp-kafka (7.4.0 and later). The older org.testcontainers.containers.KafkaContainer is deprecated.
  • Spring Boot 3.5 manages Testcontainers 1.21.x, where the module's coordinates are org.testcontainers:kafka. In Testcontainers 2.x the artifact is named testcontainers-kafka. Let Boot's dependency management pick the version.
  • @ServiceConnection on the container field (from spring-boot-testcontainers) makes Boot create a KafkaConnectionDetails bean from the running container. Connection details take precedence over spring.kafka.bootstrap-servers, and Boot supports KafkaContainer, ConfluentKafkaContainer and RedpandaContainer this way.
  • The container is a single broker in KRaft mode. The Kafka module sets the replication factor of the internal offsets and transaction topics to 1, so consumer groups and transactions work. Your own topics still cannot have three replicas there.

@EmbeddedKafka

  • Starts in the test JVM; no Docker on the build machine.
  • Broker version is the Kafka version on the test classpath, not the one in production.
  • ZooKeeper-based by default in Spring Kafka 3.3; kraft = true switches.
  • Good for wiring tests of one service: serialization, error handling, dead letters.
  • Use a different topic per test, and @DirtiesContext, as the reference recommends, to avoid shutdown races.

Testcontainers

  • Starts a container; needs Docker and takes a few seconds per container.
  • You pick the image tag, so tests can run against the broker version you deploy.
  • KRaft single node; the same image family as production.
  • Good for flows that include a database, transactions or several services' contracts.
  • Share the container between test classes (a static field in a base class, or a container bean) so the cost is paid once.

Waiting without sleeping

Three tools cover nearly every wait. Awaitility (await().atMost(...).untilAsserted(...)) polls an assertion until it passes or the time is up; it comes with spring-boot-starter-test. KafkaTestUtils.getSingleRecord(consumer, topic, timeout) and getRecords poll a test consumer for output records. ContainerTestUtils.waitForAssignment(container, partitions) blocks until a listener container has its partitions, which matters when the listener's auto.offset.reset is latest (the Kafka default): a record sent before the assignment completes is never delivered to a new group.

A harder case is proving that something did not happen, such as a second payment for a duplicate event. Waiting a fixed time proves nothing. Wait for a positive signal that the duplicate was consumed, for example the group's committed offset reaching the end of the partition (KafkaTestUtils.getCurrentOffset), and only then assert on the count.

What Spring Boot measures without any code

MeterSourceTagsRead it for
spring.kafka.listener (timer)Listener container, when Micrometer and a single MeterRegistry are presentname (container bean name, with -n per thread), result (success or failure), exceptionProcessing time per record and the failure rate of a listener
spring.kafka.template (timer)KafkaTemplatename (template bean name), result, exceptionSend latency and send failures as the application sees them
kafka.consumer.fetch.manager.records.lag.max (gauge)The Kafka consumer's own metric records-lag-max, bound by Boot's MicrometerConsumerListenerclient.id, spring.id, kafka.versionThe largest lag among the partitions this consumer instance holds
kafka.consumer.coordinator.rebalance.total, kafka.consumer.coordinator.failed.rebalance.totalConsumer metrics rebalance-total and failed-rebalance-totalas aboveRebalance storms after a deploy or when processing is too slow
kafka.producer.record.error.totalProducer metric record-error-total, bound by MicrometerProducerListeneras aboveRecords the producer gave up on after its own retries
Client meters are the Kafka metric names with the prefix kafka., the group without -metrics, and dashes turned into dots. Open /actuator/metrics to list what your client version emits.

Observations and tracing

Two Boot properties, both false by default, switch the template and the listener containers from plain timers to Micrometer observations: spring.kafka.template.observation-enabled and spring.kafka.listener.observation-enabled. An observation produces the timer (the names stay spring.kafka.template and spring.kafka.listener, with an extra .active long task timer) and, when a tracing bridge such as micrometer-tracing-bridge-otel is on the classpath, a span of the same name. The tags change to messaging conventions, for example messaging.kafka.consumer.group, messaging.source.name and spring.kafka.listener.id on the listener.

Propagation works through record headers. The template's observation writes the trace context into the headers of the outgoing record; with Boot's default W3C propagation that is a traceparent header. The listener container's observation reads it from the incoming record and continues the same trace. Boot also puts traceId and spanId into the logging MDC and prints them as a correlation id. Two limits are stated in the reference: observations are not supported for batch listeners, and Boot samples only 10% of traces unless you change management.tracing.sampling.probability.

Logging a record

Log

  • Topic, partition and offset of the record being handled.
  • The key, if it is an opaque identifier such as an order id.
  • Your business identifiers: eventId, orderId.
  • The outcome and its reason: processed, duplicate ignored, retried (attempt n), dead-lettered and why.
  • Trace and span id, which Boot adds once tracing is present.

Never log

  • The payload. It contains names, addresses, emails, amounts tied to a person, sometimes card data.
  • A key that is itself personal data, such as an email address. Log a hash or omit it.
  • Header values you do not own: they can carry tokens.
  • Exception messages that embed the input, such as a JSON parser quoting the offending text, without checking what they contain.
  • Credentials from client configuration; avoid dumping consumer or producer configs at INFO with secrets in custom properties.

The record's coordinates can be added once, for every log line, with a RecordInterceptor: put them into the MDC in intercept and remove them in afterRecord. Spring Boot attaches a RecordInterceptor bean to the default container factory. With structured logging (logging.structured.format.console=ecs, gelf or logstash), every MDC entry becomes a field of the JSON log line.

Architecture

Drawing diagram…

Flowchart. The order service sends through KafkaTemplate to the topic orders.order-placed with a traceparent header. The payment service's listener container receives it, a record interceptor sets MDC fields and the listener runs. Dotted lines show the template producing a timer and a span, the container producing a timer, a span in the same trace and client meters, and the listener producing a JSON log line. Timers go to the meter registry, spans to a tracing backend and log lines to a log store.

flowchart LR
  OS["Order service"] --> TPL["KafkaTemplate send"]
  TPL -->|record with traceparent header| TOP["orders.order-placed"]
  TOP --> CON["Listener container in payment service"]
  CON --> INT["RecordInterceptor sets MDC fields"]
  INT --> LIS["OrderPlacedListener"]
  TPL -.-> M1["Timer spring.kafka.template"]
  TPL -.-> S1["Span spring.kafka.template"]
  CON -.-> M2["Timer spring.kafka.listener"]
  CON -.-> S2["Span spring.kafka.listener, same trace"]
  CON -.-> M3["Client meters kafka.consumer"]
  LIS -.-> LOG["JSON log line with topic, partition, offset, key, traceId"]
  M1 --> REG["MeterRegistry and Actuator"]
  M2 --> REG
  M3 --> REG
  S1 --> TR["Tracing backend"]
  S2 --> TR
  LOG --> LG["Log store"]
One record from the order service to the payment listener, and the three signals produced along the way.
  • The solid line is the record's path. Everything dotted is produced by Spring or Micrometer around your code, not by it.
  • The trace survives the topic only because the context travels inside the record as a header. Anything that copies a record without its headers, such as a hand-written republisher, breaks the trace at that point.
  • The meter registry holds two kinds of Kafka meters: Spring's timers around your listener and template, and the clients' own internal metrics.
  • The log line is the only signal you write yourself. The interceptor makes sure it always carries the coordinates, and tracing adds the ids that link it to the other two.

Message flow

Drawing diagram…

Sequence diagram. The test sends the same event twice to a Kafka container. The listener container polls both records. The first inserts a payment into a PostgreSQL container, the second is rejected by a unique constraint. The container commits offset 2. The test polls the committed offset with Awaitility until it is 2, then counts payments for the order and gets 1.

sequenceDiagram
  participant T as Test method
  participant B as Kafka container
  participant C as Listener container
  participant L as OrderPlacedListener
  participant D as PostgreSQL container
  T->>B: send OrderPlaced for order 7001
  T->>B: send the same OrderPlaced again
  B-->>T: both acknowledged, offsets 0 and 1
  C->>B: poll
  B-->>C: offsets 0 and 1
  C->>L: record 0
  L->>D: insert payment for order 7001
  C->>L: record 1
  L->>D: insert rejected by unique order id
  C->>B: commit offset 2
  loop Awaitility, every 100 ms up to 30 s
    T->>B: committed offset of payment-service
    B-->>T: 2
  end
  T->>D: count payments for order 7001
  D-->>T: 1
The duplicate-delivery test: wait for proof that both records were consumed, then assert on the database.
  1. The test and the listener run on different threads and talk only through the broker and the database, exactly as in production.
  2. The wait is on the committed offset, a positive fact that both records were handled. Asserting the count right after sending would pass even if the listener had not started yet.
  3. The second insert is rejected by the database, not by Kafka. Producer idempotence would not have helped: the test sent the event twice on purpose, as two separate records.
  4. If the listener crashed on the duplicate instead of ignoring it, the committed offset would never reach 2 and the test would fail with a timeout that names the condition.

In a real system

The payment service of the example project consumes orders.order-placed (12 partitions in production, keyed by orderId) in group payment-service, writes to its payments table and publishes payments.payment-result. A typical test suite for it looks like this:

TestLayerAsserts
Zero amount or unknown currency is rejectedUnitInvalidOrderException is thrown and the processor is never called
A provider outage is not swallowedUnitThe listener lets PaymentProviderUnavailableException propagate
Undecodable bytes end in the dead-letter topicBroker (embedded or container)The original bytes and the origin headers are in orders.order-placed-dlt; the next record is processed
The same event delivered twice creates one paymentTestcontainers with PostgreSQLCommitted offset is past both records and payments has one row for the order
Provider down, then upTestcontainersThe listener is invoked again after the back-off and the payment is created once
Trace context reaches the outputBrokerThe record in payments.payment-result carries a traceparent header

In production the same service is watched through a small set of signals: the failure count of spring.kafka.listener for the container payment-orders, the consumer's records.lag.max together with lag measured from the group's committed offsets, the count of records arriving in the dead-letter topic, and rebalance totals after each deploy. When an alert fires, the on-call engineer takes a failing record's topic-partition@offset from a log line, opens the trace linked by its traceId, and sees the whole order from the HTTP request to the listener that failed.

Java and Spring Boot

Test and observability code for the payment service: the dependencies, a plain unit test, a Testcontainers integration test for duplicate delivery, the observability configuration and the MDC interceptor. The listener and error handling under test are the ones from the previous lessons and the example project at /kafka/project.

payment-service/pom.xml (dependencies)XML
<dependencies>  <!-- Observability -->  <dependency>    <groupId>org.springframework.boot</groupId>    <artifactId>spring-boot-starter-actuator</artifactId>  </dependency>  <dependency>    <groupId>io.micrometer</groupId>    <artifactId>micrometer-registry-prometheus</artifactId>    <scope>runtime</scope>  </dependency>  <dependency>    <groupId>io.micrometer</groupId>    <artifactId>micrometer-tracing-bridge-otel</artifactId>  </dependency>  <dependency>    <groupId>io.opentelemetry</groupId>    <artifactId>opentelemetry-exporter-otlp</artifactId>  </dependency>  <!-- Tests -->  <dependency>    <groupId>org.springframework.boot</groupId>    <artifactId>spring-boot-starter-test</artifactId>    <scope>test</scope>  </dependency>  <dependency>    <groupId>org.springframework.kafka</groupId>    <artifactId>spring-kafka-test</artifactId>    <scope>test</scope>  </dependency>  <dependency>    <groupId>org.springframework.boot</groupId>    <artifactId>spring-boot-testcontainers</artifactId>    <scope>test</scope>  </dependency>  <dependency>    <groupId>org.testcontainers</groupId>    <artifactId>junit-jupiter</artifactId>    <scope>test</scope>  </dependency>  <dependency>    <groupId>org.testcontainers</groupId>    <artifactId>kafka</artifactId>    <scope>test</scope>  </dependency>  <dependency>    <groupId>org.testcontainers</groupId>    <artifactId>postgresql</artifactId>    <scope>test</scope>  </dependency></dependencies>

Why it is written this way

  • No versions: the Spring Boot 3.5 parent manages all of these, including Testcontainers 1.21.x and Micrometer Tracing.
  • spring-boot-starter-test brings JUnit 5, AssertJ, Mockito and Awaitility. spring-kafka-test brings KafkaTestUtils, ContainerTestUtils and the embedded brokers.
  • org.testcontainers:kafka is the module name in the Testcontainers 1.x line that Boot 3.5 manages. If you move to Testcontainers 2.x it becomes testcontainers-kafka.
  • The tracing bridge turns observations into spans; the OTLP exporter sends them to a collector. Without a bridge, observation-enabled still gives you timers but no traces and no header propagation.

Configuration

Defaults are from the Spring Boot 3.5 application properties appendix, the Spring for Apache Kafka 3.3 reference and Javadoc, and the kafka-clients 3.9 configuration reference.

spring.kafka.listener.observation-enabledDefault false
What it does
Wraps each listener invocation in a Micrometer observation: a timer, and with a tracing bridge a span that continues the trace found in the record's headers.
When to change it
Enable it in services that take part in traced flows. It has no effect on batch listeners, which keep the plain timers.
spring.kafka.template.observation-enabledDefault false
What it does
Wraps each KafkaTemplate send in an observation and writes the trace context into the outgoing record's headers.
When to change it
Enable it together with the listener property. A producer that does not propagate leaves every consumer starting a new, unconnected trace.
management.tracing.sampling.probabilityDefault 0.1
What it does
The share of traces that are recorded and exported.
When to change it
Raise it to 1.0 in tests and while diagnosing. In production pick a value your tracing backend can afford; metrics and logs still cover every record.
management.endpoints.web.exposure.includeDefault health
What it does
Which Actuator endpoints are reachable over HTTP.
When to change it
Add metrics and prometheus (or your registry's endpoint) on a management port that is not public. Without them the meters exist but nothing can read them.
logging.structured.format.consoleDefault none (plain text pattern)
What it does
Writes console log events as JSON in the chosen format: ecs, gelf or logstash. MDC entries are included as fields.
When to change it
Use it wherever logs are collected by a log pipeline. Keep the text pattern for local development if you prefer reading it.
ContainerProperties.micrometerEnabledDefault true
What it does
Whether the container maintains the spring.kafka.listener timers when Micrometer and a single MeterRegistry are present.
When to change it
Leave it on. If the timers are missing, check that exactly one MeterRegistry bean exists and that the listener is not using a container factory built without Boot.
@EmbeddedKafka kraftDefault false (ZooKeeper-based embedded broker) in Spring for Apache Kafka 3.3
What it does
Chooses between EmbeddedKafkaZKBroker and EmbeddedKafkaKraftBroker for the test.
When to change it
Set kraft = true so tests use the mode every Kafka 4.x cluster runs in. In Spring for Apache Kafka 4.x only the KRaft broker remains.
@EmbeddedKafka partitionsDefault 2
What it does
Partitions per topic created by the embedded broker for the names in topics.
When to change it
Match what the test needs: 1 when the test reasons about a single offset sequence, the production count when it checks partition-preserving behaviour such as dead-letter routing.
auto.offset.reset (consumer)Default latest
What it does
Where a group with no committed offset starts reading.
When to change it
The main source of tests that hang: with latest, a record sent before the listener is assigned is skipped. Use earliest for the service, or wait for assignment in the test.

Hands-on exercise

Turn a failure mode into a test and an alert

Use the delivery simulator to produce a duplicate, then write down the test that would catch a non-idempotent listener and the signal that would show it in production.

Steps · 0 of 6 done

A duplicate delivery to test for

Simulation · not a real cluster

At-least-once with a crash after processing. The record is delivered again; toggle the idempotent consumer to see what your test has to tell apart.

Take it further in the lab: Duplicate message processing

What can go wrong

Thread.sleep as synchronisation
Symptom
A test passes on a laptop and fails one build in twenty on CI with an assertion on an empty table or a missing record.
Cause
The test sleeps a fixed time and asserts. The listener's start-up, the group join and the first poll take longer on a loaded build agent.
Fix
Poll for the condition with Awaitility or KafkaTestUtils.getSingleRecord with a generous timeout. A passing test then finishes as soon as the condition holds.
The record sent before the listener was ready
Symptom
An integration test times out waiting for the listener's effect, and the listener is never invoked. Re-running often passes.
Cause
The consumer's auto.offset.reset is latest and the test produced before the container had its partitions, so the new group started after the record.
Fix
Use earliest for services that must not skip, or call ContainerTestUtils.waitForAssignment on the container before sending.
Tests that share topics or groups
Symptom
Tests pass alone and fail together: unexpected extra records, counts that are too high, a consumer that reads another test's output.
Cause
A cached Spring context or a shared broker keeps topics and committed offsets between tests.
Fix
Use a unique topic or key per test and a fresh group id for test consumers. Assert on records matched by key, not on 'the next record'.
Traces that stop at the topic
Symptom
The tracing UI shows the HTTP request and the send, then a separate trace that starts at the listener.
Cause
Observation is enabled on one side only, the listener is a batch listener, the listener uses a hand-built container factory, or something republished the record without its headers.
Fix
Enable both observation properties, set observationEnabled on custom factories, and copy headers when republishing. Check for a traceparent header on the record.
Personal data in logs and dead-letter headers
Symptom
A privacy review finds customer emails and addresses in the log store, and in the exception message header of dead-letter records.
Cause
The listener logged the event object, or an exception message quoted the input. Spring copies the exception message and stack trace into dead-letter headers.
Fix
Log identifiers only. Give your own exceptions messages built from ids. If parser messages can quote input, exclude or replace the exception message header on the recoverer, and restrict access to the dead-letter topic like the source topic.
High-cardinality metric tags
Symptom
The metrics backend slows down or rejects series after a deploy that added a tag to the listener timer.
Cause
A micrometerTagsProvider or observation convention put the record key, order id or offset into a tag, creating a time series per record.
Fix
Tags must have few distinct values: topic, listener id, result, exception class. Per-record identifiers belong in logs and trace attributes.

How to troubleshoot

You need the failure rate and latency of one listener

Check

curl -s 'http://localhost:8080/actuator/metrics/spring.kafka.listener?tag=result:failure'
How to read it
COUNT is the number of failed invocations, TOTAL_TIME the time spent in them. Without the tag filter you see the available tags and their values, including each container's name. With observations enabled, filter on the error tag instead.
Lag is suspected but the dashboard has no Kafka panel

Check

curl -s http://localhost:8080/actuator/metrics/kafka.consumer.fetch.manager.records.lag.max
How to read it
The largest lag among the partitions this instance's consumers hold. Compare with kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service; if the tool shows lag on partitions with no CONSUMER-ID, no instance is reporting for them.
No Kafka meters at all under /actuator/metrics

Check

The application's bean definitions: a custom ConsumerFactory or ProducerFactory bean, and the number of MeterRegistry beans
How to read it
Boot adds MicrometerConsumerListener and MicrometerProducerListener only to the factories it creates. On your own factories add them with addListener. Spring Kafka's timers need exactly one MeterRegistry.
A trace does not continue in the consuming service

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-placed --from-beginning --max-messages 5 --formatter-property print.headers=true
How to read it
If traceparent is missing, the producer side is not observing or has no tracing bridge. If it is present, look at the consumer: listener observation off, a batch listener, or a custom container factory.
A Testcontainers test fails before any test method runs

Check

The test log for the container start-up: a message that no Docker environment was found, an image pull error, or a start-up timeout
How to read it
Testcontainers needs a reachable Docker daemon and the image. On CI, check that the job has Docker access and can pull apache/kafka; a registry mirror or a pre-pulled image removes the network dependency.
A log line cannot be matched to a record

Check

The log event's fields kafka.topic, kafka.partition and kafka.offset, then kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-placed --partition 3 --offset 1041 --max-messages 1
How to read it
If the fields are missing, the line was written outside the listener thread or the interceptor is not attached to that container factory. With them, the second command fetches exactly that record.

Key takeaways

  • Keep the business decision in classes you can construct with new, and unit-test every branch there, including that failures are thrown and not swallowed.
  • Send real bytes through a real broker for everything that is configuration: serialization, error handling, dead letters, offsets.
  • In Spring Kafka 3.3 @EmbeddedKafka is ZooKeeper-based unless kraft = true, and runs the broker from the test classpath. Testcontainers runs the image version you deploy.
  • Never sleep. Wait for a positive condition with Awaitility or KafkaTestUtils, and before asserting that something did not happen, wait for proof that the record was consumed.
  • Boot gives you spring.kafka.listener, spring.kafka.template and the clients' own kafka.* meters. Turn on both observation-enabled properties to carry traces across topics.
  • Put topic, partition, offset and key into the MDC for every record, log identifiers and outcomes, and never log payloads.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A test sends an OrderPlaced event and immediately asserts that the payments table has no second row for a duplicate it sent just before. It always passes. A colleague removes the unique constraint on order_id to try something, and the test still passes, although the staging environment then shows double payments.

What is wrong with the test?

Scenario 2 of 3

A service's integration tests use @EmbeddedKafka with default attributes on Spring Boot 3.5 and pass. Production runs Kafka 4.1 brokers. The team now enables Kafka transactions and wants a test that exercises the transactional listener together with its PostgreSQL writes before the release.

Which test setup gives the most trustworthy result?

Scenario 3 of 3

After enabling tracing, traces for checkout show the order service's HTTP span and a spring.kafka.template span, then end. The payment service's work appears as separate traces. A console consumer shows a traceparent header on the records. The payment service declares its own ConcurrentKafkaListenerContainerFactory bean and sets spring.kafka.listener.observation-enabled=true.

What is the most likely fix?

References

References: Spring for Apache Kafka 3.3 reference: Testing Applications (embedded brokers, @EmbeddedKafka, KafkaTestUtils) ↗ · Spring for Apache Kafka 3.3 reference: Monitoring (listener and template timers, Micrometer native metrics, Micrometer Observation) ↗ · Spring for Apache Kafka 3.3 reference: Micrometer Observation Documentation (meter, span and tag names) ↗ · Spring Boot 3.5 reference: Testcontainers (service connections, supported Kafka containers) ↗ · Spring Boot 3.5 reference: Metrics (Kafka metrics auto-configuration) ↗ · Spring Boot 3.5 reference: Tracing (bridges, correlation ids, sampling) ↗ · Spring Boot 3.5 reference: Logging (structured logging, MDC fields) ↗ · Spring Boot 3.5: Common application properties (spring.kafka.*.observation-enabled, management.tracing.*) ↗ · Testcontainers for Java: Kafka module (KafkaContainer, ConfluentKafkaContainer, supported images) ↗ · Apache Kafka 4.1 documentation: Monitoring (consumer and producer client metrics) ↗ · Spring Boot 3.5: Managed dependency coordinates (spring-kafka, kafka-clients, Testcontainers, Awaitility versions) ↗ · Report a problem