Path 9 · lesson 6 of 6
Testing and observing Spring Kafka applications
Test listeners as plain classes and against a real broker with Testcontainers, then make them observable: Micrometer metrics, traces across topics and safe, searchable logs.
What you will learn
- Split Kafka code so the business decision can be unit-tested without Spring or a broker
- Choose between
@EmbeddedKafkaand Testcontainers, knowing which broker each one runs in this version - Write an integration test with a
KafkaContainer,@ServiceConnectionand Awaitility that does not depend on timing - Test the paths that hurt in production: a duplicate delivery, a bad record and a failing dependency
- Find the Micrometer meters Spring Boot exposes for Kafka clients, templates and listeners, and enable observations so traces cross a topic
- Put topic, partition, offset and key into every log line of a listener, and keep payloads and personal data out
Before you start
- Spring Kafka setup
- @KafkaListener in depth
- Error handling, retries and dead letters in Spring
- Duplicates and idempotency
- Consumer lag
- JUnit 5 and Docker available on the build machine
The concept
A Kafka listener is asynchronous in both directions. In a test, nothing returns when the work is done: a record goes in, and some time later a row, another record or a log line appears. In production it is the same problem with higher stakes: when a customer asks why an order was not charged, the only evidence is what the service measured and logged while it handled one record among millions. Testing and observability are two answers to the same question: what did this code do with this record?
For tests, the answer is layered. Each layer proves something the others cannot, and the cheap layers should carry most of the cases.
| Layer | What runs | What it proves | What it cannot prove |
|---|---|---|---|
| Plain unit test | Your listener and processor classes, constructed with new. No Spring, no broker. | Business decisions: valid or invalid, which exception type, which calls are made. Milliseconds per test. | Anything about serialization, annotations, offsets, retries or configuration. |
@EmbeddedKafka | The Spring context and a broker started inside the test JVM from the Kafka libraries on the test classpath. | Wiring: deserializers, the error handler, the dead-letter path, topic names. No Docker needed. | Behaviour of the broker version you run in production. Under Spring Boot 3.5 the embedded broker is built from the 3.9 Kafka jars. |
| Testcontainers | The Spring context and a real broker in a container, at the image version you choose, plus real dependencies such as PostgreSQL. | The service as deployed: client against a 4.x broker, transactions, the database, idempotency across both. | Multi-broker failures, production load, network partitions. Those need a staging cluster or fault-injection tooling. |
For observability, Spring Boot already produces most of the raw material. With Actuator on the classpath it binds the Kafka clients' own metrics to Micrometer, times every listener invocation and every template send, and, if you enable observations and add a tracing bridge, carries a trace across the topic in a record header. What it cannot do for you is decide what your log lines say.
Why it exists
Before containers were practical in a build, teams tested Kafka code in one of two unsatisfying ways. Either everything was mocked, and the first real record in the test environment failed to deserialize; or a shared test cluster was used, and tests failed because someone else's records were in the topic. The embedded broker fixed isolation by starting a broker per test class inside the JVM. Testcontainers fixed fidelity by starting the same image you deploy.
The observability story has the same shape. A service that only logs processing failed forces you to reproduce a problem to understand it, and an asynchronous flow through four services is hard to reproduce. Three things changed that, and each answers a different question:
| Signal | Question it answers | Example for the payment service |
|---|---|---|
| Metrics | Is the flow healthy right now, and since when not? | Listener failures per minute went from 0 to 40 at 14:02; the consumer's maximum lag is rising. |
| Traces | Where did this one business operation spend its time, and where did it stop? | The order's trace shows the order service's send, 3 seconds in the topic, then a payment listener span that ended in an error. |
| Logs | What exactly happened to this record? | orders.order-placed-3@1041 key=7c1e... dead-lettered: provider returned 422. |
Each signal needs the others. A metric tells you to look; a trace or the record's coordinates tell you where; a log line tells you what. The coordinates of a Kafka record (topic, partition, offset) are the one identifier that every tool in the chain understands, which is why they belong in every log line a listener writes.
How it works inside
The embedded broker in this version
Spring for Apache Kafka 3.3 ships two embedded brokers in spring-kafka-test: EmbeddedKafkaZKBroker, which starts an embedded ZooKeeper, and EmbeddedKafkaKraftBroker, which runs KRaft in combined controller and broker mode. The reference states that the ZooKeeper one is still the default for @EmbeddedKafka in this line; set kraft = true on the annotation (available since 3.2) to get the KRaft one. Kafka 4.0 removed ZooKeeper mode, and the Spring for Apache Kafka 4.x reference lists only the KRaft broker, so tests that rely on the old default will need the attribute when you upgrade.
Since 3.0.10 the embedded broker publishes its address as spring.kafka.bootstrap-servers, so a @SpringBootTest connects to it without extra properties. It creates the topics named in the annotation, with 2 partitions each unless you set partitions.
Testcontainers and Spring Boot
- The current class is
org.testcontainers.kafka.KafkaContainer, for theapache/kafkaandapache/kafka-nativeimages.org.testcontainers.kafka.ConfluentKafkaContaineris forconfluentinc/cp-kafka(7.4.0 and later). The olderorg.testcontainers.containers.KafkaContaineris deprecated. - Spring Boot 3.5 manages Testcontainers 1.21.x, where the module's coordinates are
org.testcontainers:kafka. In Testcontainers 2.x the artifact is namedtestcontainers-kafka. Let Boot's dependency management pick the version. @ServiceConnectionon the container field (fromspring-boot-testcontainers) makes Boot create aKafkaConnectionDetailsbean from the running container. Connection details take precedence overspring.kafka.bootstrap-servers, and Boot supportsKafkaContainer,ConfluentKafkaContainerandRedpandaContainerthis way.- The container is a single broker in KRaft mode. The Kafka module sets the replication factor of the internal offsets and transaction topics to 1, so consumer groups and transactions work. Your own topics still cannot have three replicas there.
@EmbeddedKafka
- Starts in the test JVM; no Docker on the build machine.
- Broker version is the Kafka version on the test classpath, not the one in production.
- ZooKeeper-based by default in Spring Kafka 3.3;
kraft = trueswitches. - Good for wiring tests of one service: serialization, error handling, dead letters.
- Use a different topic per test, and
@DirtiesContext, as the reference recommends, to avoid shutdown races.
Testcontainers
- Starts a container; needs Docker and takes a few seconds per container.
- You pick the image tag, so tests can run against the broker version you deploy.
- KRaft single node; the same image family as production.
- Good for flows that include a database, transactions or several services' contracts.
- Share the container between test classes (a static field in a base class, or a container bean) so the cost is paid once.
Waiting without sleeping
Three tools cover nearly every wait. Awaitility (await().atMost(...).untilAsserted(...)) polls an assertion until it passes or the time is up; it comes with spring-boot-starter-test. KafkaTestUtils.getSingleRecord(consumer, topic, timeout) and getRecords poll a test consumer for output records. ContainerTestUtils.waitForAssignment(container, partitions) blocks until a listener container has its partitions, which matters when the listener's auto.offset.reset is latest (the Kafka default): a record sent before the assignment completes is never delivered to a new group.
A harder case is proving that something did not happen, such as a second payment for a duplicate event. Waiting a fixed time proves nothing. Wait for a positive signal that the duplicate was consumed, for example the group's committed offset reaching the end of the partition (KafkaTestUtils.getCurrentOffset), and only then assert on the count.
What Spring Boot measures without any code
| Meter | Source | Tags | Read it for |
|---|---|---|---|
spring.kafka.listener (timer) | Listener container, when Micrometer and a single MeterRegistry are present | name (container bean name, with -n per thread), result (success or failure), exception | Processing time per record and the failure rate of a listener |
spring.kafka.template (timer) | KafkaTemplate | name (template bean name), result, exception | Send latency and send failures as the application sees them |
kafka.consumer.fetch.manager.records.lag.max (gauge) | The Kafka consumer's own metric records-lag-max, bound by Boot's MicrometerConsumerListener | client.id, spring.id, kafka.version | The largest lag among the partitions this consumer instance holds |
kafka.consumer.coordinator.rebalance.total, kafka.consumer.coordinator.failed.rebalance.total | Consumer metrics rebalance-total and failed-rebalance-total | as above | Rebalance storms after a deploy or when processing is too slow |
kafka.producer.record.error.total | Producer metric record-error-total, bound by MicrometerProducerListener | as above | Records the producer gave up on after its own retries |
Observations and tracing
Two Boot properties, both false by default, switch the template and the listener containers from plain timers to Micrometer observations: spring.kafka.template.observation-enabled and spring.kafka.listener.observation-enabled. An observation produces the timer (the names stay spring.kafka.template and spring.kafka.listener, with an extra .active long task timer) and, when a tracing bridge such as micrometer-tracing-bridge-otel is on the classpath, a span of the same name. The tags change to messaging conventions, for example messaging.kafka.consumer.group, messaging.source.name and spring.kafka.listener.id on the listener.
Propagation works through record headers. The template's observation writes the trace context into the headers of the outgoing record; with Boot's default W3C propagation that is a traceparent header. The listener container's observation reads it from the incoming record and continues the same trace. Boot also puts traceId and spanId into the logging MDC and prints them as a correlation id. Two limits are stated in the reference: observations are not supported for batch listeners, and Boot samples only 10% of traces unless you change management.tracing.sampling.probability.
Logging a record
Log
- Topic, partition and offset of the record being handled.
- The key, if it is an opaque identifier such as an order id.
- Your business identifiers:
eventId,orderId. - The outcome and its reason: processed, duplicate ignored, retried (attempt n), dead-lettered and why.
- Trace and span id, which Boot adds once tracing is present.
Never log
- The payload. It contains names, addresses, emails, amounts tied to a person, sometimes card data.
- A key that is itself personal data, such as an email address. Log a hash or omit it.
- Header values you do not own: they can carry tokens.
- Exception messages that embed the input, such as a JSON parser quoting the offending text, without checking what they contain.
- Credentials from client configuration; avoid dumping consumer or producer configs at INFO with secrets in custom properties.
The record's coordinates can be added once, for every log line, with a RecordInterceptor: put them into the MDC in intercept and remove them in afterRecord. Spring Boot attaches a RecordInterceptor bean to the default container factory. With structured logging (logging.structured.format.console=ecs, gelf or logstash), every MDC entry becomes a field of the JSON log line.
Architecture
Drawing diagram…
Flowchart. The order service sends through KafkaTemplate to the topic orders.order-placed with a traceparent header. The payment service's listener container receives it, a record interceptor sets MDC fields and the listener runs. Dotted lines show the template producing a timer and a span, the container producing a timer, a span in the same trace and client meters, and the listener producing a JSON log line. Timers go to the meter registry, spans to a tracing backend and log lines to a log store.
flowchart LR OS["Order service"] --> TPL["KafkaTemplate send"] TPL -->|record with traceparent header| TOP["orders.order-placed"] TOP --> CON["Listener container in payment service"] CON --> INT["RecordInterceptor sets MDC fields"] INT --> LIS["OrderPlacedListener"] TPL -.-> M1["Timer spring.kafka.template"] TPL -.-> S1["Span spring.kafka.template"] CON -.-> M2["Timer spring.kafka.listener"] CON -.-> S2["Span spring.kafka.listener, same trace"] CON -.-> M3["Client meters kafka.consumer"] LIS -.-> LOG["JSON log line with topic, partition, offset, key, traceId"] M1 --> REG["MeterRegistry and Actuator"] M2 --> REG M3 --> REG S1 --> TR["Tracing backend"] S2 --> TR LOG --> LG["Log store"]
- The solid line is the record's path. Everything dotted is produced by Spring or Micrometer around your code, not by it.
- The trace survives the topic only because the context travels inside the record as a header. Anything that copies a record without its headers, such as a hand-written republisher, breaks the trace at that point.
- The meter registry holds two kinds of Kafka meters: Spring's timers around your listener and template, and the clients' own internal metrics.
- The log line is the only signal you write yourself. The interceptor makes sure it always carries the coordinates, and tracing adds the ids that link it to the other two.
Message flow
Drawing diagram…
Sequence diagram. The test sends the same event twice to a Kafka container. The listener container polls both records. The first inserts a payment into a PostgreSQL container, the second is rejected by a unique constraint. The container commits offset 2. The test polls the committed offset with Awaitility until it is 2, then counts payments for the order and gets 1.
sequenceDiagram
participant T as Test method
participant B as Kafka container
participant C as Listener container
participant L as OrderPlacedListener
participant D as PostgreSQL container
T->>B: send OrderPlaced for order 7001
T->>B: send the same OrderPlaced again
B-->>T: both acknowledged, offsets 0 and 1
C->>B: poll
B-->>C: offsets 0 and 1
C->>L: record 0
L->>D: insert payment for order 7001
C->>L: record 1
L->>D: insert rejected by unique order id
C->>B: commit offset 2
loop Awaitility, every 100 ms up to 30 s
T->>B: committed offset of payment-service
B-->>T: 2
end
T->>D: count payments for order 7001
D-->>T: 1- The test and the listener run on different threads and talk only through the broker and the database, exactly as in production.
- The wait is on the committed offset, a positive fact that both records were handled. Asserting the count right after sending would pass even if the listener had not started yet.
- The second insert is rejected by the database, not by Kafka. Producer idempotence would not have helped: the test sent the event twice on purpose, as two separate records.
- If the listener crashed on the duplicate instead of ignoring it, the committed offset would never reach 2 and the test would fail with a timeout that names the condition.
In a real system
The payment service of the example project consumes orders.order-placed (12 partitions in production, keyed by orderId) in group payment-service, writes to its payments table and publishes payments.payment-result. A typical test suite for it looks like this:
| Test | Layer | Asserts |
|---|---|---|
| Zero amount or unknown currency is rejected | Unit | InvalidOrderException is thrown and the processor is never called |
| A provider outage is not swallowed | Unit | The listener lets PaymentProviderUnavailableException propagate |
| Undecodable bytes end in the dead-letter topic | Broker (embedded or container) | The original bytes and the origin headers are in orders.order-placed-dlt; the next record is processed |
| The same event delivered twice creates one payment | Testcontainers with PostgreSQL | Committed offset is past both records and payments has one row for the order |
| Provider down, then up | Testcontainers | The listener is invoked again after the back-off and the payment is created once |
| Trace context reaches the output | Broker | The record in payments.payment-result carries a traceparent header |
In production the same service is watched through a small set of signals: the failure count of spring.kafka.listener for the container payment-orders, the consumer's records.lag.max together with lag measured from the group's committed offsets, the count of records arriving in the dead-letter topic, and rebalance totals after each deploy. When an alert fires, the on-call engineer takes a failing record's topic-partition@offset from a log line, opens the trace linked by its traceId, and sees the whole order from the HTTP request to the listener that failed.
Java and Spring Boot
Test and observability code for the payment service: the dependencies, a plain unit test, a Testcontainers integration test for duplicate delivery, the observability configuration and the MDC interceptor. The listener and error handling under test are the ones from the previous lessons and the example project at /kafka/project.
<dependencies> <!-- Observability --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <scope>runtime</scope> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-otlp</artifactId> </dependency> <!-- Tests --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-testcontainers</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>junit-jupiter</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>kafka</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>postgresql</artifactId> <scope>test</scope> </dependency></dependencies>Why it is written this way
- No versions: the Spring Boot 3.5 parent manages all of these, including Testcontainers 1.21.x and Micrometer Tracing.
spring-boot-starter-testbrings JUnit 5, AssertJ, Mockito and Awaitility.spring-kafka-testbringsKafkaTestUtils,ContainerTestUtilsand the embedded brokers.org.testcontainers:kafkais the module name in the Testcontainers 1.x line that Boot 3.5 manages. If you move to Testcontainers 2.x it becomestestcontainers-kafka.- The tracing bridge turns observations into spans; the OTLP exporter sends them to a collector. Without a bridge,
observation-enabledstill gives you timers but no traces and no header propagation.
package com.example.payments;import static org.assertj.core.api.Assertions.assertThatThrownBy;import static org.mockito.Mockito.doThrow;import static org.mockito.Mockito.mock;import static org.mockito.Mockito.verify;import static org.mockito.Mockito.verifyNoInteractions;import java.time.Instant;import java.util.UUID;import org.junit.jupiter.api.Test;/** No Spring context, no broker: the listener is an ordinary class. */class OrderPlacedListenerTest { private final PaymentProcessor processor = mock(PaymentProcessor.class); private final OrderPlacedListener listener = new OrderPlacedListener(processor); @Test void validOrderIsHandedToTheProcessor() { OrderPlaced event = order(4_500L, "EUR"); listener.onOrderPlaced(event); verify(processor).handle(event); } @Test void zeroAmountIsRejectedBeforeAnySideEffect() { assertThatThrownBy(() -> listener.onOrderPlaced(order(0L, "EUR"))) .isInstanceOf(InvalidOrderException.class); verifyNoInteractions(processor); } @Test void providerOutageIsNotSwallowed() { OrderPlaced event = order(4_500L, "EUR"); doThrow(new PaymentProviderUnavailableException("provider timed out", null)) .when(processor).handle(event); // The exception must reach the container, or the record would be acknowledged. assertThatThrownBy(() -> listener.onOrderPlaced(event)) .isInstanceOf(PaymentProviderUnavailableException.class); } private static OrderPlaced order(long amountMinor, String currency) { return new OrderPlaced(UUID.randomUUID(), UUID.randomUUID(), "customer-7", amountMinor, currency, Instant.parse("2026-01-15T10:00:00Z")); }}Why it is written this way
- This works because the listener takes the deserialized
OrderPlacedas its parameter and its collaborators through the constructor. A listener that takesConsumerRecordandAcknowledgmentand reaches for static state is much harder to test this way. - The third test protects the contract with the container: an exception means the record is not done. A well-meant try/catch added later would turn that test red.
- The exception types are the ones the error handler classifies. Testing which type is thrown here, and what the handler does with each type in a broker test, covers the policy from both sides.
- These tests run in milliseconds. Put every branch of the business rules here and keep the broker tests for wiring and for the flows that cross systems.
package com.example.payments;import static org.assertj.core.api.Assertions.assertThat;import static org.awaitility.Awaitility.await;import static org.mockito.ArgumentMatchers.any;import static org.mockito.ArgumentMatchers.anyLong;import static org.mockito.Mockito.when;import java.time.Duration;import java.time.Instant;import java.util.UUID;import java.util.concurrent.TimeUnit;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.clients.consumer.OffsetAndMetadata;import org.junit.jupiter.api.Test;import org.springframework.beans.factory.annotation.Autowired;import org.springframework.boot.test.context.SpringBootTest;import org.springframework.boot.test.context.TestConfiguration;import org.springframework.boot.testcontainers.service.connection.ServiceConnection;import org.springframework.context.annotation.Bean;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.config.TopicBuilder;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.test.utils.KafkaTestUtils;import org.springframework.test.context.bean.override.mockito.MockitoBean;import org.testcontainers.containers.PostgreSQLContainer;import org.testcontainers.junit.jupiter.Container;import org.testcontainers.junit.jupiter.Testcontainers;import org.testcontainers.kafka.KafkaContainer;@SpringBootTest(properties = "payments.topics.replicas=1")@Testcontainersclass DuplicateDeliveryIT { private static final String TOPIC = "orders.order-placed"; @Container @ServiceConnection static final KafkaContainer kafka = new KafkaContainer("apache/kafka:4.1.2"); @Container @ServiceConnection static final PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:17"); @TestConfiguration static class Topics { /** One partition, so "both records consumed" is one committed offset to check. */ @Bean NewTopic ordersTopic() { return TopicBuilder.name(TOPIC).partitions(1).replicas(1).build(); } } @Autowired private KafkaTemplate<Object, Object> template; @Autowired private JdbcTemplate jdbc; @MockitoBean private PaymentProvider provider; @Test void sameEventDeliveredTwiceCreatesOnePayment() throws Exception { when(provider.charge(any(), anyLong(), any(), any())) .thenReturn(new ChargeResult(true, "prov-ref-1")); OrderPlaced event = new OrderPlaced(UUID.randomUUID(), UUID.randomUUID(), "customer-7", 4_500L, "EUR", Instant.parse("2026-01-15T10:00:00Z")); String key = event.orderId().toString(); // Two separate records with the same business event: what a redelivery or a // producer-side resend looks like to the consumer. template.send(TOPIC, key, event).get(10, TimeUnit.SECONDS); template.send(TOPIC, key, event).get(10, TimeUnit.SECONDS); // Positive proof that both records were handled: the group's position is past them. await().atMost(Duration.ofSeconds(30)).untilAsserted(() -> { OffsetAndMetadata committed = KafkaTestUtils.getCurrentOffset( kafka.getBootstrapServers(), "payment-service", TOPIC, 0); assertThat(committed).isNotNull(); assertThat(committed.offset()).isEqualTo(2L); }); Integer payments = jdbc.queryForObject( "SELECT count(*) FROM payments WHERE order_id = ?", Integer.class, event.orderId()); assertThat(payments).isEqualTo(1); }}Why it is written this way
org.testcontainers.kafka.KafkaContaineris the class for theapache/kafkaimage. Pin a tag that matches the broker version you run; the 3.9 client that Boot 3.5 manages talks to a 4.x broker.@ServiceConnectionreplaces@DynamicPropertySourceboilerplate: Boot builds the Kafka and JDBC connection details from the running containers, and they override the properties inapplication.yml.- The containers are static fields, so they start once for the class. To share them across classes, move the fields to a common base class or declare them as beans in a
@TestConfiguration. - A one-broker container cannot host topics with three replicas. The application reads its replica count from a property (here
payments.topics.replicas) so tests can set it to 1. - The committed offset is the position of the next record to read, so 2 means records 0 and 1 are behind the group. Only then is
count = 1evidence of idempotency and not of a slow listener. getCurrentOffsetreturns null until the group has committed for that partition, which is why the null check sits inside the retried assertion.- The card provider is a mock because it is someone else's system. Everything the service owns (serialization, listener container, error handling, SQL) is real.
- If the listener's
auto.offset.resetwerelatest, the test would first have to callContainerTestUtils.waitForAssignmenton the container from theKafkaListenerEndpointRegistry. This service usesearliest.
spring: application: name: payment-service kafka: template: observation-enabled: true listener: observation-enabled: truemanagement: endpoints: web: exposure: include: health,metrics,prometheus tracing: sampling: probability: 0.1 otlp: tracing: endpoint: http://otel-collector:4318/v1/traceslogging: structured: format: console: ecsWhy it is written this way
- The two
observation-enabledproperties are separate: one forKafkaTemplate, one for listener containers. Both default to false. They apply to Boot's auto-configured template and container factory; on a factory you build yourself, callgetContainerProperties().setObservationEnabled(true). - Enabling observations replaces Spring Kafka's plain timers with timers managed by the observation. The meter names stay the same; the tags change to the messaging conventions.
- By default only the
healthendpoint is exposed over HTTP.metricslets you inspect meters by name;prometheusis the scrape endpoint. - 0.1 is Boot's default sampling probability, written out so it is a visible decision. Set it to 1.0 in tests and local runs, or you will look for traces that were never recorded.
ecswrites each log event as one JSON object in Elastic Common Schema and includes every MDC entry as a field. Structured logging is built into Spring Boot from 3.4.
package com.example.payments;import org.apache.kafka.clients.consumer.Consumer;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.slf4j.MDC;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.listener.RecordInterceptor;@Configurationpublic class KafkaLoggingConfig { /** Spring Boot attaches a RecordInterceptor bean to the default listener container factory. */ @Bean RecordInterceptor<Object, Object> recordCoordinatesInMdc() { return new RecordInterceptor<>() { @Override public ConsumerRecord<Object, Object> intercept(ConsumerRecord<Object, Object> record, Consumer<Object, Object> consumer) { MDC.put("kafka.topic", record.topic()); MDC.put("kafka.partition", Integer.toString(record.partition())); MDC.put("kafka.offset", Long.toString(record.offset())); // The key is an order id here. If your key is personal data, hash it or leave it out. MDC.put("kafka.key", String.valueOf(record.key())); return record; } @Override public void afterRecord(ConsumerRecord<Object, Object> record, Consumer<Object, Object> consumer) { MDC.remove("kafka.topic"); MDC.remove("kafka.partition"); MDC.remove("kafka.offset"); MDC.remove("kafka.key"); } }; }}Why it is written this way
interceptruns on the consumer thread just before the listener, and the listener runs on the same thread, so every log statement in the listener and in the code it calls carries the four fields.afterRecordis called when processing of the record is complete, after success or failure. Removing the entries there stops them leaking into log lines about a later record.- The MDC is per thread. If the listener hands work to another thread or an
@Asyncmethod, the fields and the trace context do not follow unless you propagate them. - A record that fails deserialization never reaches the listener, and this interceptor is not called for it. Its coordinates are in the dead-letter headers and in the error handler's own log lines.
- Only the value passed to the listener is absent from these fields on purpose. Log identifiers from the event (
orderId,eventId) in the listener; never the event itself.
Configuration
Defaults are from the Spring Boot 3.5 application properties appendix, the Spring for Apache Kafka 3.3 reference and Javadoc, and the kafka-clients 3.9 configuration reference.
spring.kafka.listener.observation-enabledDefault false
- What it does
- Wraps each listener invocation in a Micrometer observation: a timer, and with a tracing bridge a span that continues the trace found in the record's headers.
- When to change it
- Enable it in services that take part in traced flows. It has no effect on batch listeners, which keep the plain timers.
spring.kafka.template.observation-enabledDefault false
- What it does
- Wraps each
KafkaTemplatesend in an observation and writes the trace context into the outgoing record's headers. - When to change it
- Enable it together with the listener property. A producer that does not propagate leaves every consumer starting a new, unconnected trace.
management.tracing.sampling.probabilityDefault 0.1
- What it does
- The share of traces that are recorded and exported.
- When to change it
- Raise it to 1.0 in tests and while diagnosing. In production pick a value your tracing backend can afford; metrics and logs still cover every record.
management.endpoints.web.exposure.includeDefault health
- What it does
- Which Actuator endpoints are reachable over HTTP.
- When to change it
- Add
metricsandprometheus(or your registry's endpoint) on a management port that is not public. Without them the meters exist but nothing can read them.
logging.structured.format.consoleDefault none (plain text pattern)
- What it does
- Writes console log events as JSON in the chosen format:
ecs,gelforlogstash. MDC entries are included as fields. - When to change it
- Use it wherever logs are collected by a log pipeline. Keep the text pattern for local development if you prefer reading it.
ContainerProperties.micrometerEnabledDefault true
- What it does
- Whether the container maintains the
spring.kafka.listenertimers when Micrometer and a singleMeterRegistryare present. - When to change it
- Leave it on. If the timers are missing, check that exactly one
MeterRegistrybean exists and that the listener is not using a container factory built without Boot.
@EmbeddedKafka kraftDefault false (ZooKeeper-based embedded broker) in Spring for Apache Kafka 3.3
- What it does
- Chooses between
EmbeddedKafkaZKBrokerandEmbeddedKafkaKraftBrokerfor the test. - When to change it
- Set
kraft = trueso tests use the mode every Kafka 4.x cluster runs in. In Spring for Apache Kafka 4.x only the KRaft broker remains.
@EmbeddedKafka partitionsDefault 2
- What it does
- Partitions per topic created by the embedded broker for the names in
topics. - When to change it
- Match what the test needs: 1 when the test reasons about a single offset sequence, the production count when it checks partition-preserving behaviour such as dead-letter routing.
auto.offset.reset (consumer)Default latest
- What it does
- Where a group with no committed offset starts reading.
- When to change it
- The main source of tests that hang: with
latest, a record sent before the listener is assigned is skipped. Useearliestfor the service, or wait for assignment in the test.
Hands-on exercise
Turn a failure mode into a test and an alert
Use the delivery simulator to produce a duplicate, then write down the test that would catch a non-idempotent listener and the signal that would show it in production.
Steps · 0 of 6 done
A duplicate delivery to test for
Simulation · not a real clusterAt-least-once with a crash after processing. The record is delivered again; toggle the idempotent consumer to see what your test has to tell apart.
What you should see
Without the idempotent consumer the payment effect happens twice; with it, once. The test sends the same event as two records, waits until the group's committed offset is past both, and asserts one payment row. The wait is needed in both cases, because a count of one is also what you see before the second record has been read. In production the signal is the failure count of spring.kafka.listener for that container (the result tag, or the error tag once observations are enabled), and the duplicate's log line carries topic, partition, offset, the order id and the event id, and no payload.
What can go wrong
Thread.sleep as synchronisation
- Symptom
- A test passes on a laptop and fails one build in twenty on CI with an assertion on an empty table or a missing record.
- Cause
- The test sleeps a fixed time and asserts. The listener's start-up, the group join and the first poll take longer on a loaded build agent.
- Fix
- Poll for the condition with Awaitility or
KafkaTestUtils.getSingleRecordwith a generous timeout. A passing test then finishes as soon as the condition holds.
The record sent before the listener was ready
- Symptom
- An integration test times out waiting for the listener's effect, and the listener is never invoked. Re-running often passes.
- Cause
- The consumer's
auto.offset.resetislatestand the test produced before the container had its partitions, so the new group started after the record. - Fix
- Use
earliestfor services that must not skip, or callContainerTestUtils.waitForAssignmenton the container before sending.
Tests that share topics or groups
- Symptom
- Tests pass alone and fail together: unexpected extra records, counts that are too high, a consumer that reads another test's output.
- Cause
- A cached Spring context or a shared broker keeps topics and committed offsets between tests.
- Fix
- Use a unique topic or key per test and a fresh group id for test consumers. Assert on records matched by key, not on 'the next record'.
Traces that stop at the topic
- Symptom
- The tracing UI shows the HTTP request and the send, then a separate trace that starts at the listener.
- Cause
- Observation is enabled on one side only, the listener is a batch listener, the listener uses a hand-built container factory, or something republished the record without its headers.
- Fix
- Enable both observation properties, set
observationEnabledon custom factories, and copy headers when republishing. Check for atraceparentheader on the record.
Personal data in logs and dead-letter headers
- Symptom
- A privacy review finds customer emails and addresses in the log store, and in the exception message header of dead-letter records.
- Cause
- The listener logged the event object, or an exception message quoted the input. Spring copies the exception message and stack trace into dead-letter headers.
- Fix
- Log identifiers only. Give your own exceptions messages built from ids. If parser messages can quote input, exclude or replace the exception message header on the recoverer, and restrict access to the dead-letter topic like the source topic.
High-cardinality metric tags
- Symptom
- The metrics backend slows down or rejects series after a deploy that added a tag to the listener timer.
- Cause
- A
micrometerTagsProvideror observation convention put the record key, order id or offset into a tag, creating a time series per record. - Fix
- Tags must have few distinct values: topic, listener id, result, exception class. Per-record identifiers belong in logs and trace attributes.
How to troubleshoot
You need the failure rate and latency of one listener
Check
curl -s 'http://localhost:8080/actuator/metrics/spring.kafka.listener?tag=result:failure'
- How to read it
- COUNT is the number of failed invocations, TOTAL_TIME the time spent in them. Without the tag filter you see the available tags and their values, including each container's name. With observations enabled, filter on the
errortag instead.
Lag is suspected but the dashboard has no Kafka panel
Check
curl -s http://localhost:8080/actuator/metrics/kafka.consumer.fetch.manager.records.lag.max
- How to read it
- The largest lag among the partitions this instance's consumers hold. Compare with kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service; if the tool shows lag on partitions with no CONSUMER-ID, no instance is reporting for them.
No Kafka meters at all under /actuator/metrics
Check
The application's bean definitions: a custom ConsumerFactory or ProducerFactory bean, and the number of MeterRegistry beans
- How to read it
- Boot adds
MicrometerConsumerListenerandMicrometerProducerListeneronly to the factories it creates. On your own factories add them withaddListener. Spring Kafka's timers need exactly oneMeterRegistry.
A trace does not continue in the consuming service
Check
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-placed --from-beginning --max-messages 5 --formatter-property print.headers=true
- How to read it
- If
traceparentis missing, the producer side is not observing or has no tracing bridge. If it is present, look at the consumer: listener observation off, a batch listener, or a custom container factory.
A Testcontainers test fails before any test method runs
Check
The test log for the container start-up: a message that no Docker environment was found, an image pull error, or a start-up timeout
- How to read it
- Testcontainers needs a reachable Docker daemon and the image. On CI, check that the job has Docker access and can pull
apache/kafka; a registry mirror or a pre-pulled image removes the network dependency.
A log line cannot be matched to a record
Check
The log event's fields kafka.topic, kafka.partition and kafka.offset, then kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders.order-placed --partition 3 --offset 1041 --max-messages 1
- How to read it
- If the fields are missing, the line was written outside the listener thread or the interceptor is not attached to that container factory. With them, the second command fetches exactly that record.
Key takeaways
- Keep the business decision in classes you can construct with
new, and unit-test every branch there, including that failures are thrown and not swallowed. - Send real bytes through a real broker for everything that is configuration: serialization, error handling, dead letters, offsets.
- In Spring Kafka 3.3
@EmbeddedKafkais ZooKeeper-based unlesskraft = true, and runs the broker from the test classpath. Testcontainers runs the image version you deploy. - Never sleep. Wait for a positive condition with Awaitility or
KafkaTestUtils, and before asserting that something did not happen, wait for proof that the record was consumed. - Boot gives you
spring.kafka.listener,spring.kafka.templateand the clients' ownkafka.*meters. Turn on bothobservation-enabledproperties to carry traces across topics. - Put topic, partition, offset and key into the MDC for every record, log identifiers and outcomes, and never log payloads.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A test sends an OrderPlaced event and immediately asserts that the payments table has no second row for a duplicate it sent just before. It always passes. A colleague removes the unique constraint on order_id to try something, and the test still passes, although the staging environment then shows double payments.
What is wrong with the test?
Scenario 2 of 3
A service's integration tests use @EmbeddedKafka with default attributes on Spring Boot 3.5 and pass. Production runs Kafka 4.1 brokers. The team now enables Kafka transactions and wants a test that exercises the transactional listener together with its PostgreSQL writes before the release.
Which test setup gives the most trustworthy result?
Scenario 3 of 3
After enabling tracing, traces for checkout show the order service's HTTP span and a spring.kafka.template span, then end. The payment service's work appears as separate traces. A console consumer shows a traceparent header on the records. The payment service declares its own ConcurrentKafkaListenerContainerFactory bean and sets spring.kafka.listener.observation-enabled=true.
What is the most likely fix?
References
References: Spring for Apache Kafka 3.3 reference: Testing Applications (embedded brokers, @EmbeddedKafka, KafkaTestUtils) ↗ · Spring for Apache Kafka 3.3 reference: Monitoring (listener and template timers, Micrometer native metrics, Micrometer Observation) ↗ · Spring for Apache Kafka 3.3 reference: Micrometer Observation Documentation (meter, span and tag names) ↗ · Spring Boot 3.5 reference: Testcontainers (service connections, supported Kafka containers) ↗ · Spring Boot 3.5 reference: Metrics (Kafka metrics auto-configuration) ↗ · Spring Boot 3.5 reference: Tracing (bridges, correlation ids, sampling) ↗ · Spring Boot 3.5 reference: Logging (structured logging, MDC fields) ↗ · Spring Boot 3.5: Common application properties (spring.kafka.*.observation-enabled, management.tracing.*) ↗ · Testcontainers for Java: Kafka module (KafkaContainer, ConfluentKafkaContainer, supported images) ↗ · Apache Kafka 4.1 documentation: Monitoring (consumer and producer client metrics) ↗ · Spring Boot 3.5: Managed dependency coordinates (spring-kafka, kafka-clients, Testcontainers, Awaitility versions) ↗ · Report a problem