Path 9 · lesson 1 of 6
Spring Kafka setup: what Boot builds and what you configure
Wire a Spring Boot service to Kafka: the beans Boot creates from spring.kafka properties, producer and consumer configuration, topics as code, profiles and a local KRaft broker.
What you will learn
- Name the five Kafka beans Spring Boot creates and say what makes each of them back off
- Write an application.yml with common, producer, consumer and listener settings, and pass client properties that have no dedicated Spring Boot key
- Declare topics with NewTopic beans and explain why production clusters usually switch off every form of automatic topic creation
- Separate local and production settings with profiles without duplicating the shared configuration
- Run a single-node KRaft broker locally with the official apache/kafka image and connect a Spring Boot service to it
Before you start
The concept
Spring for Apache Kafka (spring-kafka) is a library on top of the plain Kafka Java client. It does not replace the client: every send still goes through a KafkaProducer and every read through a KafkaConsumer. What it adds is lifecycle and threading. A factory owns the client objects, a template sends, and a listener container runs the poll loop and calls your method.
Spring Boot adds one more layer: with spring-kafka on the classpath it reads spring.kafka.* properties and creates those objects as beans. There is no Boot starter for Kafka; the dependency itself switches the auto-configuration on. Most services need no Kafka @Configuration class for clients at all.
| Bean Boot creates | What it does | Boot backs off when |
|---|---|---|
ProducerFactory (DefaultKafkaProducerFactory) | Creates and owns the producer. By default one producer is shared by all threads | You declare any ProducerFactory bean |
KafkaTemplate | Sends records through the producer factory and returns a future per send | You declare any KafkaTemplate bean |
ConsumerFactory (DefaultKafkaConsumerFactory) | Creates a consumer for each listener thread from the consumer properties | You declare any ConsumerFactory bean |
ConcurrentKafkaListenerContainerFactory named kafkaListenerContainerFactory | Builds the listener container behind each @KafkaListener, using spring.kafka.listener.* | A bean with that name exists |
KafkaAdmin | Creates the topics you declare as NewTopic beans when the context starts | You declare a KafkaAdmin bean |
Boot also enables @KafkaListener processing, so @EnableKafka is only needed in applications without Boot. The path's next lessons use these beans: the listener in depth, serialisation and error handling.
Why it exists
Without Spring you write the same code in every service: build a Properties object, create a KafkaProducer once and share it, start threads that each own a KafkaConsumer, loop on poll, commit offsets at the right moment, catch errors without killing the loop, and close everything in order on shutdown. Each of those steps has a well-known wrong version, such as a producer per request or a commit before processing.
The factories, template and container exist to make the correct version the default. Auto-configuration then removes the wiring code, which has a second benefit: configuration becomes data. The same jar runs against a laptop broker and a production cluster, and the difference is a profile and a few environment variables, not code.
The cost is that behaviour now comes from three layers, and you need to know which layer decided a value:
- Kafka client defaults, for every key nobody sets (for example
max.poll.records500). - Spring Boot and Spring Kafka choices: Boot defaults the serializers and deserializers to the String ones and
bootstrap-serverstolocalhost:9092; the listener container setsenable.auto.committo false unless you set it yourself. - Your properties: dedicated keys such as
spring.kafka.producer.acks, and raw client keys under apropertiesmap, which win when both are given.
How it works inside
From a YAML key to a client setting
Boot binds spring.kafka.* to a KafkaProperties object and builds three maps from it: producer, consumer and admin properties. Keys directly under spring.kafka (bootstrap servers, client id, security, SSL and the common properties map) go into all three. Keys under producer, consumer or admin go into that client type only and override the common value.
Boot has dedicated keys for the settings Kafka marks as high importance, a selection of the others, and those without a default. Everything else goes through the properties maps, which accept any client key. Write those keys in brackets in YAML so the dots stay part of the key: "[linger.ms]": 10. A value set this way overrides the dedicated Boot key for the same setting, so set each one in a single place.
| You write | Reaches | Example |
|---|---|---|
spring.kafka.<key> | Producer, consumer and admin | spring.kafka.bootstrap-servers |
spring.kafka.properties.* | Producer, consumer and admin, any client key | sasl.mechanism |
spring.kafka.producer.<key> | The producer factory | spring.kafka.producer.acks |
spring.kafka.producer.properties.* | The producer factory, any producer key | delivery.timeout.ms, spring.json.add.type.headers |
spring.kafka.consumer.<key> | The consumer factory | spring.kafka.consumer.group-id |
spring.kafka.consumer.properties.* | The consumer factory, any consumer key | partition.assignment.strategy |
spring.kafka.listener.* | The listener container factory. These are Spring settings, not Kafka client settings | spring.kafka.listener.concurrency |
spring.kafka.admin.* | KafkaAdmin | spring.kafka.admin.auto-create |
When the clients are actually created
- Admin: when the context starts,
KafkaAdminconnects and creates missingNewTopicbeans. If the broker is unreachable it logs the failure and the application keeps starting, unlessspring.kafka.admin.fail-fastis true. - Consumers: each
@KafkaListenercontainer starts late in context startup and creates one consumer per thread. They join the group immediately. - Producer: created on the first
send. A wrongbootstrap-serverstherefore shows up at the first request, as a send that blocks formax.block.ms, not at startup.
Topics as code, and why auto-creation is switched off
There are three different mechanisms that can create a topic, and they are often confused.
| Mechanism | Controlled by | What you get |
|---|---|---|
| Broker auto-creation on first use | Broker auto.create.topics.enable (default true) together with the consumer's allow.auto.create.topics (default true) | A topic with the broker defaults: num.partitions 1 and default.replication.factor 1 unless the operators changed them |
NewTopic beans through KafkaAdmin | spring.kafka.admin.auto-create (default true) | A topic with exactly the partitions, replication factor and topic configs you declared |
| Platform tooling | Your organisation's pipeline (Terraform, an operator, a GitOps repository) | Reviewed topics with owners, quotas and access rules |
Broker auto-creation is the dangerous one. A typo in a topic name creates a new topic with one partition and no replication, the producer happily writes to it, and nobody reads it. A partition count chosen by accident is also hard to correct later, because adding partitions changes which partition a key maps to. Production clusters therefore usually set auto.create.topics.enable=false and restrict who may create topics with ACLs.
NewTopic beans are explicit and versioned with the code, which makes them a good fit for development, tests and teams that own their topics. Know what KafkaAdmin does with a topic that already exists: it leaves the configuration alone unless spring.kafka.admin.modify-topic-configs is true, but it does increase the partition count if the bean declares more partitions than the topic has. Raising a number in code is therefore a production change to key-to-partition mapping.
When to declare your own beans
Stay with properties
- One cluster, one key type and one value format per direction
- Settings that are plain client or container properties
- Different values per environment: use profiles and environment variables
- A cross-cutting bean Boot picks up by type:
CommonErrorHandler,RecordMessageConverter,ConsumerAwareRebalanceListener
Declare a bean
- A second listener container factory with another acknowledgement mode or batch delivery
- Serializers built in code, for example with a customised
ObjectMapper - A second cluster, or a second template with different serializers
- Programmatic topic sets that depend on configuration
When you do declare a bean, remember the back-off rules in the first table. An extra listener container factory under another name is additive: Boot still creates kafkaListenerContainerFactory. A KafkaTemplate, ProducerFactory or ConsumerFactory bean of your own replaces Boot's, for every user of that type. For the factories the Spring reference adds a detail: declare the bean method with wildcard generic types, as Boot itself does, otherwise the pieces Boot still auto-configures may not find it. Boot's ConcurrentKafkaListenerContainerFactoryConfigurer lets a custom container factory start from everything in spring.kafka.listener.*, which the code below uses.
Architecture
Drawing diagram…
A flowchart. application.yml and the active profile are bound to KafkaProperties, which yields producer, consumer and admin properties. Producer properties feed the ProducerFactory, which feeds KafkaTemplate. Consumer properties feed the ConsumerFactory, which together with the listener settings feeds the listener container factory. Admin properties feed KafkaAdmin. Application code consists of a publisher that calls the template, a KafkaListener method invoked by the container, and NewTopic beans handed to KafkaAdmin. Template, container and admin all connect to the Kafka cluster.
flowchart TB
Yml["application.yml plus the active profile"] --> KP["KafkaProperties"]
KP --> PP["Producer properties"]
KP --> CP["Consumer properties"]
KP --> AP["Admin properties"]
subgraph Auto["Beans created by Spring Boot"]
PF["ProducerFactory"]
KT["KafkaTemplate"]
CF["ConsumerFactory"]
LCF["kafkaListenerContainerFactory"]
KA["KafkaAdmin"]
end
PP --> PF
PF --> KT
CP --> CF
CF --> LCF
KP --> LCF
AP --> KA
subgraph Yours["Your code"]
Pub["Publisher calls KafkaTemplate send"]
Lis["Method annotated KafkaListener"]
NT["NewTopic beans"]
end
Pub --> KT
LCF --> Lis
NT --> KA
KT --> Brk["Kafka cluster"]
LCF --> Brk
KA --> Brk- Everything in the
Autobox is replaceable, but each bean has its own condition. Replacing one does not replace the others, which is how half-custom configurations end up with two different sets of settings. - The listener container factory takes input from two places: the consumer properties (Kafka client behaviour) and
spring.kafka.listener.*(Spring's threads and acknowledgement mode). KafkaAdminis a separate client with its own connection. An application that is allowed to read and write a topic is not necessarily allowed to create it.- Your code depends on three small surfaces. That is what makes it testable against an embedded broker or Testcontainers with no code change; see testing and observability.
Message flow
Drawing diagram…
A sequence diagram of application startup. The application context initialises KafkaAdmin with the NewTopic beans. KafkaAdmin asks the broker to describe the topics, learns that orders.placed exists and payments.completed does not, and creates payments.completed with 6 partitions and replication factor 3. The context then starts the listener containers, which join the group payment-service, receive an assignment and begin polling. The context is ready before any producer exists. On the first send, KafkaTemplate creates the producer, fetches metadata, produces the record and receives the acknowledgement.
sequenceDiagram participant App as Application context participant Adm as KafkaAdmin participant Con as Listener container participant Tpl as KafkaTemplate participant B as Kafka broker App->>Adm: initialise with NewTopic beans Adm->>B: describe topics orders.placed and payments.completed B-->>Adm: orders.placed exists, payments.completed unknown Adm->>B: create payments.completed, 6 partitions, replication factor 3 B-->>Adm: created App->>Con: start containers, late lifecycle phase Con->>B: join group payment-service B-->>Con: assignment orders.placed 0 to 5 Con->>B: poll Note over App,Tpl: context is ready, no producer exists yet App->>Tpl: first send Tpl->>B: create producer, fetch metadata B-->>Tpl: partition leaders Tpl->>B: produce record B-->>Tpl: acknowledged
- Topics are handled before listeners start, so a listener on a topic declared in the same application finds it.
- Listener containers start in a late lifecycle phase, after ordinary beans are ready, so the first record does not arrive in a half-built application.
- The producer is lazy. Health at startup says nothing about whether sends will work; the first send pays for the connection and metadata fetch.
- An existing topic is not recreated. Only a missing topic, or a lower partition count than declared, leads to a change on the cluster.
In a real system
Take the payment service of the example project. It consumes orders.placed in the group payment-service and publishes the outcome to payments.completed, keyed by order id so that everything about one order stays in one partition. Both topics have 6 partitions, replication factor 3 and min.insync.replicas=2 in production.
| Setting | Local profile | Production profile |
|---|---|---|
| Brokers | localhost:9092, one container | Three or more bootstrap addresses from the environment |
| Security | Plaintext | SASL_SSL with credentials from a secret store |
| Topics | Created by NewTopic beans, replication factor 1 | Created by the platform pipeline; spring.kafka.admin.auto-create=false |
| A topic the service reads is missing | Startup fails; start the owning service first or create the topic by hand | Startup fails, and the deployment with it (missing-topics-fatal=true) |
| First start of a new group | auto-offset-reset=earliest | The same: a new payment consumer must not skip existing orders |
| Listener threads | 1 | 3 per instance, two instances for six partitions |
The rule behind the split: guarantees are shared, addresses and sizes are per environment. If acks or the acknowledgement mode differed between local and production, tests on a laptop would prove nothing about production behaviour.
Java and Spring Boot
The setup of the payment service: build file, local broker, shared and per-profile configuration, topic declarations and one additional container factory. It was written against the Spring Boot 3.5 and Spring for Apache Kafka 3.3 references and has not been compiled or run in this environment.
<parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>3.5.16</version> <relativePath/></parent><properties> <java.version>21</java.version></properties><dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka-test</artifactId> <scope>test</scope> </dependency></dependencies>Why it is written this way
- No version on
spring-kafka. The Boot parent manages it (3.3.16 in Boot 3.5.16) together with thekafka-clientsversion it was built against (3.9.2). - There is no
spring-boot-starter-kafkain Boot 3.x. The plainspring-kafkadependency is what activates the auto-configuration. spring-boot-starter-webbrings Jackson, which the JSON serializer and deserializer need.- The full build file of each service is in the example project.
services: broker: image: apache/kafka:4.4.0 container_name: broker ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker,controller KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093 KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"Why it is written this way
- The official
apache/kafkaimage runs in KRaft mode: one container is both broker and controller (KAFKA_PROCESS_ROLES: broker,controller). There is no ZooKeeper container. - The image maps environment variables to broker settings: prefix
KAFKA_, upper case, dots become underscores.KAFKA_AUTO_CREATE_TOPICS_ENABLEisauto.create.topics.enable. - The image documentation warns that as soon as you override any setting, none of the image's default configuration is used. That is why the listener and quorum settings are all spelled out.
KAFKA_ADVERTISED_LISTENERSis the address the broker tells clients to use after bootstrap.localhost:9092is right for an application on the host. A client in another container would need a second listener advertised under the container name; the image documentation has that example.- The three replication settings are 1 because the internal offsets and transaction topics default to replication factor 3, which a single broker cannot satisfy.
- Auto-creation is switched off locally too, so a misspelt topic fails on the laptop the same way it would in production.
- The tools are inside the container:
docker exec --workdir /opt/kafka/bin -it broker ./kafka-topics.sh --bootstrap-server localhost:9092 --list.
spring: application: name: payment-service kafka: bootstrap-servers: localhost:9092 client-id: payment-service producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer acks: all compression-type: lz4 properties: "[enable.idempotence]": true "[linger.ms]": 10 "[max.block.ms]": 5000 "[delivery.timeout.ms]": 30000 "[request.timeout.ms]": 10000 "[spring.json.add.type.headers]": false consumer: group-id: payment-service auto-offset-reset: earliest max-poll-records: 100 key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer properties: "[spring.deserializer.value.delegate.class]": org.springframework.kafka.support.serializer.JsonDeserializer "[spring.json.value.default.type]": com.example.payments.events.OrderPlaced "[spring.json.use.type.headers]": false "[partition.assignment.strategy]": org.apache.kafka.clients.consumer.CooperativeStickyAssignor listener: ack-mode: record concurrency: 1 missing-topics-fatal: trueapp: topics: orders-placed: orders.placed payments-completed: payments.completed partitions: 6 replicas: 3 min-insync-replicas: 2Why it is written this way
- The shared file holds what must be identical everywhere: serializers,
acks, idempotence, the group id and the acknowledgement mode. Profiles below change only addresses, security and sizes. acks: allandenable.idempotence: trueare already the client defaults. Writing them down documents the intent and turns a later conflicting edit into a startup error. The reasoning is in acknowledgements, retries and idempotence.linger.ms,max.block.msand the timeouts have no dedicated Boot key, so they go underpropertiesin bracket notation.group-idis the consumer group. Every instance of this service uses the same one so that they share the partitions; a different service uses a different group and gets its own copy of every record.auto-offset-reset: earliestonly applies when the group has no committed offset for a partition (a brand-new group, or offsets that expired). The client default islatest, which makes a new group silently skip everything already in the topic.- The value deserializer is
ErrorHandlingDeserializerwrappingJsonDeserializer, so a record that cannot be parsed reaches the error handler instead of failing insidepollforever. Serialisation explains eachspring.json.*line. CooperativeStickyAssignorreplaces the 3.9 client's default list (RangeAssignor, thenCooperativeStickyAssignor), so a rebalance revokes only the partitions that move. All members of a group must be able to agree on an assignor, so change it for a running group with the rolling procedure in consumer rebalancing.ack-mode: recordcommits after each record the listener returns from. The default isbatch. The trade-off is covered in the next lesson.- The
app.topicsblock is this application's own configuration, bound to a record below. Topic names are data, not string literals in annotations.
spring: kafka: admin: auto-create: true fail-fast: trueapp: topics: partitions: 3 replicas: 1 min-insync-replicas: 1Why it is written this way
- Active with
--spring.profiles.active=localorSPRING_PROFILES_ACTIVE=local. A profile file only lists differences; everything else comes fromapplication.yml. - One broker cannot hold three replicas. With the production values every
acks=allsend would fail withNotEnoughReplicasException, so the local profile sets replication factor 1 andmin.insync.replicas1. fail-fast: truemakes a broker that is not running an immediate startup error instead of a log line followed by a hanging first request.missing-topics-fatalstays true locally.orders.placedis declared by the order service, so start that service first or create the topic withkafka-topics.sh --create; the error message names the missing topic.
spring: kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS} security: protocol: SASL_SSL properties: "[sasl.mechanism]": SCRAM-SHA-512 "[sasl.jaas.config]": ${KAFKA_SASL_JAAS_CONFIG} admin: auto-create: false listener: concurrency: 3Why it is written this way
KAFKA_BOOTSTRAP_SERVERShas no fallback on purpose. A production instance that silently falls back tolocalhost:9092is worse than one that refuses to start.- List several brokers in the variable. The list is only used for the first connection, but with a single address startup depends on that one broker being up.
- Security settings sit under
spring.kafkaandspring.kafka.properties, so producer, consumer and admin all receive them. The mechanism and the JAAS line depend on how your cluster is secured; see Kafka security. admin.auto-create: falseleaves theNewTopicbeans as documentation of what the service expects. The platform pipeline creates the topics, andmissing-topics-fatal: truefrom the shared file makes a missing one a failed deployment.concurrency: 3gives each instance three consumer threads. Two instances then cover the six partitions with one thread each.
package com.example.payments.config;import org.springframework.boot.context.properties.ConfigurationProperties;@ConfigurationProperties("app.topics")public record TopicProperties( String ordersPlaced, String paymentsCompleted, int partitions, int replicas, int minInsyncReplicas) {}Why it is written this way
- A record bound from
app.topics.*. Relaxed binding mapsorders-placedtoordersPlaced. - Publishers inject this record for topic names. Listener annotations can refer to the same keys with a placeholder such as
${app.topics.orders-placed}.
package com.example.payments.config;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.boot.context.properties.EnableConfigurationProperties;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configuration@EnableConfigurationProperties(TopicProperties.class)public class KafkaTopicConfig { @Bean NewTopic paymentsCompletedTopic(TopicProperties topics) { return TopicBuilder.name(topics.paymentsCompleted()) .partitions(topics.partitions()) .replicas(topics.replicas()) .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, String.valueOf(topics.minInsyncReplicas())) .build(); }}Why it is written this way
- The service declares the topic it owns (the one it publishes to).
orders.placedbelongs to the order service and is declared there; two applications declaring the same topic with different settings is a source of surprises. - The durability settings live next to the producer that relies on them.
acks=allprotects a record only together with replication factor 3 andmin.insync.replicas=2. KafkaAdmincreates the topic if it is missing. If it exists, its configuration is left alone (unlessspring.kafka.admin.modify-topic-configs=true), but a higher partition count in the bean is applied to the existing topic.- With
spring.kafka.admin.auto-create=falsethis bean is not applied to the cluster.
package com.example.payments.config;import org.springframework.boot.autoconfigure.kafka.ConcurrentKafkaListenerContainerFactoryConfigurer;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;import org.springframework.kafka.core.ConsumerFactory;@Configurationpublic class KafkaListenerFactoriesConfig { /** * A second container factory for listeners that receive a whole poll as a list. * Boot's default factory, kafkaListenerContainerFactory, is still created. */ @Bean ConcurrentKafkaListenerContainerFactory<Object, Object> batchKafkaListenerContainerFactory( ConcurrentKafkaListenerContainerFactoryConfigurer configurer, ConsumerFactory<Object, Object> consumerFactory) { ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>(); configurer.configure(factory, consumerFactory); factory.setBatchListener(true); return factory; }}Why it is written this way
- This is the one kind of custom bean most services eventually need: an additional container factory. It has a different bean name, so Boot's default factory does not back off.
configurer.configure(...)applies everything Boot would apply to its own factory:spring.kafka.listener.*, the error handler bean, converters and interceptors. Only the difference is set afterwards. Building the factory by hand would silently drop those settings.- It reuses Boot's
ConsumerFactory, so the consumer block ofapplication.ymlapplies to both factories. - A listener selects it with
@KafkaListener(containerFactory = "batchKafkaListenerContainerFactory"). - There is deliberately no
ProducerFactory,KafkaTemplateorConsumerFactorybean here. Declaring one would replace Boot's for the whole application.
Configuration
Defaults are from the Spring Boot 3.5 application properties appendix and the Boot source where the appendix lists none; Kafka client defaults are those of the 3.9 client and broker defaults those of Kafka 4.4.
spring.kafka.bootstrap-serversDefault none listed in the appendix; Boot's KafkaProperties initialises it to localhost:9092
- What it does
- Host and port pairs for the first connection. Applies to producer, consumer and admin unless overridden under
producer,consumeroradmin. - When to change it
- Give several brokers. In production take it from the environment with no fallback, so a missing value stops the start.
spring.kafka.producer.key-serializer / value-serializerDefault none listed in the appendix; Boot's KafkaProperties uses StringSerializer for both
- What it does
- Serializer classes for record keys and values.
- When to change it
- Set them explicitly. With the String default, sending an object fails at the first send with a
ClassCastExceptionwrapped in a serialisation error.
spring.kafka.consumer.key-deserializer / value-deserializerDefault none listed in the appendix; Boot's KafkaProperties uses StringDeserializer for both
- What it does
- Deserializer classes for record keys and values.
- When to change it
- Wrap any deserializer that can fail in
ErrorHandlingDeserializerso that a bad record is handled instead of blocking the partition.
spring.kafka.consumer.group-idDefault none
- What it does
- The consumer group for listeners that do not set one on the annotation.
- When to change it
- One group per service and purpose. A listener without any group id fails to start.
spring.kafka.consumer.auto-offset-resetDefault none in Spring Boot; Kafka client default latest
- What it does
- Where to start when the group has no committed offset for a partition, or the committed offset no longer exists:
earliest,latestornone. - When to change it
- Use
earliestfor services that must process everything. It is not a replay switch: it has no effect once offsets are committed.
spring.kafka.consumer.enable-auto-commitDefault none in Spring Boot; Spring's listener container sets enable.auto.commit=false unless you set it
- What it does
- Whether the Kafka client commits offsets on a timer in the background.
- When to change it
- Leave it unset. The container commits according to
spring.kafka.listener.ack-mode, after your listener has run.
spring.kafka.producer.properties.* / spring.kafka.consumer.properties.* / spring.kafka.properties.*Default empty
- What it does
- Any Kafka client property (or Spring serializer property) by its real name. Values here override the dedicated Boot key for the same setting.
- When to change it
- Use bracket notation in YAML (
"[linger.ms]": 10). Check theProducerConfig valuesandConsumerConfig valueslog blocks to confirm what was applied.
spring.kafka.listener.concurrencyDefault none in Spring Boot; the container default is 1
- What it does
- Number of consumer threads (and consumers) per
@KafkaListenerin this application instance. - When to change it
- Size it so that instances multiplied by concurrency does not exceed the partition count; extra consumers sit idle.
spring.kafka.listener.missing-topics-fatalDefault false
- What it does
- Whether a listener container refuses to start when one of its topics does not exist.
- When to change it
- Set true where topics are provisioned outside the application.
spring.kafka.admin.auto-createDefault true
- What it does
- Whether
KafkaAdmincreatesNewTopicbeans on the cluster during context start. - When to change it
- Set false in environments where a pipeline owns topics or the application has no permission to create them.
spring.kafka.admin.fail-fastDefault false
- What it does
- Whether startup fails when
KafkaAdmincannot reach the broker. - When to change it
- Useful locally and in tests. In production decide whether a service that also serves HTTP should refuse to start when Kafka is down.
spring.kafka.admin.modify-topic-configsDefault false
- What it does
- Whether
KafkaAdminchanges the configuration of topics that already exist to match theNewTopicbeans. - When to change it
- Leave false unless the application is the single owner of the topic's configuration.
auto.create.topics.enable (broker)Default true
- What it does
- Whether the broker creates a topic when a client first asks for one that does not exist, using
num.partitions(default 1) anddefault.replication.factor(default 1). - When to change it
- Set false on any shared cluster. Create topics deliberately, with a chosen partition count and replication factor.
allow.auto.create.topics (consumer)Default true
- What it does
- Whether a consumer's subscription may trigger broker auto-creation. It only has an effect when the broker allows auto-creation.
- When to change it
- Set false under
spring.kafka.consumer.propertiesif you cannot change the broker setting.
Hands-on exercise
Read your NewTopic declaration on a real cluster shape
See what the production topic declaration of this lesson means on three brokers, and why the same numbers cannot work on the one-broker local setup.
Steps · 0 of 6 done
Cluster explorer: the payment service's topics
Simulation · not a real clusterThree brokers and the two topics exactly as the production profile expects them. Stop brokers and watch leaders, in-sync replicas and whether acks=all writes are accepted. This is a deterministic simulation, not a real cluster.
What you should see
With one broker down every partition still has a leader and two in-sync replicas, so acks=all writes continue. With two down, one replica is left, which is below min.insync.replicas 2, and acks=all writes are refused although reads still work. On a single local broker the only values that can work are replication factor 1 and min.insync.replicas 1, which is exactly what the local profile overrides; the describe output should show 3 partitions and replication factor 1.
What can go wrong
Declaring one custom bean and losing Boot's configuration
- Symptom
- After adding a second
KafkaTemplatefor another value type, the original publisher fails at startup with no matching bean, or sends with different settings thanapplication.ymlsays. - Cause
- Boot's template is created only when no
KafkaTemplatebean exists. The new bean made Boot back off, and a factory built by hand from a fresh map does not contain thespring.kafka.producer.*settings. - Fix
- When you replace a bean, declare every variant you need explicitly, and build factories from Boot's
KafkaProperties(for examplebuildProducerProperties()) plus your changes. Prefer properties, customisers and additional container factories, which do not replace anything.
The same setting in two places
- Symptom
- Changing
spring.kafka.producer.ackshas no effect; theProducerConfig valueslog block shows another value. - Cause
- The key is also set under
spring.kafka.producer.properties(or the commonspring.kafka.properties). Values in apropertiesmap override the dedicated Boot key. - Fix
- Set each Kafka setting once. Use the dedicated key where one exists and the
propertiesmap only for the rest.
Topics created by accident
- Symptom
- A topic with one partition and replication factor 1 exists that nobody declared; a service writes to it and no consumer ever reads the records.
- Cause
- A misspelt topic name, with broker
auto.create.topics.enableleft at true. The first metadata request created the topic with broker defaults. - Fix
- Disable broker auto-creation, take topic names from configuration instead of repeating string literals, and set
spring.kafka.listener.missing-topics-fatal=true.
Production replication settings on a single local broker
- Symptom
- Locally the application starts, but topic creation logs an
InvalidReplicationFactorException, or every send fails withNotEnoughReplicasExceptionafterdelivery.timeout.ms. - Cause
- Replication factor 3 cannot be placed on one broker, and
min.insync.replicas=2can never be met by one replica. - Fix
- Override replication factor and
min.insync.replicasto 1 in the local profile, as this lesson does, or run three brokers locally.
A raised partition count applied by KafkaAdmin
- Symptom
- After a release that changed
partitionsfrom 6 to 12 in configuration, events for the same order start to appear out of order for a while. - Cause
KafkaAdminincreases the partition count of an existing topic when theNewTopicbean declares more. Keys now hash to different partitions, so old and new events for one key sit in different partitions.- Fix
- Treat the partition count as an operational change with a migration plan, not a property edit. Where the pipeline owns topics, keep
spring.kafka.admin.auto-create=false. See keys and partitioning.
Advertised listener that the client cannot reach
- Symptom
- The application connects to the bootstrap address, then logs repeated connection failures to another host name or to
localhost, and sends time out. - Cause
- After bootstrap, clients connect to the addresses the broker advertises. A broker in a container that advertises
localhost:9092is unreachable from another container; one that advertises its container name is unreachable from the host. - Fix
- Advertise an address that is valid from where the client runs. For both host and container clients, configure two listeners with different advertised addresses, as in the image documentation's multi-listener example.
How to troubleshoot
Not sure which values the clients really run with
Check
The `ProducerConfig values:`, `ConsumerConfig values:` and `AdminClientConfig values:` blocks the Kafka client logs at INFO when each client is created
- How to read it
- They show the merged result of client defaults, Boot keys and
propertiesoverrides. A value you set that does not appear there was set under the wrong key or overridden. The producer block appears only after the first send.
Startup fails with an error that no group id was found for a listener
Check
`spring.kafka.consumer.group-id` in the active configuration, and the `groupId` and `id` attributes of the `@KafkaListener`
- How to read it
- A subscribing consumer needs a group. Set the property, or
groupIdon the annotation. If the property is set but only in a profile file, the profile is probably not active.
The application starts but never receives or sends anything, and the log repeats that a connection to a node could not be established
Check
docker exec --workdir /opt/kafka/bin -it broker ./kafka-broker-api-versions.sh --bootstrap-server localhost:9092
- How to read it
- If this works inside the container but the application cannot connect, compare
spring.kafka.bootstrap-serverswith the broker's advertised listeners and the published port. The host name in the log line is the advertised one.
Need to confirm what a topic really looks like on the cluster
Check
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic payments.completed
- How to read it
- Compare
PartitionCount,ReplicationFactorandConfigswith theNewTopicbean. A difference in configs on an existing topic is expected, becauseKafkaAdmindoes not modify them by default.
A new consumer group starts and processes nothing although the topic has records
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- If
CURRENT-OFFSETequalsLOG-END-OFFSETright after the first start, the group began at the end:auto.offset.resetwaslatest. Settingearliestnow changes nothing for this group, because offsets are already committed; reset them with--reset-offsetswhile the group is stopped.
Which profile and which configuration files were used?
Check
The startup log line `The following 1 profile is active: "prod"` and, with Actuator, the `/actuator/env` endpoint
- How to read it
- If the line says no active profile was set, only
application.ymlapplied, which in this lesson meanslocalhost:9092and no topic overrides.
Key takeaways
spring-kafkawraps the Kafka client; Spring Boot buildsProducerFactory,KafkaTemplate,ConsumerFactory, the listener container factory andKafkaAdminfromspring.kafka.*.- Each auto-configured bean backs off separately when you declare your own. Add container factories freely; replace templates and factories only knowingly.
- Any client setting without a dedicated Boot key goes under a
propertiesmap, and that map wins over the dedicated key. - Boot 3.5 runs a 3.9 client against 4.x brokers. Defaults belong to the client version, so write down the settings you rely on and read the config blocks in the log.
- Declare topics deliberately, as
NewTopicbeans or in a pipeline, and switch off broker auto-creation. Remember thatKafkaAdminraises partition counts. - Share guarantees across environments and vary only addresses, security and sizes through profiles.
- Admin and listeners connect at startup; the producer connects on the first send.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A service uses Boot's auto-configured KafkaTemplate with JsonSerializer and acks=all from application.yml. A developer adds a @Bean KafkaTemplate<String, byte[]> built from a new DefaultKafkaProducerFactory with only bootstrap servers and ByteArraySerializer in its map, for one audit topic. After the release, order events fail to send with a serialisation error.
What happened?
Scenario 2 of 3
A new inventory service is deployed with group id inventory-service on a cluster where auto.create.topics.enable is true. Its listener is annotated with topics = "order.placed" (the real topic is orders.placed, 6 partitions). The service starts cleanly, reports healthy and processes nothing for two days.
Which combination would have turned this into a failed deployment?
Scenario 3 of 3
Locally you run the single-broker compose file. You start the service without a profile, so app.topics has replicas 3 and min-insync-replicas 2, and spring.kafka.admin.fail-fast is false. The application starts. The first payment result send blocks and then fails.
What is the most direct correction?
References
References: Spring Boot 3.5 reference: Messaging, Apache Kafka Support (auto-configuration, additional properties) ↗ · Spring Boot 3.5: Common application properties (spring.kafka.*) ↗ · Spring Boot 3.5: Managed dependency coordinates (spring-kafka 3.3.16, kafka-clients 3.9.2) ↗ · Spring for Apache Kafka 3.3 reference: Configuring Topics (KafkaAdmin, NewTopic, TopicBuilder) ↗ · Spring for Apache Kafka 3.3 reference: Message Listener Containers (enable.auto.commit, missingTopicsFatal) ↗ · Spring for Apache Kafka 3.3 reference: Serialization (overriding Boot's factories with wildcard generics) ↗ · Spring Boot 3.5 source: KafkaAutoConfiguration (bean conditions) ↗ · Apache Kafka official Docker image: usage and environment variables ↗ · Apache Kafka Docker image usage guide and example compose files ↗ · Apache Kafka 4.4: Broker configs (auto.create.topics.enable, num.partitions, default.replication.factor) ↗ · Apache Kafka 3.9: Consumer configs (defaults of the client Spring Boot 3.5 manages) ↗ · Report a problem