Path 2 · lesson 1 of 8
Clusters, brokers and topics
What a broker does, how brokers form a cluster, how a client gets from a bootstrap address to the right partition leader, and what a topic really is.
What you will learn
- List the jobs a single broker does and the ones it leaves to the controller
- Trace how a client goes from
bootstrap.serversto the broker that leads a partition - Read the output of
kafka-topics.sh --describeand say which broker serves which partition - Predict which clients notice when one broker stops, and which do not
- Declare a topic with an explicit partition count and replication factor from Spring Boot
Before you start
The concept
A broker is one Kafka server process. It owns some disks, listens on a TCP port, and does four things: it appends records that producers send to it, it serves records to consumers that ask for them, it copies partitions from other brokers so that there is more than one copy, and it tells any client that asks where everything in the cluster lives.
A cluster is a set of brokers that share one cluster id and one source of truth about the cluster's state. That source of truth is the controller: a small group of servers (usually three or five) that agree, through a Raft-based quorum called KRaft, on which brokers are alive, which topics exist and which broker leads each partition. Brokers hold the data. Controllers hold the map.
A topic is a name for a stream of records, such as orders. It is not one file on one machine. A topic is split into partitions, each partition is an independent append-only log, and each partition is stored on several brokers (its replicas). One replica is the leader and takes all the writes for that partition. A topic is therefore better thought of as a name that groups a set of logs spread across the cluster.
| Thing | What it is | Who decides it |
|---|---|---|
| Broker | A server that stores partition replicas and answers produce, fetch and metadata requests | You, when you start it with a node.id |
| Cluster | Brokers plus a controller quorum that share a cluster id | You, when you format storage with the same cluster id |
| Topic | A named set of partitions with a replication factor and settings | Whoever creates it (or the broker, if auto-creation is on) |
| Partition leader | The one replica that accepts writes for a partition right now | The controller |
Click around a three-broker cluster
Simulation · not a real clusterClick each broker to see what it stores and leads, then click the topics to see how their partitions are spread across the brokers.
Why it exists
A message broker that lives on one server has a hard ceiling: one machine's disk, one machine's network card, and an outage whenever that machine restarts. The usual way to grow past one server is to put several behind a load balancer or a routing tier. For a log that fails in two ways. The router becomes the bottleneck that every byte passes through, and it hides where the data is, so the client cannot keep a stream of related records going to the same place in order.
Kafka made the opposite choice: there is no routing tier. Clients are told the layout of the cluster and connect directly to the broker that leads the partition they need. That is why every broker can answer a metadata request, and why a Kafka client holds connections to many brokers at once. The cost of this design is that every broker must be individually reachable from every client, which is the source of the most common connection problem in Kafka (see the pitfalls).
Splitting a topic into partitions is what lets a topic be larger and busier than one machine. A single partition has to fit on one server, but a topic with twelve partitions can spread its reads and writes over up to twelve leaders. And someone has to make cluster-wide decisions consistently: if two brokers both believed they led orders-0, the log would fork. Giving those decisions to one controller quorum, instead of letting brokers negotiate among themselves, is what keeps a single answer to the question of who leads what.
Routing tier in front of the servers
- Clients know one address
- Every request takes an extra hop through the router
- The router must scale with total traffic
- The client cannot choose where a record lands
Kafka: cluster-aware clients
- Clients learn every broker's address from metadata
- Requests go straight to the partition leader
- Adding brokers adds capacity with no shared choke point
- The producer picks the partition, so a key can keep its order
How it works inside
From bootstrap address to partition leader
bootstrap.servers is only a way in. The client connects to one address from that list and sends a Metadata request. The broker answers from its own copy of the cluster metadata with the list of live brokers (id, host, port, rack) and, for each topic asked about, every partition with its leader, its replicas and its in-sync replicas. From then on the client ignores the bootstrap list and opens connections to the brokers named in the response.
- Connect to any address in
bootstrap.serversand ask for metadata about the topics the client uses. - Cache the answer: brokers and their addresses, and the leader of each partition.
- Send each produce request to the leader of the target partition. Consumers send fetch requests to the leader as well (or to a nearby follower when follower fetching is configured, covered in replication).
- Refresh the cache when a broker answers with an error that means the map is stale, such as
NOT_LEADER_OR_FOLLOWER, and in any case everymetadata.max.age.ms(5 minutes by default).
What a broker keeps on disk
For every partition replica it hosts, a broker has a directory under one of its log.dirs, named after the topic and the partition number: orders-0, orders-1, payments-0. Inside are the segment files of that partition's log (covered in log segments and retention). A broker that hosts no replica of orders-2 has no such directory and will answer a produce request for it with an error telling the client to refresh its metadata. This is also why topic names are limited to 249 characters: the name has to fit in a directory name together with the partition number.
Where the controller fits
Each server is started with process.roles set to broker, controller, or both (combined mode, meant for development). The controllers form a quorum; one is the active controller and the others are hot standbys. Brokers register with the active controller and keep a session alive by heartbeating every broker.heartbeat.interval.ms (2 seconds). When a broker's heartbeats stop for broker.session.timeout.ms (9 seconds), the controller treats it as gone and picks new leaders for the partitions it led.
Every decision the controller makes (a topic created, a leader changed, a broker registered) is a record in an internal metadata log. Brokers follow that log and keep the result in memory, which is how any broker can answer a client's metadata request without asking the controller. Producers and consumers never talk to the controller. The quorum and the metadata log are the subject of KRaft and the controller.
Broker
- Stores partition replicas on disk
- Serves produce and fetch requests
- Replicates partitions it follows
- Answers metadata requests from its copy of the cluster state
- Hosts group coordinators for consumer groups
Controller
- Stores no topic data
- Tracks which brokers are alive
- Creates and deletes topics, assigns replicas to brokers
- Elects a leader for each partition
- Writes every change to the metadata log
Architecture
Drawing diagram…
A producer first asks one broker for metadata, then sends records straight to the broker leading orders partition 0. A consumer fetches from the leader of partition 2. Three controllers form a quorum whose metadata log all three brokers follow.
flowchart LR
subgraph APPS["Client applications"]
P["Order service producer"]
C["Payment service consumer"]
end
subgraph CL["Kafka cluster"]
subgraph B1["Broker 1"]
A0["orders-0 leader"]
A2["orders-2 follower"]
end
subgraph B2["Broker 2"]
A1["orders-1 leader"]
A0f["orders-0 follower"]
end
subgraph B3["Broker 3"]
A2l["orders-2 leader"]
A1f["orders-1 follower"]
end
end
subgraph Q["Controller quorum"]
K1["Controller 1 active"]
K2["Controller 2 standby"]
K3["Controller 3 standby"]
end
P -->|"1 metadata request"| A1
P -->|"2 produce to leader"| A0
C -->|"fetch from leader"| A2l
A0 -. "replicated to" .-> A0f
K1 -. "metadata log" .-> B1
K1 -. "metadata log" .-> B2
K1 -. "metadata log" .-> B3- Three brokers each hold replicas of several partitions and lead some of them. Leadership is spread so that no single broker takes all the writes for a topic.
- The producer's first request can go to any broker. Its produce requests go to the broker that leads the target partition, with no router in between.
- The controllers are separate servers. They do not sit on the data path: if all three brokers are healthy, a produce request never touches a controller.
- The dotted lines from the active controller are the metadata log. Brokers replay it, which is how broker 2 can tell the producer that broker 1 leads
orders-0.
Message flow
Drawing diagram…
The producer asks broker 2 for metadata, then produces to broker 1, the leader of orders partition 0. After leadership moves, broker 1 rejects the request, the producer refreshes its metadata and resends the same records to broker 3.
sequenceDiagram participant P as Producer participant B2 as Broker 2 participant B1 as Broker 1 participant B3 as Broker 3 Note over P: bootstrap.servers lists broker 2 and broker 3 P->>B2: Connect and send Metadata request for orders B2-->>P: Brokers 1 2 3 with addresses and the leader of each partition P->>B1: Produce to orders-0 B1-->>P: Acknowledged at offset 41 Note over B1,B3: Later the controller moves leadership of orders-0 to broker 3 P->>B1: Produce to orders-0 B1-->>P: Error NOT_LEADER_OR_FOLLOWER P->>B2: Metadata request for orders B2-->>P: Leader of orders-0 is now broker 3 P->>B3: Produce to orders-0 again B3-->>P: Acknowledged at offset 42
- The producer never sends records to the bootstrap broker unless that broker happens to lead the partition.
- Broker 1 was never in
bootstrap.servers. The client learned its address from the metadata response. - A leadership change is not announced to clients. The client finds out when the old leader refuses a request, refreshes its metadata from any broker, and retries. The producer does this inside
send(); the application only sees a slightly slower acknowledgement. - Offsets continue from where they were: the new leader already had the records up to offset 41 as a follower.
In a real system
A typical order platform runs six brokers spread over three availability zones and three dedicated controllers, one per zone. The services share a handful of topics, each created explicitly by the team that owns it:
| Topic | Key | Partitions | Replication factor | Written by | Read by |
|---|---|---|---|---|---|
orders | order id | 12 | 3 | Order service | Payment, inventory and notification services |
payments | order id | 12 | 3 | Payment service | Order and notification services |
inventory-reservations | SKU | 6 | 3 | Inventory service | Order service |
With twelve partitions and a replication factor of three, orders has 36 partition replicas, six per broker on average, and each broker leads about two partitions. Every service is configured with the same three bootstrap addresses, one broker per zone, so that a new instance can start while any single zone is down. Auto-creation of topics is switched off on the brokers, so a typo in a topic name fails loudly instead of producing a one-partition, unreplicated topic that nobody owns.
When broker 4 is restarted for a kernel patch, the only producers that notice are the ones writing to the two orders partitions it led, and what they notice is a few retried batches. The example project uses the same topics.
Java and Spring Boot
Spring Boot creates a KafkaAdmin for you. Declaring NewTopic beans makes it create missing topics at startup, and the same admin configuration can be used to look at what the cluster reports. The runner below prints the brokers and the leader of each orders partition, which is exactly the metadata a producer works from.
spring: kafka: # Three brokers, one per availability zone. Only used for the first connection. bootstrap-servers: kafka-a1.internal:9092,kafka-b1.internal:9092,kafka-c1.internal:9092 client-id: order-service admin: # Fail the application start if no broker can be reached, instead of logging and carrying on. fail-fast: trueWhy it is written this way
bootstrap-serverslists more than one broker so that a new instance can start while one of them is down. It does not need to list every broker.client-idshows up in broker request logs and quota metrics, which makes it possible to tell which service a connection belongs to.spring.kafka.admin.fail-fastdefaults to false: by default a missing broker at startup is only logged and topics are not created.
package com.example.orders.kafka;import org.apache.kafka.clients.admin.NewTopic;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class TopicConfig { public static final String ORDERS = "orders"; @Bean public NewTopic ordersTopic() { return TopicBuilder.name(ORDERS) .partitions(12) .replicas(3) .build(); }}Why it is written this way
- Partition count and replication factor are written down in code. Leaving them out would apply the broker defaults, which are 1 and 1 unless an operator changed them.
KafkaAdminonly creates topics that are missing. If the topic exists with fewer partitions it adds partitions, which changes where keys land; it never reduces them and never changes the replication factor.- Many teams create topics from infrastructure code instead and keep this bean only for local development. Either way the numbers should be explicit.
package com.example.orders.kafka;import java.util.List;import java.util.Map;import org.apache.kafka.clients.admin.Admin;import org.apache.kafka.clients.admin.TopicDescription;import org.apache.kafka.common.Node;import org.apache.kafka.common.TopicPartitionInfo;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.boot.ApplicationArguments;import org.springframework.boot.ApplicationRunner;import org.springframework.kafka.core.KafkaAdmin;import org.springframework.stereotype.Component;@Componentpublic class ClusterReport implements ApplicationRunner { private static final Logger log = LoggerFactory.getLogger(ClusterReport.class); private final KafkaAdmin kafkaAdmin; public ClusterReport(KafkaAdmin kafkaAdmin) { this.kafkaAdmin = kafkaAdmin; } @Override public void run(ApplicationArguments args) throws Exception { try (Admin admin = Admin.create(kafkaAdmin.getConfigurationProperties())) { var cluster = admin.describeCluster(); log.info("Connected to cluster {}", cluster.clusterId().get()); for (Node broker : cluster.nodes().get()) { log.info("Broker {} at {}:{} rack {}", broker.id(), broker.host(), broker.port(), broker.rack()); } Map<String, TopicDescription> topics = admin.describeTopics(List.of(TopicConfig.ORDERS)).allTopicNames().get(); for (TopicPartitionInfo partition : topics.get(TopicConfig.ORDERS).partitions()) { List<Integer> replicas = partition.replicas().stream().map(Node::id).toList(); List<Integer> inSync = partition.isr().stream().map(Node::id).toList(); Integer leader = partition.leader() == null ? null : partition.leader().id(); log.info("orders-{} leader={} replicas={} isr={}", partition.partition(), leader, replicas, inSync); } } }}Why it is written this way
- The host and port printed for each broker are the advertised addresses. If they are not reachable from where this application runs, that is the bug to fix before anything else.
partition.leader()can be null while a partition has no leader, so the code does not assume one.describeCluster().controller()is deliberately not used: when an admin client connects throughbootstrap.servers, Kafka returns an arbitrary broker there, not the active controller.- The admin client is created once and closed. Creating a client per request is a common way to exhaust connections.
Configuration
The settings that decide how clients find the cluster and what a topic looks like when nobody specified it. Defaults are from the Kafka 4.4 configuration reference.
bootstrap.serversDefault none (required)
- What it does
- Client setting. Host and port pairs used for the first connection only; the client then discovers every broker from metadata.
- When to change it
- List two or three brokers in different racks or zones, or one DNS name that resolves to several brokers. Listing all brokers is unnecessary.
advertised.listenersDefault null (falls back to `listeners`)
- What it does
- Broker setting. The addresses the broker publishes to clients and other brokers in metadata responses.
- When to change it
- Set it whenever the address the broker binds to is not the address clients must use: containers, NAT, cloud load balancers. It must be reachable from every client.
metadata.max.age.msDefault 300000 (5 minutes)
- What it does
- Client setting. How long a client may use cached metadata before refreshing it even if nothing has failed, to notice new brokers and partitions.
- When to change it
- Leadership changes trigger a refresh by themselves. Lower it only if clients must notice new partitions or topics matching a pattern sooner.
metadata.recovery.strategyDefault rebootstrap
- What it does
- Client setting. What to do when none of the brokers the client knows about is reachable:
rebootstrapstarts again frombootstrap.servers. - When to change it
- The default was
nonein 3.x clients, including the kafka-clients 3.9 that Spring Boot 3.5 manages. Set it torebootstrapthere if all broker addresses can change at once, for example on a full cluster replacement.
process.rolesDefault none (required in KRaft)
- What it does
- Server setting. Whether this process is a
broker, acontroller, or both. - When to change it
- Use separate broker and controller servers in production. Combined mode is for development.
auto.create.topics.enableDefault true
- What it does
- Broker setting. Create a topic the first time a client asks for metadata about a name that does not exist.
- When to change it
- Turn it off in shared and production clusters so that topics are created on purpose, with a chosen partition count and replication factor.
num.partitionsDefault 1
- What it does
- Broker setting. Partition count for topics created without one, including auto-created topics.
- When to change it
- Do not rely on it. State the partition count when creating a topic.
default.replication.factorDefault 1
- What it does
- Broker setting. Replication factor for topics created without one.
- When to change it
- Raise it to 3 on any cluster with three or more brokers, so that a forgotten setting does not produce an unreplicated topic.
offsets.topic.replication.factorDefault 3
- What it does
- Broker setting. Replication factor of the internal
__consumer_offsetstopic. The topic cannot be created until the cluster has at least this many brokers. - When to change it
- Lower it to 1 only on a single-broker development cluster, where consumer groups otherwise cannot start.
Hands-on exercise
Find who serves what, then take a broker away
Read a cluster the way a client does: work out which broker leads each partition, then stop one broker and see exactly which partitions change and which do not.
Steps · 0 of 6 done
Four brokers, two topics
Simulation · not a real clusterClick brokers, topics, partitions and replicas to inspect them. Stop a broker to see which partitions get a new leader.
What you should see
Only the partitions the stopped broker was leading get a new leader, chosen from their remaining replicas. Partitions it was following keep their leader and simply have one replica fewer in sync. Partitions with no replica on that broker do not change at all. After the restart the broker comes back as a follower and catches up before it can lead again.
What can go wrong
Bootstrap works, everything after it fails
- Symptom
- The client logs that it connected and fetched the cluster id, then every send or poll times out, often with connection errors naming a host you never configured (
localhost, a container name, a private address). - Cause
- The brokers advertise addresses that the client cannot resolve or reach. After bootstrap the client uses the addresses from metadata, which come from
advertised.listeners. - Fix
- Set
advertised.listenerson each broker to an address reachable from the clients. Where internal and external clients need different addresses, define two listeners and advertise each separately.
One bootstrap address
- Symptom
- Running instances are fine, but a newly deployed instance cannot start during a broker restart, and a rolling deployment stalls.
- Cause
bootstrap.serversnames a single broker. Clients that are already running have the full broker list in memory; a starting client has only that one address.- Fix
- List several brokers in different zones, or a DNS name that resolves to several brokers.
Topics created by accident
- Symptom
- A topic appears with one partition and replication factor 1, often with a misspelled name. A consumer of the correctly named topic receives nothing.
- Cause
auto.create.topics.enableis true by default, so a client asking for metadata about an unknown name causes the broker to create it with the broker defaults.- Fix
- Disable auto-creation on shared clusters and create topics explicitly. On the consumer side,
spring.kafka.listener.missing-topics-fatal=truemakes a listener fail at startup when its topic does not exist.
Treating the cluster as one endpoint
- Symptom
- Firewall rules or a proxy allow traffic to one broker or one load balancer address only. Some partitions work and others time out, and which ones changes after every leader election.
- Cause
- Clients must reach the leader of every partition they use, and leaders move between brokers.
- Fix
- Open the client network path to every broker, on the advertised addresses. A load balancer can serve the bootstrap address but cannot stand in for the brokers.
How to troubleshoot
Client connects, then times out talking to an unexpected host
Check
kafka-broker-api-versions.sh --bootstrap-server kafka-a1.internal:9092
- How to read it
- The tool prints one block per broker, headed by the broker's advertised host, port, id and rack. Run it from the machine where the client runs: every address listed must resolve and accept connections from there.
Not sure which broker serves a partition, or whether a topic was created as intended
Check
kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --topic orders
- How to read it
- The first line gives
PartitionCountandReplicationFactor. Each following line is one partition withLeader(a broker id),ReplicasandIsr. A topic showingPartitionCount: 1 ReplicationFactor: 1that nobody created on purpose was auto-created.
Producers or consumers hang on some partitions only
Check
kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --unavailable-partitions
- How to read it
- Lists partitions that currently have no leader (
Leader: none). Requests for those partitions cannot be served until a replica is available again; other partitions of the same topic are unaffected.
Topic creation or leader elections are not happening
Check
kafka-metadata-quorum.sh --bootstrap-server kafka-a1.internal:9092 describe --status
- How to read it
- Shows the controller quorum: the current leader id, the voters and how far each is behind. No leader, or a majority of voters missing, means the cluster can still serve existing leaders but cannot change anything.
One broker is missing from the cluster
Check
kafka.controller:type=KafkaController,name=ActiveBrokerCount and name=FencedBrokerCount
- How to read it
- Active brokers are registered and heartbeating. A fenced broker is registered but not allowed to lead or be in sync, usually because it is still starting up or its heartbeats are not arriving.
Key takeaways
- A broker stores partition replicas and serves them. The controller quorum decides which brokers are alive and who leads each partition, and stays off the data path.
bootstrap.serversis only the first hop. After one metadata request the client talks directly to partition leaders at their advertised addresses.- Every broker must be reachable from every client.
advertised.listenersis what clients actually connect to. - A topic is a name for a set of partitions spread over brokers. Each partition has one leader at a time.
- A broker failure affects only the partitions that broker led, and clients recover by refreshing metadata and retrying.
- Create topics on purpose: the broker defaults are one partition and one replica.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A new service runs in Kubernetes and reaches Kafka through kafka.shared.example.com:9092. Its log shows the cluster id being fetched. Thirty seconds later every send fails with a timeout, and the client log repeats: connection to node 2 (broker-2.kafka.svc:9092) could not be established. Services inside the Kafka namespace work normally.
What needs to change?
Scenario 2 of 3
A six-broker cluster hosts orders with 12 partitions and replication factor 3. Every service lists brokers 1 and 2 in bootstrap.servers. During a mistaken maintenance run, brokers 1 and 2 are both stopped for ten minutes. All orders partitions still have a leader on brokers 3 to 6.
What happens to the order service during those ten minutes?
Scenario 3 of 3
Broker 5 crashes. It hosted replicas of orders partitions 1, 4, 7 and 10, and was the leader of partitions 4 and 10. The topic has replication factor 3 and all other brokers are healthy. Four producers each write to exactly one of those four partitions.
Which producers see anything, and what do they see?
References
References: Apache Kafka documentation: Introduction (servers, clients, topics, partitions) ↗ · Apache Kafka documentation: Design (the producer, load balancing, replication) ↗ · Apache Kafka documentation: KRaft (process roles, controllers) ↗ · Apache Kafka documentation: Broker configs ↗ · Apache Kafka documentation: Producer configs (bootstrap.servers, metadata settings) ↗ · Apache Kafka documentation: Basic Kafka operations (topics, graceful shutdown) ↗ · Spring for Apache Kafka reference: Configuring topics ↗ · Report a problem