Skip to content

Path 2 · lesson 1 of 8

Foundation14 minApache Kafka 4.4 (KRaft only) · Java 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 (kafka-clients 3.9)

Clusters, brokers and topics

What a broker does, how brokers form a cluster, how a client gets from a bootstrap address to the right partition leader, and what a topic really is.

What you will learn

  • List the jobs a single broker does and the ones it leaves to the controller
  • Trace how a client goes from bootstrap.servers to the broker that leads a partition
  • Read the output of kafka-topics.sh --describe and say which broker serves which partition
  • Predict which clients notice when one broker stops, and which do not
  • Declare a topic with an explicit partition count and replication factor from Spring Boot

Before you start

The concept

A broker is one Kafka server process. It owns some disks, listens on a TCP port, and does four things: it appends records that producers send to it, it serves records to consumers that ask for them, it copies partitions from other brokers so that there is more than one copy, and it tells any client that asks where everything in the cluster lives.

A cluster is a set of brokers that share one cluster id and one source of truth about the cluster's state. That source of truth is the controller: a small group of servers (usually three or five) that agree, through a Raft-based quorum called KRaft, on which brokers are alive, which topics exist and which broker leads each partition. Brokers hold the data. Controllers hold the map.

A topic is a name for a stream of records, such as orders. It is not one file on one machine. A topic is split into partitions, each partition is an independent append-only log, and each partition is stored on several brokers (its replicas). One replica is the leader and takes all the writes for that partition. A topic is therefore better thought of as a name that groups a set of logs spread across the cluster.

ThingWhat it isWho decides it
BrokerA server that stores partition replicas and answers produce, fetch and metadata requestsYou, when you start it with a node.id
ClusterBrokers plus a controller quorum that share a cluster idYou, when you format storage with the same cluster id
TopicA named set of partitions with a replication factor and settingsWhoever creates it (or the broker, if auto-creation is on)
Partition leaderThe one replica that accepts writes for a partition right nowThe controller

Click around a three-broker cluster

Simulation · not a real cluster

Click each broker to see what it stores and leads, then click the topics to see how their partitions are spread across the brokers.

Why it exists

A message broker that lives on one server has a hard ceiling: one machine's disk, one machine's network card, and an outage whenever that machine restarts. The usual way to grow past one server is to put several behind a load balancer or a routing tier. For a log that fails in two ways. The router becomes the bottleneck that every byte passes through, and it hides where the data is, so the client cannot keep a stream of related records going to the same place in order.

Kafka made the opposite choice: there is no routing tier. Clients are told the layout of the cluster and connect directly to the broker that leads the partition they need. That is why every broker can answer a metadata request, and why a Kafka client holds connections to many brokers at once. The cost of this design is that every broker must be individually reachable from every client, which is the source of the most common connection problem in Kafka (see the pitfalls).

Splitting a topic into partitions is what lets a topic be larger and busier than one machine. A single partition has to fit on one server, but a topic with twelve partitions can spread its reads and writes over up to twelve leaders. And someone has to make cluster-wide decisions consistently: if two brokers both believed they led orders-0, the log would fork. Giving those decisions to one controller quorum, instead of letting brokers negotiate among themselves, is what keeps a single answer to the question of who leads what.

Routing tier in front of the servers

  • Clients know one address
  • Every request takes an extra hop through the router
  • The router must scale with total traffic
  • The client cannot choose where a record lands

Kafka: cluster-aware clients

  • Clients learn every broker's address from metadata
  • Requests go straight to the partition leader
  • Adding brokers adds capacity with no shared choke point
  • The producer picks the partition, so a key can keep its order

How it works inside

From bootstrap address to partition leader

bootstrap.servers is only a way in. The client connects to one address from that list and sends a Metadata request. The broker answers from its own copy of the cluster metadata with the list of live brokers (id, host, port, rack) and, for each topic asked about, every partition with its leader, its replicas and its in-sync replicas. From then on the client ignores the bootstrap list and opens connections to the brokers named in the response.

  1. Connect to any address in bootstrap.servers and ask for metadata about the topics the client uses.
  2. Cache the answer: brokers and their addresses, and the leader of each partition.
  3. Send each produce request to the leader of the target partition. Consumers send fetch requests to the leader as well (or to a nearby follower when follower fetching is configured, covered in replication).
  4. Refresh the cache when a broker answers with an error that means the map is stale, such as NOT_LEADER_OR_FOLLOWER, and in any case every metadata.max.age.ms (5 minutes by default).

What a broker keeps on disk

For every partition replica it hosts, a broker has a directory under one of its log.dirs, named after the topic and the partition number: orders-0, orders-1, payments-0. Inside are the segment files of that partition's log (covered in log segments and retention). A broker that hosts no replica of orders-2 has no such directory and will answer a produce request for it with an error telling the client to refresh its metadata. This is also why topic names are limited to 249 characters: the name has to fit in a directory name together with the partition number.

Where the controller fits

Each server is started with process.roles set to broker, controller, or both (combined mode, meant for development). The controllers form a quorum; one is the active controller and the others are hot standbys. Brokers register with the active controller and keep a session alive by heartbeating every broker.heartbeat.interval.ms (2 seconds). When a broker's heartbeats stop for broker.session.timeout.ms (9 seconds), the controller treats it as gone and picks new leaders for the partitions it led.

Every decision the controller makes (a topic created, a leader changed, a broker registered) is a record in an internal metadata log. Brokers follow that log and keep the result in memory, which is how any broker can answer a client's metadata request without asking the controller. Producers and consumers never talk to the controller. The quorum and the metadata log are the subject of KRaft and the controller.

Broker

  • Stores partition replicas on disk
  • Serves produce and fetch requests
  • Replicates partitions it follows
  • Answers metadata requests from its copy of the cluster state
  • Hosts group coordinators for consumer groups

Controller

  • Stores no topic data
  • Tracks which brokers are alive
  • Creates and deletes topics, assigns replicas to brokers
  • Elects a leader for each partition
  • Writes every change to the metadata log

Architecture

Drawing diagram…

A producer first asks one broker for metadata, then sends records straight to the broker leading orders partition 0. A consumer fetches from the leader of partition 2. Three controllers form a quorum whose metadata log all three brokers follow.

flowchart LR
  subgraph APPS["Client applications"]
    P["Order service producer"]
    C["Payment service consumer"]
  end
  subgraph CL["Kafka cluster"]
    subgraph B1["Broker 1"]
      A0["orders-0 leader"]
      A2["orders-2 follower"]
    end
    subgraph B2["Broker 2"]
      A1["orders-1 leader"]
      A0f["orders-0 follower"]
    end
    subgraph B3["Broker 3"]
      A2l["orders-2 leader"]
      A1f["orders-1 follower"]
    end
  end
  subgraph Q["Controller quorum"]
    K1["Controller 1 active"]
    K2["Controller 2 standby"]
    K3["Controller 3 standby"]
  end
  P -->|"1 metadata request"| A1
  P -->|"2 produce to leader"| A0
  C -->|"fetch from leader"| A2l
  A0 -. "replicated to" .-> A0f
  K1 -. "metadata log" .-> B1
  K1 -. "metadata log" .-> B2
  K1 -. "metadata log" .-> B3
Brokers store partition replicas; the controller quorum decides leaders; clients talk to leaders directly.
  • Three brokers each hold replicas of several partitions and lead some of them. Leadership is spread so that no single broker takes all the writes for a topic.
  • The producer's first request can go to any broker. Its produce requests go to the broker that leads the target partition, with no router in between.
  • The controllers are separate servers. They do not sit on the data path: if all three brokers are healthy, a produce request never touches a controller.
  • The dotted lines from the active controller are the metadata log. Brokers replay it, which is how broker 2 can tell the producer that broker 1 leads orders-0.

Message flow

Drawing diagram…

The producer asks broker 2 for metadata, then produces to broker 1, the leader of orders partition 0. After leadership moves, broker 1 rejects the request, the producer refreshes its metadata and resends the same records to broker 3.

sequenceDiagram
  participant P as Producer
  participant B2 as Broker 2
  participant B1 as Broker 1
  participant B3 as Broker 3
  Note over P: bootstrap.servers lists broker 2 and broker 3
  P->>B2: Connect and send Metadata request for orders
  B2-->>P: Brokers 1 2 3 with addresses and the leader of each partition
  P->>B1: Produce to orders-0
  B1-->>P: Acknowledged at offset 41
  Note over B1,B3: Later the controller moves leadership of orders-0 to broker 3
  P->>B1: Produce to orders-0
  B1-->>P: Error NOT_LEADER_OR_FOLLOWER
  P->>B2: Metadata request for orders
  B2-->>P: Leader of orders-0 is now broker 3
  P->>B3: Produce to orders-0 again
  B3-->>P: Acknowledged at offset 42
Bootstrap once, then talk to leaders; a stale map is corrected by one error and one metadata refresh.
  1. The producer never sends records to the bootstrap broker unless that broker happens to lead the partition.
  2. Broker 1 was never in bootstrap.servers. The client learned its address from the metadata response.
  3. A leadership change is not announced to clients. The client finds out when the old leader refuses a request, refreshes its metadata from any broker, and retries. The producer does this inside send(); the application only sees a slightly slower acknowledgement.
  4. Offsets continue from where they were: the new leader already had the records up to offset 41 as a follower.

In a real system

A typical order platform runs six brokers spread over three availability zones and three dedicated controllers, one per zone. The services share a handful of topics, each created explicitly by the team that owns it:

TopicKeyPartitionsReplication factorWritten byRead by
ordersorder id123Order servicePayment, inventory and notification services
paymentsorder id123Payment serviceOrder and notification services
inventory-reservationsSKU63Inventory serviceOrder service

With twelve partitions and a replication factor of three, orders has 36 partition replicas, six per broker on average, and each broker leads about two partitions. Every service is configured with the same three bootstrap addresses, one broker per zone, so that a new instance can start while any single zone is down. Auto-creation of topics is switched off on the brokers, so a typo in a topic name fails loudly instead of producing a one-partition, unreplicated topic that nobody owns.

When broker 4 is restarted for a kernel patch, the only producers that notice are the ones writing to the two orders partitions it led, and what they notice is a few retried batches. The example project uses the same topics.

Java and Spring Boot

Spring Boot creates a KafkaAdmin for you. Declaring NewTopic beans makes it create missing topics at startup, and the same admin configuration can be used to look at what the cluster reports. The runner below prints the brokers and the leader of each orders partition, which is exactly the metadata a producer works from.

src/main/resources/application.ymlYAML
spring:  kafka:    # Three brokers, one per availability zone. Only used for the first connection.    bootstrap-servers: kafka-a1.internal:9092,kafka-b1.internal:9092,kafka-c1.internal:9092    client-id: order-service    admin:      # Fail the application start if no broker can be reached, instead of logging and carrying on.      fail-fast: true

Why it is written this way

  • bootstrap-servers lists more than one broker so that a new instance can start while one of them is down. It does not need to list every broker.
  • client-id shows up in broker request logs and quota metrics, which makes it possible to tell which service a connection belongs to.
  • spring.kafka.admin.fail-fast defaults to false: by default a missing broker at startup is only logged and topics are not created.

Configuration

The settings that decide how clients find the cluster and what a topic looks like when nobody specified it. Defaults are from the Kafka 4.4 configuration reference.

bootstrap.serversDefault none (required)
What it does
Client setting. Host and port pairs used for the first connection only; the client then discovers every broker from metadata.
When to change it
List two or three brokers in different racks or zones, or one DNS name that resolves to several brokers. Listing all brokers is unnecessary.
advertised.listenersDefault null (falls back to `listeners`)
What it does
Broker setting. The addresses the broker publishes to clients and other brokers in metadata responses.
When to change it
Set it whenever the address the broker binds to is not the address clients must use: containers, NAT, cloud load balancers. It must be reachable from every client.
metadata.max.age.msDefault 300000 (5 minutes)
What it does
Client setting. How long a client may use cached metadata before refreshing it even if nothing has failed, to notice new brokers and partitions.
When to change it
Leadership changes trigger a refresh by themselves. Lower it only if clients must notice new partitions or topics matching a pattern sooner.
metadata.recovery.strategyDefault rebootstrap
What it does
Client setting. What to do when none of the brokers the client knows about is reachable: rebootstrap starts again from bootstrap.servers.
When to change it
The default was none in 3.x clients, including the kafka-clients 3.9 that Spring Boot 3.5 manages. Set it to rebootstrap there if all broker addresses can change at once, for example on a full cluster replacement.
process.rolesDefault none (required in KRaft)
What it does
Server setting. Whether this process is a broker, a controller, or both.
When to change it
Use separate broker and controller servers in production. Combined mode is for development.
auto.create.topics.enableDefault true
What it does
Broker setting. Create a topic the first time a client asks for metadata about a name that does not exist.
When to change it
Turn it off in shared and production clusters so that topics are created on purpose, with a chosen partition count and replication factor.
num.partitionsDefault 1
What it does
Broker setting. Partition count for topics created without one, including auto-created topics.
When to change it
Do not rely on it. State the partition count when creating a topic.
default.replication.factorDefault 1
What it does
Broker setting. Replication factor for topics created without one.
When to change it
Raise it to 3 on any cluster with three or more brokers, so that a forgotten setting does not produce an unreplicated topic.
offsets.topic.replication.factorDefault 3
What it does
Broker setting. Replication factor of the internal __consumer_offsets topic. The topic cannot be created until the cluster has at least this many brokers.
When to change it
Lower it to 1 only on a single-broker development cluster, where consumer groups otherwise cannot start.

Hands-on exercise

Find who serves what, then take a broker away

Read a cluster the way a client does: work out which broker leads each partition, then stop one broker and see exactly which partitions change and which do not.

Steps · 0 of 6 done

Four brokers, two topics

Simulation · not a real cluster

Click brokers, topics, partitions and replicas to inspect them. Stop a broker to see which partitions get a new leader.

Take it further in the lab: Broker failure and leader election

What can go wrong

Bootstrap works, everything after it fails
Symptom
The client logs that it connected and fetched the cluster id, then every send or poll times out, often with connection errors naming a host you never configured (localhost, a container name, a private address).
Cause
The brokers advertise addresses that the client cannot resolve or reach. After bootstrap the client uses the addresses from metadata, which come from advertised.listeners.
Fix
Set advertised.listeners on each broker to an address reachable from the clients. Where internal and external clients need different addresses, define two listeners and advertise each separately.
One bootstrap address
Symptom
Running instances are fine, but a newly deployed instance cannot start during a broker restart, and a rolling deployment stalls.
Cause
bootstrap.servers names a single broker. Clients that are already running have the full broker list in memory; a starting client has only that one address.
Fix
List several brokers in different zones, or a DNS name that resolves to several brokers.
Topics created by accident
Symptom
A topic appears with one partition and replication factor 1, often with a misspelled name. A consumer of the correctly named topic receives nothing.
Cause
auto.create.topics.enable is true by default, so a client asking for metadata about an unknown name causes the broker to create it with the broker defaults.
Fix
Disable auto-creation on shared clusters and create topics explicitly. On the consumer side, spring.kafka.listener.missing-topics-fatal=true makes a listener fail at startup when its topic does not exist.
Treating the cluster as one endpoint
Symptom
Firewall rules or a proxy allow traffic to one broker or one load balancer address only. Some partitions work and others time out, and which ones changes after every leader election.
Cause
Clients must reach the leader of every partition they use, and leaders move between brokers.
Fix
Open the client network path to every broker, on the advertised addresses. A load balancer can serve the bootstrap address but cannot stand in for the brokers.

How to troubleshoot

Client connects, then times out talking to an unexpected host

Check

kafka-broker-api-versions.sh --bootstrap-server kafka-a1.internal:9092
How to read it
The tool prints one block per broker, headed by the broker's advertised host, port, id and rack. Run it from the machine where the client runs: every address listed must resolve and accept connections from there.
Not sure which broker serves a partition, or whether a topic was created as intended

Check

kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --topic orders
How to read it
The first line gives PartitionCount and ReplicationFactor. Each following line is one partition with Leader (a broker id), Replicas and Isr. A topic showing PartitionCount: 1 ReplicationFactor: 1 that nobody created on purpose was auto-created.
Producers or consumers hang on some partitions only

Check

kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --unavailable-partitions
How to read it
Lists partitions that currently have no leader (Leader: none). Requests for those partitions cannot be served until a replica is available again; other partitions of the same topic are unaffected.
Topic creation or leader elections are not happening

Check

kafka-metadata-quorum.sh --bootstrap-server kafka-a1.internal:9092 describe --status
How to read it
Shows the controller quorum: the current leader id, the voters and how far each is behind. No leader, or a majority of voters missing, means the cluster can still serve existing leaders but cannot change anything.
One broker is missing from the cluster

Check

kafka.controller:type=KafkaController,name=ActiveBrokerCount and name=FencedBrokerCount
How to read it
Active brokers are registered and heartbeating. A fenced broker is registered but not allowed to lead or be in sync, usually because it is still starting up or its heartbeats are not arriving.

Key takeaways

  • A broker stores partition replicas and serves them. The controller quorum decides which brokers are alive and who leads each partition, and stays off the data path.
  • bootstrap.servers is only the first hop. After one metadata request the client talks directly to partition leaders at their advertised addresses.
  • Every broker must be reachable from every client. advertised.listeners is what clients actually connect to.
  • A topic is a name for a set of partitions spread over brokers. Each partition has one leader at a time.
  • A broker failure affects only the partitions that broker led, and clients recover by refreshing metadata and retrying.
  • Create topics on purpose: the broker defaults are one partition and one replica.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A new service runs in Kubernetes and reaches Kafka through kafka.shared.example.com:9092. Its log shows the cluster id being fetched. Thirty seconds later every send fails with a timeout, and the client log repeats: connection to node 2 (broker-2.kafka.svc:9092) could not be established. Services inside the Kafka namespace work normally.

What needs to change?

Scenario 2 of 3

A six-broker cluster hosts orders with 12 partitions and replication factor 3. Every service lists brokers 1 and 2 in bootstrap.servers. During a mistaken maintenance run, brokers 1 and 2 are both stopped for ten minutes. All orders partitions still have a leader on brokers 3 to 6.

What happens to the order service during those ten minutes?

Scenario 3 of 3

Broker 5 crashes. It hosted replicas of orders partitions 1, 4, 7 and 10, and was the leader of partitions 4 and 10. The topic has replication factor 3 and all other brokers are healthy. Four producers each write to exactly one of those four partitions.

Which producers see anything, and what do they see?

References

References: Apache Kafka documentation: Introduction (servers, clients, topics, partitions) ↗ · Apache Kafka documentation: Design (the producer, load balancing, replication) ↗ · Apache Kafka documentation: KRaft (process roles, controllers) ↗ · Apache Kafka documentation: Broker configs ↗ · Apache Kafka documentation: Producer configs (bootstrap.servers, metadata settings) ↗ · Apache Kafka documentation: Basic Kafka operations (topics, graceful shutdown) ↗ · Spring for Apache Kafka reference: Configuring topics ↗ · Report a problem