Skip to content

Path 8 · lesson 2 of 5

Intermediate26 minApache Kafka 4.4 (operations, hardware and OS, configuration references) · Spring Boot 3.5 with Spring for Apache Kafka 3.3, which manages kafka-clients 3.9

Partition sizing and consumer parallelism

How to choose a partition count from a target rate and a processing time, what too many partitions cost, why the count is hard to change, and how to add parallelism safely.

What you will learn

  • Calculate a partition count from a peak rate, a per-record processing time, a utilisation target and expected growth
  • List what each extra partition costs on brokers and clients, using what the Kafka documentation states
  • Explain why partitions can be added but not removed, and what adding them does to keyed ordering
  • Map partitions to consumer threads across application instances with Spring's listener concurrency
  • Process records of one partition in parallel while keeping the order of records that share a key

Before you start

The concept

A partition is the unit of parallelism for a consumer group. Within one group a partition is read by exactly one consumer at a time, so a topic with 12 partitions can keep at most 12 consumers of that group busy. A thirteenth consumer joins the group, is assigned nothing and waits as a spare.

That makes the partition count a capacity decision. Too few and the group cannot be scaled past a ceiling, however many instances you start. Too many and you pay for them on every broker and client, for the lifetime of the topic. The count is also close to permanent: Kafka can add partitions to a topic and cannot remove them, and adding them changes which partition a key goes to.

So the count is chosen from numbers, in this order: the peak rate the topic must carry, how fast one consumer thread processes a record, how much spare capacity you want at peak, and how much the rate will grow before you are willing to migrate to a new topic.

Consumers beyond the partition count are idle

Simulation · not a real cluster

Six partitions and six consumers. Add a seventh and an eighth consumer and look at what they are assigned. Then raise the partition count.

Why it exists

The cap exists because of the ordering guarantee. Kafka promises order within a partition, and a consumer group keeps that promise by giving each partition one reader. If two consumers of a group read the same partition, records for one order could be processed out of sequence, and one committed position could not describe what both had done. Parallelism therefore comes from having more partitions, each with its own ordered stream.

The same rule explains why the number is hard to change later. A keyed record goes to the partition given by a hash of its key modulo the partition count. Change the count and the same key gives a different result.

Watch keys move when a partition is added

Simulation · not a real cluster

Six order ids are spread over six partitions. Note where each key sits, then add a partition and see which keys change place.

Because of this, the usual practice is to create a keyed topic with enough partitions for its expected lifetime, and to treat a later increase as a migration to be planned, not a setting to turn up during an incident.

How it works inside

The sizing calculation

  1. Per-thread rate. One consumer thread processing records one at a time handles 1,000 / (processing time in ms) records per second. Measure the processing time under load, including the database call, at a high percentile and not only the average.
  2. Capacity needed. Divide the peak produce rate by the utilisation you want at peak. A target of 70% means capacity = peak / 0.7, which leaves room for bursts and for catching up after a pause.
  3. Threads needed. Capacity needed / per-thread rate, rounded up. This is the minimum partition count for that consumer group.
  4. Slowest group decides. Repeat for every consumer group that reads the topic. The group that needs the most threads sets the count, unless it gets its parallelism another way (see below).
  5. Growth. Redo the calculation with the peak you expect at the end of the topic's planned life, and use that result.
  6. Round up to a convenient number. A count with many divisors, such as 12, 24 or 48, divides evenly over 2, 3, 4 or 6 instances, so each thread gets the same number of partitions.

Check the produce side as well: the bytes per second per partition must be something one partition leader can append and its followers can replicate. For business events this is almost never the limit. For a high-volume stream, measure it with kafka-producer-perf-test.sh on a test topic instead of assuming a figure.

What each partition costs

WhereCostWhat the Kafka documentation says
Broker: open filesEvery log segment of every partition replica is an open fileA broker needs at least partitions × (partition size / segment size) file descriptors plus its connections; at least 100000 allowed descriptors is the recommended starting point
Broker: memory mapsEach segment has an offset index and a time index, both memory-mappedEach segment uses 2 map areas. With the common Linux default vm.max_map_count of about 65535, 50,000 partitions on one broker need 100,000 map areas and are likely to crash it
Broker: failure handlingWhen a broker fails, a new leader is elected for every partition it ledThe design guide calls this the critical window of unavailability and describes how the controller batches the leadership changes to keep it short for many partitions
Producer: memory and batchingThe producer keeps a batch per partition it is writing toA buffer of batch.size is allocated per batch. The same traffic spread over more partitions fills each batch more slowly, so batches are smaller
Consumer: fetch sizeA fetch can return data for every assigned partitionUp to max.partition.fetch.bytes (1 MiB) per partition, within fetch.max.bytes (50 MiB) per request
ReplicationEach follower fetches every partition it replicatesnum.replica.fetchers (default 1) threads per source broker carry all of it
These are the costs the documentation supports. It does not give a single maximum partition count, because the limit depends on hardware, segment settings and load.

What else happens when you add partitions

  • Existing data stays where it is. Old records are not moved to the new partitions. The new partitions start empty.
  • Clients find out late. Producers and consumers learn about new partitions when they refresh metadata (metadata.max.age.ms, default 5 minutes). Until then traffic is uneven.
  • Records can be skipped. The operations guide warns that a consumer with auto.offset.reset=latest may miss records produced to a new partition before the group discovers it, because the partition has no committed offset and the consumer starts at its end. For such topics the 4.4 guide recommends by_duration:<duration>, a setting of newer clients; with an older client use earliest and idempotent processing.
  • The group rebalances to hand out the new partitions.
  • Never do it to internal topics such as __consumer_offsets or __transaction_state: the guide warns this breaks the coordinator mapping.

Consumers, threads and instances

A group member is one KafkaConsumer, which is used by one thread. In Spring, concurrency on a listener container creates that many consumers and threads inside one application instance. The total number of members in the group is instances × concurrency, and Kafka spreads the partitions over all of them.

Setup for 24 partitionsMembersResult
6 instances × concurrency 424One partition per thread: full parallelism
3 instances × concurrency 412Two partitions per thread: half the capacity, same ordering
8 instances × concurrency 43224 threads work, 8 are idle spares
6 instances × concurrency 4, one instance down20Its 4 partitions move to other threads, which now own two each
Idle spares are not wasted if you want fast takeover, but they add no throughput.

More parallelism than partitions

Sometimes one group needs far more concurrency than the others, typically because each record makes a slow remote call. Raising the partition count for everyone is one answer. The other is to process several records of the same partition at once without breaking the order that matters, which is the order of records with the same key.

The safe pattern: take a batch from poll(), split it into a fixed number of lanes by a hash of the key, run the lanes in parallel with each lane processing its records in sequence, and return only when every lane has finished. Records with the same key always share a lane, so their order holds. The offsets are committed after the whole batch, so the committed position never passes an unfinished record.

Architecture

Drawing diagram…

Flowchart: peak rate and target utilisation give capacity needed, processing time gives the per-thread rate, together they give threads needed. The largest result over all consumer groups, adjusted for growth, becomes the partition count, which is checked against per-broker cost before the topic is created.

flowchart TD
  RATE["Peak produce rate"] --> NEED["Capacity needed = peak / target utilisation"]
  PROC["Processing time per record"] --> PER["Per thread rate = 1000 / ms"]
  NEED --> THREADS["Threads needed = capacity / per thread rate"]
  PER --> THREADS
  THREADS --> GROUPS["Take the largest result over all consumer groups"]
  GROWTH["Expected growth over the topic lifetime"] --> GROUPS
  GROUPS --> COUNT["Partition count, rounded up to a divisible number"]
  COUNT --> CHECK{"Per broker cost acceptable"}
  CHECK -->|"yes"| CREATE["Create the topic"]
  CHECK -->|"no"| ALT["Faster processing or parallel lanes inside a partition"]
  ALT --> PROC
The partition count is the output of a calculation, checked against what the brokers can carry.
  • Two measured inputs drive everything: the peak rate and the processing time per record. If either is a guess, the result is a guess.
  • The utilisation target turns a rate that just keeps up into one with room for bursts and recovery.
  • The loop at the bottom is the alternative to more partitions: make each record cheaper, or process several keys of one partition in parallel.
  • The decision is made once per topic, at creation, because lowering the count is impossible and raising it disturbs keyed ordering.

Message flow

Drawing diagram…

Sequence diagram: an order's first event goes to partition 3 and is read by consumer 1. The operator raises the partition count from 6 to 8. The order's next event hashes to partition 7 and, after a rebalance, is read by consumer 2.

sequenceDiagram
  participant Ops as Operator
  participant K as Kafka cluster
  participant P as Order producer
  participant C1 as Consumer 1
  participant C2 as Consumer 2
  P->>K: OrderPlaced key o-1001 to partition 3 of 6
  K-->>C1: partition 3 delivers OrderPlaced
  Ops->>K: alter topic to 8 partitions
  Note over P: metadata refresh shows 8 partitions
  P->>K: OrderCancelled key o-1001 to partition 7 of 8
  K->>C1: rebalance
  K->>C2: rebalance and assign partition 7
  K-->>C2: partition 7 delivers OrderCancelled
  Note over C1,C2: two consumers hold events of one order
What adding partitions does to one key: its later events land in a different partition, read by a different consumer.
  1. Before the change, every event of order o-1001 goes to one partition and reaches one consumer in order.
  2. After the change the hash is taken modulo 8 instead of 6. For this key the result differs, so the next event goes elsewhere. (The partition numbers in the picture are illustrative.)
  3. If consumer 1 is behind and consumer 2 is not, the cancellation can be processed before the placement. Nothing in Kafka prevents this, since order is per partition.
  4. A safe increase therefore needs a quiet period for the affected keys, consumers that tolerate reordering, or a new topic with the new count and a controlled switch.

In a real system

The example project sends order events to orders.events, keyed by order id, and three groups read it: payment-service, inventory-service and notification-service. All figures below are assumptions for this example. Replace them with your own measurements.

Worked example: sizing orders.events

StepArithmeticResult
Peak rate todayAssumed from the busiest hour1,200 records/s
Peak rate at end of planned lifeAssumed growth to1,600 records/s
Capacity needed at 70% utilisation1,600 / 0.72,286 records/s
Payment: 10 ms per record1,000 / 10 = 100 per thread; 2,286 / 100 = 22.923 threads
Inventory: 4 ms per record1,000 / 4 = 250 per thread; 2,286 / 250 = 9.110 threads
Notification: 25 ms per record (email API)1,000 / 25 = 40 per thread; 2,286 / 40 = 57.258 threads
Count from payment, rounded to a divisible number23 → 2424 partitions
Notification with 24 partitions3 lanes per partition: 24 × 3 × 402,880 records/s, enough
Bytes per partition at peak1,600 × 1 KB / 2467 KB/s
Payment sets the count at 24. Notification would need 58 partitions by itself, so it gets its extra parallelism inside each partition instead.

Sizing the topic for the notification group would have given 60 partitions, which the other two groups do not need and every broker would pay for. Parallel lanes confine the cost to the one service with the slow call. Payment runs as 6 instances with concurrency 4, inventory as 3 instances with concurrency 4 (12 threads, two partitions each), and notification as 6 instances with concurrency 4 and 3 lanes.

The payment group at today's peak

Simulation · not a real cluster

24 partitions, 24 consumers, 1,200 records per second at 10 ms each. Check the headroom, then raise the rate to the planned 1,600.

Java and Spring Boot

The topic is declared in code with its partition count, the listener's concurrency comes from configuration, and the notification listener shows key-ordered parallel processing inside a partition.

src/main/java/com/example/order/OrderTopicConfig.javaJava
package com.example.order;import org.apache.kafka.clients.admin.NewTopic;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class OrderTopicConfig {    @Bean    public NewTopic orderEvents() {        return TopicBuilder.name("orders.events")                .partitions(24)                .replicas(3)                .config("min.insync.replicas", "2")                .build();    }}

Why it is written this way

  • Spring's KafkaAdmin creates the topic at startup if it does not exist.
  • It also adds partitions if the existing topic has fewer than this bean declares (documented in the Spring for Apache Kafka reference under configuring topics). Changing 24 to 48 here and deploying is therefore a repartitioning of a keyed topic, with the ordering consequences described above. Treat this number as a schema, not a tuning value.
  • The broker default num.partitions is 1. A topic created automatically, or without an explicit count, gets one partition and no parallelism.

Configuration

Settings that decide how many partitions a topic gets and how consumers spread over them. Defaults are from the Kafka 4.4 references and the Spring Boot 3.5 application properties.

num.partitions (broker)Default 1
What it does
Partition count for topics created automatically or created without an explicit count.
When to change it
Do not rely on it. Create each topic with a calculated count, and consider disabling automatic topic creation so a typo cannot create a one-partition topic.
spring.kafka.listener.concurrencyDefault none (one consumer per listener)
What it does
Number of consumers, each with its own thread, that a listener container runs in one application instance.
When to change it
Set instances × concurrency to the partition count. Higher values add idle members. It can also be set per listener with the concurrency attribute of @KafkaListener.
max.poll.records (consumer)Default 500
What it does
Maximum records per poll(). Bounds the size of one batch of work.
When to change it
Lower it when processing is slow, so that batch size × processing time stays far below max.poll.interval.ms.
max.poll.interval.ms (consumer)Default 300000
What it does
Longest allowed time between two calls to poll() before the consumer is considered failed and its partitions are reassigned.
When to change it
Size batches to fit well inside it. Raising it hides slow processing and delays detection of a stuck consumer.
metadata.max.age.ms (producer and consumer)Default 300000
What it does
How often a client refreshes cluster metadata even without an error. With the classic group protocol this is how a consumer discovers new partitions.
When to change it
Explains why traffic to new partitions starts unevenly. Lowering it speeds discovery at the cost of more metadata requests.
max.partition.fetch.bytes (consumer)Default 1048576
What it does
Maximum data per partition in one fetch response. Not an absolute limit: a larger first batch is still returned so the consumer can progress.
When to change it
With many partitions per consumer, this times the partition count bounds what one fetch can hold in memory, within fetch.max.bytes (52428800).
group.protocol (consumer)Default classic
What it does
Which rebalance protocol the consumer uses: classic, or consumer for the newer protocol from KIP-848.
When to change it
Affects how fast new partitions and members are picked up: with consumer the broker pushes assignments on the next heartbeat. Needs a client and brokers that support it.

Hands-on exercise

Size a topic, then find the limit that partitions cannot fix

Reproduce the worked example in the throughput model, see what partitions and consumers each contribute, and find the point where another resource becomes the limit.

Steps · 0 of 6 done

From exactly enough to planned headroom

Simulation · not a real cluster

A simple queueing model with illustrative numbers. It starts at 100% utilisation: 12 consumers × 100 records per second for 1,200 arriving.

Take it further in the lab: Consumer group assignment

What can go wrong

Scaling consumers past the partition count
Symptom
Lag is growing, the deployment is scaled from 6 to 12 instances, and the consume rate does not change. Half the instances show no assigned partitions.
Cause
The topic has 6 partitions. A group can use at most one consumer per partition, so the new members are idle.
Fix
Compare members with partitions before scaling (kafka-consumer-groups.sh --describe --members). Beyond the cap, make processing faster, add key-ordered parallelism inside the partition, or plan a partition increase.
Adding partitions to a keyed topic during an incident
Symptom
After the partition count was raised, a small number of orders end up in impossible states, such as cancelled and then paid.
Cause
The key to partition mapping changed. Later events for existing keys went to different partitions and were processed in a different order relative to the earlier ones.
Fix
Size keyed topics up front. If an increase is unavoidable, make consumers tolerate reordering (version checks, idempotent state transitions) or migrate to a new topic with a controlled cutover.
Choosing a huge count to be safe
Symptom
Brokers approach file descriptor or memory map limits, restarts and failovers take longer, and producer batches are tiny because each partition receives little traffic.
Cause
Every partition replica costs open files, map areas, replication work and client memory, whether or not it carries traffic.
Fix
Calculate the count from rate, processing time, headroom and growth. Check partitions per broker against ulimit -n and vm.max_map_count.
Changing the NewTopic partition count in code
Symptom
A routine deployment is followed by a rebalance and by keys appearing in new partitions. Nobody ran a topic command.
Cause
Spring's KafkaAdmin increases the partition count of an existing topic when the NewTopic bean declares more.
Fix
Review changes to topic declarations like schema changes. Where topics are managed centrally, do not declare them in the service at all.
Parallelising with an unbounded executor
Symptom
Throughput rises, then records are lost on restart and events for one key are occasionally applied in the wrong order.
Cause
The listener returned before the work finished, so offsets were committed early, and records with the same key ran concurrently.
Fix
Use lanes keyed by a hash of the record key and wait for all of them before returning, as in the notification listener above.

How to troubleshoot

You need to know how many partitions a topic has and where they are

Check

bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders.events
How to read it
The first line shows PartitionCount and ReplicationFactor; each following line shows a partition's leader, replicas and in-sync replicas.
Consumers were added and throughput did not rise

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
How to read it
Each member is listed with its assigned partitions. Members with none are above the partition cap, or subscribed to something else.
Lag grows on all partitions at a similar rate

Check

Compare the produce rate with threads × 1,000 / processing ms, using the `orders.events.processing` timer from the previous lesson
How to read it
If the produce rate is above the ceiling, the group is undersized: add consumers up to the partition count, then reduce processing time. If it is below, look for rebalances or a slow dependency.
A broker logs OutOfMemoryError: Map failed or Too many open files

Check

sysctl vm.max_map_count; ulimit -n for the broker process; kafka.server:type=ReplicaManager,name=PartitionCount
How to read it
The broker hosts more partition segments than the operating system limits allow. Each segment needs 2 map areas and file descriptors. Raise the limits and reduce partitions per broker.
After adding partitions, some consumers never read the first records of the new partitions

Check

The consumer's `auto.offset.reset` setting and the group's committed offsets for the new partitions
How to read it
With latest, a consumer that discovers the partition after records were written starts at the end and skips them. Use earliest (or by_duration on a client that supports it) for topics whose count may grow.

Key takeaways

  • A consumer group can use at most one consumer per partition, so the partition count caps the group's parallelism.
  • Partitions = (peak rate / target utilisation) / (1,000 / processing ms), taken for the slowest group, with growth, rounded up to a divisible number.
  • Every partition costs open files, memory maps, replication work and client memory on every broker that hosts a replica.
  • Partitions can be added and never removed, and adding them changes the key to partition mapping, which breaks per-key ordering across the change.
  • Instances × listener concurrency is the number of group members. Members beyond the partition count are idle.
  • When one group needs more concurrency than the topic has partitions, use key-hashed lanes inside the batch and commit only after all of them finish.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A new topic will carry shipment events keyed by shipment id. The expected peak is 3,000 records per second. The only consumer group takes 6 ms per record, measured at p95 under load. The team wants to run at no more than 75% utilisation at peak and expects no significant growth.

Which partition count fits these numbers?

Scenario 2 of 3

An inventory topic has 8 partitions and is keyed by product id. The group runs 8 threads and lag is growing at about 200 records per second on every partition during a sale. Someone suggests raising the topic to 16 partitions right now and doubling the instances. Inventory updates for one product must be applied in order.

What is the main risk of doing that during the sale?

Scenario 3 of 3

A webhook delivery service reads a topic with 12 partitions. Each record triggers an HTTP call that takes about 200 ms, so one thread delivers 5 per second and 12 threads deliver 60. The peak is 150 records per second. Order matters only between webhooks for the same customer, which is the record key. Other groups on the topic are fast and need only 12 partitions.

Which change gets the service to the capacity it needs with the least side effect?

References

References: Apache Kafka 4.4: Basic Kafka operations (modifying topics: adding partitions, side effects, no reduction) ↗ · Apache Kafka 4.4: Hardware and OS (file descriptors, vm.max_map_count) ↗ · Apache Kafka 4.4: Design (leader election and the controller) ↗ · Apache Kafka 4.4: Consumer configs (max.poll.records, max.poll.interval.ms, max.partition.fetch.bytes, group.protocol) ↗ · Apache Kafka 4.4: Broker configs (num.partitions, num.replica.fetchers) ↗ · KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum ↗ · Spring for Apache Kafka: Message listener containers (concurrency) ↗ · Spring for Apache Kafka: Configuring topics (KafkaAdmin adds partitions) ↗ · Report a problem