Path 8 · lesson 2 of 5
Partition sizing and consumer parallelism
How to choose a partition count from a target rate and a processing time, what too many partitions cost, why the count is hard to change, and how to add parallelism safely.
What you will learn
- Calculate a partition count from a peak rate, a per-record processing time, a utilisation target and expected growth
- List what each extra partition costs on brokers and clients, using what the Kafka documentation states
- Explain why partitions can be added but not removed, and what adding them does to keyed ordering
- Map partitions to consumer threads across application instances with Spring's listener concurrency
- Process records of one partition in parallel while keeping the order of records that share a key
Before you start
The concept
A partition is the unit of parallelism for a consumer group. Within one group a partition is read by exactly one consumer at a time, so a topic with 12 partitions can keep at most 12 consumers of that group busy. A thirteenth consumer joins the group, is assigned nothing and waits as a spare.
That makes the partition count a capacity decision. Too few and the group cannot be scaled past a ceiling, however many instances you start. Too many and you pay for them on every broker and client, for the lifetime of the topic. The count is also close to permanent: Kafka can add partitions to a topic and cannot remove them, and adding them changes which partition a key goes to.
So the count is chosen from numbers, in this order: the peak rate the topic must carry, how fast one consumer thread processes a record, how much spare capacity you want at peak, and how much the rate will grow before you are willing to migrate to a new topic.
Consumers beyond the partition count are idle
Simulation · not a real clusterSix partitions and six consumers. Add a seventh and an eighth consumer and look at what they are assigned. Then raise the partition count.
Why it exists
The cap exists because of the ordering guarantee. Kafka promises order within a partition, and a consumer group keeps that promise by giving each partition one reader. If two consumers of a group read the same partition, records for one order could be processed out of sequence, and one committed position could not describe what both had done. Parallelism therefore comes from having more partitions, each with its own ordered stream.
The same rule explains why the number is hard to change later. A keyed record goes to the partition given by a hash of its key modulo the partition count. Change the count and the same key gives a different result.
Watch keys move when a partition is added
Simulation · not a real clusterSix order ids are spread over six partitions. Note where each key sits, then add a partition and see which keys change place.
Because of this, the usual practice is to create a keyed topic with enough partitions for its expected lifetime, and to treat a later increase as a migration to be planned, not a setting to turn up during an incident.
How it works inside
The sizing calculation
- Per-thread rate. One consumer thread processing records one at a time handles 1,000 / (processing time in ms) records per second. Measure the processing time under load, including the database call, at a high percentile and not only the average.
- Capacity needed. Divide the peak produce rate by the utilisation you want at peak. A target of 70% means capacity = peak / 0.7, which leaves room for bursts and for catching up after a pause.
- Threads needed. Capacity needed / per-thread rate, rounded up. This is the minimum partition count for that consumer group.
- Slowest group decides. Repeat for every consumer group that reads the topic. The group that needs the most threads sets the count, unless it gets its parallelism another way (see below).
- Growth. Redo the calculation with the peak you expect at the end of the topic's planned life, and use that result.
- Round up to a convenient number. A count with many divisors, such as 12, 24 or 48, divides evenly over 2, 3, 4 or 6 instances, so each thread gets the same number of partitions.
Check the produce side as well: the bytes per second per partition must be something one partition leader can append and its followers can replicate. For business events this is almost never the limit. For a high-volume stream, measure it with kafka-producer-perf-test.sh on a test topic instead of assuming a figure.
What each partition costs
| Where | Cost | What the Kafka documentation says |
|---|---|---|
| Broker: open files | Every log segment of every partition replica is an open file | A broker needs at least partitions × (partition size / segment size) file descriptors plus its connections; at least 100000 allowed descriptors is the recommended starting point |
| Broker: memory maps | Each segment has an offset index and a time index, both memory-mapped | Each segment uses 2 map areas. With the common Linux default vm.max_map_count of about 65535, 50,000 partitions on one broker need 100,000 map areas and are likely to crash it |
| Broker: failure handling | When a broker fails, a new leader is elected for every partition it led | The design guide calls this the critical window of unavailability and describes how the controller batches the leadership changes to keep it short for many partitions |
| Producer: memory and batching | The producer keeps a batch per partition it is writing to | A buffer of batch.size is allocated per batch. The same traffic spread over more partitions fills each batch more slowly, so batches are smaller |
| Consumer: fetch size | A fetch can return data for every assigned partition | Up to max.partition.fetch.bytes (1 MiB) per partition, within fetch.max.bytes (50 MiB) per request |
| Replication | Each follower fetches every partition it replicates | num.replica.fetchers (default 1) threads per source broker carry all of it |
What else happens when you add partitions
- Existing data stays where it is. Old records are not moved to the new partitions. The new partitions start empty.
- Clients find out late. Producers and consumers learn about new partitions when they refresh metadata (
metadata.max.age.ms, default 5 minutes). Until then traffic is uneven. - Records can be skipped. The operations guide warns that a consumer with
auto.offset.reset=latestmay miss records produced to a new partition before the group discovers it, because the partition has no committed offset and the consumer starts at its end. For such topics the 4.4 guide recommendsby_duration:<duration>, a setting of newer clients; with an older client useearliestand idempotent processing. - The group rebalances to hand out the new partitions.
- Never do it to internal topics such as
__consumer_offsetsor__transaction_state: the guide warns this breaks the coordinator mapping.
Consumers, threads and instances
A group member is one KafkaConsumer, which is used by one thread. In Spring, concurrency on a listener container creates that many consumers and threads inside one application instance. The total number of members in the group is instances × concurrency, and Kafka spreads the partitions over all of them.
| Setup for 24 partitions | Members | Result |
|---|---|---|
| 6 instances × concurrency 4 | 24 | One partition per thread: full parallelism |
| 3 instances × concurrency 4 | 12 | Two partitions per thread: half the capacity, same ordering |
| 8 instances × concurrency 4 | 32 | 24 threads work, 8 are idle spares |
| 6 instances × concurrency 4, one instance down | 20 | Its 4 partitions move to other threads, which now own two each |
More parallelism than partitions
Sometimes one group needs far more concurrency than the others, typically because each record makes a slow remote call. Raising the partition count for everyone is one answer. The other is to process several records of the same partition at once without breaking the order that matters, which is the order of records with the same key.
The safe pattern: take a batch from poll(), split it into a fixed number of lanes by a hash of the key, run the lanes in parallel with each lane processing its records in sequence, and return only when every lane has finished. Records with the same key always share a lane, so their order holds. The offsets are committed after the whole batch, so the committed position never passes an unfinished record.
Architecture
Drawing diagram…
Flowchart: peak rate and target utilisation give capacity needed, processing time gives the per-thread rate, together they give threads needed. The largest result over all consumer groups, adjusted for growth, becomes the partition count, which is checked against per-broker cost before the topic is created.
flowchart TD
RATE["Peak produce rate"] --> NEED["Capacity needed = peak / target utilisation"]
PROC["Processing time per record"] --> PER["Per thread rate = 1000 / ms"]
NEED --> THREADS["Threads needed = capacity / per thread rate"]
PER --> THREADS
THREADS --> GROUPS["Take the largest result over all consumer groups"]
GROWTH["Expected growth over the topic lifetime"] --> GROUPS
GROUPS --> COUNT["Partition count, rounded up to a divisible number"]
COUNT --> CHECK{"Per broker cost acceptable"}
CHECK -->|"yes"| CREATE["Create the topic"]
CHECK -->|"no"| ALT["Faster processing or parallel lanes inside a partition"]
ALT --> PROC- Two measured inputs drive everything: the peak rate and the processing time per record. If either is a guess, the result is a guess.
- The utilisation target turns a rate that just keeps up into one with room for bursts and recovery.
- The loop at the bottom is the alternative to more partitions: make each record cheaper, or process several keys of one partition in parallel.
- The decision is made once per topic, at creation, because lowering the count is impossible and raising it disturbs keyed ordering.
Message flow
Drawing diagram…
Sequence diagram: an order's first event goes to partition 3 and is read by consumer 1. The operator raises the partition count from 6 to 8. The order's next event hashes to partition 7 and, after a rebalance, is read by consumer 2.
sequenceDiagram participant Ops as Operator participant K as Kafka cluster participant P as Order producer participant C1 as Consumer 1 participant C2 as Consumer 2 P->>K: OrderPlaced key o-1001 to partition 3 of 6 K-->>C1: partition 3 delivers OrderPlaced Ops->>K: alter topic to 8 partitions Note over P: metadata refresh shows 8 partitions P->>K: OrderCancelled key o-1001 to partition 7 of 8 K->>C1: rebalance K->>C2: rebalance and assign partition 7 K-->>C2: partition 7 delivers OrderCancelled Note over C1,C2: two consumers hold events of one order
- Before the change, every event of order
o-1001goes to one partition and reaches one consumer in order. - After the change the hash is taken modulo 8 instead of 6. For this key the result differs, so the next event goes elsewhere. (The partition numbers in the picture are illustrative.)
- If consumer 1 is behind and consumer 2 is not, the cancellation can be processed before the placement. Nothing in Kafka prevents this, since order is per partition.
- A safe increase therefore needs a quiet period for the affected keys, consumers that tolerate reordering, or a new topic with the new count and a controlled switch.
In a real system
The example project sends order events to orders.events, keyed by order id, and three groups read it: payment-service, inventory-service and notification-service. All figures below are assumptions for this example. Replace them with your own measurements.
Worked example: sizing orders.events
| Step | Arithmetic | Result |
|---|---|---|
| Peak rate today | Assumed from the busiest hour | 1,200 records/s |
| Peak rate at end of planned life | Assumed growth to | 1,600 records/s |
| Capacity needed at 70% utilisation | 1,600 / 0.7 | 2,286 records/s |
| Payment: 10 ms per record | 1,000 / 10 = 100 per thread; 2,286 / 100 = 22.9 | 23 threads |
| Inventory: 4 ms per record | 1,000 / 4 = 250 per thread; 2,286 / 250 = 9.1 | 10 threads |
| Notification: 25 ms per record (email API) | 1,000 / 25 = 40 per thread; 2,286 / 40 = 57.2 | 58 threads |
| Count from payment, rounded to a divisible number | 23 → 24 | 24 partitions |
| Notification with 24 partitions | 3 lanes per partition: 24 × 3 × 40 | 2,880 records/s, enough |
| Bytes per partition at peak | 1,600 × 1 KB / 24 | 67 KB/s |
Sizing the topic for the notification group would have given 60 partitions, which the other two groups do not need and every broker would pay for. Parallel lanes confine the cost to the one service with the slow call. Payment runs as 6 instances with concurrency 4, inventory as 3 instances with concurrency 4 (12 threads, two partitions each), and notification as 6 instances with concurrency 4 and 3 lanes.
The payment group at today's peak
Simulation · not a real cluster24 partitions, 24 consumers, 1,200 records per second at 10 ms each. Check the headroom, then raise the rate to the planned 1,600.
Java and Spring Boot
The topic is declared in code with its partition count, the listener's concurrency comes from configuration, and the notification listener shows key-ordered parallel processing inside a partition.
package com.example.order;import org.apache.kafka.clients.admin.NewTopic;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class OrderTopicConfig { @Bean public NewTopic orderEvents() { return TopicBuilder.name("orders.events") .partitions(24) .replicas(3) .config("min.insync.replicas", "2") .build(); }}Why it is written this way
- Spring's
KafkaAdmincreates the topic at startup if it does not exist. - It also adds partitions if the existing topic has fewer than this bean declares (documented in the Spring for Apache Kafka reference under configuring topics). Changing 24 to 48 here and deploying is therefore a repartitioning of a keyed topic, with the ordering consequences described above. Treat this number as a schema, not a tuning value.
- The broker default
num.partitionsis 1. A topic created automatically, or without an explicit count, gets one partition and no parallelism.
spring: kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS} consumer: group-id: payment-service auto-offset-reset: earliest max-poll-records: 100 listener: concurrency: 4Why it is written this way
concurrency: 4gives four consumers in this instance. With 6 instances the group has 24 members for 24 partitions.- Total members above the partition count are idle. Size instances × concurrency to the partition count, plus spares only if you want them.
max-poll-records: 100at 10 ms per record keeps one batch at about a second, far from the 5 minutemax.poll.interval.ms, and limits how long a record waits behind others in its batch.auto-offset-reset: earliestmeans a partition with no committed offset, including a newly added one, is read from its start.
package com.example.notification;import java.util.List;import java.util.Map;import java.util.Objects;import java.util.concurrent.CompletableFuture;import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;import java.util.stream.Collectors;import jakarta.annotation.PreDestroy;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderNotificationListener { private static final int LANES = 3; private final NotificationSender sender; private final ExecutorService workers = Executors.newVirtualThreadPerTaskExecutor(); public OrderNotificationListener(NotificationSender sender) { this.sender = sender; } @KafkaListener(id = "notification-service", topics = "orders.events", batch = "true", concurrency = "4") public void onBatch(List<ConsumerRecord<String, OrderPlaced>> records) { Map<Integer, List<ConsumerRecord<String, OrderPlaced>>> lanes = records.stream() .collect(Collectors.groupingBy(r -> Math.floorMod(Objects.hashCode(r.key()), LANES))); List<CompletableFuture<Void>> running = lanes.values().stream() .map(lane -> CompletableFuture.runAsync(() -> lane.forEach(r -> sender.send(r.value())), workers)) .toList(); CompletableFuture.allOf(running.toArray(CompletableFuture[]::new)).join(); } @PreDestroy void shutdown() { workers.close(); }}Why it is written this way
batch = "true"makes this a batch listener: it receives everything onepoll()returned.- Records are grouped by a hash of the key.
groupingBykeeps the records of each lane in their original order, and each lane runs as one sequential task, so two events of the same order are never processed at once or out of order. join()blocks the consumer thread until every lane is done. Only then does the container commit the batch, so the committed position never runs ahead of unfinished work.- If one lane fails,
join()throws and the container's error handler redelivers the batch. Records that had already succeeded in other lanes are processed again, sosender.sendmust be idempotent (for example by recording the event id). - Virtual threads (Java 21) suit this because the work is a blocking remote call. The lanes add concurrent calls to the email API: 24 partitions × 3 lanes is up to 72 at once, which the API and any connection pool must accept.
- Keep the batch time below
max.poll.interval.ms: withmax.poll.recordsat 500, 3 lanes and 25 ms per record a full batch takes about 500 / 3 × 25 ms ≈ 4.2 seconds.
Configuration
Settings that decide how many partitions a topic gets and how consumers spread over them. Defaults are from the Kafka 4.4 references and the Spring Boot 3.5 application properties.
num.partitions (broker)Default 1
- What it does
- Partition count for topics created automatically or created without an explicit count.
- When to change it
- Do not rely on it. Create each topic with a calculated count, and consider disabling automatic topic creation so a typo cannot create a one-partition topic.
spring.kafka.listener.concurrencyDefault none (one consumer per listener)
- What it does
- Number of consumers, each with its own thread, that a listener container runs in one application instance.
- When to change it
- Set instances × concurrency to the partition count. Higher values add idle members. It can also be set per listener with the
concurrencyattribute of@KafkaListener.
max.poll.records (consumer)Default 500
- What it does
- Maximum records per
poll(). Bounds the size of one batch of work. - When to change it
- Lower it when processing is slow, so that batch size × processing time stays far below
max.poll.interval.ms.
max.poll.interval.ms (consumer)Default 300000
- What it does
- Longest allowed time between two calls to
poll()before the consumer is considered failed and its partitions are reassigned. - When to change it
- Size batches to fit well inside it. Raising it hides slow processing and delays detection of a stuck consumer.
metadata.max.age.ms (producer and consumer)Default 300000
- What it does
- How often a client refreshes cluster metadata even without an error. With the classic group protocol this is how a consumer discovers new partitions.
- When to change it
- Explains why traffic to new partitions starts unevenly. Lowering it speeds discovery at the cost of more metadata requests.
max.partition.fetch.bytes (consumer)Default 1048576
- What it does
- Maximum data per partition in one fetch response. Not an absolute limit: a larger first batch is still returned so the consumer can progress.
- When to change it
- With many partitions per consumer, this times the partition count bounds what one fetch can hold in memory, within
fetch.max.bytes(52428800).
group.protocol (consumer)Default classic
- What it does
- Which rebalance protocol the consumer uses:
classic, orconsumerfor the newer protocol from KIP-848. - When to change it
- Affects how fast new partitions and members are picked up: with
consumerthe broker pushes assignments on the next heartbeat. Needs a client and brokers that support it.
Hands-on exercise
Size a topic, then find the limit that partitions cannot fix
Reproduce the worked example in the throughput model, see what partitions and consumers each contribute, and find the point where another resource becomes the limit.
Steps · 0 of 6 done
From exactly enough to planned headroom
Simulation · not a real clusterA simple queueing model with illustrative numbers. It starts at 100% utilisation: 12 consumers × 100 records per second for 1,200 arriving.
What you should see
At 12 and 12 the group just keeps up with 0% headroom. Partitions alone add nothing: 18 partitions with 12 consumers still gives a ceiling of 1,200, and because six consumers now own two partitions each, those six fall behind. With 18 consumers the ceiling is 1,800 (33% headroom at 1,200) and with 24 it is 2,400 (50%). With 24 consumers on 18 partitions, 6 are reported idle. At 1,600 records per second, 24 consumers leave 33%. With a pool of 10 the ceiling drops to 1,000 whatever the partition count, and the model names the database pool as the limit.
What can go wrong
Scaling consumers past the partition count
- Symptom
- Lag is growing, the deployment is scaled from 6 to 12 instances, and the consume rate does not change. Half the instances show no assigned partitions.
- Cause
- The topic has 6 partitions. A group can use at most one consumer per partition, so the new members are idle.
- Fix
- Compare members with partitions before scaling (
kafka-consumer-groups.sh --describe --members). Beyond the cap, make processing faster, add key-ordered parallelism inside the partition, or plan a partition increase.
Adding partitions to a keyed topic during an incident
- Symptom
- After the partition count was raised, a small number of orders end up in impossible states, such as cancelled and then paid.
- Cause
- The key to partition mapping changed. Later events for existing keys went to different partitions and were processed in a different order relative to the earlier ones.
- Fix
- Size keyed topics up front. If an increase is unavoidable, make consumers tolerate reordering (version checks, idempotent state transitions) or migrate to a new topic with a controlled cutover.
Choosing a huge count to be safe
- Symptom
- Brokers approach file descriptor or memory map limits, restarts and failovers take longer, and producer batches are tiny because each partition receives little traffic.
- Cause
- Every partition replica costs open files, map areas, replication work and client memory, whether or not it carries traffic.
- Fix
- Calculate the count from rate, processing time, headroom and growth. Check partitions per broker against
ulimit -nandvm.max_map_count.
Changing the NewTopic partition count in code
- Symptom
- A routine deployment is followed by a rebalance and by keys appearing in new partitions. Nobody ran a topic command.
- Cause
- Spring's
KafkaAdminincreases the partition count of an existing topic when theNewTopicbean declares more. - Fix
- Review changes to topic declarations like schema changes. Where topics are managed centrally, do not declare them in the service at all.
Parallelising with an unbounded executor
- Symptom
- Throughput rises, then records are lost on restart and events for one key are occasionally applied in the wrong order.
- Cause
- The listener returned before the work finished, so offsets were committed early, and records with the same key ran concurrently.
- Fix
- Use lanes keyed by a hash of the record key and wait for all of them before returning, as in the notification listener above.
How to troubleshoot
You need to know how many partitions a topic has and where they are
Check
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders.events
- How to read it
- The first line shows
PartitionCountandReplicationFactor; each following line shows a partition's leader, replicas and in-sync replicas.
Consumers were added and throughput did not rise
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service --members --verbose
- How to read it
- Each member is listed with its assigned partitions. Members with none are above the partition cap, or subscribed to something else.
Lag grows on all partitions at a similar rate
Check
Compare the produce rate with threads × 1,000 / processing ms, using the `orders.events.processing` timer from the previous lesson
- How to read it
- If the produce rate is above the ceiling, the group is undersized: add consumers up to the partition count, then reduce processing time. If it is below, look for rebalances or a slow dependency.
A broker logs OutOfMemoryError: Map failed or Too many open files
Check
sysctl vm.max_map_count; ulimit -n for the broker process; kafka.server:type=ReplicaManager,name=PartitionCount
- How to read it
- The broker hosts more partition segments than the operating system limits allow. Each segment needs 2 map areas and file descriptors. Raise the limits and reduce partitions per broker.
After adding partitions, some consumers never read the first records of the new partitions
Check
The consumer's `auto.offset.reset` setting and the group's committed offsets for the new partitions
- How to read it
- With
latest, a consumer that discovers the partition after records were written starts at the end and skips them. Useearliest(orby_durationon a client that supports it) for topics whose count may grow.
Key takeaways
- A consumer group can use at most one consumer per partition, so the partition count caps the group's parallelism.
- Partitions = (peak rate / target utilisation) / (1,000 / processing ms), taken for the slowest group, with growth, rounded up to a divisible number.
- Every partition costs open files, memory maps, replication work and client memory on every broker that hosts a replica.
- Partitions can be added and never removed, and adding them changes the key to partition mapping, which breaks per-key ordering across the change.
- Instances × listener concurrency is the number of group members. Members beyond the partition count are idle.
- When one group needs more concurrency than the topic has partitions, use key-hashed lanes inside the batch and commit only after all of them finish.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A new topic will carry shipment events keyed by shipment id. The expected peak is 3,000 records per second. The only consumer group takes 6 ms per record, measured at p95 under load. The team wants to run at no more than 75% utilisation at peak and expects no significant growth.
Which partition count fits these numbers?
Scenario 2 of 3
An inventory topic has 8 partitions and is keyed by product id. The group runs 8 threads and lag is growing at about 200 records per second on every partition during a sale. Someone suggests raising the topic to 16 partitions right now and doubling the instances. Inventory updates for one product must be applied in order.
What is the main risk of doing that during the sale?
Scenario 3 of 3
A webhook delivery service reads a topic with 12 partitions. Each record triggers an HTTP call that takes about 200 ms, so one thread delivers 5 per second and 12 threads deliver 60. The peak is 150 records per second. Order matters only between webhooks for the same customer, which is the record key. Other groups on the topic are fast and need only 12 partitions.
Which change gets the service to the capacity it needs with the least side effect?
References
References: Apache Kafka 4.4: Basic Kafka operations (modifying topics: adding partitions, side effects, no reduction) ↗ · Apache Kafka 4.4: Hardware and OS (file descriptors, vm.max_map_count) ↗ · Apache Kafka 4.4: Design (leader election and the controller) ↗ · Apache Kafka 4.4: Consumer configs (max.poll.records, max.poll.interval.ms, max.partition.fetch.bytes, group.protocol) ↗ · Apache Kafka 4.4: Broker configs (num.partitions, num.replica.fetchers) ↗ · KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum ↗ · Spring for Apache Kafka: Message listener containers (concurrency) ↗ · Spring for Apache Kafka: Configuring topics (KafkaAdmin adds partitions) ↗ · Report a problem