Path 3 · lesson 2 of 5
Keys and partitioning: who decides where a record goes
How the producer picks a partition, why the key is what gives you ordering, how to choose a key, and what skew and adding partitions do to both.
What you will learn
- Predict which partition a record goes to from its partition field, its key and the topic's partition count
- Explain why records with the same key stay in order and name the things that break that
- Choose a key for an event from the ordering the consumers actually need
- Recognise skew caused by a hot or low-cardinality key and pick a fix that keeps the ordering you rely on
- Plan a partition count change knowing that it moves keys to different partitions
- Decide when a custom partitioner is justified and when a different key is the better answer
Before you start
The concept
A topic is split into partitions, and each partition is an independent, ordered log. A record is appended to exactly one of them, and the producer makes that choice before the record leaves your process. The broker does not route records; it appends what it is given to the partition named in the request.
The choice follows a fixed order of precedence:
| The record has | Partition used | Typical use |
|---|---|---|
| An explicit partition number | That partition, no questions asked | Rare: tools, tests, replaying to the original partition |
| A key (and no partition) | murmur2(serialised key bytes) made non-negative, modulo the number of partitions | The normal case for business events |
| Neither | The producer's current sticky partition, which changes after about batch.size bytes | Logs, metrics, clickstream where order between records does not matter |
partitioner.class is configured, it replaces the last two rows. An explicit partition still wins.The second row is the one that matters for correctness. The hash is deterministic, so two records with the same key bytes and the same partition count always go to the same partition. Since a partition is ordered, those records are stored and read in the order the broker appended them. The key is how you tell Kafka which records must stay in order relative to each other.
Keys to partitions
Simulation · not a real clusterSend records for the four orders and see that each order always lands in the same partition. Then send records without a key.
Why it exists
Partitions exist so a topic can be written and read in parallel. Parallelism and ordering pull in opposite directions: one log gives a total order and no parallelism, while spreading records at random gives parallelism and no order at all.
Most systems do not need a total order. They need the events of one order, one account or one device to be in sequence, and they do not care how order 1001 interleaves with order 1002. Hashing a key gives exactly that: each entity gets a private ordered lane, and different entities are processed in parallel by different consumers in a group.
Records without a key have no ordering requirement, so the producer is free to optimise for something else. Early clients rotated through partitions record by record, which produced many tiny batches. Since Kafka 2.4 the producer sticks to one partition for a while so batches fill up, and since 3.3 it switches after a fixed number of bytes rather than on each new batch, which keeps the spread even when one broker is slower than the others.
Keyed records
- Partition is a pure function of key bytes and partition count
- Same key, same partition, order preserved
- Distribution is only as even as the keys are
- Changing the partition count changes the mapping
Records without a key
- Producer picks a partition and stays on it for about one batch
- No ordering relationship between records
- Bytes spread evenly over time, adapting to slow brokers
- Changing the partition count has no semantic effect
How it works inside
The key hash
The Java client computes the partition from the serialised key: the 32-bit murmur2 hash of the key bytes, with the sign bit cleared, modulo the topic's partition count. Three consequences follow directly.
- The serialiser is part of the mapping. The string
"42"and the long42serialise to different bytes and usually land in different partitions. Every producer writing to a topic must agree on the key type and its serialisation. - The modulus is the total number of partitions, not the number that are currently reachable. If the leader of the target partition is unavailable, a keyed record waits in the buffer and is retried; it is never diverted to another partition.
- The client library is part of the mapping too. Clients built on librdkafka (Python, Go, .NET and others) default to a CRC32-based partitioner; they must be set to
murmur2_randomto agree with Java producers.
No key: sticky, then uniform
For a record with no key and no partition, the built-in logic keeps a current partition per topic and sends everything there until roughly batch.size bytes have been produced to it, then picks another. With partitioner.adaptive.partitioning.enable=true (the default) the next partition is chosen with a probability weighted against brokers whose queues in the accumulator are long, so a slow broker receives less. partitioner.availability.timeout.ms (default 0, off) can additionally exclude partitions whose broker has not accepted a batch for that long. These settings affect only records without a key.
What breaks per-key ordering
- Changing the partition count. The modulus changes, so most keys map to a different partition from then on. Kafka does not move existing records. For a while a key has old events in one partition and new events in another, and a consumer group may read the new ones first.
- Producers that disagree on key serialisation or hash function. The same entity is then in two partitions at once.
- Retries without idempotence while several requests are in flight, which can swap two batches inside one partition. See Acks, retries and idempotence.
- Several producers writing events for the same key. Each producer's records are ordered among themselves, but the interleaving between producers is decided by arrival at the broker, not by your business clock.
- Reprocessing on the consumer side, for example a retry topic or a rewind, which replays older records after newer ones have been handled. The log is still ordered; your processing is not.
Choosing a key
Start from the consumer: which events must it see in sequence? Use the identifier of that entity. For order lifecycle events that is the order id; for account balance changes, the account id; for device telemetry that feeds a per-device state machine, the device id. Then check the key against three questions.
- Is it specific enough? A key is the smallest unit that needs ordering. Using
customerIdwhen only per-order ordering is needed forces all of a customer's orders through one partition for no benefit. - Does it have enough distinct values with no dominant one? A key such as country code or tenant id concentrates traffic in a few partitions. One tenant with 40% of the traffic means one partition with at least 40% of the traffic, whatever the partition count.
- Is it stable? The key must not change during the entity's life. An email address or a status field is a poor key.
Hot partition key
Simulation, not a real clusterThe animated Build lesson shows one hot key overloading a single partition while the others stay idle.
Step 1 · The pattern
The key picks the partition
Every message has a key. Kafka hashes it to choose a partition, so messages with the same key always share a partition and stay in order.
- message
- consumed
Custom partitioners, and when not to write one
A class implementing org.apache.kafka.clients.producer.Partitioner and named in partitioner.class receives the topic, the key and value (as objects and as bytes) and the cluster metadata, and returns a partition number. It replaces the built-in logic for every record that has no explicit partition, so the sticky and adaptive behaviour for unkeyed records, and the partitioner.* settings, no longer apply.
Write one only when the routing rule cannot be expressed as a key: for example, isolating one very large tenant on dedicated partitions so it cannot delay everyone else. In most other cases a better key does the job with no code to maintain. A custom partitioner is a hidden contract: every producer to the topic, in every language, must implement the same rule, and nothing checks that they do.
Architecture
Drawing diagram…
Decision flowchart. If the record has an explicit partition it is used. Otherwise, if a custom partitioner is configured it decides. Otherwise, if a key is present and keys are not ignored, the murmur2 hash of the serialised key modulo the partition count gives the partition. Otherwise the record goes to the current sticky partition, which is replaced after about batch.size bytes by a new partition chosen with a preference for brokers with short queues.
flowchart TD
R["Record passed to send"] --> Q1{"Partition set on the record"}
Q1 -->|yes| P1["Use that partition"]
Q1 -->|no| Q2{"partitioner.class configured"}
Q2 -->|yes| P2["Custom partitioner returns the partition"]
Q2 -->|no| Q3{"Key present and keys not ignored"}
Q3 -->|yes| H["murmur2 of serialised key bytes"]
H --> M["Modulo partition count"]
M --> P3["Same key gives same partition"]
Q3 -->|no| S["Current sticky partition for the topic"]
S --> SW{"About batch.size bytes produced to it"}
SW -->|no| S2["Stay on it"]
SW -->|yes| A["Pick next partition, favouring brokers with short queues"]- An explicit partition bypasses everything, including a custom partitioner. That makes it a sharp tool: the caller takes over responsibility for distribution and ordering.
- The keyed branch has no state and no randomness. Given the key bytes and the partition count, any Java producer anywhere computes the same answer.
- The unkeyed branch has state per producer instance. Two producers will be on different sticky partitions at the same moment, which is fine because these records carry no ordering requirement.
partitioner.ignore.keys=truesends keyed records down the unkeyed branch. It is for topics where the key is needed for compaction or for consumers, but not for placement.
Message flow
Drawing diagram…
Sequence diagram. With six partitions, two events for order-1001 are hashed to partition 1 and consumed in order. After the topic is altered to eight partitions and the producer refreshes metadata, the next event for order-1001 hashes to partition 4. A consumer of partition 4 receives OrderShipped with no earlier events for that order in the same partition.
sequenceDiagram participant OS as Order service participant P as Producer participant P1 as Partition 1 of orders participant P4 as Partition 4 of orders participant C as Payment consumer Note over P: topic has 6 partitions OS->>P: OrderPlaced key order-1001 P->>P1: hash of key mod 6 gives 1, append at offset 40 OS->>P: OrderPaid key order-1001 P->>P1: same key, append at offset 41 C->>P1: fetch P1-->>C: OrderPlaced then OrderPaid Note over P1,P4: topic is altered to 8 partitions P->>P: metadata refresh sees 8 partitions OS->>P: OrderShipped key order-1001 P->>P4: hash of key mod 8 gives 4, append at offset 0 C->>P4: fetch P4-->>C: OrderShipped with nothing before it
- Before the change, both events share a partition, so any consumer of that partition sees
OrderPlacedbeforeOrderPaid. - The alter command does not touch stored records. Offsets 40 and 41 stay in partition 1.
- The producer starts using the new count when it refreshes metadata, which can take up to
metadata.max.age.ms. Several producers switch at different moments, so for a few minutes the same key can be written to both the old and the new partition. - After the change, the order's history is split across two partitions. If the consumer of partition 1 is behind,
OrderShippedcan be processed beforeOrderPaid.
In a real system
A typical payments platform publishes every state change of a payment (PaymentAuthorised, PaymentCaptured, PaymentRefunded) to one topic, payments.events.v1, with 12 partitions. A ledger service consumes it and applies each change to the payment's row.
| Candidate key | What you get | Verdict |
|---|---|---|
| No key | Even spread, but Captured can be read before Authorised | Wrong: the ledger needs per-payment order |
merchantId | Per-merchant order, but the largest merchant fills one partition and its consumer falls behind | Stronger ordering than needed, at the cost of skew |
paymentId | Per-payment order; millions of distinct values spread evenly | Right: the smallest unit that needs ordering |
paymentId plus event type | Each event type of a payment may land in a different partition | Wrong: destroys the ordering the key was for |
With paymentId as the key, 12 ledger consumers can each own one partition, and each payment is always handled by one consumer at a time. A merchant-level report that needs all of a merchant's payments together is a different consumer with a different need; it reads the same topic and groups by merchant in its own store rather than forcing the topic's key to serve it.
The team sized the topic at 12 partitions on purpose, above current need, because growing it later would reshuffle keys. When a topic with keyed data really must grow, the usual approach is a new topic with the new count: producers switch to it, and consumers finish the old topic before starting the new one.
Java and Spring Boot
The key is chosen where the record is built. The first file shows the normal case, an entity id as the key. The other two show a custom partitioner for the one situation that tends to justify it, with the configuration that activates it.
package com.example.payments.events;import java.nio.charset.StandardCharsets;import java.util.concurrent.CompletableFuture;import org.apache.kafka.clients.producer.ProducerRecord;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.support.SendResult;import org.springframework.stereotype.Component;@Componentpublic class PaymentEventPublisher { public static final String TOPIC = "payments.events.v1"; private static final Logger log = LoggerFactory.getLogger(PaymentEventPublisher.class); private final KafkaTemplate<String, PaymentEvent> kafkaTemplate; public PaymentEventPublisher(KafkaTemplate<String, PaymentEvent> kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; } public CompletableFuture<SendResult<String, PaymentEvent>> publish(PaymentEvent event) { // Key = the entity whose events must stay in order. Never the event id, never the event type. String key = event.paymentId(); ProducerRecord<String, PaymentEvent> record = new ProducerRecord<>(TOPIC, key, event); record.headers().add("event-type", event.type().getBytes(StandardCharsets.UTF_8)); return kafkaTemplate.send(record).whenComplete((result, ex) -> { if (ex != null) { log.error("{} not published paymentId={}", event.type(), key, ex); } else { log.debug("{} paymentId={} -> partition {} offset {}", event.type(), key, result.getRecordMetadata().partition(), result.getRecordMetadata().offset()); } }); } /** All state changes of one payment share this shape and this topic. */ public record PaymentEvent(String eventId, String paymentId, String merchantId, String type, long amountMinor) { }}Why it is written this way
- All event types of a payment go to one topic with one key. Putting each type in its own topic would lose the order between them, because ordering does not span topics.
- The key is a
Stringserialised withStringSerializer. Any other service producing to this topic must also send the payment id as a UTF-8 string, or the same payment ends up in two partitions. - No partition is passed. Hard-coding a partition number in application code ties the code to the current partition count.
- Logging the partition at debug level is useful when investigating ordering: the same
paymentIdshould always show the same partition.
package com.example.payments.events;import java.util.Arrays;import java.util.Map;import java.util.Set;import java.util.stream.Collectors;import org.apache.kafka.clients.producer.Partitioner;import org.apache.kafka.common.Cluster;import org.apache.kafka.common.utils.Utils;/** * Keys look like "merchantId:paymentId". Merchants listed in * "dedicated.merchants" get the LAST partition of the topic to themselves; * everyone else is hashed over the remaining partitions. * Per-key ordering is kept in both cases. */public class DedicatedMerchantPartitioner implements Partitioner { public static final String DEDICATED_MERCHANTS_CONFIG = "dedicated.merchants"; private Set<String> dedicatedMerchants = Set.of(); @Override public void configure(Map<String, ?> configs) { Object value = configs.get(DEDICATED_MERCHANTS_CONFIG); if (value != null) { dedicatedMerchants = Arrays.stream(value.toString().split(",")) .map(String::trim) .filter(s -> !s.isEmpty()) .collect(Collectors.toUnmodifiableSet()); } } @Override public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) { int partitions = cluster.partitionsForTopic(topic).size(); if (keyBytes == null) { throw new IllegalArgumentException("Records on " + topic + " must have a key"); } if (partitions == 1) { return 0; } String merchantId = key.toString().split(":", 2)[0]; if (dedicatedMerchants.contains(merchantId)) { return partitions - 1; } // Same hash as the built-in logic, over one partition fewer. return Utils.toPositive(Utils.murmur2(keyBytes)) % (partitions - 1); } @Override public void close() { // nothing to release }}Why it is written this way
- The rule cannot be expressed with a key alone: it reserves a partition for named merchants. That is the kind of requirement that justifies a partitioner.
- It still hashes the key for everyone else, so a payment keeps its partition and its ordering. A partitioner that returns a random or rotating partition for keyed records silently removes per-key ordering.
Utils.murmur2andUtils.toPositiveare the functions the client uses internally.Utilsis not part of the documented public API, so cover this class with a test that pins the expected partition for known keys.- Unkeyed records are rejected rather than spread, because with a custom partitioner the built-in sticky logic is no longer available.
- Every producer to the topic must use this class with the same merchant list. A producer without it will place the dedicated merchant's payments elsewhere.
spring: kafka: bootstrap-servers: localhost:9092 producer: client-id: payment-service key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: spring.json.add.type.headers: false # Only when the routing rule cannot be a key. Remove both lines to get the built-in behaviour. partitioner.class: com.example.payments.events.DedicatedMerchantPartitioner dedicated.merchants: merchant-774Why it is written this way
partitioner.classis a plain producer property, so it goes underspring.kafka.producer.properties.- Custom properties such as
dedicated.merchantsare passed to the partitioner'sconfiguremethod along with the rest of the producer configuration. - With this partitioner the key in the publisher above would be
merchantId + ":" + paymentId. Changing the key format is itself a remapping of every key, so do it on a new topic.
Configuration
Settings that take part in partition choice. Defaults are from the Apache Kafka 4.x producer and broker configuration reference. The three partitioner.* tuning settings affect only records without a key and have no effect when a custom partitioner is set.
partitioner.classDefault null
- What it does
- Class that chooses the partition. Unset means the built-in logic: hash of the key when there is one, sticky partition otherwise.
- When to change it
- Leave unset unless a routing rule cannot be expressed as a key.
RoundRobinPartitionerships with the client but ignores keys, so it gives up per-key ordering.
partitioner.ignore.keysDefault false
- What it does
- When true, the key is not used for placement and keyed records are spread like unkeyed ones.
- When to change it
- Use only when the key exists for another reason, such as compaction or consumer logic, and you do not need per-key ordering.
partitioner.adaptive.partitioning.enableDefault true
- What it does
- For unkeyed records, send more to partitions on brokers that are keeping up and less to slow ones.
- When to change it
- Turn off only if you need unkeyed records spread strictly evenly regardless of broker speed.
partitioner.availability.timeout.msDefault 0
- What it does
- For unkeyed records, treat a partition as unavailable if its broker has not accepted a ready batch for this long. 0 disables the check.
- When to change it
- Consider a non-zero value if unkeyed traffic should route around a stalled broker quickly. It does nothing for keyed records.
batch.sizeDefault 16384
- What it does
- Besides bounding a batch, it is the number of bytes the producer sends to one sticky partition before switching.
- When to change it
- At low volume a small topic can look uneven over short windows because the producer stays on one partition for a whole batch. That is expected.
metadata.max.age.msDefault 300000 (5 minutes)
- What it does
- How often the producer refreshes metadata without being prompted by an error, which is how it learns that a topic has more partitions.
- When to change it
- Explains the window after a partition increase in which producers disagree about the count. Do not lower it to fix that; plan the change instead.
num.partitions (broker)Default 1
- What it does
- Partition count for topics created automatically, without an explicit count.
- When to change it
- Create topics explicitly with a count chosen for the next few years of keyed traffic. The count can be raised later but never lowered.
Hands-on exercise
Move a key by adding a partition
See that a key always maps to the same partition, that unkeyed records do not, and that one extra partition sends existing keys somewhere new.
Steps · 0 of 6 done
Partitioner: keys, no keys, and a resize
Simulation · not a real clusterSend keyed and unkeyed records, then add a partition and send again. This is a deterministic simulation of the producer's partition choice.
What you should see
Each key keeps its partition for as long as the count is 4. Unkeyed records cluster on one partition at a time. After the fifth partition is added, most keys map to a different partition while their earlier records stay where they were, so one key's history is now split across two partitions.
What can go wrong
Adding partitions to a keyed topic
- Symptom
- Shortly after a partition increase, consumers report impossible sequences such as a shipment for an order that is not yet paid. Nothing is lost.
- Cause
- The hash is taken modulo the partition count. The new count moved most keys, existing records stayed in the old partitions, and the consumers of the old and new partitions progress independently.
- Fix
- Size keyed topics generously up front. If a topic must grow, migrate to a new topic with the new count, or pause producers, let consumers drain to zero lag, then alter and resume. Consumers that check a version or sequence number on the entity tolerate the remaining window.
A low-cardinality or dominant key
- Symptom
- One partition's lag grows while the others are idle. Adding consumers or partitions does not help.
- Cause
- All records with one key go to one partition by design. A key with few values, or one value carrying a large share of the traffic, concentrates the load, and one partition is read by one consumer in a group.
- Fix
- Key by the smallest entity that needs ordering (order id rather than tenant id). If one entity really is that large, isolate it on its own partitions or topic. See Hot partitions and backpressure.
Producers that hash differently
- Symptom
- Events for the same entity are found in two partitions even though the partition count never changed.
- Cause
- Two services produce to the topic with different key serialisation (string versus number, different string formats) or with a client whose default partitioner is not murmur2, such as librdkafka's default.
- Fix
- Write the key type and format into the topic's contract. Set
partitioner=murmur2_randomin librdkafka-based clients that share a topic with Java producers.
Using the event id or a timestamp as the key
- Symptom
- Distribution is perfectly even, yet consumers see an entity's events out of order.
- Cause
- The key is unique per record, so related records hash to unrelated partitions. An even spread was achieved by giving up ordering.
- Fix
- Key by the entity id and carry the event id in a header or in the payload for deduplication.
A custom partitioner that forgets keys
- Symptom
- After a partitioner was added to balance load, per-key ordering problems appear across the topic.
- Cause
- The partitioner returns a rotating or load-based partition for every record, including keyed ones, and it also replaced the built-in handling of unkeyed records.
- Fix
- Remove it and fix the key, or make the partitioner deterministic per key. Test it with fixed keys and a fixed partition count.
How to troubleshoot
You need to confirm how many partitions a topic has and whether that changed
Check
kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic payments.events.v1
- How to read it
- The first line shows
PartitionCount. If it differs from the count the topic was designed with, keys were remapped at the time of the change.
Suspected skew: one consumer is always behind
Check
kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic payments.events.v1
- How to read it
- Prints
topic:partition:end offsetper partition. Run it twice a minute apart and compare the growth per partition. One partition growing several times faster than the rest points at a hot key, not at a slow consumer.
Lag is concentrated on one partition of a consumer group
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ledger-service
- How to read it
- Compare the
LAGcolumn across partitions. Even lag means the group is short of capacity overall. Lag on one partition with the others near zero means that partition receives more than one consumer can handle.
You want to see which keys land in which partition
Check
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments.events.v1 --from-beginning --max-messages 200 --formatter-property print.key=true --formatter-property print.partition=true
- How to read it
- Each line shows the partition and key. The same key appearing with two partition numbers means a partition count change or producers that hash differently. Tools before Kafka 4.2 spell the option
--property.
Unkeyed records look unevenly spread in a short test
Check
Producer metric `batch-size-avg` and the number of records sent in the test
- How to read it
- The producer stays on one partition for about
batch.sizebytes. A test that sends less than a few batches per partition will look lopsided. Over a longer run the byte counts even out.
Key takeaways
- The producer chooses the partition: explicit partition first, then a custom partitioner if set, then the key hash, then the sticky partition for unkeyed records.
- For keyed records the partition is murmur2 of the serialised key bytes modulo the partition count. Same bytes and same count give the same partition.
- Ordering exists only within a partition, so the key defines which records are ordered relative to each other.
- Key by the smallest entity that needs ordering, with many distinct values and no dominant one.
- Changing the partition count remaps keys and leaves old records where they are. Plan it as a migration.
- Adding partitions or consumers cannot split the load of a single hot key.
- A custom partitioner is a contract every producer must share. Prefer a better key.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
The orders.events topic has 6 partitions and is keyed by order id. An inventory consumer relies on seeing each order's events in sequence. Traffic has tripled, the six consumers are saturated, and someone proposes running kafka-topics.sh --alter --partitions 12 this afternoon while producers keep running.
What should the plan account for?
Scenario 2 of 3
A multi-tenant topic with 24 partitions is keyed by tenantId. One tenant produces 40% of all records. Partition 17 has a lag of 2 million and growing; the other 23 partitions are at zero. The consumers only need events of the same invoice to be in order.
Which change removes the bottleneck without losing the ordering that is needed?
Scenario 3 of 3
A Java service and a Python service (confluent-kafka, built on librdkafka) both publish events for the same customers to customer.events, keyed by customer id as a UTF-8 string. The partition count has never changed. A consumer finds customer c-881 in partitions 2 and 9, and its events are applied out of order.
What is the most direct fix?
References
References: Apache Kafka documentation: Producer Configs (partitioner.class, partitioner.ignore.keys, partitioner.adaptive.partitioning.enable) ↗ · Apache Kafka documentation: Basic Kafka Operations, Modifying topics (effects of adding partitions) ↗ · KIP-794: Strictly Uniform Sticky Partitioner ↗ · KIP-480: Sticky Partitioner ↗ · Partitioner interface Javadoc ↗ · Apache Kafka client source: BuiltInPartitioner (key hash and sticky logic) ↗ · librdkafka configuration reference (partitioner property) ↗ · Spring for Apache Kafka 3.3 reference: Sending Messages ↗ · Report a problem