Skip to content

Path 3 · lesson 2 of 5

Intermediate24 minApache Kafka 4.x (producer configs checked against the 4.0, 4.1 and 4.4 reference; partition logic read in the 4.1 client source) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 (kafka-clients 3.9)

Keys and partitioning: who decides where a record goes

How the producer picks a partition, why the key is what gives you ordering, how to choose a key, and what skew and adding partitions do to both.

What you will learn

  • Predict which partition a record goes to from its partition field, its key and the topic's partition count
  • Explain why records with the same key stay in order and name the things that break that
  • Choose a key for an event from the ordering the consumers actually need
  • Recognise skew caused by a hot or low-cardinality key and pick a fix that keeps the ordering you rely on
  • Plan a partition count change knowing that it moves keys to different partitions
  • Decide when a custom partitioner is justified and when a different key is the better answer

Before you start

The concept

A topic is split into partitions, and each partition is an independent, ordered log. A record is appended to exactly one of them, and the producer makes that choice before the record leaves your process. The broker does not route records; it appends what it is given to the partition named in the request.

The choice follows a fixed order of precedence:

The record hasPartition usedTypical use
An explicit partition numberThat partition, no questions askedRare: tools, tests, replaying to the original partition
A key (and no partition)murmur2(serialised key bytes) made non-negative, modulo the number of partitionsThe normal case for business events
NeitherThe producer's current sticky partition, which changes after about batch.size bytesLogs, metrics, clickstream where order between records does not matter
If a custom partitioner.class is configured, it replaces the last two rows. An explicit partition still wins.

The second row is the one that matters for correctness. The hash is deterministic, so two records with the same key bytes and the same partition count always go to the same partition. Since a partition is ordered, those records are stored and read in the order the broker appended them. The key is how you tell Kafka which records must stay in order relative to each other.

Keys to partitions

Simulation · not a real cluster

Send records for the four orders and see that each order always lands in the same partition. Then send records without a key.

Why it exists

Partitions exist so a topic can be written and read in parallel. Parallelism and ordering pull in opposite directions: one log gives a total order and no parallelism, while spreading records at random gives parallelism and no order at all.

Most systems do not need a total order. They need the events of one order, one account or one device to be in sequence, and they do not care how order 1001 interleaves with order 1002. Hashing a key gives exactly that: each entity gets a private ordered lane, and different entities are processed in parallel by different consumers in a group.

Records without a key have no ordering requirement, so the producer is free to optimise for something else. Early clients rotated through partitions record by record, which produced many tiny batches. Since Kafka 2.4 the producer sticks to one partition for a while so batches fill up, and since 3.3 it switches after a fixed number of bytes rather than on each new batch, which keeps the spread even when one broker is slower than the others.

Keyed records

  • Partition is a pure function of key bytes and partition count
  • Same key, same partition, order preserved
  • Distribution is only as even as the keys are
  • Changing the partition count changes the mapping

Records without a key

  • Producer picks a partition and stays on it for about one batch
  • No ordering relationship between records
  • Bytes spread evenly over time, adapting to slow brokers
  • Changing the partition count has no semantic effect

How it works inside

The key hash

The Java client computes the partition from the serialised key: the 32-bit murmur2 hash of the key bytes, with the sign bit cleared, modulo the topic's partition count. Three consequences follow directly.

  • The serialiser is part of the mapping. The string "42" and the long 42 serialise to different bytes and usually land in different partitions. Every producer writing to a topic must agree on the key type and its serialisation.
  • The modulus is the total number of partitions, not the number that are currently reachable. If the leader of the target partition is unavailable, a keyed record waits in the buffer and is retried; it is never diverted to another partition.
  • The client library is part of the mapping too. Clients built on librdkafka (Python, Go, .NET and others) default to a CRC32-based partitioner; they must be set to murmur2_random to agree with Java producers.

No key: sticky, then uniform

For a record with no key and no partition, the built-in logic keeps a current partition per topic and sends everything there until roughly batch.size bytes have been produced to it, then picks another. With partitioner.adaptive.partitioning.enable=true (the default) the next partition is chosen with a probability weighted against brokers whose queues in the accumulator are long, so a slow broker receives less. partitioner.availability.timeout.ms (default 0, off) can additionally exclude partitions whose broker has not accepted a batch for that long. These settings affect only records without a key.

What breaks per-key ordering

  • Changing the partition count. The modulus changes, so most keys map to a different partition from then on. Kafka does not move existing records. For a while a key has old events in one partition and new events in another, and a consumer group may read the new ones first.
  • Producers that disagree on key serialisation or hash function. The same entity is then in two partitions at once.
  • Retries without idempotence while several requests are in flight, which can swap two batches inside one partition. See Acks, retries and idempotence.
  • Several producers writing events for the same key. Each producer's records are ordered among themselves, but the interleaving between producers is decided by arrival at the broker, not by your business clock.
  • Reprocessing on the consumer side, for example a retry topic or a rewind, which replays older records after newer ones have been handled. The log is still ordered; your processing is not.

Choosing a key

Start from the consumer: which events must it see in sequence? Use the identifier of that entity. For order lifecycle events that is the order id; for account balance changes, the account id; for device telemetry that feeds a per-device state machine, the device id. Then check the key against three questions.

  1. Is it specific enough? A key is the smallest unit that needs ordering. Using customerId when only per-order ordering is needed forces all of a customer's orders through one partition for no benefit.
  2. Does it have enough distinct values with no dominant one? A key such as country code or tenant id concentrates traffic in a few partitions. One tenant with 40% of the traffic means one partition with at least 40% of the traffic, whatever the partition count.
  3. Is it stable? The key must not change during the entity's life. An email address or a status field is a poor key.

Hot partition key

Simulation, not a real cluster

The animated Build lesson shows one hot key overloading a single partition while the others stay idle.

Step 1 · The pattern

The key picks the partition

Every message has a key. Kafka hashes it to choose a partition, so messages with the same key always share a partition and stay in order.

  • message
  • consumed
Open the full animated lesson

Custom partitioners, and when not to write one

A class implementing org.apache.kafka.clients.producer.Partitioner and named in partitioner.class receives the topic, the key and value (as objects and as bytes) and the cluster metadata, and returns a partition number. It replaces the built-in logic for every record that has no explicit partition, so the sticky and adaptive behaviour for unkeyed records, and the partitioner.* settings, no longer apply.

Write one only when the routing rule cannot be expressed as a key: for example, isolating one very large tenant on dedicated partitions so it cannot delay everyone else. In most other cases a better key does the job with no code to maintain. A custom partitioner is a hidden contract: every producer to the topic, in every language, must implement the same rule, and nothing checks that they do.

Architecture

Drawing diagram…

Decision flowchart. If the record has an explicit partition it is used. Otherwise, if a custom partitioner is configured it decides. Otherwise, if a key is present and keys are not ignored, the murmur2 hash of the serialised key modulo the partition count gives the partition. Otherwise the record goes to the current sticky partition, which is replaced after about batch.size bytes by a new partition chosen with a preference for brokers with short queues.

flowchart TD
  R["Record passed to send"] --> Q1{"Partition set on the record"}
  Q1 -->|yes| P1["Use that partition"]
  Q1 -->|no| Q2{"partitioner.class configured"}
  Q2 -->|yes| P2["Custom partitioner returns the partition"]
  Q2 -->|no| Q3{"Key present and keys not ignored"}
  Q3 -->|yes| H["murmur2 of serialised key bytes"]
  H --> M["Modulo partition count"]
  M --> P3["Same key gives same partition"]
  Q3 -->|no| S["Current sticky partition for the topic"]
  S --> SW{"About batch.size bytes produced to it"}
  SW -->|no| S2["Stay on it"]
  SW -->|yes| A["Pick next partition, favouring brokers with short queues"]
The producer's partition decision, in the order the Java client evaluates it.
  • An explicit partition bypasses everything, including a custom partitioner. That makes it a sharp tool: the caller takes over responsibility for distribution and ordering.
  • The keyed branch has no state and no randomness. Given the key bytes and the partition count, any Java producer anywhere computes the same answer.
  • The unkeyed branch has state per producer instance. Two producers will be on different sticky partitions at the same moment, which is fine because these records carry no ordering requirement.
  • partitioner.ignore.keys=true sends keyed records down the unkeyed branch. It is for topics where the key is needed for compaction or for consumers, but not for placement.

Message flow

Drawing diagram…

Sequence diagram. With six partitions, two events for order-1001 are hashed to partition 1 and consumed in order. After the topic is altered to eight partitions and the producer refreshes metadata, the next event for order-1001 hashes to partition 4. A consumer of partition 4 receives OrderShipped with no earlier events for that order in the same partition.

sequenceDiagram
  participant OS as Order service
  participant P as Producer
  participant P1 as Partition 1 of orders
  participant P4 as Partition 4 of orders
  participant C as Payment consumer
  Note over P: topic has 6 partitions
  OS->>P: OrderPlaced key order-1001
  P->>P1: hash of key mod 6 gives 1, append at offset 40
  OS->>P: OrderPaid key order-1001
  P->>P1: same key, append at offset 41
  C->>P1: fetch
  P1-->>C: OrderPlaced then OrderPaid
  Note over P1,P4: topic is altered to 8 partitions
  P->>P: metadata refresh sees 8 partitions
  OS->>P: OrderShipped key order-1001
  P->>P4: hash of key mod 8 gives 4, append at offset 0
  C->>P4: fetch
  P4-->>C: OrderShipped with nothing before it
The same key before and after a partition count change. Partition numbers are illustrative.
  1. Before the change, both events share a partition, so any consumer of that partition sees OrderPlaced before OrderPaid.
  2. The alter command does not touch stored records. Offsets 40 and 41 stay in partition 1.
  3. The producer starts using the new count when it refreshes metadata, which can take up to metadata.max.age.ms. Several producers switch at different moments, so for a few minutes the same key can be written to both the old and the new partition.
  4. After the change, the order's history is split across two partitions. If the consumer of partition 1 is behind, OrderShipped can be processed before OrderPaid.

In a real system

A typical payments platform publishes every state change of a payment (PaymentAuthorised, PaymentCaptured, PaymentRefunded) to one topic, payments.events.v1, with 12 partitions. A ledger service consumes it and applies each change to the payment's row.

Candidate keyWhat you getVerdict
No keyEven spread, but Captured can be read before AuthorisedWrong: the ledger needs per-payment order
merchantIdPer-merchant order, but the largest merchant fills one partition and its consumer falls behindStronger ordering than needed, at the cost of skew
paymentIdPer-payment order; millions of distinct values spread evenlyRight: the smallest unit that needs ordering
paymentId plus event typeEach event type of a payment may land in a different partitionWrong: destroys the ordering the key was for

With paymentId as the key, 12 ledger consumers can each own one partition, and each payment is always handled by one consumer at a time. A merchant-level report that needs all of a merchant's payments together is a different consumer with a different need; it reads the same topic and groups by merchant in its own store rather than forcing the topic's key to serve it.

The team sized the topic at 12 partitions on purpose, above current need, because growing it later would reshuffle keys. When a topic with keyed data really must grow, the usual approach is a new topic with the new count: producers switch to it, and consumers finish the old topic before starting the new one.

Java and Spring Boot

The key is chosen where the record is built. The first file shows the normal case, an entity id as the key. The other two show a custom partitioner for the one situation that tends to justify it, with the configuration that activates it.

src/main/java/com/example/payments/events/PaymentEventPublisher.javaJava
package com.example.payments.events;import java.nio.charset.StandardCharsets;import java.util.concurrent.CompletableFuture;import org.apache.kafka.clients.producer.ProducerRecord;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.support.SendResult;import org.springframework.stereotype.Component;@Componentpublic class PaymentEventPublisher {    public static final String TOPIC = "payments.events.v1";    private static final Logger log = LoggerFactory.getLogger(PaymentEventPublisher.class);    private final KafkaTemplate<String, PaymentEvent> kafkaTemplate;    public PaymentEventPublisher(KafkaTemplate<String, PaymentEvent> kafkaTemplate) {        this.kafkaTemplate = kafkaTemplate;    }    public CompletableFuture<SendResult<String, PaymentEvent>> publish(PaymentEvent event) {        // Key = the entity whose events must stay in order. Never the event id, never the event type.        String key = event.paymentId();        ProducerRecord<String, PaymentEvent> record = new ProducerRecord<>(TOPIC, key, event);        record.headers().add("event-type", event.type().getBytes(StandardCharsets.UTF_8));        return kafkaTemplate.send(record).whenComplete((result, ex) -> {            if (ex != null) {                log.error("{} not published paymentId={}", event.type(), key, ex);            } else {                log.debug("{} paymentId={} -> partition {} offset {}", event.type(), key,                        result.getRecordMetadata().partition(), result.getRecordMetadata().offset());            }        });    }    /** All state changes of one payment share this shape and this topic. */    public record PaymentEvent(String eventId, String paymentId, String merchantId, String type, long amountMinor) {    }}

Why it is written this way

  • All event types of a payment go to one topic with one key. Putting each type in its own topic would lose the order between them, because ordering does not span topics.
  • The key is a String serialised with StringSerializer. Any other service producing to this topic must also send the payment id as a UTF-8 string, or the same payment ends up in two partitions.
  • No partition is passed. Hard-coding a partition number in application code ties the code to the current partition count.
  • Logging the partition at debug level is useful when investigating ordering: the same paymentId should always show the same partition.

Configuration

Settings that take part in partition choice. Defaults are from the Apache Kafka 4.x producer and broker configuration reference. The three partitioner.* tuning settings affect only records without a key and have no effect when a custom partitioner is set.

partitioner.classDefault null
What it does
Class that chooses the partition. Unset means the built-in logic: hash of the key when there is one, sticky partition otherwise.
When to change it
Leave unset unless a routing rule cannot be expressed as a key. RoundRobinPartitioner ships with the client but ignores keys, so it gives up per-key ordering.
partitioner.ignore.keysDefault false
What it does
When true, the key is not used for placement and keyed records are spread like unkeyed ones.
When to change it
Use only when the key exists for another reason, such as compaction or consumer logic, and you do not need per-key ordering.
partitioner.adaptive.partitioning.enableDefault true
What it does
For unkeyed records, send more to partitions on brokers that are keeping up and less to slow ones.
When to change it
Turn off only if you need unkeyed records spread strictly evenly regardless of broker speed.
partitioner.availability.timeout.msDefault 0
What it does
For unkeyed records, treat a partition as unavailable if its broker has not accepted a ready batch for this long. 0 disables the check.
When to change it
Consider a non-zero value if unkeyed traffic should route around a stalled broker quickly. It does nothing for keyed records.
batch.sizeDefault 16384
What it does
Besides bounding a batch, it is the number of bytes the producer sends to one sticky partition before switching.
When to change it
At low volume a small topic can look uneven over short windows because the producer stays on one partition for a whole batch. That is expected.
metadata.max.age.msDefault 300000 (5 minutes)
What it does
How often the producer refreshes metadata without being prompted by an error, which is how it learns that a topic has more partitions.
When to change it
Explains the window after a partition increase in which producers disagree about the count. Do not lower it to fix that; plan the change instead.
num.partitions (broker)Default 1
What it does
Partition count for topics created automatically, without an explicit count.
When to change it
Create topics explicitly with a count chosen for the next few years of keyed traffic. The count can be raised later but never lowered.

Hands-on exercise

Move a key by adding a partition

See that a key always maps to the same partition, that unkeyed records do not, and that one extra partition sends existing keys somewhere new.

Steps · 0 of 6 done

Partitioner: keys, no keys, and a resize

Simulation · not a real cluster

Send keyed and unkeyed records, then add a partition and send again. This is a deterministic simulation of the producer's partition choice.

Take it further in the lab: Partition key and ordering

What can go wrong

Adding partitions to a keyed topic
Symptom
Shortly after a partition increase, consumers report impossible sequences such as a shipment for an order that is not yet paid. Nothing is lost.
Cause
The hash is taken modulo the partition count. The new count moved most keys, existing records stayed in the old partitions, and the consumers of the old and new partitions progress independently.
Fix
Size keyed topics generously up front. If a topic must grow, migrate to a new topic with the new count, or pause producers, let consumers drain to zero lag, then alter and resume. Consumers that check a version or sequence number on the entity tolerate the remaining window.
A low-cardinality or dominant key
Symptom
One partition's lag grows while the others are idle. Adding consumers or partitions does not help.
Cause
All records with one key go to one partition by design. A key with few values, or one value carrying a large share of the traffic, concentrates the load, and one partition is read by one consumer in a group.
Fix
Key by the smallest entity that needs ordering (order id rather than tenant id). If one entity really is that large, isolate it on its own partitions or topic. See Hot partitions and backpressure.
Producers that hash differently
Symptom
Events for the same entity are found in two partitions even though the partition count never changed.
Cause
Two services produce to the topic with different key serialisation (string versus number, different string formats) or with a client whose default partitioner is not murmur2, such as librdkafka's default.
Fix
Write the key type and format into the topic's contract. Set partitioner=murmur2_random in librdkafka-based clients that share a topic with Java producers.
Using the event id or a timestamp as the key
Symptom
Distribution is perfectly even, yet consumers see an entity's events out of order.
Cause
The key is unique per record, so related records hash to unrelated partitions. An even spread was achieved by giving up ordering.
Fix
Key by the entity id and carry the event id in a header or in the payload for deduplication.
A custom partitioner that forgets keys
Symptom
After a partitioner was added to balance load, per-key ordering problems appear across the topic.
Cause
The partitioner returns a rotating or load-based partition for every record, including keyed ones, and it also replaced the built-in handling of unkeyed records.
Fix
Remove it and fix the key, or make the partitioner deterministic per key. Test it with fixed keys and a fixed partition count.

How to troubleshoot

You need to confirm how many partitions a topic has and whether that changed

Check

kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic payments.events.v1
How to read it
The first line shows PartitionCount. If it differs from the count the topic was designed with, keys were remapped at the time of the change.
Suspected skew: one consumer is always behind

Check

kafka-get-offsets.sh --bootstrap-server localhost:9092 --topic payments.events.v1
How to read it
Prints topic:partition:end offset per partition. Run it twice a minute apart and compare the growth per partition. One partition growing several times faster than the rest points at a hot key, not at a slow consumer.
Lag is concentrated on one partition of a consumer group

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group ledger-service
How to read it
Compare the LAG column across partitions. Even lag means the group is short of capacity overall. Lag on one partition with the others near zero means that partition receives more than one consumer can handle.
You want to see which keys land in which partition

Check

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments.events.v1 --from-beginning --max-messages 200 --formatter-property print.key=true --formatter-property print.partition=true
How to read it
Each line shows the partition and key. The same key appearing with two partition numbers means a partition count change or producers that hash differently. Tools before Kafka 4.2 spell the option --property.
Unkeyed records look unevenly spread in a short test

Check

Producer metric `batch-size-avg` and the number of records sent in the test
How to read it
The producer stays on one partition for about batch.size bytes. A test that sends less than a few batches per partition will look lopsided. Over a longer run the byte counts even out.

Key takeaways

  • The producer chooses the partition: explicit partition first, then a custom partitioner if set, then the key hash, then the sticky partition for unkeyed records.
  • For keyed records the partition is murmur2 of the serialised key bytes modulo the partition count. Same bytes and same count give the same partition.
  • Ordering exists only within a partition, so the key defines which records are ordered relative to each other.
  • Key by the smallest entity that needs ordering, with many distinct values and no dominant one.
  • Changing the partition count remaps keys and leaves old records where they are. Plan it as a migration.
  • Adding partitions or consumers cannot split the load of a single hot key.
  • A custom partitioner is a contract every producer must share. Prefer a better key.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

The orders.events topic has 6 partitions and is keyed by order id. An inventory consumer relies on seeing each order's events in sequence. Traffic has tripled, the six consumers are saturated, and someone proposes running kafka-topics.sh --alter --partitions 12 this afternoon while producers keep running.

What should the plan account for?

Scenario 2 of 3

A multi-tenant topic with 24 partitions is keyed by tenantId. One tenant produces 40% of all records. Partition 17 has a lag of 2 million and growing; the other 23 partitions are at zero. The consumers only need events of the same invoice to be in order.

Which change removes the bottleneck without losing the ordering that is needed?

Scenario 3 of 3

A Java service and a Python service (confluent-kafka, built on librdkafka) both publish events for the same customers to customer.events, keyed by customer id as a UTF-8 string. The partition count has never changed. A consumer finds customer c-881 in partitions 2 and 9, and its events are applied out of order.

What is the most direct fix?

References

References: Apache Kafka documentation: Producer Configs (partitioner.class, partitioner.ignore.keys, partitioner.adaptive.partitioning.enable) ↗ · Apache Kafka documentation: Basic Kafka Operations, Modifying topics (effects of adding partitions) ↗ · KIP-794: Strictly Uniform Sticky Partitioner ↗ · KIP-480: Sticky Partitioner ↗ · Partitioner interface Javadoc ↗ · Apache Kafka client source: BuiltInPartitioner (key hash and sticky logic) ↗ · librdkafka configuration reference (partitioner property) ↗ · Spring for Apache Kafka 3.3 reference: Sending Messages ↗ · Report a problem