Step 1 · The pattern
The key picks the partition
Every message has a key. Kafka hashes it to choose a partition, so messages with the same key always share a partition and stay in order.
- message
- consumed
In short
Hot partition key: how it works and how it breaks
Kafka picks a partition for each message by hashing its key. Messages with the same key always land in the same partition, which is how per-key ordering is guaranteed. The side effect is that the spread of traffic across partitions is exactly the spread of traffic across key values. Key by something with a few dominant values, such as country, tenant or event type, and whichever value is biggest sends most of the topic through one partition. That partition's consumer falls behind while the others have nothing to do, and adding consumers or partitions does not move the hot key off its partition. The fix is in the key. Choose the most fine-grained thing you actually need ordering for, usually an order, user or device ID. A big tenant then spreads across every partition, and each individual order still sees its own events in sequence.
How this comes up in interviews
"One partition of a multi-tenant topic has huge lag while the rest are almost empty. How do you diagnose it and what do you change?"
Then they ask
- If you key by order ID instead of tenant ID, which ordering guarantee do you lose, and when does that matter?
- Why does adding partitions or consumers not move the hot key off its partition?
- How would you change the key on a topic that is already carrying production traffic?
A strong answer covers
- The key is hashed to pick the partition, so traffic per partition mirrors traffic per key value.
- A key with a few dominant values, such as tenant or country, sends most of the topic through one partition.
- More consumers or partitions do not help, because one key still maps to one partition.
- Fix it in the key: choose the most fine-grained thing you actually need ordering for.
- The cost is scope: events for different orders of the same tenant are no longer ordered relative to each other.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Apache Kafka documentation: producer partitioning ↗ · Discord: How Discord stores trillions of messages ↗ · Slack: Scaling Slack's job queue ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.