Step 1 · The pattern
A log and a reader
The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.
- message
- consumed
In short
Partitions & consumer groups: how it works and how it breaks
A Kafka topic is split into partitions, and each partition is an ordered, append-only log. Producers choose a partition for every message, usually by hashing a key, so all messages with the same key stay in order. Consumers that share a group ID divide the partitions between them, and the rule that matters is this: within a group, a partition is read by exactly one consumer at a time. That rule is what preserves order. It also sets a ceiling on parallelism. If a topic has one partition, a group of three consumers has one worker and two spectators. To process more in parallel you need more partitions, and then the group spreads across them automatically. Partitions are easy to add and awkward to remove, and adding them changes which partition a key maps to, so it pays to choose the count with future traffic in mind.
How this comes up in interviews
"Your order-events topic is falling behind at peak. You add more consumers and nothing improves. What is going on, and how do you fix it?"
Then they ask
- How would you choose the partition count for a new topic, and what makes changing it later awkward?
- What happens to in-flight work when a consumer joins or leaves the group?
- Two services need every event from the same topic. How do you set that up?
A strong answer covers
- Within a group, a partition is read by exactly one consumer at a time, which is what preserves order.
- The partition count is the ceiling on parallelism. Extra consumers beyond it sit idle as standbys.
- Producers pick a partition by hashing the key, so messages with the same key stay in order.
- Partitions are easy to add and awkward to remove, and adding them changes which partition a key maps to.
- Choose the count with future traffic in mind, then let the group spread across the partitions.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: Apache Kafka documentation: design ↗ · Kafka: a distributed messaging system for log processing (NetDB 2011) ↗ · Jay Kreps: The Log ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.