Path 2 · lesson 2 of 8
Partitions and offsets
A partition is an ordered, append-only log and an offset is a position in it. Learn the four offsets that matter and why the partition is the unit of ordering, parallelism and placement.
What you will learn
- Describe a partition as a log and say what an offset does and does not identify
- Tell apart the log start offset, high watermark, log end offset and a group's committed offset, and say who moves each
- Work out from those numbers what a consumer can read right now and how far behind it is
- Explain why ordering, consumer parallelism and placement on brokers are all decided per partition
- Name the things that break per-key ordering, including adding partitions to a keyed topic
Before you start
The concept
A partition is a log: a sequence of records that only ever grows at the end. A record is appended, given the next number in the sequence, and never moved or modified afterwards. That number is the record's offset. Offsets start at 0 and belong to one partition, so the full address of a record is topic, partition and offset: orders-3 at offset 181.
The partition leader assigns offsets as it appends, so the offset order is the order in which the broker wrote the records. Every replica of the partition holds the same records at the same offsets. Reading does not remove anything: a consumer asks for records starting at an offset, gets a batch, and asks again from the next offset. Old records leave only when retention deletes whole segments from the front of the log, or compaction removes superseded values.
Because a reader's position in a partition is a single number, the state Kafka needs to keep per consumer group is tiny: one offset per partition. That is the design decision that lets many independent groups read the same log cheaply, and lets any of them rewind and read it again.
One partition as a log
Simulation · not a real clusterAppend records and watch each one take the next offset. Notice that the log is stored as segments and that old segments leave from the front, which moves the log start offset.
Why it exists
One log per topic would be the simplest model, and it would give every record in the topic a single global order. It would also mean one leader broker taking every write, one disk holding everything, and one consumer at a time in each group reading it, because two readers sharing a log cannot both keep a single position. Partitions are how Kafka gives up global order in exchange for scale, while keeping order where it is needed.
| The partition is the unit of | What that means | Consequence for you |
|---|---|---|
| Ordering | Records in one partition are read in the order the leader appended them. Kafka keeps no order between partitions. | Records that must stay in order need the same key, so that they go to the same partition. |
| Parallelism | Within a consumer group, each partition is read by exactly one consumer at a time. | A group cannot do more work in parallel than it has partitions. Extra consumers sit idle. |
| Placement | A partition replica lives entirely on one broker. Different partitions of a topic live on different brokers. | A topic can be bigger than one machine, but a single partition cannot. Leaders are spread per partition. |
| Replication and failure | Each partition has its own leader, followers and in-sync set. | A broker failure affects the partitions it led, not the whole topic. |
How it works inside
How a record gets its partition
The producer chooses the partition before it sends. With a key, the built-in partitioner hashes the key bytes and takes the result modulo the partition count, so equal keys go to the same partition for as long as the count does not change. Without a key, the producer sticks to one partition until it has filled a batch for it, then moves on, which spreads records evenly without any ordering promise. The details are in keys and partitioning.
Keys to partitions
Simulation · not a real clusterSend records for a few order ids and see each id stay on one partition. Then add a partition and watch where the same ids go next.
The four offsets
Four positions describe the state of a partition and of one group reading it. They are easy to confuse because tools often label two of them the same way.
| Offset | Kept by | Moves when | What it tells you |
|---|---|---|---|
| Log start offset | Each replica | Retention deletes the oldest segment, or records are deleted on request | The earliest offset that can still be read. Anything below it is gone. |
| High watermark | The leader, passed on to followers | Every in-sync replica has copied the records below it | The end of the log as consumers see it. Consumers are only given records below the high watermark. |
| Log end offset | Each replica, for its own copy | A record is appended | The offset the next appended record will get. On the leader, records between the high watermark and here are written but not yet fully replicated. |
| Committed offset | The group coordinator, per group and partition, in __consumer_offsets | A consumer in the group commits | The offset of the next record this group will read after a restart or rebalance. Not the last record processed, and not an acknowledgement of any single record. |
Two more positions appear in client code. A running consumer has an in-memory position per partition, the next offset it will fetch, which runs ahead of the committed offset between commits. And a consumer with isolation.level=read_committed reads only up to the last stable offset, which is held back below the high watermark while a transaction is still open.
Lag for a group on a partition is the end of the log minus the committed offset. kafka-consumer-groups.sh prints the committed offset as CURRENT-OFFSET and the end as LOG-END-OFFSET; the end it reports is the one a client can ask for, which is the high watermark. Records above the high watermark are not counted as lag because no consumer could have read them yet.
When a position falls off the log
If a group's committed offset is lower than the log start offset, because the group was stopped for longer than the topic's retention, the offset it asks for no longer exists. The broker answers with an out-of-range error and the consumer applies auto.offset.reset: earliest jumps to the log start offset, latest jumps to the end and skips everything in between, none throws. The same setting applies when a group has no committed offset at all, which also happens when its offsets expire: the coordinator drops a group's offsets offsets.retention.minutes (7 days) after the group has become empty.
Changing the partition count
Partitions can be added to a topic but never removed. Adding them does not move existing records. It changes the divisor in the key hash, so from that moment many keys map to a different partition than the one holding their history. The Kafka documentation also warns that a consumer with auto.offset.reset=latest can miss the first records written to a new partition, because producers may start writing before the group has discovered it. Choosing the partition count is covered in partition sizing.
Partitions & consumer groups
Simulation, not a real clusterAnimated walk-through of partitions being shared out among the consumers of a group.
Step 1 · The pattern
A log and a reader
The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.
- message
- consumed
Architecture
Drawing diagram…
A partition log drawn left to right. Offsets below 100 were deleted, 100 is the log start offset, 100 to 181 can be read by consumers, 182 is the high watermark, 182 to 189 are appended but not yet fully replicated, and 190 is the log end offset. A group's committed offset points at 150.
flowchart LR
subgraph LOG["Partition orders-3 on its leader"]
direction LR
D["Offsets 0 to 99, deleted by retention"] --> S["Offset 100, log start offset"]
S --> R["Offsets 100 to 181, readable by consumers"]
R --> H["Offset 182, high watermark"]
H --> U["Offsets 182 to 189, not yet on every in-sync replica"]
U --> E["Offset 190, log end offset"]
end
PR["Producer"] -->|"next append gets 190"| E
CG["Group payment-service, committed offset 150"] -. "resumes here" .-> R
CF["Any consumer"] -->|"fetches below 182"| R
FO["Follower replicas"] -->|"fetch from their own log end"| U- The log only grows at the right-hand end and only shrinks at the left-hand end. Nothing in the middle changes.
- The producer moves the log end offset. Followers copying the log are what let the leader move the high watermark up behind it.
- Consumers of every group share the same readable range, 100 to 181. Each group has its own committed offset inside that range.
- The group
payment-servicewould resume at 150 and has 32 records it can read now: its lag is 182 minus 150. - If retention moved the log start offset past 150 while the group was stopped, the committed offset would point at nothing and
auto.offset.resetwould decide where the group continues.
Message flow
Drawing diagram…
A producer appends three records at offsets 190 to 192. A follower fetches them, and its next fetch from 193 lets the leader raise the high watermark. The consumer then fetches the three records, processes them and commits offset 193 to the group coordinator.
sequenceDiagram participant P as Producer participant L as Leader of orders-3 participant F as Follower participant C as Consumer in payment-service participant G as Group coordinator P->>L: Produce 3 records with key order-42 Note over L: Appended at offsets 190 191 192, log end offset is 193 F->>L: Fetch from offset 190 L-->>F: Records 190 to 192 F->>L: Fetch from offset 193 Note over L: Every in-sync replica has 190 to 192, high watermark moves to 193 L-->>P: Acknowledged, base offset 190 C->>L: Fetch from offset 190 L-->>C: Records 190 to 192 Note over C: Process the three records C->>G: Commit offset 193 for orders-3 G-->>C: Commit stored in __consumer_offsets
- The leader assigns the offsets. The producer learns them from the acknowledgement and does not choose them.
- The follower's second fetch request, starting at 193, is how the leader learns that the follower has everything up to 192.
- The consumer could not have been given offset 190 before the high watermark passed it, however early it asked.
- The consumer commits 193, the offset of the next record it wants, not 192, the last one it handled. A group that commits the last processed offset reads that record again after every restart.
- The commit goes to the group coordinator, which is a broker chosen by the group id, and not necessarily the leader of
orders-3.
In a real system
A typical order platform has an orders topic with 12 partitions, keyed by order id. The order service writes OrderCreated, OrderPaid and OrderShipped events. All events for order 42 hash to the same partition, so every consumer sees them in the order they were written. Events for order 42 and order 43 are probably in different partitions and have no defined order relative to each other, which is fine: nothing in the business depends on it.
The payment service runs four instances in the group payment-service, so each instance reads three partitions. The notification service is a separate group with its own committed offsets on the same twelve logs; it can be stopped for an hour and catch up without the payment service noticing. An on-call engineer looking at one partition sees something like this:
| Group | Partition | Committed offset | End offset | Lag |
|---|---|---|---|---|
payment-service | orders-3 | 8,412,950 | 8,412,982 | 32 |
notification-service | orders-3 | 8,390,100 | 8,412,982 | 22,882 |
fraud-replay (started today) | orders-3 | none yet | 8,412,982 | depends on auto.offset.reset |
The choice of key is a statement about which records need a relative order. Keying by customer id instead of order id would also keep one customer's orders in sequence, at the price of putting a very active customer's whole traffic on one partition. The hot partition lesson shows what that looks like.
Java and Spring Boot
A listener can see the partition and offset of every record it receives, and the Admin API can read the log start offset, the end offset and a group's committed offsets. Together they are enough to reproduce what kafka-consumer-groups.sh prints, which is a useful thing to expose on an internal diagnostics endpoint.
package com.example.payments.kafka;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderEventListener { private static final Logger log = LoggerFactory.getLogger(OrderEventListener.class); @KafkaListener(topics = "orders", groupId = "payment-service") public void onOrderEvent(ConsumerRecord<String, String> record) { log.info("orders-{} offset={} key={} timestamp={}", record.partition(), record.offset(), record.key(), record.timestamp()); // Business handling goes here. Records with the same key arrive in offset order, // because they are in the same partition and one thread reads that partition. }}Why it is written this way
- Taking a
ConsumerRecordinstead of only the payload gives access to the partition and offset. Logging both on every failure makes a record findable later. - Partition plus offset identifies this record in this topic on this cluster. It is a good value for log correlation and a poor business identifier, because a re-published or mirrored event gets a different one.
- The container commits offsets after the listener returns (the default
BATCHack mode commits after the records from a poll have been processed), so the committed offset is always the next record to read.
package com.example.payments.kafka;import java.util.Comparator;import java.util.List;import java.util.Map;import java.util.stream.Collectors;import org.apache.kafka.clients.admin.Admin;import org.apache.kafka.clients.admin.ListOffsetsResult.ListOffsetsResultInfo;import org.apache.kafka.clients.admin.OffsetSpec;import org.apache.kafka.clients.consumer.OffsetAndMetadata;import org.apache.kafka.common.TopicPartition;import org.springframework.kafka.core.KafkaAdmin;import org.springframework.stereotype.Component;@Componentpublic class PartitionOffsetReport { public record PartitionOffsets(int partition, long logStartOffset, long endOffset, Long committedOffset, Long lag) { } private final KafkaAdmin kafkaAdmin; public PartitionOffsetReport(KafkaAdmin kafkaAdmin) { this.kafkaAdmin = kafkaAdmin; } public List<PartitionOffsets> report(String topic, String groupId) throws Exception { try (Admin admin = Admin.create(kafkaAdmin.getConfigurationProperties())) { List<TopicPartition> partitions = admin.describeTopics(List.of(topic)) .allTopicNames().get().get(topic).partitions().stream() .map(info -> new TopicPartition(topic, info.partition())) .toList(); Map<TopicPartition, OffsetSpec> earliest = partitions.stream() .collect(Collectors.toMap(tp -> tp, tp -> OffsetSpec.earliest())); Map<TopicPartition, OffsetSpec> latest = partitions.stream() .collect(Collectors.toMap(tp -> tp, tp -> OffsetSpec.latest())); Map<TopicPartition, ListOffsetsResultInfo> start = admin.listOffsets(earliest).all().get(); Map<TopicPartition, ListOffsetsResultInfo> end = admin.listOffsets(latest).all().get(); Map<TopicPartition, OffsetAndMetadata> committed = admin.listConsumerGroupOffsets(groupId).partitionsToOffsetAndMetadata().get(); return partitions.stream() .sorted(Comparator.comparingInt(TopicPartition::partition)) .map(tp -> { long endOffset = end.get(tp).offset(); OffsetAndMetadata position = committed.get(tp); Long committedOffset = position == null ? null : position.offset(); Long lag = committedOffset == null ? null : Math.max(0, endOffset - committedOffset); return new PartitionOffsets(tp.partition(), start.get(tp).offset(), endOffset, committedOffset, lag); }) .toList(); } }}Why it is written this way
OffsetSpec.earliest()returns the log start offset andOffsetSpec.latest()returns the end of the log as a consumer sees it, which is the high watermark for the default isolation level.- A group that has never committed for a partition has no entry, so the committed offset and the lag are nullable instead of being reported as zero.
- The committed offset is compared with the end offset directly: both mean the next offset, so the difference is the number of records still to read.
- If the committed offset is lower than the log start offset, the group has lost records to retention. That is worth an explicit alert; the lag number alone does not show it.
- This creates and closes an admin client per call, which is fine for an occasional diagnostic. A scheduled reporter should keep one admin client open.
Configuration
Settings that decide how many partitions exist, how long positions and records stay valid, and where a consumer starts when its position is missing. Defaults are from the Kafka 4.4 configuration reference.
num.partitionsDefault 1
- What it does
- Broker setting. Partition count for topics created without an explicit count.
- When to change it
- Always pass a partition count when creating a topic. The count can be raised later but never lowered, and raising it moves keys.
auto.offset.resetDefault latest
- What it does
- Consumer setting. Where to start when the group has no committed offset for a partition, or the committed offset is no longer in the log. Values are
earliest,latest,by_duration:<duration>(newer clients) andnone. - When to change it
- Use
earliestwhen skipping records is worse than reprocessing. Usenonewhen either would be a bug you want to hear about. Withlatest, a new group, an expired group and a newly added partition all start at the end silently.
offsets.retention.minutesDefault 10080 (7 days)
- What it does
- Broker setting. How long committed offsets are kept after a group has lost all its members, or since the last commit for a partition the group no longer subscribes to.
- When to change it
- Raise it if a consumer may legitimately be switched off for longer than a week, such as a batch job that runs monthly.
retention.msDefault 604800000 (7 days)
- What it does
- Topic setting. How long records are kept before their segment may be deleted, which is what moves the log start offset.
- When to change it
- Set it longer than the longest outage a consumer has to survive without losing records, and keep the disk to match.
isolation.levelDefault read_uncommitted
- What it does
- Consumer setting.
read_committedhides records from aborted transactions and limits reads to the last stable offset. - When to change it
- Set
read_committedwhen the topic is written by transactional producers. Expect reads to pause behind an open transaction.
offsets.topic.num.partitionsDefault 50
- What it does
- Broker setting. Partition count of
__consumer_offsets. A group's offsets live in the partition selected by hashing the group id, and that partition's leader is the group's coordinator. - When to change it
- Leave it alone. The documentation says it should not change after deployment, because the mapping from group to coordinator depends on it.
Hands-on exercise
Read the offsets of a partition
Use the cluster explorer to connect the picture of a partition to the numbers: which broker leads it, where its log ends, how far consumers can read, and what changes when its leader stops.
Steps · 0 of 7 done
Six partitions on three brokers
Simulation · not a real clusterClick a partition to see its leader, replicas, log end offset and high watermark. Stop a broker to see what happens to the partitions it led.
What you should see
Each partition has its own leader and its own offsets, unrelated to the other partitions. A follower's log end offset is at or slightly behind the leader's, and the high watermark is never above the lowest log end offset among the in-sync replicas. When a leader stops, a follower takes over with the same records at the same offsets up to the high watermark, so consumers continue from where they were. Only the partitions that broker led change leader.
What can go wrong
Expecting order across a topic
- Symptom
- A consumer sometimes handles
OrderShippedbeforeOrderCreatedfor the same order, or two instances process related events at the same time. - Cause
- The related events were written with different keys, or with no key, and landed in different partitions. Kafka keeps no order between partitions.
- Fix
- Give events that must stay in order the same key. If the order must hold across entities, either use one key for the whole group or make the consumer tolerate reordering.
Adding partitions to a keyed topic
- Symptom
- After the partition count was raised, some entities show events out of order, and stateful consumers find part of an entity's history arriving on a different instance.
- Cause
- The key hash is taken modulo the partition count, so existing keys now map to different partitions while their older records stay where they were.
- Fix
- Size keyed topics generously at creation. If the count must change, do it when consumers have no backlog, or create a new topic with the new count and migrate producers and consumers to it.
More consumers than partitions
- Symptom
- Lag keeps growing after the group was scaled from 6 to 10 instances. Four instances show no assigned partitions.
- Cause
- Each partition is read by one consumer in a group, so a 6-partition topic supports at most 6 active consumers in that group.
- Fix
- Make each consumer faster (batching, fewer round trips, a larger database pool), or increase the partition count with the ordering consequences in mind.
A committed offset that is no longer in the log
- Symptom
- A consumer that was stopped for a long time restarts and either skips straight to new records, leaving a silent gap, or reprocesses days of history.
- Cause
- Retention deleted the records at the committed offset, or the group's offsets expired. The consumer fell back to
auto.offset.reset. - Fix
- Choose
auto.offset.resetdeliberately for each consumer, keep retention longer than the longest planned downtime, and alert when a committed offset drops below the log start offset.
Treating offsets as portable
- Symptom
- After switching to a mirrored cluster, or copying a topic, consumers restart from the wrong place when their old offsets are applied.
- Cause
- An offset is a position in one partition of one cluster. A copy of the topic has its own offsets for the same records.
- Fix
- Translate positions by timestamp or with the offset translation the mirroring tool provides, and never store an offset without the cluster, topic and partition it belongs to.
How to troubleshoot
Need to know how far behind a group is on each partition
Check
kafka-consumer-groups.sh --bootstrap-server kafka-a1.internal:9092 --describe --group payment-service
- How to read it
- One row per partition.
CURRENT-OFFSETis the committed offset (the next record the group will read),LOG-END-OFFSETis the end of the log, andLAGis the difference.CONSUMER-IDshows which member owns the partition; a dash means nobody does.
Need the first and last offsets of each partition
Check
kafka-get-offsets.sh --bootstrap-server kafka-a1.internal:9092 --topic orders --time earliest (then again with --time latest)
- How to read it
- Prints
topic:partition:offset.earliestis the log start offset andlatestis the end. The difference is roughly how many records the partition holds now. Older versions of the tool take -2 and -1 instead of the names.
One partition lags while the others are fine
Check
Compare `latest` minus `earliest` across partitions with kafka-get-offsets.sh, and the per-partition LAG column of the group
- How to read it
- A partition that grows much faster than its siblings is receiving a hot key. Even growth with one lagging partition points at the consumer that owns it: look for a slow or stuck instance.
A consumer jumped forwards or backwards after a restart
Check
The consumer's log at startup for an offset reset message naming the partition, and the value of `auto.offset.reset`
- How to read it
- A reset means there was no usable committed offset: the group is new, its offsets expired, or the position was below the log start offset. A group id that changes on every deploy produces the same effect.
Records were produced but a consumer does not receive them yet
Check
kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --topic orders
- How to read it
- Consumers only read below the high watermark. If the partition's
Isrlist is shorter thanReplicas, replication is degraded; whether the high watermark can still advance is the subject of the in-sync replicas lesson.
Key takeaways
- A partition is an append-only log. An offset is a position in one partition and means nothing outside it.
- Order is guaranteed within a partition, in the order the leader appended. Records that must stay in order need the same key.
- A group's parallelism is capped by the partition count, and a partition must fit on one broker.
- Log start offset: oldest readable record. High watermark: how far consumers can read. Log end offset: where the next append goes.
- A committed offset is the next record a group will read. Lag is the end of the log minus that.
- Partitions can be added and never removed, and adding them moves keys.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
Partition orders-3 has log start offset 100, high watermark 182 and, on the leader, log end offset 190. The group payment-service has committed offset 150 for it. Its only consumer is restarted now, and no new records are produced.
What does the consumer read after the restart?
Scenario 2 of 3
orders has 6 partitions and is keyed by order id. The payment group has 6 instances, each at full CPU, and a backlog of 400,000 records that is still growing during a sale. Someone proposes raising the topic to 12 partitions right now and doubling the instances. Open orders receive several events within a few minutes.
What is the main risk of doing this now?
Scenario 3 of 3
A shipments topic has 4 partitions. The group reading it runs 4 instances and lag is rising by about 2,000 records a minute on every partition. Each instance spends most of its time waiting on a carrier API that takes 80 ms per record. The team plans to scale the group to 8 instances.
What will scaling to 8 instances do?
References
References: Apache Kafka documentation: Design (consumer position, replication) ↗ · Apache Kafka documentation: Implementation, Log ↗ · Apache Kafka documentation: Implementation, consumer offset tracking ↗ · Apache Kafka documentation: Basic Kafka operations (modifying topics, checking consumer position) ↗ · Apache Kafka documentation: Consumer configs (auto.offset.reset, isolation.level) ↗ · KafkaConsumer Javadoc (endOffsets, position, committed) ↗ · Spring for Apache Kafka reference: Message listeners ↗ · Report a problem