Skip to content

Path 2 · lesson 2 of 8

Foundation16 minApache Kafka 4.4 (KRaft only) · Java 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 (kafka-clients 3.9)

Partitions and offsets

A partition is an ordered, append-only log and an offset is a position in it. Learn the four offsets that matter and why the partition is the unit of ordering, parallelism and placement.

What you will learn

  • Describe a partition as a log and say what an offset does and does not identify
  • Tell apart the log start offset, high watermark, log end offset and a group's committed offset, and say who moves each
  • Work out from those numbers what a consumer can read right now and how far behind it is
  • Explain why ordering, consumer parallelism and placement on brokers are all decided per partition
  • Name the things that break per-key ordering, including adding partitions to a keyed topic

Before you start

The concept

A partition is a log: a sequence of records that only ever grows at the end. A record is appended, given the next number in the sequence, and never moved or modified afterwards. That number is the record's offset. Offsets start at 0 and belong to one partition, so the full address of a record is topic, partition and offset: orders-3 at offset 181.

The partition leader assigns offsets as it appends, so the offset order is the order in which the broker wrote the records. Every replica of the partition holds the same records at the same offsets. Reading does not remove anything: a consumer asks for records starting at an offset, gets a batch, and asks again from the next offset. Old records leave only when retention deletes whole segments from the front of the log, or compaction removes superseded values.

Because a reader's position in a partition is a single number, the state Kafka needs to keep per consumer group is tiny: one offset per partition. That is the design decision that lets many independent groups read the same log cheaply, and lets any of them rewind and read it again.

One partition as a log

Simulation · not a real cluster

Append records and watch each one take the next offset. Notice that the log is stored as segments and that old segments leave from the front, which moves the log start offset.

Why it exists

One log per topic would be the simplest model, and it would give every record in the topic a single global order. It would also mean one leader broker taking every write, one disk holding everything, and one consumer at a time in each group reading it, because two readers sharing a log cannot both keep a single position. Partitions are how Kafka gives up global order in exchange for scale, while keeping order where it is needed.

The partition is the unit ofWhat that meansConsequence for you
OrderingRecords in one partition are read in the order the leader appended them. Kafka keeps no order between partitions.Records that must stay in order need the same key, so that they go to the same partition.
ParallelismWithin a consumer group, each partition is read by exactly one consumer at a time.A group cannot do more work in parallel than it has partitions. Extra consumers sit idle.
PlacementA partition replica lives entirely on one broker. Different partitions of a topic live on different brokers.A topic can be bigger than one machine, but a single partition cannot. Leaders are spread per partition.
Replication and failureEach partition has its own leader, followers and in-sync set.A broker failure affects the partitions it led, not the whole topic.

How it works inside

How a record gets its partition

The producer chooses the partition before it sends. With a key, the built-in partitioner hashes the key bytes and takes the result modulo the partition count, so equal keys go to the same partition for as long as the count does not change. Without a key, the producer sticks to one partition until it has filled a batch for it, then moves on, which spreads records evenly without any ordering promise. The details are in keys and partitioning.

Keys to partitions

Simulation · not a real cluster

Send records for a few order ids and see each id stay on one partition. Then add a partition and watch where the same ids go next.

The four offsets

Four positions describe the state of a partition and of one group reading it. They are easy to confuse because tools often label two of them the same way.

OffsetKept byMoves whenWhat it tells you
Log start offsetEach replicaRetention deletes the oldest segment, or records are deleted on requestThe earliest offset that can still be read. Anything below it is gone.
High watermarkThe leader, passed on to followersEvery in-sync replica has copied the records below itThe end of the log as consumers see it. Consumers are only given records below the high watermark.
Log end offsetEach replica, for its own copyA record is appendedThe offset the next appended record will get. On the leader, records between the high watermark and here are written but not yet fully replicated.
Committed offsetThe group coordinator, per group and partition, in __consumer_offsetsA consumer in the group commitsThe offset of the next record this group will read after a restart or rebalance. Not the last record processed, and not an acknowledgement of any single record.

Two more positions appear in client code. A running consumer has an in-memory position per partition, the next offset it will fetch, which runs ahead of the committed offset between commits. And a consumer with isolation.level=read_committed reads only up to the last stable offset, which is held back below the high watermark while a transaction is still open.

Lag for a group on a partition is the end of the log minus the committed offset. kafka-consumer-groups.sh prints the committed offset as CURRENT-OFFSET and the end as LOG-END-OFFSET; the end it reports is the one a client can ask for, which is the high watermark. Records above the high watermark are not counted as lag because no consumer could have read them yet.

When a position falls off the log

If a group's committed offset is lower than the log start offset, because the group was stopped for longer than the topic's retention, the offset it asks for no longer exists. The broker answers with an out-of-range error and the consumer applies auto.offset.reset: earliest jumps to the log start offset, latest jumps to the end and skips everything in between, none throws. The same setting applies when a group has no committed offset at all, which also happens when its offsets expire: the coordinator drops a group's offsets offsets.retention.minutes (7 days) after the group has become empty.

Changing the partition count

Partitions can be added to a topic but never removed. Adding them does not move existing records. It changes the divisor in the key hash, so from that moment many keys map to a different partition than the one holding their history. The Kafka documentation also warns that a consumer with auto.offset.reset=latest can miss the first records written to a new partition, because producers may start writing before the group has discovered it. Choosing the partition count is covered in partition sizing.

Partitions & consumer groups

Simulation, not a real cluster

Animated walk-through of partitions being shared out among the consumers of a group.

Step 1 · The pattern

A log and a reader

The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.

  • message
  • consumed
Open the full animated lesson

Architecture

Drawing diagram…

A partition log drawn left to right. Offsets below 100 were deleted, 100 is the log start offset, 100 to 181 can be read by consumers, 182 is the high watermark, 182 to 189 are appended but not yet fully replicated, and 190 is the log end offset. A group's committed offset points at 150.

flowchart LR
  subgraph LOG["Partition orders-3 on its leader"]
    direction LR
    D["Offsets 0 to 99, deleted by retention"] --> S["Offset 100, log start offset"]
    S --> R["Offsets 100 to 181, readable by consumers"]
    R --> H["Offset 182, high watermark"]
    H --> U["Offsets 182 to 189, not yet on every in-sync replica"]
    U --> E["Offset 190, log end offset"]
  end
  PR["Producer"] -->|"next append gets 190"| E
  CG["Group payment-service, committed offset 150"] -. "resumes here" .-> R
  CF["Any consumer"] -->|"fetches below 182"| R
  FO["Follower replicas"] -->|"fetch from their own log end"| U
One partition, with the positions that producers, consumers, followers and retention each move.
  • The log only grows at the right-hand end and only shrinks at the left-hand end. Nothing in the middle changes.
  • The producer moves the log end offset. Followers copying the log are what let the leader move the high watermark up behind it.
  • Consumers of every group share the same readable range, 100 to 181. Each group has its own committed offset inside that range.
  • The group payment-service would resume at 150 and has 32 records it can read now: its lag is 182 minus 150.
  • If retention moved the log start offset past 150 while the group was stopped, the committed offset would point at nothing and auto.offset.reset would decide where the group continues.

Message flow

Drawing diagram…

A producer appends three records at offsets 190 to 192. A follower fetches them, and its next fetch from 193 lets the leader raise the high watermark. The consumer then fetches the three records, processes them and commits offset 193 to the group coordinator.

sequenceDiagram
  participant P as Producer
  participant L as Leader of orders-3
  participant F as Follower
  participant C as Consumer in payment-service
  participant G as Group coordinator
  P->>L: Produce 3 records with key order-42
  Note over L: Appended at offsets 190 191 192, log end offset is 193
  F->>L: Fetch from offset 190
  L-->>F: Records 190 to 192
  F->>L: Fetch from offset 193
  Note over L: Every in-sync replica has 190 to 192, high watermark moves to 193
  L-->>P: Acknowledged, base offset 190
  C->>L: Fetch from offset 190
  L-->>C: Records 190 to 192
  Note over C: Process the three records
  C->>G: Commit offset 193 for orders-3
  G-->>C: Commit stored in __consumer_offsets
Append moves the log end offset, replication moves the high watermark, and a commit moves the group's offset.
  1. The leader assigns the offsets. The producer learns them from the acknowledgement and does not choose them.
  2. The follower's second fetch request, starting at 193, is how the leader learns that the follower has everything up to 192.
  3. The consumer could not have been given offset 190 before the high watermark passed it, however early it asked.
  4. The consumer commits 193, the offset of the next record it wants, not 192, the last one it handled. A group that commits the last processed offset reads that record again after every restart.
  5. The commit goes to the group coordinator, which is a broker chosen by the group id, and not necessarily the leader of orders-3.

In a real system

A typical order platform has an orders topic with 12 partitions, keyed by order id. The order service writes OrderCreated, OrderPaid and OrderShipped events. All events for order 42 hash to the same partition, so every consumer sees them in the order they were written. Events for order 42 and order 43 are probably in different partitions and have no defined order relative to each other, which is fine: nothing in the business depends on it.

The payment service runs four instances in the group payment-service, so each instance reads three partitions. The notification service is a separate group with its own committed offsets on the same twelve logs; it can be stopped for an hour and catch up without the payment service noticing. An on-call engineer looking at one partition sees something like this:

GroupPartitionCommitted offsetEnd offsetLag
payment-serviceorders-38,412,9508,412,98232
notification-serviceorders-38,390,1008,412,98222,882
fraud-replay (started today)orders-3none yet8,412,982depends on auto.offset.reset
Three groups, one log. Each group's position is independent of the others.

The choice of key is a statement about which records need a relative order. Keying by customer id instead of order id would also keep one customer's orders in sequence, at the price of putting a very active customer's whole traffic on one partition. The hot partition lesson shows what that looks like.

Java and Spring Boot

A listener can see the partition and offset of every record it receives, and the Admin API can read the log start offset, the end offset and a group's committed offsets. Together they are enough to reproduce what kafka-consumer-groups.sh prints, which is a useful thing to expose on an internal diagnostics endpoint.

src/main/java/com/example/payments/kafka/OrderEventListener.javaJava
package com.example.payments.kafka;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderEventListener {    private static final Logger log = LoggerFactory.getLogger(OrderEventListener.class);    @KafkaListener(topics = "orders", groupId = "payment-service")    public void onOrderEvent(ConsumerRecord<String, String> record) {        log.info("orders-{} offset={} key={} timestamp={}",                record.partition(), record.offset(), record.key(), record.timestamp());        // Business handling goes here. Records with the same key arrive in offset order,        // because they are in the same partition and one thread reads that partition.    }}

Why it is written this way

  • Taking a ConsumerRecord instead of only the payload gives access to the partition and offset. Logging both on every failure makes a record findable later.
  • Partition plus offset identifies this record in this topic on this cluster. It is a good value for log correlation and a poor business identifier, because a re-published or mirrored event gets a different one.
  • The container commits offsets after the listener returns (the default BATCH ack mode commits after the records from a poll have been processed), so the committed offset is always the next record to read.

Configuration

Settings that decide how many partitions exist, how long positions and records stay valid, and where a consumer starts when its position is missing. Defaults are from the Kafka 4.4 configuration reference.

num.partitionsDefault 1
What it does
Broker setting. Partition count for topics created without an explicit count.
When to change it
Always pass a partition count when creating a topic. The count can be raised later but never lowered, and raising it moves keys.
auto.offset.resetDefault latest
What it does
Consumer setting. Where to start when the group has no committed offset for a partition, or the committed offset is no longer in the log. Values are earliest, latest, by_duration:<duration> (newer clients) and none.
When to change it
Use earliest when skipping records is worse than reprocessing. Use none when either would be a bug you want to hear about. With latest, a new group, an expired group and a newly added partition all start at the end silently.
offsets.retention.minutesDefault 10080 (7 days)
What it does
Broker setting. How long committed offsets are kept after a group has lost all its members, or since the last commit for a partition the group no longer subscribes to.
When to change it
Raise it if a consumer may legitimately be switched off for longer than a week, such as a batch job that runs monthly.
retention.msDefault 604800000 (7 days)
What it does
Topic setting. How long records are kept before their segment may be deleted, which is what moves the log start offset.
When to change it
Set it longer than the longest outage a consumer has to survive without losing records, and keep the disk to match.
isolation.levelDefault read_uncommitted
What it does
Consumer setting. read_committed hides records from aborted transactions and limits reads to the last stable offset.
When to change it
Set read_committed when the topic is written by transactional producers. Expect reads to pause behind an open transaction.
offsets.topic.num.partitionsDefault 50
What it does
Broker setting. Partition count of __consumer_offsets. A group's offsets live in the partition selected by hashing the group id, and that partition's leader is the group's coordinator.
When to change it
Leave it alone. The documentation says it should not change after deployment, because the mapping from group to coordinator depends on it.

Hands-on exercise

Read the offsets of a partition

Use the cluster explorer to connect the picture of a partition to the numbers: which broker leads it, where its log ends, how far consumers can read, and what changes when its leader stops.

Steps · 0 of 7 done

Six partitions on three brokers

Simulation · not a real cluster

Click a partition to see its leader, replicas, log end offset and high watermark. Stop a broker to see what happens to the partitions it led.

Take it further in the lab: Partition key and ordering

What can go wrong

Expecting order across a topic
Symptom
A consumer sometimes handles OrderShipped before OrderCreated for the same order, or two instances process related events at the same time.
Cause
The related events were written with different keys, or with no key, and landed in different partitions. Kafka keeps no order between partitions.
Fix
Give events that must stay in order the same key. If the order must hold across entities, either use one key for the whole group or make the consumer tolerate reordering.
Adding partitions to a keyed topic
Symptom
After the partition count was raised, some entities show events out of order, and stateful consumers find part of an entity's history arriving on a different instance.
Cause
The key hash is taken modulo the partition count, so existing keys now map to different partitions while their older records stay where they were.
Fix
Size keyed topics generously at creation. If the count must change, do it when consumers have no backlog, or create a new topic with the new count and migrate producers and consumers to it.
More consumers than partitions
Symptom
Lag keeps growing after the group was scaled from 6 to 10 instances. Four instances show no assigned partitions.
Cause
Each partition is read by one consumer in a group, so a 6-partition topic supports at most 6 active consumers in that group.
Fix
Make each consumer faster (batching, fewer round trips, a larger database pool), or increase the partition count with the ordering consequences in mind.
A committed offset that is no longer in the log
Symptom
A consumer that was stopped for a long time restarts and either skips straight to new records, leaving a silent gap, or reprocesses days of history.
Cause
Retention deleted the records at the committed offset, or the group's offsets expired. The consumer fell back to auto.offset.reset.
Fix
Choose auto.offset.reset deliberately for each consumer, keep retention longer than the longest planned downtime, and alert when a committed offset drops below the log start offset.
Treating offsets as portable
Symptom
After switching to a mirrored cluster, or copying a topic, consumers restart from the wrong place when their old offsets are applied.
Cause
An offset is a position in one partition of one cluster. A copy of the topic has its own offsets for the same records.
Fix
Translate positions by timestamp or with the offset translation the mirroring tool provides, and never store an offset without the cluster, topic and partition it belongs to.

How to troubleshoot

Need to know how far behind a group is on each partition

Check

kafka-consumer-groups.sh --bootstrap-server kafka-a1.internal:9092 --describe --group payment-service
How to read it
One row per partition. CURRENT-OFFSET is the committed offset (the next record the group will read), LOG-END-OFFSET is the end of the log, and LAG is the difference. CONSUMER-ID shows which member owns the partition; a dash means nobody does.
Need the first and last offsets of each partition

Check

kafka-get-offsets.sh --bootstrap-server kafka-a1.internal:9092 --topic orders --time earliest (then again with --time latest)
How to read it
Prints topic:partition:offset. earliest is the log start offset and latest is the end. The difference is roughly how many records the partition holds now. Older versions of the tool take -2 and -1 instead of the names.
One partition lags while the others are fine

Check

Compare `latest` minus `earliest` across partitions with kafka-get-offsets.sh, and the per-partition LAG column of the group
How to read it
A partition that grows much faster than its siblings is receiving a hot key. Even growth with one lagging partition points at the consumer that owns it: look for a slow or stuck instance.
A consumer jumped forwards or backwards after a restart

Check

The consumer's log at startup for an offset reset message naming the partition, and the value of `auto.offset.reset`
How to read it
A reset means there was no usable committed offset: the group is new, its offsets expired, or the position was below the log start offset. A group id that changes on every deploy produces the same effect.
Records were produced but a consumer does not receive them yet

Check

kafka-topics.sh --bootstrap-server kafka-a1.internal:9092 --describe --topic orders
How to read it
Consumers only read below the high watermark. If the partition's Isr list is shorter than Replicas, replication is degraded; whether the high watermark can still advance is the subject of the in-sync replicas lesson.

Key takeaways

  • A partition is an append-only log. An offset is a position in one partition and means nothing outside it.
  • Order is guaranteed within a partition, in the order the leader appended. Records that must stay in order need the same key.
  • A group's parallelism is capped by the partition count, and a partition must fit on one broker.
  • Log start offset: oldest readable record. High watermark: how far consumers can read. Log end offset: where the next append goes.
  • A committed offset is the next record a group will read. Lag is the end of the log minus that.
  • Partitions can be added and never removed, and adding them moves keys.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

Partition orders-3 has log start offset 100, high watermark 182 and, on the leader, log end offset 190. The group payment-service has committed offset 150 for it. Its only consumer is restarted now, and no new records are produced.

What does the consumer read after the restart?

Scenario 2 of 3

orders has 6 partitions and is keyed by order id. The payment group has 6 instances, each at full CPU, and a backlog of 400,000 records that is still growing during a sale. Someone proposes raising the topic to 12 partitions right now and doubling the instances. Open orders receive several events within a few minutes.

What is the main risk of doing this now?

Scenario 3 of 3

A shipments topic has 4 partitions. The group reading it runs 4 instances and lag is rising by about 2,000 records a minute on every partition. Each instance spends most of its time waiting on a carrier API that takes 80 ms per record. The team plans to scale the group to 8 instances.

What will scaling to 8 instances do?

References

References: Apache Kafka documentation: Design (consumer position, replication) ↗ · Apache Kafka documentation: Implementation, Log ↗ · Apache Kafka documentation: Implementation, consumer offset tracking ↗ · Apache Kafka documentation: Basic Kafka operations (modifying topics, checking consumer position) ↗ · Apache Kafka documentation: Consumer configs (auto.offset.reset, isolation.level) ↗ · KafkaConsumer Javadoc (endOffsets, position, committed) ↗ · Spring for Apache Kafka reference: Message listeners ↗ · Report a problem