Skip to content

Path 1 · lesson 1 of 6

Foundation20 minApache Kafka 4.4 (KRaft) · Java 21 · Spring Boot 3.5 · Spring for Apache Kafka 3.3 (kafka-clients 3.9)

What Kafka is and why it exists

Kafka is a distributed commit log: services append records to it, and any number of readers consume them at their own pace. See the problem it was built for and the smallest working picture.

What you will learn

  • Describe Kafka as an append-only, partitioned, replicated log and name its moving parts
  • Explain the data-pipeline problem LinkedIn had and why existing brokers and log collectors did not solve it
  • Trace one record from a producer, into a partition at an offset, and out to two independent consumer groups
  • Predict what a new consumer group reads, and what a restarted one reads, from where its offset is
  • Write and run a minimal Spring Boot producer and listener against a local broker

Before you start

  • Comfortable with Java and a little Spring Boot (a REST controller, an application.yml)
  • A rough idea of what a message broker or a queue is for

The concept

Apache Kafka is a distributed commit log. A commit log is the simplest storage structure there is: an ordered sequence of records where the only way to write is to add to the end, and existing records never change. Kafka runs that structure across several servers, keeps copies of it, and lets many programs write to it and read from it at the same time.

Everything else in Kafka follows from that one idea. Because writes only go to the end, every record gets a position, and that position never changes. Because nothing is modified in place, a reader needs to remember just one thing: how far along the log it has got. And because reading does not remove anything, a second reader, or a tenth, can read the same records without the first one noticing.

The vocabulary, in one pass

TermWhat it isIn the order example
RecordOne entry in the log: a key, a value, a timestamp and optional headers. The docs also say event or message.Key order-1042, value: the JSON of an OrderPlaced event
TopicA named stream of records. You write to a topic and read from a topic.orders
PartitionA topic is split into partitions. Each partition is one log, with its own order.orders has 3: orders-0, orders-1, orders-2
OffsetThe position of a record inside its partition: 0, 1, 2 and so on.The event lands at orders-1, offset 57
BrokerA Kafka server. It stores partitions on disk and answers produce and fetch requests.3 brokers, each holding a copy of every partition
ProducerA client that appends records to a topic.The order service
ConsumerA client that reads records from a topic, in offset order per partition.The payment service
Consumer groupA set of consumers sharing a group.id. They split the partitions between them and share one saved position per partition.payment-service, analytics

Two details matter from day one. First, order is per partition, not per topic: Kafka guarantees that anyone reading a partition sees its records in the order the broker appended them, and it makes no promise about how records in different partitions relate. Records with the same key go to the same partition, which is how you keep everything about one order in sequence. Second, reading does not delete. Records stay until the topic's retention policy removes them, seven days by default, whether or not anyone has read them.

Why it exists

Kafka was built at LinkedIn around 2010 and described in the 2011 paper Kafka: a Distributed Messaging System for Log Processing (opens in a new tab). The problem was activity data: page views, clicks, searches, plus operational metrics from every machine. That data had stopped being something you only analysed overnight. Search relevance, recommendations, ad reporting and abuse detection all needed it within seconds, and its volume was far larger than the transactional data in the databases.

The paper explains why neither kind of existing tool fitted.

Message brokers of the time

  • Built around rich per-message delivery guarantees that log data did not need and that made the broker complex
  • Throughput was not the main design goal: the paper notes that JMS had no way for a producer to batch several messages into one request
  • Little support for spreading one stream across many machines
  • Assumed messages are consumed almost immediately, and slowed down badly when a backlog built up, which is exactly what a nightly batch load creates

Log collectors of the time

  • Good at shipping log files from servers into Hadoop or a warehouse
  • Designed for offline consumption, not for an application reading within seconds
  • Mostly pushed data at the consumer, so a slow consumer was flooded instead of setting its own pace
  • Exposed storage details such as per-minute files to whoever read the data

Jay Kreps, one of the authors, later described the wider problem in The Log: What every software engineer should know about real-time data's unifying abstraction (opens in a new tab). LinkedIn had many specialised data systems, and each new one needed a custom pipeline to and from each of the others. With N systems that heads toward N×N pipelines, each one built and operated separately. The fix was to put one log in the middle: every source writes to it once, every destination reads from it, and adding a system means one new connection, not N.

What a log in the middle buys you

  • Decoupling. The producer writes a record and is done. It does not know which services read it, how many there are, or whether they are running. Adding a reader requires no change to the writer.
  • Buffering. The log absorbs the difference between how fast data is produced and how fast each reader can take it. A reader that is slow, or down for maintenance, falls behind and catches up later. Nobody upstream has to slow down or hold data for it.
  • Replay. Because records stay in the log, a reader can go back to an earlier offset and read again: to recover from a bug in its own code, to rebuild a cache or search index, or to feed a system that did not exist when the records were written.
  • Many readers of one stream. Each consumer group has its own position. Payments can be at the end of the log while a nightly export is six hours behind, and neither affects the other.

How it works inside

A partition is a file you only append to

On a broker, each partition is a directory of segment files. A produce request appends the new records to the end of the newest segment and the broker assigns each one the next offset. Appending to the end of a file and reading a file from a position are sequential operations, which disks and the operating system's page cache handle very efficiently. This is why Kafka's cost per record does not grow with the amount of data already stored, and why keeping a week of records is normal, not a problem to avoid.

Consumers pull, and keep their own place

A consumer asks the broker for records from a given offset onward, processes what comes back, and asks again from where it stopped. The broker does not push, and it does not track which individual records each reader has handled. The reader's entire state is one number per partition: the offset of the next record it wants. A consumer group saves that number back to Kafka by committing it, so a restarted consumer continues where the group left off.

Partitions are how it scales

One log on one machine has a limit. Splitting a topic into partitions lets different brokers hold different partitions and lets several consumers in a group read in parallel, one partition each. The producer picks the partition: records with the same key are hashed to the same partition, so everything about order-1042 stays in one ordered sequence, while different orders spread across partitions. The price is that there is no single order across the whole topic.

Where does a record go?

Simulation · not a real cluster

Send records for a few order ids and watch which partition each key lands in. The same key always lands in the same partition while the partition count stays the same.

Copies on several brokers

Each partition is replicated to several brokers, commonly three. One replica is the leader and takes the writes; the others follow by copying from it. If the leader's broker fails, a follower that was keeping up takes over. How that works, and what it does and does not guarantee, is the subject of the architecture path.

Partitions & consumer groups

Simulation, not a real cluster

An animated walk through partitions and consumer groups from the Build section. Watch how the partitions are divided between the consumers of one group.

Step 1 · The pattern

A log and a reader

The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.

  • message
  • consumed
Open the full animated lesson

Architecture

Drawing diagram…

The order service writes keyed records into three partitions of the orders topic. The payment-service group splits the partitions between two consumers, and the analytics group reads all three partitions with one consumer.

flowchart LR
  OS["Order service - producer"]
  subgraph T["Topic orders on a 3-broker cluster"]
    P0["Partition 0 - offsets 0 to 41"]
    P1["Partition 1 - offsets 0 to 57"]
    P2["Partition 2 - offsets 0 to 38"]
  end
  subgraph G1["Group payment-service"]
    C1["Consumer A"]
    C2["Consumer B"]
  end
  subgraph G2["Group analytics"]
    C3["Consumer C"]
  end
  OS -->|"key order-1042"| P1
  OS -->|"key order-1043"| P0
  OS -->|"key order-1044"| P2
  P0 --> C1
  P1 --> C1
  P2 --> C2
  P0 --> C3
  P1 --> C3
  P2 --> C3
One topic, three partitions, two consumer groups that each read every record independently.
  • The order service appends each record to one partition, chosen from the record's key. It never talks to a consumer.
  • Inside the payment-service group the three partitions are divided between two consumers, so each record is handled by one of them. A partition is read by only one consumer of a group at a time.
  • The analytics group has its own saved offsets. It receives every record too, at its own pace, and nothing it does changes what payment-service sees.
  • All three partitions are stored, and replicated, on the brokers. The consumers hold no data: stop them all and the records are still there.

Message flow

Drawing diagram…

The order service produces a record that the broker appends at offset 57 and acknowledges. The payment-service group fetches it, processes it and commits offset 58. Two hours later the analytics group fetches from offset 0, receives records 0 to 57 and commits offset 58.

sequenceDiagram
  participant O as Order service
  participant B as Broker - leader of orders-1
  participant P as payment-service group
  participant A as analytics group
  O->>B: Produce OrderPlaced with key order-1042
  B->>B: Append to orders-1 at offset 57
  B-->>O: Acknowledged - partition 1 offset 57
  P->>B: Fetch orders-1 from offset 57
  B-->>P: Record at offset 57
  P->>P: Charge the card
  P->>B: Commit offset 58 for payment-service
  Note over A: Starts two hours later
  A->>B: Fetch orders-1 from offset 0
  B-->>A: Records 0 to 57
  A->>B: Commit offset 58 for analytics
One record is appended once, then read by two groups at different times.
  1. The acknowledgement goes to the producer as soon as the brokers have stored the record. The producer is finished at that point, whether or not any consumer exists.
  2. The payment group fetches from its saved position, does its work, then commits 58: the offset of the next record it will read.
  3. The analytics group starts later with no saved position. Configured to start from the earliest offset, it reads the whole partition, including records the payment group handled hours ago. The record at offset 57 was never removed.

In a real system

A typical online shop. When a customer checks out, the order service saves the order and appends an OrderPlaced record to the orders topic. Several teams care about that fact for different reasons.

ElementChoiceReason
TopicordersOne stream for events about orders
KeyThe order id, e.g. order-1042All events about one order go to one partition and stay in order
ValueJSON of the event: order id, customer id, amount, currencyReadable by every consumer without calling the order service
Partitions6Lets up to six consumers of one group work in parallel
Replication factor3The topic survives the loss of a broker
Retention7 days (the default)A reader can be down, or a bug can go unnoticed, for days and nothing is lost
Consumer groupspayment-service, inventory-service, notification-service, analyticsEach team reads every order, at its own pace

Now the four benefits become concrete. The order service does not change when the fraud team adds a fifth group next quarter (decoupling). During a sale, orders arrive faster than the notification service can send emails; its lag grows and it catches up afterwards while checkout is unaffected (buffering). When the analytics job is found to have miscounted since Tuesday, it is fixed and re-run from Tuesday's offsets (replay). And all of this happens on one stream that the order service writes exactly once (many readers).

The academy's example project builds exactly this system: Order Service → Kafka → Payment Service → Inventory Service → Notification Service.

Java and Spring Boot

The smallest useful pair: one class that publishes an event and one that receives it, in a single Spring Boot application. Add the spring-kafka dependency (Spring Boot manages its version) and start a local broker with docker run -p 9092:9092 apache/kafka:4.4.0.

src/main/java/com/example/orders/OrderPlaced.javaJava
package com.example.orders;/** * A fact: an order was placed. Immutable, named in the past tense. * amountMinor is the amount in the currency's minor unit (cents), to avoid floating point. */public record OrderPlaced(String orderId, String customerId, long amountMinor, String currency) {}

Why it is written this way

  • A record is a good fit for an event: it is immutable, and an event describes something that has already happened and cannot change.
  • The order id is inside the value as well as being the key. Consumers should not have to depend on the key to understand the event.

Configuration

Only a handful of settings are needed to get a first record through. Defaults are from the Apache Kafka 4.4 documentation; the client defaults listed are the same in the 3.9 client that Spring Boot 3.5 manages.

bootstrap.serversDefault none (required)
What it does
One or more broker addresses the client uses for its first connection. From there it discovers the whole cluster and which broker leads each partition.
When to change it
List two or three brokers in production so that the client can still start when one of them is down. In Spring Boot: spring.kafka.bootstrap-servers.
group.idDefault null
What it does
The consumer group this consumer belongs to. Consumers with the same value split the partitions and share committed offsets.
When to change it
Use one stable name per application that should process each record once, such as payment-service. A different name means an independent reader with its own position.
auto.offset.resetDefault latest
What it does
Where to start when the group has no committed offset for a partition, or the committed offset no longer exists on the broker. earliest starts at the oldest retained record, latest at the end.
When to change it
Choose earliest when a new consumer must not miss records written before it first started. It has no effect once the group has a committed offset.
acksDefault all
What it does
How many replicas must have the record before the broker acknowledges the produce request. all waits for every replica that is currently in sync.
When to change it
Leave it at all. It is only a real durability guarantee together with replication and min.insync.replicas, which the producers path explains.
retention.ms (topic)Default 604800000 (7 days)
What it does
How long a record is kept before its segment may be deleted, when the topic uses the default delete cleanup policy. It does not depend on whether anyone has read the record.
When to change it
Raise it when consumers may need to replay further back or may be offline for longer; lower it when disk matters more than history.
num.partitions (broker)Default 1
What it does
The partition count given to a topic that is created without one, including topics the broker auto-creates on first use.
When to change it
Do not rely on it. Create topics explicitly with a partition count chosen for the parallelism you need, because changing it later moves keys to different partitions.
auto.create.topics.enable (broker)Default true
What it does
Whether the broker creates a topic the first time a client asks for one that does not exist.
When to change it
Convenient locally. Many teams turn it off in production so that a typo in a topic name fails loudly and does not silently create a one-partition, one-replica topic.

Hands-on exercise

Find your record in the cluster

Use the cluster explorer to see that a topic is a set of partition logs spread over brokers, and that the data stays put when a broker stops.

Steps · 0 of 6 done

A three-broker cluster holding the orders topic

Simulation · not a real cluster

Click a broker, the topic or a partition to inspect it. Then stop the broker that leads a partition and see who takes over.

Take it further in the lab: Producer publishing messages

What can go wrong

A new consumer sees nothing
Symptom
The producer logs successful sends, the listener starts without errors, and it never receives the records that were already written.
Cause
The group has no committed offset, and auto.offset.reset defaults to latest, so it starts at the end of each partition and only sees records written after it joined.
Fix
Set auto.offset.reset to earliest for consumers that must read existing records. Remember it only applies while the group has no committed offset; to move an existing group, reset its offsets.
Expecting one order across the whole topic
Symptom
With three partitions, a consumer logs order-1043 before order-1042 although they were sent the other way round.
Cause
Ordering exists inside a partition only. The two orders have different keys, hashed to different partitions, and each partition is read independently.
Fix
Give records that must stay in order the same key. If you truly need one total order, that means one partition, and with it one consumer per group.
Two copies of a service with different group ids
Symptom
After scaling the payment service to two instances, every order is charged twice.
Cause
Each instance was given its own group.id, for example by including the hostname. Kafka treats them as two independent readers and delivers every record to both.
Fix
All instances of one service share one fixed group id. Different group ids are for different services that each need the full stream.
Treating the topic as a database table
Symptom
Code that needs the current state of order 1042 reads the topic from the beginning and filters, and gets slower every week.
Cause
A log can only be read in sequence from an offset. There is no lookup by key and no query.
Fix
Consume the topic into a store built for lookups (a database table, a cache) and query that. Kafka carries the changes; something else serves the reads.

How to troubleshoot

Not sure the topic exists or how it is laid out

Check

bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
How to read it
The first line shows PartitionCount and ReplicationFactor. Each following line is one partition with its Leader, Replicas and Isr. A topic with one partition and one replica that you did not create on purpose was auto-created by the broker.
Not sure the records reached Kafka at all

Check

bin/kafka-console-consumer.sh --topic orders --from-beginning --bootstrap-server localhost:9092
How to read it
This reads the topic from the oldest retained record without touching your application's group. If records appear here, the producer side works and the problem is in your consumer or its position.
The listener is running but seems stuck or behind

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
One row per partition. CURRENT-OFFSET is the group's committed position, LOG-END-OFFSET is the end of the partition, and LAG is the difference. Lag of 0 on every partition means the group has committed up to the end of the log.
The application hangs at startup or the first send fails after a minute

Check

The client log for a line such as `Connection to node -1 (localhost/127.0.0.1:9092) could not be established`
How to read it
The client cannot reach the address in bootstrap.servers. Check that the broker is running and that the address is reachable from where the application runs, which inside a container is not localhost.

Key takeaways

  • Kafka is an append-only log, split into partitions and copied across brokers. Records get a permanent offset inside their partition.
  • Producers and consumers never talk to each other. The producer is done when the brokers have stored the record.
  • Reading does not delete. Records leave when retention removes them, seven days by default, not when someone consumes them.
  • A consumer group's state is one offset per partition: the next record to read. Same group id means shared work; different group id means an independent reader.
  • Order is guaranteed inside a partition only. The key decides the partition, so the key is how you choose what stays in order.
  • It was built for many systems reading the same high-volume stream at different speeds, and that is still what it is best at.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

The orders topic has 6 partitions and 7-day retention. The payment-service group has consumed it for months with zero lag. A fraud team now wants to score every order, starting with the orders from the last three days, without any risk to payments.

How should the fraud service get the data?

Scenario 2 of 3

Two instances of the notification service run with group.id set to notification- plus the pod name. Customers report receiving every order confirmation email twice. The topic has 3 partitions and the producer logs show each order was sent once.

What is the fix?

Scenario 3 of 3

A loyalty-points consumer shipped a bug on Monday at 09:00 that awarded half the correct points. It is now Wednesday. The fix is ready. The orders topic keeps 7 days of records, and the points table can be recalculated per order.

What is the most direct way to repair the data?

References

References: Apache Kafka documentation: Introduction ↗ · Apache Kafka 4.4 documentation: Design ↗ · Apache Kafka 4.4 documentation: Quickstart ↗ · Apache Kafka 4.4 documentation: Consumer configs ↗ · Apache Kafka 4.4 documentation: Topic configs ↗ · Apache Kafka 4.4 documentation: Broker configs ↗ · Kreps, Narkhede, Rao: Kafka: a Distributed Messaging System for Log Processing (NetDB 2011) ↗ · Jay Kreps: The Log: What every software engineer should know about real-time data's unifying abstraction (LinkedIn Engineering, 2013) ↗ · Spring for Apache Kafka 3.3 reference: Quick Tour ↗ · Spring Boot 3.5 reference: Apache Kafka Support ↗ · Report a problem