Path 1 · lesson 1 of 6
What Kafka is and why it exists
Kafka is a distributed commit log: services append records to it, and any number of readers consume them at their own pace. See the problem it was built for and the smallest working picture.
What you will learn
- Describe Kafka as an append-only, partitioned, replicated log and name its moving parts
- Explain the data-pipeline problem LinkedIn had and why existing brokers and log collectors did not solve it
- Trace one record from a producer, into a partition at an offset, and out to two independent consumer groups
- Predict what a new consumer group reads, and what a restarted one reads, from where its offset is
- Write and run a minimal Spring Boot producer and listener against a local broker
Before you start
- Comfortable with Java and a little Spring Boot (a REST controller, an application.yml)
- A rough idea of what a message broker or a queue is for
The concept
Apache Kafka is a distributed commit log. A commit log is the simplest storage structure there is: an ordered sequence of records where the only way to write is to add to the end, and existing records never change. Kafka runs that structure across several servers, keeps copies of it, and lets many programs write to it and read from it at the same time.
Everything else in Kafka follows from that one idea. Because writes only go to the end, every record gets a position, and that position never changes. Because nothing is modified in place, a reader needs to remember just one thing: how far along the log it has got. And because reading does not remove anything, a second reader, or a tenth, can read the same records without the first one noticing.
The vocabulary, in one pass
| Term | What it is | In the order example |
|---|---|---|
| Record | One entry in the log: a key, a value, a timestamp and optional headers. The docs also say event or message. | Key order-1042, value: the JSON of an OrderPlaced event |
| Topic | A named stream of records. You write to a topic and read from a topic. | orders |
| Partition | A topic is split into partitions. Each partition is one log, with its own order. | orders has 3: orders-0, orders-1, orders-2 |
| Offset | The position of a record inside its partition: 0, 1, 2 and so on. | The event lands at orders-1, offset 57 |
| Broker | A Kafka server. It stores partitions on disk and answers produce and fetch requests. | 3 brokers, each holding a copy of every partition |
| Producer | A client that appends records to a topic. | The order service |
| Consumer | A client that reads records from a topic, in offset order per partition. | The payment service |
| Consumer group | A set of consumers sharing a group.id. They split the partitions between them and share one saved position per partition. | payment-service, analytics |
Two details matter from day one. First, order is per partition, not per topic: Kafka guarantees that anyone reading a partition sees its records in the order the broker appended them, and it makes no promise about how records in different partitions relate. Records with the same key go to the same partition, which is how you keep everything about one order in sequence. Second, reading does not delete. Records stay until the topic's retention policy removes them, seven days by default, whether or not anyone has read them.
Why it exists
Kafka was built at LinkedIn around 2010 and described in the 2011 paper Kafka: a Distributed Messaging System for Log Processing (opens in a new tab). The problem was activity data: page views, clicks, searches, plus operational metrics from every machine. That data had stopped being something you only analysed overnight. Search relevance, recommendations, ad reporting and abuse detection all needed it within seconds, and its volume was far larger than the transactional data in the databases.
The paper explains why neither kind of existing tool fitted.
Message brokers of the time
- Built around rich per-message delivery guarantees that log data did not need and that made the broker complex
- Throughput was not the main design goal: the paper notes that JMS had no way for a producer to batch several messages into one request
- Little support for spreading one stream across many machines
- Assumed messages are consumed almost immediately, and slowed down badly when a backlog built up, which is exactly what a nightly batch load creates
Log collectors of the time
- Good at shipping log files from servers into Hadoop or a warehouse
- Designed for offline consumption, not for an application reading within seconds
- Mostly pushed data at the consumer, so a slow consumer was flooded instead of setting its own pace
- Exposed storage details such as per-minute files to whoever read the data
Jay Kreps, one of the authors, later described the wider problem in The Log: What every software engineer should know about real-time data's unifying abstraction (opens in a new tab). LinkedIn had many specialised data systems, and each new one needed a custom pipeline to and from each of the others. With N systems that heads toward N×N pipelines, each one built and operated separately. The fix was to put one log in the middle: every source writes to it once, every destination reads from it, and adding a system means one new connection, not N.
What a log in the middle buys you
- Decoupling. The producer writes a record and is done. It does not know which services read it, how many there are, or whether they are running. Adding a reader requires no change to the writer.
- Buffering. The log absorbs the difference between how fast data is produced and how fast each reader can take it. A reader that is slow, or down for maintenance, falls behind and catches up later. Nobody upstream has to slow down or hold data for it.
- Replay. Because records stay in the log, a reader can go back to an earlier offset and read again: to recover from a bug in its own code, to rebuild a cache or search index, or to feed a system that did not exist when the records were written.
- Many readers of one stream. Each consumer group has its own position. Payments can be at the end of the log while a nightly export is six hours behind, and neither affects the other.
How it works inside
A partition is a file you only append to
On a broker, each partition is a directory of segment files. A produce request appends the new records to the end of the newest segment and the broker assigns each one the next offset. Appending to the end of a file and reading a file from a position are sequential operations, which disks and the operating system's page cache handle very efficiently. This is why Kafka's cost per record does not grow with the amount of data already stored, and why keeping a week of records is normal, not a problem to avoid.
Consumers pull, and keep their own place
A consumer asks the broker for records from a given offset onward, processes what comes back, and asks again from where it stopped. The broker does not push, and it does not track which individual records each reader has handled. The reader's entire state is one number per partition: the offset of the next record it wants. A consumer group saves that number back to Kafka by committing it, so a restarted consumer continues where the group left off.
Partitions are how it scales
One log on one machine has a limit. Splitting a topic into partitions lets different brokers hold different partitions and lets several consumers in a group read in parallel, one partition each. The producer picks the partition: records with the same key are hashed to the same partition, so everything about order-1042 stays in one ordered sequence, while different orders spread across partitions. The price is that there is no single order across the whole topic.
Where does a record go?
Simulation · not a real clusterSend records for a few order ids and watch which partition each key lands in. The same key always lands in the same partition while the partition count stays the same.
Copies on several brokers
Each partition is replicated to several brokers, commonly three. One replica is the leader and takes the writes; the others follow by copying from it. If the leader's broker fails, a follower that was keeping up takes over. How that works, and what it does and does not guarantee, is the subject of the architecture path.
Partitions & consumer groups
Simulation, not a real clusterAn animated walk through partitions and consumer groups from the Build section. Watch how the partitions are divided between the consumers of one group.
Step 1 · The pattern
A log and a reader
The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.
- message
- consumed
Architecture
Drawing diagram…
The order service writes keyed records into three partitions of the orders topic. The payment-service group splits the partitions between two consumers, and the analytics group reads all three partitions with one consumer.
flowchart LR
OS["Order service - producer"]
subgraph T["Topic orders on a 3-broker cluster"]
P0["Partition 0 - offsets 0 to 41"]
P1["Partition 1 - offsets 0 to 57"]
P2["Partition 2 - offsets 0 to 38"]
end
subgraph G1["Group payment-service"]
C1["Consumer A"]
C2["Consumer B"]
end
subgraph G2["Group analytics"]
C3["Consumer C"]
end
OS -->|"key order-1042"| P1
OS -->|"key order-1043"| P0
OS -->|"key order-1044"| P2
P0 --> C1
P1 --> C1
P2 --> C2
P0 --> C3
P1 --> C3
P2 --> C3- The order service appends each record to one partition, chosen from the record's key. It never talks to a consumer.
- Inside the
payment-servicegroup the three partitions are divided between two consumers, so each record is handled by one of them. A partition is read by only one consumer of a group at a time. - The
analyticsgroup has its own saved offsets. It receives every record too, at its own pace, and nothing it does changes whatpayment-servicesees. - All three partitions are stored, and replicated, on the brokers. The consumers hold no data: stop them all and the records are still there.
Message flow
Drawing diagram…
The order service produces a record that the broker appends at offset 57 and acknowledges. The payment-service group fetches it, processes it and commits offset 58. Two hours later the analytics group fetches from offset 0, receives records 0 to 57 and commits offset 58.
sequenceDiagram participant O as Order service participant B as Broker - leader of orders-1 participant P as payment-service group participant A as analytics group O->>B: Produce OrderPlaced with key order-1042 B->>B: Append to orders-1 at offset 57 B-->>O: Acknowledged - partition 1 offset 57 P->>B: Fetch orders-1 from offset 57 B-->>P: Record at offset 57 P->>P: Charge the card P->>B: Commit offset 58 for payment-service Note over A: Starts two hours later A->>B: Fetch orders-1 from offset 0 B-->>A: Records 0 to 57 A->>B: Commit offset 58 for analytics
- The acknowledgement goes to the producer as soon as the brokers have stored the record. The producer is finished at that point, whether or not any consumer exists.
- The payment group fetches from its saved position, does its work, then commits 58: the offset of the next record it will read.
- The analytics group starts later with no saved position. Configured to start from the earliest offset, it reads the whole partition, including records the payment group handled hours ago. The record at offset 57 was never removed.
In a real system
A typical online shop. When a customer checks out, the order service saves the order and appends an OrderPlaced record to the orders topic. Several teams care about that fact for different reasons.
| Element | Choice | Reason |
|---|---|---|
| Topic | orders | One stream for events about orders |
| Key | The order id, e.g. order-1042 | All events about one order go to one partition and stay in order |
| Value | JSON of the event: order id, customer id, amount, currency | Readable by every consumer without calling the order service |
| Partitions | 6 | Lets up to six consumers of one group work in parallel |
| Replication factor | 3 | The topic survives the loss of a broker |
| Retention | 7 days (the default) | A reader can be down, or a bug can go unnoticed, for days and nothing is lost |
| Consumer groups | payment-service, inventory-service, notification-service, analytics | Each team reads every order, at its own pace |
Now the four benefits become concrete. The order service does not change when the fraud team adds a fifth group next quarter (decoupling). During a sale, orders arrive faster than the notification service can send emails; its lag grows and it catches up afterwards while checkout is unaffected (buffering). When the analytics job is found to have miscounted since Tuesday, it is fixed and re-run from Tuesday's offsets (replay). And all of this happens on one stream that the order service writes exactly once (many readers).
The academy's example project builds exactly this system: Order Service → Kafka → Payment Service → Inventory Service → Notification Service.
Java and Spring Boot
The smallest useful pair: one class that publishes an event and one that receives it, in a single Spring Boot application. Add the spring-kafka dependency (Spring Boot manages its version) and start a local broker with docker run -p 9092:9092 apache/kafka:4.4.0.
package com.example.orders;/** * A fact: an order was placed. Immutable, named in the past tense. * amountMinor is the amount in the currency's minor unit (cents), to avoid floating point. */public record OrderPlaced(String orderId, String customerId, long amountMinor, String currency) {}Why it is written this way
- A record is a good fit for an event: it is immutable, and an event describes something that has already happened and cannot change.
- The order id is inside the value as well as being the key. Consumers should not have to depend on the key to understand the event.
package com.example.orders;import java.util.concurrent.CompletableFuture;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.support.SendResult;import org.springframework.stereotype.Component;@Componentpublic class OrderEventPublisher { private static final Logger log = LoggerFactory.getLogger(OrderEventPublisher.class); static final String TOPIC = "orders"; private final KafkaTemplate<String, OrderPlaced> kafkaTemplate; public OrderEventPublisher(KafkaTemplate<String, OrderPlaced> kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; } public CompletableFuture<SendResult<String, OrderPlaced>> publish(OrderPlaced event) { // topic, key, value: the key decides the partition. return kafkaTemplate.send(TOPIC, event.orderId(), event) .whenComplete((result, ex) -> { if (ex != null) { log.error("OrderPlaced {} was NOT stored by Kafka", event.orderId(), ex); return; } var metadata = result.getRecordMetadata(); log.info("OrderPlaced {} stored at {}-{} offset {}", event.orderId(), metadata.topic(), metadata.partition(), metadata.offset()); }); }}Why it is written this way
sendis asynchronous: it returns aCompletableFutureat once and the record is sent in the background. Until that future completes you do not know whether Kafka has the record.- The key is the order id, so every event for one order goes to the same partition and is read in the order it was written.
- The callback logs the partition and offset the broker assigned. Those two numbers are the record's permanent address.
- Failures are logged here to keep the example small. A real service must decide what a failed send means for the order: see Acks, retries and idempotence.
package com.example.orders;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderEventListener { private static final Logger log = LoggerFactory.getLogger(OrderEventListener.class); @KafkaListener(topics = "orders", groupId = "payment-service") public void onOrderPlaced(ConsumerRecord<String, OrderPlaced> record) { OrderPlaced event = record.value(); log.info("payment-service read {} from partition {} offset {}: charge {} {}", event.orderId(), record.partition(), record.offset(), event.amountMinor(), event.currency()); }}Why it is written this way
groupIdnames the consumer group. Start a second copy of the application with the same group id and the two copies share the partitions. Use a different group id and that copy gets every record as well.- Taking a
ConsumerRecordexposes the partition and offset, which is useful while learning. A listener can also take theOrderPlacedvalue directly. - Spring's listener container runs the poll loop and commits offsets after the listener returns. It sets the client's
enable.auto.committo false and does the committing itself.
package com.example.orders;import org.apache.kafka.clients.admin.NewTopic;import org.springframework.boot.ApplicationRunner;import org.springframework.boot.SpringApplication;import org.springframework.boot.autoconfigure.SpringBootApplication;import org.springframework.context.annotation.Bean;import org.springframework.kafka.config.TopicBuilder;@SpringBootApplicationpublic class OrdersApplication { public static void main(String[] args) { SpringApplication.run(OrdersApplication.class, args); } /** Created on startup if it does not exist. One replica is only acceptable on a local single broker. */ @Bean NewTopic ordersTopic() { return TopicBuilder.name("orders").partitions(3).replicas(1).build(); } /** Publishes three events at startup so there is something to read. */ @Bean ApplicationRunner demo(OrderEventPublisher publisher) { return args -> { publisher.publish(new OrderPlaced("order-1042", "customer-7", 4999, "EUR")); publisher.publish(new OrderPlaced("order-1043", "customer-9", 1250, "EUR")); publisher.publish(new OrderPlaced("order-1042", "customer-7", 4999, "EUR")); }; }}Why it is written this way
- Declaring the topic as a
NewTopicbean makes the partition count an explicit decision. Relying on broker auto-creation gives you whatever the broker defaults are: one partition and one replica. order-1042is sent twice on purpose. Both records land in the same partition, and the log shows two entries: Kafka stores what it is given and does not notice that the business event is the same.
spring: kafka: bootstrap-servers: localhost:9092 producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer properties: "[spring.json.add.type.headers]": false consumer: group-id: payment-service auto-offset-reset: earliest key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer properties: "[spring.json.value.default.type]": com.example.orders.OrderPlaced "[spring.json.trusted.packages]": com.example.ordersWhy it is written this way
- Kafka stores bytes. The serializers turn the key and value into bytes on the way in, and the deserializers turn them back on the way out. Both sides must agree on the format.
auto-offset-reset: earliestonly matters when the group has no committed offset yet. It makes a brand-new group start from the oldest retained record; without it the default islatestand the group would miss everything written before it first connected.- Type headers are switched off and the consumer is told which class to build, so the consumer does not depend on the producer's Java class name.
Configuration
Only a handful of settings are needed to get a first record through. Defaults are from the Apache Kafka 4.4 documentation; the client defaults listed are the same in the 3.9 client that Spring Boot 3.5 manages.
bootstrap.serversDefault none (required)
- What it does
- One or more broker addresses the client uses for its first connection. From there it discovers the whole cluster and which broker leads each partition.
- When to change it
- List two or three brokers in production so that the client can still start when one of them is down. In Spring Boot:
spring.kafka.bootstrap-servers.
group.idDefault null
- What it does
- The consumer group this consumer belongs to. Consumers with the same value split the partitions and share committed offsets.
- When to change it
- Use one stable name per application that should process each record once, such as
payment-service. A different name means an independent reader with its own position.
auto.offset.resetDefault latest
- What it does
- Where to start when the group has no committed offset for a partition, or the committed offset no longer exists on the broker.
earlieststarts at the oldest retained record,latestat the end. - When to change it
- Choose
earliestwhen a new consumer must not miss records written before it first started. It has no effect once the group has a committed offset.
acksDefault all
- What it does
- How many replicas must have the record before the broker acknowledges the produce request.
allwaits for every replica that is currently in sync. - When to change it
- Leave it at
all. It is only a real durability guarantee together with replication andmin.insync.replicas, which the producers path explains.
retention.ms (topic)Default 604800000 (7 days)
- What it does
- How long a record is kept before its segment may be deleted, when the topic uses the default
deletecleanup policy. It does not depend on whether anyone has read the record. - When to change it
- Raise it when consumers may need to replay further back or may be offline for longer; lower it when disk matters more than history.
num.partitions (broker)Default 1
- What it does
- The partition count given to a topic that is created without one, including topics the broker auto-creates on first use.
- When to change it
- Do not rely on it. Create topics explicitly with a partition count chosen for the parallelism you need, because changing it later moves keys to different partitions.
auto.create.topics.enable (broker)Default true
- What it does
- Whether the broker creates a topic the first time a client asks for one that does not exist.
- When to change it
- Convenient locally. Many teams turn it off in production so that a typo in a topic name fails loudly and does not silently create a one-partition, one-replica topic.
Hands-on exercise
Find your record in the cluster
Use the cluster explorer to see that a topic is a set of partition logs spread over brokers, and that the data stays put when a broker stops.
Steps · 0 of 6 done
A three-broker cluster holding the orders topic
Simulation · not a real clusterClick a broker, the topic or a partition to inspect it. Then stop the broker that leads a partition and see who takes over.
What you should see
Each partition has exactly one leader and two followers, and leadership is spread over the brokers. Stopping a leader's broker moves leadership of that partition to another broker that holds a copy, and the partition stays available. In the Spring Boot run, both order-1042 records report the same partition with consecutive or increasing offsets, and order-1043 may be in a different partition.
What can go wrong
A new consumer sees nothing
- Symptom
- The producer logs successful sends, the listener starts without errors, and it never receives the records that were already written.
- Cause
- The group has no committed offset, and
auto.offset.resetdefaults tolatest, so it starts at the end of each partition and only sees records written after it joined. - Fix
- Set
auto.offset.resettoearliestfor consumers that must read existing records. Remember it only applies while the group has no committed offset; to move an existing group, reset its offsets.
Expecting one order across the whole topic
- Symptom
- With three partitions, a consumer logs
order-1043beforeorder-1042although they were sent the other way round. - Cause
- Ordering exists inside a partition only. The two orders have different keys, hashed to different partitions, and each partition is read independently.
- Fix
- Give records that must stay in order the same key. If you truly need one total order, that means one partition, and with it one consumer per group.
Two copies of a service with different group ids
- Symptom
- After scaling the payment service to two instances, every order is charged twice.
- Cause
- Each instance was given its own
group.id, for example by including the hostname. Kafka treats them as two independent readers and delivers every record to both. - Fix
- All instances of one service share one fixed group id. Different group ids are for different services that each need the full stream.
Treating the topic as a database table
- Symptom
- Code that needs the current state of order 1042 reads the topic from the beginning and filters, and gets slower every week.
- Cause
- A log can only be read in sequence from an offset. There is no lookup by key and no query.
- Fix
- Consume the topic into a store built for lookups (a database table, a cache) and query that. Kafka carries the changes; something else serves the reads.
How to troubleshoot
Not sure the topic exists or how it is laid out
Check
bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
- How to read it
- The first line shows
PartitionCountandReplicationFactor. Each following line is one partition with itsLeader,ReplicasandIsr. A topic with one partition and one replica that you did not create on purpose was auto-created by the broker.
Not sure the records reached Kafka at all
Check
bin/kafka-console-consumer.sh --topic orders --from-beginning --bootstrap-server localhost:9092
- How to read it
- This reads the topic from the oldest retained record without touching your application's group. If records appear here, the producer side works and the problem is in your consumer or its position.
The listener is running but seems stuck or behind
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
- How to read it
- One row per partition.
CURRENT-OFFSETis the group's committed position,LOG-END-OFFSETis the end of the partition, andLAGis the difference. Lag of 0 on every partition means the group has committed up to the end of the log.
The application hangs at startup or the first send fails after a minute
Check
The client log for a line such as `Connection to node -1 (localhost/127.0.0.1:9092) could not be established`
- How to read it
- The client cannot reach the address in
bootstrap.servers. Check that the broker is running and that the address is reachable from where the application runs, which inside a container is notlocalhost.
Key takeaways
- Kafka is an append-only log, split into partitions and copied across brokers. Records get a permanent offset inside their partition.
- Producers and consumers never talk to each other. The producer is done when the brokers have stored the record.
- Reading does not delete. Records leave when retention removes them, seven days by default, not when someone consumes them.
- A consumer group's state is one offset per partition: the next record to read. Same group id means shared work; different group id means an independent reader.
- Order is guaranteed inside a partition only. The key decides the partition, so the key is how you choose what stays in order.
- It was built for many systems reading the same high-volume stream at different speeds, and that is still what it is best at.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
The orders topic has 6 partitions and 7-day retention. The payment-service group has consumed it for months with zero lag. A fraud team now wants to score every order, starting with the orders from the last three days, without any risk to payments.
How should the fraud service get the data?
Scenario 2 of 3
Two instances of the notification service run with group.id set to notification- plus the pod name. Customers report receiving every order confirmation email twice. The topic has 3 partitions and the producer logs show each order was sent once.
What is the fix?
Scenario 3 of 3
A loyalty-points consumer shipped a bug on Monday at 09:00 that awarded half the correct points. It is now Wednesday. The fix is ready. The orders topic keeps 7 days of records, and the points table can be recalculated per order.
What is the most direct way to repair the data?
References
References: Apache Kafka documentation: Introduction ↗ · Apache Kafka 4.4 documentation: Design ↗ · Apache Kafka 4.4 documentation: Quickstart ↗ · Apache Kafka 4.4 documentation: Consumer configs ↗ · Apache Kafka 4.4 documentation: Topic configs ↗ · Apache Kafka 4.4 documentation: Broker configs ↗ · Kreps, Narkhede, Rao: Kafka: a Distributed Messaging System for Log Processing (NetDB 2011) ↗ · Jay Kreps: The Log: What every software engineer should know about real-time data's unifying abstraction (LinkedIn Engineering, 2013) ↗ · Spring for Apache Kafka 3.3 reference: Quick Tour ↗ · Spring Boot 3.5 reference: Apache Kafka Support ↗ · Report a problem