Path 2 of 10
Kafka Architecture and Internals
Brokers, topics, partitions, replication, the log on disk, and how KRaft runs the cluster.
0 / 8 lessons completed ยท about 157 min in total
After this path you can
Outcomes
- Trace a record from producer to disk to consumer
- Predict what happens when a broker fails
- Read partition, replica and ISR state
Lessons
- 1Clusters, brokers and topicsWhat a broker does, how brokers form a cluster, how a client gets from a bootstrap address to the right partition leader, and what a topic really is.Foundation14 min
- 2Partitions and offsetsA partition is an ordered, append-only log and an offset is a position in it. Learn the four offsets that matter and why the partition is the unit of ordering, parallelism and placement.Foundation16 min
- 3Replication: leaders and followersHow each partition is copied to several brokers, what the leader and followers each do, what the leader epoch is for, and what a client sees when leadership moves.Intermediate17 min
- 4In-sync replicas, min.insync.replicas and the high watermarkExactly what in sync means, how the ISR shrinks and grows, how the high watermark follows it, and what acks=all with min.insync.replicas really guarantees.Intermediate18 min
- 5Log segments and retentionHow a partition is stored as segment files with indexes, when segments roll, how time and size retention delete them, and what a consumer sees when its offset is gone.Intermediate22 min
- 6Log compactionHow a compacted topic keeps at least the latest record per key, how the log cleaner does it, what tombstones are, and what compaction does not promise.Intermediate22 min
- 7KRaft: the controller quorum and the metadata logHow a Kafka 4 cluster manages itself: a Raft quorum of controllers keeps a replicated metadata log, brokers follow it, and none of it sits in the data path.Intermediate24 min
- 8Partition leadership and leader electionWho picks a new partition leader when a broker fails, why leader epochs keep replicas consistent, what unclean election costs, and what clients do meanwhile.Intermediate24 min
Related labs
Simulations, not a real cluster.