Skip to content

Path 2 of 10

Kafka Architecture and Internals

Brokers, topics, partitions, replication, the log on disk, and how KRaft runs the cluster.

0 / 8 lessons completed ยท about 157 min in total

After this path you can

Outcomes

  • Trace a record from producer to disk to consumer
  • Predict what happens when a broker fails
  • Read partition, replica and ISR state

Lessons

  1. 1Clusters, brokers and topicsWhat a broker does, how brokers form a cluster, how a client gets from a bootstrap address to the right partition leader, and what a topic really is.Foundation14 min
  2. 2Partitions and offsetsA partition is an ordered, append-only log and an offset is a position in it. Learn the four offsets that matter and why the partition is the unit of ordering, parallelism and placement.Foundation16 min
  3. 3Replication: leaders and followersHow each partition is copied to several brokers, what the leader and followers each do, what the leader epoch is for, and what a client sees when leadership moves.Intermediate17 min
  4. 4In-sync replicas, min.insync.replicas and the high watermarkExactly what in sync means, how the ISR shrinks and grows, how the high watermark follows it, and what acks=all with min.insync.replicas really guarantees.Intermediate18 min
  5. 5Log segments and retentionHow a partition is stored as segment files with indexes, when segments roll, how time and size retention delete them, and what a consumer sees when its offset is gone.Intermediate22 min
  6. 6Log compactionHow a compacted topic keeps at least the latest record per key, how the log cleaner does it, what tombstones are, and what compaction does not promise.Intermediate22 min
  7. 7KRaft: the controller quorum and the metadata logHow a Kafka 4 cluster manages itself: a Raft quorum of controllers keeps a replicated metadata log, brokers follow it, and none of it sits in the data path.Intermediate24 min
  8. 8Partition leadership and leader electionWho picks a new partition leader when a broker fails, why leader epochs keep replicas consistent, what unclean election costs, and what clients do meanwhile.Intermediate24 min

Related labs

Simulations, not a real cluster.

  1. Lab 1Producer publishing messagesFoundation20 min
  2. Lab 2Partition key and orderingFoundation20 min
  3. Lab 3Consumer group assignmentFoundation20 min
  4. Lab 9Broker failure and leader electionAdvanced30 min
  5. Lab 10Consumer lag during high loadIntermediate30 min