Skip to content

Path 8 of 10

Performance and Scalability

Throughput, latency, partition counts, hot partitions, backpressure and scaling the cluster.

0 / 5 lessons completed ยท about 132 min in total

After this path you can

Outcomes

  • Find the real bottleneck before tuning
  • Size partitions and consumers for a target rate
  • Tune producers and consumers for a high-volume stream

Lessons

  1. 1Throughput and latencyWhat sets each of them in a Kafka pipeline, why raising one usually costs the other, where the milliseconds go end to end, and how to measure with percentiles.Intermediate24 min
  2. 2Partition sizing and consumer parallelismHow to choose a partition count from a target rate and a processing time, what too many partitions cost, why the count is hard to change, and how to add parallelism safely.Intermediate26 min
  3. 3Tuning producers and consumer fetchingA procedure for tuning batching, compression, buffers, fetch sizes and large messages: calculate first, change one setting at a time, and measure.Advanced26 min
  4. 4Hot partitions and backpressureWhy one partition falls behind while the rest are idle, how to find and fix a skewed key, and how a pull-based consumer protects a slow database instead of flooding it.Advanced28 min
  5. 5Cluster resources and scalingHow to size disk, network, memory and CPU for a Kafka cluster from throughput and retention, and what adding brokers, consumers or tiered storage really changes.Advanced28 min

Related labs

Simulations, not a real cluster.

  1. Lab 1Producer publishing messagesFoundation20 min
  2. Lab 2Partition key and orderingFoundation20 min
  3. Lab 3Consumer group assignmentFoundation20 min
  4. Lab 4Adding and removing consumersIntermediate25 min
  5. Lab 9Broker failure and leader electionAdvanced30 min
  6. Lab 10Consumer lag during high loadIntermediate30 min
  7. Lab 11Hot partitionAdvanced35 min
  8. Lab 13Database unavailable while consuming Kafka messagesAdvanced35 min