Skip to content

Path 10 of 10

Production Kafka Engineering

Monitoring, failures, upgrades, security, multi-region replication and schema evolution.

0 / 6 lessons completed ยท about 183 min in total

After this path you can

Outcomes

  • Read the handful of metrics that matter
  • Run upgrades and broker failures without an outage
  • Evolve schemas without breaking consumers

Lessons

  1. 1Monitoring Kafka: the signals that matter firstWhich broker, controller and client metrics tell you a cluster is unhealthy, how they reach a dashboard, and which of them deserve to wake someone up.Advanced26 min
  2. 2Broker failures and incident responseWhat a cluster does when a broker crashes, returns, fills its disk or merely slows down, and a calm sequence for handling the incident, including reassignment.Advanced30 min
  3. 3Rolling upgrades and scaling a clusterHow to restart, upgrade, grow and shrink a Kafka cluster one broker at a time without refusing a write, and which steps cannot be undone.Advanced28 min
  4. 4Kafka security: TLS, SASL and ACLsHow a connection is encrypted, how a client proves who it is, and how ACLs decide what that principal may do, with the exceptions each failure produces.Advanced32 min
  5. 5Multi-region Kafka and MirrorMaker 2Why regions get separate clusters, how MirrorMaker 2 copies topics and translates consumer offsets between them, and what a failover and a failback really involve.Advanced34 min
  6. 6Schema registries and schema evolutionWhat a schema registry stores, what each compatibility mode really promises, which order to deploy in, and how to evolve events safely with or without a registry.Advanced33 min

Related labs

Simulations, not a real cluster.

  1. Lab 1Producer publishing messagesFoundation20 min
  2. Lab 2Partition key and orderingFoundation20 min
  3. Lab 5Duplicate message processingIntermediate25 min
  4. Lab 6Consumer crash before offset commitIntermediate25 min
  5. Lab 7Poison message and dead-letter queueIntermediate25 min
  6. Lab 9Broker failure and leader electionAdvanced30 min
  7. Lab 10Consumer lag during high loadIntermediate30 min
  8. Lab 15Schema evolutionAdvanced35 min