Skip to content
All outages

Incident · Hard · Kafka / streaming

One cluster, two truths

  • SEV1
  • 21:30 UTC
  • ~10% OF JOB WRITES FAIL
90s

The situation

Your background jobs run on a Kafka cluster. Tonight about one in ten job writes fails, and clients disagree about which broker leads which partition. Every broker process is up and the network checks are clean. The only change in progress is routine: the ZooKeeper hosts behind the cluster are being replaced with new ones as part of an upgrade.

The evidence

Job enqueue failures (%)illustrative
  • failed writes
  • brokers up
Logs and signalsReconstructed
21:12zk upgrade: replacement hosts joining
21:28zk: leader elected among the new hosts
21:29zk: old ensemble still has its own leader
21:30broker-4: session opened to new zk hosts
21:30broker-4: elected as controller
21:31controllers reported by brokers: 2
21:32job client: not leader for partition · retrying
-- brokers down: 0 · packet loss: 0%

Your call

What is the cause?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem