Incident · Hard · Kafka / streaming
One cluster, two truths
- SEV1
- 21:30 UTC
- ~10% OF JOB WRITES FAIL
90s
The situation
Your background jobs run on a Kafka cluster. Tonight about one in ten job writes fails, and clients disagree about which broker leads which partition. Every broker process is up and the network checks are clean. The only change in progress is routine: the ZooKeeper hosts behind the cluster are being replaced with new ones as part of an upgrade.
The evidence
- failed writes
- brokers up
Logs and signalsReconstructed
21:12zk upgrade: replacement hosts joining21:28zk: leader elected among the new hosts21:29zk: old ensemble still has its own leader21:30broker-4: session opened to new zk hosts21:30broker-4: elected as controller21:31controllers reported by brokers: 221:32job client: not leader for partition · retrying-- brokers down: 0 · packet loss: 0%
Your call
What is the cause?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.