Kafka & events · Production incident
A broker died and took orders with it
A broker fails and some acknowledged records are gone. What do you configure so that a broker failure cannot lose an acknowledged write?
Thursday 08:40 · checkout-serviceIllustrative scenario, not a real incident
Reconciliation: 340 orders confirmed to customers last night never reached fulfilment
The checkout saves each order to its own database, publishes it to the orders topic, and confirms to the customer once the send succeeds. Fulfilment reads the topic.
At 02:14 one of the six brokers lost power. The cluster kept serving and the checkout logged no failed sends. The morning reconciliation found 340 orders in the checkout database that fulfilment never received.
What the dashboards and logs show
- Missing orders
- 340, all sent between 02:13:58 and 02:14:03
- Checkout send errors
- none overnight
- Broker 3
- down at 02:14, back at 02:51
- Where they were sent
- partitions 4 and 9 of `orders`, out of 12
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
340 customers hold a confirmation for an order that fulfilment has never seen. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.