Skip to content
Interview rounds

Kafka & events · Production incident

A broker died and took orders with it

A broker fails and some acknowledged records are gone. What do you configure so that a broker failure cannot lose an acknowledged write?

Platform-SRESenior14 min
Interview-style scenarioLast reviewed: Checked against official documentation

Thursday 08:40 · checkout-serviceIllustrative scenario, not a real incident

Reconciliation: 340 orders confirmed to customers last night never reached fulfilment

The checkout saves each order to its own database, publishes it to the orders topic, and confirms to the customer once the send succeeds. Fulfilment reads the topic.

At 02:14 one of the six brokers lost power. The cluster kept serving and the checkout logged no failed sends. The morning reconciliation found 340 orders in the checkout database that fulfilment never received.

What the dashboards and logs show

Missing orders
340, all sent between 02:13:58 and 02:14:03
Checkout send errors
none overnight
Broker 3
down at 02:14, back at 02:51
Where they were sent
partitions 4 and 9 of `orders`, out of 12

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

340 customers hold a confirmation for an order that fulfilment has never seen. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.