Kafka & events · Production incident
Two consumers, one partition
Two consumers in one group acted on the same records at the same time. Who owns the partition, and how do you stop the other one's side effects?
Tuesday 11:40 · payout-workerIllustrative scenario, not a real incident
Finance: 23 sellers were paid twice this morning, within three minutes
The payout worker reads payout-approved with six consumers. For each record it calls the bank's transfer API, writes a row to payouts_sent, and commits its offsets after each batch.
Each of the 23 payouts was sent once by one pod and once by another, between 11:03 and 11:05. No pod restarted and nothing was deployed. The platform team was restarting brokers one at a time from 10:55 to 11:10.
What the dashboards and logs show
- Double payouts
- 23, all from partition 7 of `payout-approved`
- Senders
- payout-2 and payout-5, for the same 23 records
- Pods
- 6 of 6 running, no restarts today
- The topic
- each of the 23 records appears once
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Sellers have been paid twice, and the next long pause would do it again. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.