Kafka & events · Production incident
Order saved, event missing
A service saves to its database and publishes an event to Kafka as two separate writes, and they drift apart. How do you keep them consistent?
Monday 09:40 · order-serviceIllustrative scenario, not a real incident
Fulfilment: 46 paid orders from Sunday night were never sent to the warehouse
The order service stores each order in PostgreSQL and publishes order-created to Kafka. The shipping service consumes that topic and creates a shipment for the warehouse.
All 46 orders exist in the database with status paid. The shipping service has no record of them, and customers are asking where their parcels are.
What the dashboards and logs show
- Orders affected
- 46, all placed between 02:10 and 02:19 on Sunday night
- Orders table
- all 46 present, paid, and returned 201 to the customer
- order-created topic
- no record for any of the 46 order ids
- Shipping service
- no errors, no lag
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
Forty-six customers have paid and nothing is being shipped. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.