Kafka & events · Production incident
A producer release breaks its readers
After a producer release, one consumer of an event fails on every record and another reports odd figures. How should the event be changed safely?
Tuesday 11:40 · order-placedIllustrative scenario, not a real incident
Invoicing: every order-placed event has failed since 11:00; lag is 2,400 and rising
The orders service publishes an order-placed event for each order, encoded with Avro. Two other teams read it: invoicing creates an invoice, and analytics feeds the revenue dashboard.
At 11:00 the orders team released a new version of their service. Since then invoicing has logged an error for every event and created no invoices. Analytics has no errors, and its dashboard shows every order since 11:00 in pounds.
What the dashboards and logs show
- Invoicing errors
- one for each event since 11:00, none before
- Invoices created since 11:00
- 0
- Analytics errors
- 0
- Revenue by currency since 11:00
- 100% GBP, against about 70% on a normal day
- Deploys today
- orders service at 11:00; neither consumer
Clarify · optional · scored under "Hypothesis before change"
Ask the interviewer
0 of 3 asked
Each question reveals a fact. What you ask before you act is part of your score.
First move · scored under "Mitigate before root cause"
No invoices are being created and the dashboard is filling with pounds. What do you do first?
Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.