Skip to content
Interview rounds

Kafka & events · Production incident

A producer release breaks its readers

After a producer release, one consumer of an event fails on every record and another reports odd figures. How should the event be changed safely?

BackendSenior14 min
Interview-style scenarioLast reviewed: Checked against official documentation

Tuesday 11:40 · order-placedIllustrative scenario, not a real incident

Invoicing: every order-placed event has failed since 11:00; lag is 2,400 and rising

The orders service publishes an order-placed event for each order, encoded with Avro. Two other teams read it: invoicing creates an invoice, and analytics feeds the revenue dashboard.

At 11:00 the orders team released a new version of their service. Since then invoicing has logged an error for every event and created no invoices. Analytics has no errors, and its dashboard shows every order since 11:00 in pounds.

What the dashboards and logs show

Invoicing errors
one for each event since 11:00, none before
Invoices created since 11:00
0
Analytics errors
0
Revenue by currency since 11:00
100% GBP, against about 70% on a normal day
Deploys today
orders service at 11:00; neither consumer

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

No invoices are being created and the dashboard is filling with pounds. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.