Skip to content
Interview rounds

Databases & caching · Production incident

The query that got ten times slower overnight

A production query is suddenly ten times slower and nothing was deployed. How do you find out why and bring it back?

BackendSenior12 min
Interview-style scenarioLast reviewed: Checked against official documentation

Wednesday 09:40 · orders database (PostgreSQL)Illustrative scenario, not a real incident

Order search p99 has been 900 ms since 09:10. Last week it was 80 ms

The order search endpoint runs one query against the orders table, which holds about 40 million rows. It filters by customer and status and returns the 20 newest.

Since 09:10 that query has been more than ten times slower. Database CPU has gone from 35% to 85%, and other endpoints that share the database are starting to slow down as well.

What the dashboards and logs show

Order search p99
900 ms (was 80 ms)
Database CPU
85% (was 35%)
Releases
none since Monday
Request rate
the same as last Wednesday

Clarify · optional · scored under "Hypothesis before change"

Ask the interviewer

0 of 3 asked

Each question reveals a fact. What you ask before you act is part of your score.

First move · scored under "Mitigate before root cause"

Other endpoints are being dragged down while you look. What do you do first?

Your rubric score, the model answer, the follow-up questions and the related lessons appear once you have answered the last step.