Skip to content
Practice

SEV-1 simulationPlatform and OS · Real outage · Reconstructed data · 15–20 minutes

Incident #009 — Bigger boxes didn't help

Customer impact

Players online have fallen to half of normal. The Consul cluster everything depends on is slow and unstable.

SignalCurrent value
Players online≈50% of normal
Consul KV write p50≈2 s (normal <300 ms)
Consul leaderout of sync with voters
Schedulingfailing
Vaultdegraded
Consul server CPUhigh on the leader

Your task

You join the incident on the first evening. Work out why Consul is slow, using the service map, metrics, logs, traces and recent changes. Choose what to do, then name the cause and the permanent fix.

16:40Impact continuing · +0 minabout 0 thousand player sessions lost lost since you joined

Investigate · each first look costs 3 simulated minutes

What do you want to look at?

Nothing open yet. Pick a view above. You do not have to open them all before you act.

Step 1 · mitigate · you can act at any time

Consul is slow and half the players have dropped. What do you do?