Three replicas of an API run on a three-node cluster; two can carry the load, the third is spare. Two replicas happen to sit on node 1. At 10 seconds a cluster upgrade drains each node in turn. A replacement replica takes 15 seconds to be ready.
From 10 to 120 seconds, answer at least 99% of requests and finish upgrading all three nodes.
ready00:00 / 02:00Success · 10 s–p99 · 10 s–
Goal · measured 00:10–02:00
0 changes · par 1
Requests answered ≥ 99.0%not measured yet
Nodes upgraded ≥ 100%not measured yet
Service map
Node upgrade · pausedSelect a component to inspect
SYSTEM BOUNDARYno PDBUsers180 req/sCLIENTCluster3 nodesSERVICEapi replicas3/3 ready · nodes 1,1,2SERVICE
All components healthy.requestretrytrial call (half-open)failed
What changed
Turn on a fault or change a defence while the run plays. Its effect shows here: the 5 seconds before against 5 seconds after it settles.
Service map
Node upgrade · pausedSelect a component to inspect
SYSTEM BOUNDARYno PDBUsers180 req/sCLIENTCluster3 nodesSERVICEapi replicas3/3 ready · nodes 1,1,2SERVICE
SYSTEM BOUNDARYno PDBUsers180 req/sCLIENTCluster3 nodesSERVICEapi replicas3/3 ready · nodes 1,1,2SERVICE
All components healthy.requestretrytrial call (half-open)failed
What changed
Turn on a fault or change a defence while the run plays. Its effect shows here: the 5 seconds before against 5 seconds after it settles.
Dashboard
Requests per second
100/s
Press Start
00:0001:00
Answered–
Failed–
Latency
100 ms
Press Start
00:0001:00
p50–
p95–
p99–
Events and logs
Nothing yet. Faults, timeouts, retries, breaker changes, shedding, cache expiry and your own changes appear here, and as markers on the charts.
Slowest recent request
Press Start to see requests.
This is a simulation. Its numbers come from a model of a typical system, not from measuring real servers. The same link and seed always give the same run. How the Chaos Lab works.