Skip to content
Chaos Lab

Challenge 35 · intermediate · Search fan-out

The P99 problem

A search fans out to 20 index shards and waits for all of them. Each shard answers in about 8 ms, but from 5 seconds on, each one pauses for 400 ms about 1% of the time (garbage collection, compaction).

From 5 to 60 seconds, keep p99 search latency at or under 100 ms and answer at least 99% of searches.

ready00:00 / 01:00Success · 10 s–p99 · 10 s–

Goal · measured 00:05–01:00

0 changes · par 1
  • p99 search latency ≤ 100 msnot measured yet
  • Searches answered ≥ 99.0%not measured yet

This is a simulation. Its numbers come from a model of a typical system, not from measuring real servers. The same link and seed always give the same run. How the Chaos Lab works.