Step 1 · The pattern
Services, and what they run on
A request goes through three services in a row: WEB, then API, then STOCK. Each runs on resources: CPU, memory, a pool of workers. A batch host works flat out next to them, but no request ever touches it.
- request
- answered
- batch job
In short
RED and USE methods: how it works and how it breaks
RED and USE are two short checklists for two kinds of thing. RED is for services, seen from the caller's side: Rate (requests per second), Errors (how many fail) and Duration (how long they take). USE is for resources such as CPUs, memory, disks, pools and queues: Utilisation (how busy), Saturation (how much work is waiting) and Errors (the resource's own error events). In an incident the order matters. Read RED for each service along the request path first. Services above the fault inherit its errors and slowness, and services below it look normal with less traffic, so the last bad hop is the place to open. Then run USE on that one service's resources to find what is full. Starting with host dashboards asks which machine is busy. There is always one, and it is often innocent.
How this comes up in interviews
"Checkout is failing and you have dashboards for every service and every host. Where do you look first, and in what order?"
Then they ask
- A service further down the path shows no errors and half its usual traffic. What does that tell you?
- One host is at very high CPU during the incident. How do you decide whether it matters?
- The last bad hop shows idle CPU. Which of its resources do you check next, and for what?
A strong answer covers
- RED is for services, seen from the caller's side: rate, errors, duration.
- USE is for resources: utilisation, saturation and the resource's own error events.
- Read RED for each service along the request path first. Services above the fault inherit its errors and slowness.
- Services below the fault look normal with less traffic, so the last bad hop is the place to open.
- Then run USE on that one service's resources to find what is full, such as a worker pool with a queue.
Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.
On one page: the track cheat sheet.
Sources: The USE Method (Brendan Gregg) ↗ · The RED Method: How to Instrument Your Services (Grafana Labs, 2018) ↗ · Report a problem
Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.