Skip to content

Resilience patterns · 9 of 9

Step 1 · The pattern

A nightly job

Users send 5 queries a second; each takes 150 ms, and the database runs 3 at a time. At midnight one nightly job starts: a report that holds a connection for about 3 seconds. It takes one slot, and users barely notice. (All numbers are illustrative.)

  • user query
  • batch job
  • served

In short

Scheduled job stampede: how it works and how it breaks

Daily jobs are usually scheduled for midnight, so reports, clean-ups, exports and emails all start in the same minute and hit the same database. Worse, when the app is scaled out and every copy carries the same cron line, each job runs once per copy. Batch work fills the database's connections and users' ordinary queries queue behind it, time out and retry. The fix has three parts. Run each job once, from a scheduler or behind a lock, not from every copy of the app. Spread start times: give each job its own minute, for example by hashing its name, instead of 00:00 for all. And make jobs safe to run twice, because schedulers can launch a job twice or skip it. Give batch work its own connection limit, or a replica, so it can never starve users.

What this simulation simplifies

  • Midnight is compressed to a single moment and each batch job holds one database connection for about three seconds; real jobs run for minutes or hours and may hold many connections.
  • The database is modelled as three connections with a queue; real databases also share CPU, disk and locks between batch and user work.
  • A user query that finds six already running or queued is refused immediately, standing in for a client timeout.

How this comes up in interviews

"Your service has periodic jobs: nightly reports, hourly clean-ups, a daily email. How do you run them reliably at scale?"

Then they ask

  1. The app runs on twenty servers. Where does the cron schedule live?
  2. The scheduler fails over halfway through a run. What can go wrong?
  3. A job sometimes takes longer than its interval. What do you do?

A strong answer covers

  • Run each job once, from a central scheduler or with a lock or leader election, not from every server's crontab.
  • Spread start times (hash or jitter) instead of scheduling everything at midnight.
  • Make jobs idempotent, and decide per job whether skipping or double running is the safer failure.
  • Prevent overlapping runs (for example, a forbid concurrency policy) and alert on runs that take too long.
  • Isolate batch work from user traffic: a separate pool, a replica, or a rate limit.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last reviewed: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.