Skip to content
All outages

Incident · Medium · Kafka / streaming · Redis

Too full to empty

  • SEV1
  • JOB QUEUE FULL
  • DATABASE RECOVERED
90s

The situation

Background jobs wait in an in-memory Redis queue until workers pick them up. A slowdown in the database made every job take longer, and the queue grew until Redis hit its memory limit. New jobs are now refused, so anything that needs one fails. The database problem has been fixed and the workers are healthy and idle. The queue is not draining at all.

The evidence

Job queueillustrative
  • queue memory (%)
  • jobs completed
Logs and signalsReconstructed
10:05database: resource contention · jobs slow
10:20queue depth climbing
10:41redis: maxmemory reached
10:41enqueue: refused · callers failing
10:58database: recovered
11:00workers: dequeue → error, out of memory
11:00workers idle · queue depth unchanged

Your call

Why is the queue not draining?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Engineering write-up