Incident · Medium · Kafka / streaming · Redis
Too full to empty
- SEV1
- JOB QUEUE FULL
- DATABASE RECOVERED
90s
The situation
Background jobs wait in an in-memory Redis queue until workers pick them up. A slowdown in the database made every job take longer, and the queue grew until Redis hit its memory limit. New jobs are now refused, so anything that needs one fails. The database problem has been fixed and the workers are healthy and idle. The queue is not draining at all.
The evidence
- queue memory (%)
- jobs completed
Logs and signalsReconstructed
10:05database: resource contention · jobs slow10:20queue depth climbing10:41redis: maxmemory reached10:41enqueue: refused · callers failing10:58database: recovered11:00workers: dequeue → error, out of memory11:00workers idle · queue depth unchanged
Your call
Why is the queue not draining?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.