Skip to content
All outages

Editor's note: A job queue, not Kafka: the lesson carries over.

Incident · Easy · Kafka / streaming

The message that kills its reader

  • SEV1
  • 15:40
  • APP CRASH LOOP
90s

The situation

Your app is one binary that serves web requests and also works through a queue of background events. At 15:40 the whole process dies. You restart it, it serves traffic for a short while, then dies again. And again. There was no deploy. The platform's logs cut off just before each crash, so there is no stack trace to read.

The evidence

Healthy app instancesillustrative
  • instances up
Logs and signalsReconstructed
15:40platform: app crashed
15:42manual restart · up
15:44platform: app crashed
15:45logs: lines dropped by rate limit
15:50queue: oldest unacked message from 15:39
15:51queue: same message redelivered
-- deploys today: 0 · database: healthy

Your call

What keeps taking the app down?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem