Skip to content
All outages

Editor's note: A log pipeline, not Kafka: the lesson carries over.

Incident · Medium · Kafka / streaming

Reverted in five minutes

  • SEV1
  • LOG DELIVERY
  • ~55% OF LOGS LOST
90s

The situation

Your pipeline delivers logs to customers who have set up an export job. A forwarder at the edge picks out those customers' events, and a buffering tier keeps one buffer per job until a pusher ships it. A change produced a blank forwarder config. You caught it and reverted in under five minutes. Yet the buffering tier is now so overloaded it barely responds, and more than half of all logs are not being delivered.

The evidence

Buffering tierillustrative
  • buffers (× normal)
  • logs delivered (%)
Logs and signalsReconstructed
T+0 forwarder config regenerated: 0 export jobs
T+0 forwarder: no usable config → failsafe: forward all
T+2m buffer tier: new buffers ~40× normal
T+4m config change reverted
T+6m forwarder: config OK · filtering restored
T+30mbuffer tier: unresponsive · admin calls time out
-- buffer tier overload limits: not configured

Your call

Why did a five-minute mistake cause hours of loss?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Official postmortem