Incident · Medium · Kafka / streaming
Reverted in five minutes
- SEV1
- LOG DELIVERY
- ~55% OF LOGS LOST
90s
The situation
Your pipeline delivers logs to customers who have set up an export job. A forwarder at the edge picks out those customers' events, and a buffering tier keeps one buffer per job until a pusher ships it. A change produced a blank forwarder config. You caught it and reverted in under five minutes. Yet the buffering tier is now so overloaded it barely responds, and more than half of all logs are not being delivered.
The evidence
- buffers (× normal)
- logs delivered (%)
Logs and signalsReconstructed
T+0 forwarder config regenerated: 0 export jobsT+0 forwarder: no usable config → failsafe: forward allT+2m buffer tier: new buffers ~40× normalT+4m config change revertedT+6m forwarder: config OK · filtering restoredT+30mbuffer tier: unresponsive · admin calls time out-- buffer tier overload limits: not configured
Your call
Why did a five-minute mistake cause hours of loss?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.