Incident · Hard · Kafka / streaming
The offset that went backwards
- SEV2
- DUPLICATE EMAILS
- CONSUMERS REWOUND
90s
The situation
Members receive the same emails again, some of them days old. The consumer that sends email restarted a short while ago, after the broker holding its group's offsets changed. Producers show no retries. On the consumer's dashboard, the committed position has jumped backwards.
The evidence
- offset
Logs and signalsReconstructed
11:02offsets partition moved to new broker · loading cache (17 min)11:05stale-offset cleanup runs against half-loaded cache11:05cleanup: tombstone written for a current checkpoint11:20consumer: fetch committed offset → −111:20consumer: no offset → reset to earliest · reprocessing-- producer retries: 0
Your call
Why were messages processed twice?
Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.