Skip to content
All outages

Incident · Hard · Kafka / streaming

The offset that went backwards

  • SEV2
  • DUPLICATE EMAILS
  • CONSUMERS REWOUND
90s

The situation

Members receive the same emails again, some of them days old. The consumer that sends email restarted a short while ago, after the broker holding its group's offsets changed. Producers show no retries. On the consumer's dashboard, the committed position has jumped backwards.

The evidence

Consumer committed offsetillustrative
  • offset
Logs and signalsReconstructed
11:02offsets partition moved to new broker · loading cache (17 min)
11:05stale-offset cleanup runs against half-loaded cache
11:05cleanup: tombstone written for a current checkpoint
11:20consumer: fetch committed offset → −1
11:20consumer: no offset → reset to earliest · reprocessing
-- producer retries: 0

Your call

Why were messages processed twice?

Charts and logs are reconstructions. Aim for 90 seconds; going over costs nothing.

Last updated: Engineering write-up