Skip to content

Kafka / streaming · 5 of 9

Step 1 · The pattern

A second copy, far away

Producers write to the log in region A and the consumer reads it there. A replication job copies every message to region B, so the data survives if region A is lost.

  • message
  • copy to region B
  • processed

In short

Offsets across regions: how it works and how it breaks

A consumer remembers its place in a topic as an offset, which is a position in one particular log. To survive losing a region, teams copy topics to a second region with a replication job. The copy holds the same messages under different offsets, because each log numbers from its own start and copied messages can arrive in a different order. An offset committed in one region is therefore meaningless in the other. A consumer that fails over cannot start from the newest message, because its unprocessed backlog would be silently skipped. Starting from the oldest replays days of data. The fix is to translate. While copying, the replication job records checkpoints that pair a source offset with its destination offset, and a sync job keeps each consumer's position translated. On failover the consumer resumes slightly before the translated position: a few messages repeat and none are lost.

How this comes up in interviews

"Your Kafka topics are replicated to a second region. The primary region fails. Where do consumers start reading in the other region?"

Then they ask

  1. Why can the consumer not reuse the offset it committed in the failed region?
  2. The translated position is never exact. Which way do you round, and what must the handler tolerate?
  3. What happens to messages written shortly before the failure that the replication job had not copied?

A strong answer covers

  • An offset is a position in one particular log. The copy numbers messages from its own start.
  • Starting from the newest message silently skips the unprocessed backlog. Starting from the oldest replays days of data.
  • The replication job records checkpoints pairing source offsets with destination offsets, and a sync job keeps positions translated.
  • Resume slightly before the translated position: a few messages repeat and none are lost.
  • Repeats are the at-least-once deal consumers already handle, so an idempotent handler makes them harmless.

Points to cover, not a script. Say them in your own words and tie them to a system you have worked on.

On one page: the track cheat sheet.

Last updated: Official documentation

Diagrams are our own simulations: the timings, rates and counts in them are illustrative, not measurements. Captions are original wording.