Skip to content

Path 10 · lesson 5 of 6

Advanced34 minApache Kafka 4.4 documentation (MirrorMaker 2 on Kafka Connect, KRaft only) · KIP-382 · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 (kafka-clients 3.9). MirrorMaker defaults checked against the Kafka 4.4 MirrorMaker configuration reference

Multi-region Kafka and MirrorMaker 2

Why regions get separate clusters, how MirrorMaker 2 copies topics and translates consumer offsets between them, and what a failover and a failback really involve.

What you will learn

  • Explain why one cluster stretched across distant regions is discouraged, and how that differs from a cluster spread over nearby zones
  • Choose between an active-passive and an active-active layout and name what each costs
  • Describe what the three MirrorMaker 2 connectors do and which internal topics they write
  • Predict the name of a mirrored topic and subscribe consumers so that they find it after a failover
  • State the data that can be lost and the data that will be processed twice in a failover, and size both
  • Write a failover and failback runbook that deals with consumer offsets explicitly

Before you start

The concept

Replication inside a Kafka cluster protects against the loss of a broker, a disk or, with rack awareness, a whole availability zone. It does not protect against the loss of the region the cluster lives in, and it does not put data near users on another continent. For that you need the data in a second cluster, and something that copies it there.

MirrorMaker 2 (MM2) is the tool that ships with Apache Kafka for this. It is built on Kafka Connect: it consumes from a source cluster and produces to a target cluster, continuously. The Kafka documentation calls each direction a replication flow, written source->target. This is a different mechanism from the replication inside a cluster. Followers copy a leader's log byte for byte under the control of the ISR. MM2 is an ordinary consumer and producer that happens to connect two clusters.

LayoutFlowsWho writes whereTypical use
Active-passiveprimary->drAll producers write to the primary. The second cluster only receives copies until a failoverDisaster recovery
Active-activewest->east and east->westProducers in each region write to their local cluster. Each cluster also holds the other's records, in separate topicsUsers in several regions, each served locally
Aggregationa->central, b->centralRegional clusters take local writes. A central cluster collects all of themAnalytics over all regions

Three facts follow from MM2 being a consumer and a producer, and every multi-region design has to be built around them.

  1. Replication is asynchronous. A record is acknowledged to the producer by the source cluster before MM2 has copied it. If the source region is lost, the records not yet copied are not in the target. The recovery point objective (RPO) is the replication delay, never zero.
  2. Offsets differ between clusters. The copy of a record gets whatever offset is next in the target partition. Offset 181,150 in the source may be 181,100 in the target, or anything else. A committed offset from one cluster means nothing in the other.
  3. Topic names differ by default. Topic orders.placed from the cluster with the alias primary arrives in the target as primary.orders.placed. A consumer that subscribes to orders.placed in the target does not see it.

Offsets across regions

Simulation, not a real cluster

The animated lesson shows the same record getting a different offset in each region, a region failing, and what a consumer does with an offset that belongs to the other cluster.

Step 1 · The pattern

A second copy, far away

Producers write to the log in region A and the consumer reads it there. A replication job copies every message to region B, so the data survives if region A is lost.

  • message
  • copy to region B
  • processed
Open the full animated lesson

Why it exists

Why not one cluster across both regions?

It looks simpler: one cluster, brokers in two regions, replicas on both sides, and Kafka's own replication does the rest with no second tool. The Kafka documentation advises against it for distant locations, for two reasons that come straight from how replication works.

  • Every write pays the distance. With acks=all the leader answers only when the in-sync followers have the record. If a follower is 80 ms away, every produce request waits for that round trip. A follower that cannot keep up within replica.lag.time.max.ms (30 seconds by default) is dropped from the ISR, and a wide-area link that stutters makes the ISR shrink and grow continuously.
  • A network split stops one side. The KRaft controller quorum needs a majority. With two locations, one of them holds the minority and loses the ability to elect leaders when the link breaks. The cluster does not stay available in both places.

Stretch cluster over nearby zones

  • One cluster, brokers in three availability zones a few milliseconds apart
  • broker.rack spreads the replicas of each partition across zones
  • Replication is synchronous: with acks=all, losing a zone loses no acknowledged record
  • Same topics, same offsets, no failover procedure for clients
  • Needs low, stable latency and a third location for the controller majority

Separate clusters with MirrorMaker 2

  • One independent cluster per region, each fully functional alone
  • Replication is asynchronous: writes are fast locally, and the mirror lags by the replication delay
  • Survives the loss of a region and of the link between regions
  • Topic names and offsets differ, and consumers must be moved deliberately
  • Failover loses the unreplicated tail and replays a window. Failback is a project

These are not competing answers to one question. A zone failure and a region failure are different events. Many production systems use both: a cluster spread over three zones in each region, and MirrorMaker between regions. The pattern the Kafka documentation recommends is a local cluster per data centre, with applications talking only to their local cluster and mirroring between clusters, so that each site keeps working when the link is down and the mirror catches up afterwards.

How it works inside

Three connectors

ConnectorWhat it doesWhat it writes
MirrorSourceConnectorConsumes the selected topics from the source and produces them to the target, keeping the partition number. Also creates remote topics and new partitions, and syncs topic configuration and ACLsRemote topics in the target. Offset syncs to mm2-offset-syncs.<target>.internal, by default in the source cluster
MirrorCheckpointConnectorReads the committed offsets of consumer groups in the source and translates them to target offsetsCheckpoints to <source>.checkpoints.internal in the target. Optionally the translated offsets straight into the target's __consumer_offsets
MirrorHeartbeatConnectorEmits a heartbeat record at a fixed interval (every second by default), which is mirrored like any topicThe heartbeats topic, seen in the target as <source>.heartbeats

In the dedicated mode described in the Kafka documentation, you write one properties file that names the clusters and enables flows, and start connect-mirror-maker.sh. The driver creates these connectors for each enabled flow. Several MirrorMaker processes with the same configuration find each other and share the work, like any Connect cluster. The recommendation is to run them next to the target cluster (consume from remote, produce to local), because a producer copes worse with a slow, unreliable link than a consumer does.

Topic naming and loops

The default DefaultReplicationPolicy names a mirrored topic <source alias>.<topic>. The prefix has a job: it keeps records from different clusters out of the same partition, and it lets MM2 detect cycles. In an active-active pair, west sends orders.placed to east as west.orders.placed. The flow back from east skips any topic whose name already carries the west prefix, so nothing bounces forever. A consumer that wants everything subscribes to the local topic and the remote ones; the merge happens in the consumer, never in the topic.

IdentityReplicationPolicy keeps the name unchanged: orders.placed stays orders.placed. That makes failover simpler for clients, and it suits one-way replication and migrations. Its own documentation carries the warning: MirrorMaker cannot prevent cycles with this policy. Enable flows in both directions with identical names and each cluster re-mirrors what it just received.

How offsets are translated

  1. While copying a partition, the source connector periodically records a pair: this source offset corresponds to this target offset. That is an offset sync. offset.lag.max (default 100) bounds how far the mapping may drift before a new sync is written.
  2. The checkpoint connector reads each group's committed offset in the source, finds the sync that most closely precedes it, and derives the matching target offset. It emits the result as a checkpoint every emit.checkpoints.interval.seconds (default 60).
  3. If sync.group.offsets.enabled is true, it also writes the translated offset into the target cluster's __consumer_offsets for that group, as long as the group has no active consumers in the target. The default is false.
  4. A consumer that starts in the target with the same group id then finds a committed offset for the remote topic and resumes from it.

Because translation uses a sync at or before the committed position, the translated offset points at or behind where the group really was. The checkpoint itself is up to an interval old. Both errors go the same way: after a failover the group resumes a little earlier than it stopped, and processes some records a second time. Translation is unavailable for a partition when no usable sync exists, for example when replication started after the group's position.

What a failover delivers

EffectCauseSizeWho has to deal with it
Records missing in the targetAcknowledged in the source, not yet mirrored when the source was lostThe replication delay at that moment, times the produce rateThe business: accept the loss, or reconcile from the source when it returns
Records processed twiceTranslated offsets lag the real position. MM2 may also write a record twice after its own retries unless exactly-once source support is enabledCheckpoint interval plus the sync granularity, times the consume rateEvery consumer: it must be idempotent
Records in an unexpected orderThe same key can appear in a local topic and in a remote topic, and nothing orders the two against each otherAny key written in both regions around the switchConsumers that depend on per-key order: use a version or sequence in the event

Within one mirrored partition the order of the source partition is kept, and the partition number is kept too. What is lost is any ordering between orders.placed and primary.orders.placed. During normal active-passive operation that does not matter, because only one of them receives writes. Around a failover both do.

Failback

Failing over is one decision under pressure. Coming back is harder, because by then the two clusters have diverged in both directions.

  • The old primary still holds the tail that was never mirrored. When it returns and replication resumes, those records reach the other cluster late, after newer events for the same orders.
  • The cluster you failed over to has taken new writes in its own local topics. The old primary has none of them. Getting them there needs a flow in the opposite direction, which produces dr.orders.placed on the primary, a topic its consumers were never subscribed to.
  • The committed offsets of every group in the old primary are frozen at the moment of the outage. Work done since then happened against other topics with other offsets. A consumer that simply reconnects to the primary replays from the stale commit.
  • Producers and consumers must move back in an order that does not leave a gap: stop producers on one side, let replication drain, move the consumers with translated offsets, then start producers on the other.

There is no single command that does this. Teams choose one of two paths: rehearse a real failback runbook regularly, or do not fail back at all and treat the surviving cluster as the new primary, rebuilding the other as its mirror. The second is often the safer plan.

Architecture

Drawing diagram…

A flowchart with two regions. In the primary region a producer writes to orders.placed and a consumer group commits offsets. In the second region the MirrorSourceConnector copies the topic to primary.orders.placed and writes offset pairs, and the MirrorCheckpointConnector combines committed offsets with those pairs to write checkpoints and translated offsets. A standby consumer group can read both the remote and the local topic.

flowchart LR
  subgraph PRI["Region eu-west, cluster alias primary"]
    P["Order Service producer"] --> T1["orders.placed, 12 partitions"]
    T1 --> C1["notification-service group"]
    C1 --> CO1["__consumer_offsets"]
    OS["mm2-offset-syncs.dr.internal"]
  end
  subgraph DRR["Region eu-central, cluster alias dr"]
    MSC["MirrorSourceConnector"]
    MCC["MirrorCheckpointConnector"]
    T2["primary.orders.placed, 12 partitions"]
    CK["primary.checkpoints.internal"]
    CO2["__consumer_offsets"]
    T3["orders.placed, local, empty until failover"]
    C2["notification-service group, standby"]
  end
  T1 -- "consume" --> MSC
  MSC -- "produce" --> T2
  MSC -- "offset pairs" --> OS
  CO1 -- "committed offsets" --> MCC
  OS -- "read syncs" --> MCC
  MCC --> CK
  MCC -- "translated offsets" --> CO2
  T2 -.-> C2
  T3 -.-> C2
Active-passive: MirrorMaker runs in the target region, copies the topic under a prefixed name and translates the group's offsets.
  • MirrorMaker sits in the target region. It reaches across the link as a consumer and produces locally, which is the placement the Kafka documentation recommends.
  • Two different things are copied by two different connectors: records by the source connector, and the consumer group's position by the checkpoint connector. The second only works because the first leaves offset pairs behind.
  • The standby group has two dotted inputs. After a failover, the backlog is in primary.orders.placed and new orders arrive in the local orders.placed. A consumer subscribed to only one of them misses half the picture.

Message flow

Drawing diagram…

A sequence diagram in which a producer sends an order to the primary cluster, MirrorMaker copies it to the second cluster at a different offset and records the offset pair, the consumer's committed offset is translated and written to the second cluster, a second order is acknowledged but not mirrored before the region fails, and the consumer resumes in the second cluster from the translated offset.

sequenceDiagram
  participant P as Producer
  participant A as Cluster primary
  participant M as MirrorMaker 2
  participant D as Cluster dr
  participant C as notification-service
  P->>A: Send order 7001 to orders.placed p3
  A-->>P: Ack at offset 181150
  M->>A: Fetch orders.placed p3
  M->>D: Produce to primary.orders.placed p3, lands at offset 181100
  M->>A: Offset sync 181150 maps to 181100
  C->>A: Commit offset 181151 for the group
  M->>A: Read committed offsets of the group
  M->>D: Checkpoint and group offset 181101 for primary.orders.placed p3
  P->>A: Send order 7002, acked at offset 181151
  Note over A: Region eu-west is lost before 7002 is mirrored
  C->>D: Start in dr, fetch committed offsets
  D-->>C: 181101 for primary.orders.placed p3
  Note over C: Resumes at or slightly before its true position
  C->>D: Poll primary.orders.placed and orders.placed
  Note over C: Order 7002 never arrives from the mirror
A record is acknowledged, mirrored, and the group's position translated; the region then fails with one record not yet copied.
  1. The ack to the producer comes before the mirror step. Order 7002 is the RPO made concrete: acknowledged, durable in the primary, and absent from the second cluster.
  2. The same record has offset 181,150 in one cluster and 181,100 in the other. The consumer's commit of 181,151 has to become 181,101, and only the offset sync makes that calculation possible.
  3. In this picture the translation is exact. In a real run the checkpoint is up to a minute old and the sync may be up to offset.lag.max records behind, so the group resumes earlier and repeats some work.
  4. After the failover the consumer polls both the mirrored topic and the local one. New orders are produced to the local orders.placed in the second cluster.

In a real system

Incident: the failover that re-sent a week of emails

A typical order platform, active-passive. The primary cluster is in eu-west, alias primary. The disaster recovery cluster is in eu-central, alias dr. MirrorMaker 2 runs in eu-central with the flow primary->dr and default settings otherwise. The topic is orders.placed, 12 partitions, keyed by order id, 7 days of retention, about 300,000 orders a day. The Payment Service (group payment-service) and the Notification Service (group notification-service) consume it, with auto.offset.reset=earliest. Standby copies of both services exist in eu-central, scaled to zero.

TimeWhat happenedWhy
09:40A network failure isolates eu-west. Producers and consumers there stopThe region is unreachable. About 2 seconds of acknowledged orders had not been mirrored
10:05The team declares a failover. The standby services in eu-central are scaled up against the dr clusterThe runbook says: start the services in the other region
10:06The services run and consume nothing. Lag on primary.orders.placed is not even reported for their groupsThey subscribe to orders.placed. In dr the data is in primary.orders.placed
10:20A hotfix changes the subscription to a pattern covering both names. The services start consuming from offset 0sync.group.offsets.enabled was never set. Neither group has committed offsets in dr, so auto.offset.reset=earliest applies
10:20 to 12:302.1 million orders from the last 7 days are reprocessedEverything the mirrored topic retains is new to a group without offsets
10:25Customers start receiving order confirmations for purchases made days agoThe Notification Service sends an email per event and keeps no record of what it has sent
12:30The backlog is drained. Live orders, produced to the local orders.placed since 10:10, have waited behind the replayOne consumer group, one stream of work: the replay delayed current orders by over two hours

The Payment Service came through it with nothing worse than wasted work: it stores each order id it has charged and skips repeats, so 2.1 million redeliveries produced no second charge. The Notification Service had no such check. The difference between the two services, on the same topic in the same incident, was consumer idempotency.

Three separate gaps lined up. Any one of them fixed would have reduced the damage.

  • Topic names: the standby consumers had never been run against the dr cluster, so nobody noticed they were subscribed to a topic that the mirror does not fill.
  • Offsets: translated offsets were available as checkpoints the whole time. Nothing applied them. With sync.group.offsets.enabled=true the groups would have resumed within about a minute of their true position.
  • Idempotency: even with synced offsets a failover replays a window. A consumer with a side effect that cannot be repeated safely is broken for every failover, large or small.

The 2 seconds of orders stranded in eu-west surfaced four hours later, when the region returned. They were paid for by reconciliation between the order database and the payment records, not by Kafka. Real cases of the same family in the Break library: The offset that went backwards, where a group lost its committed position and reprocessed days of messages, and Forty-three seconds, a database failover across regions that left each side holding writes the other lacked.

Java and Spring Boot

A MirrorMaker 2 configuration for the active-passive pair above with offset syncing switched on, a Spring consumer that finds its topic under either name and is safe to replay, and a small tool that applies checkpoints by hand for clusters where automatic syncing is off. The MirrorMaker file uses the dedicated-mode syntax from the Kafka documentation.

mm2.propertiesProperties
# Cluster aliases. The alias becomes the prefix of mirrored topics.clusters = primary, drprimary.bootstrap.servers = kafka-1.eu-west.example:9093,kafka-2.eu-west.example:9093dr.bootstrap.servers = kafka-1.eu-central.example:9093,kafka-2.eu-central.example:9093# One direction only: active-passive.primary->dr.enabled = truedr->primary.enabled = false# What to mirror.primary->dr.topics = orders.*, payments.*primary->dr.groups = payment-service.*, notification-service.*# Remote topics are created with this replication factor (default 2).replication.factor = 3# Consumer group offsets: emit checkpoints and write translated offsets# into the target's __consumer_offsets (off by default).primary->dr.emit.checkpoints.interval.seconds = 30primary->dr.sync.group.offsets.enabled = trueprimary->dr.sync.group.offsets.interval.seconds = 30# Pick up new topics and partitions sooner than the default 600 seconds.primary->dr.refresh.topics.interval.seconds = 120# Do not mirror records of aborted transactions.primary.consumer.isolation.level = read_committed# Spread the work over several tasks (the Connect default is 1).tasks.max = 4# Start in eu-central, next to the target cluster:#   bin/connect-mirror-maker.sh mm2.properties --clusters dr

Why it is written this way

  • The aliases primary and dr are part of every mirrored topic name (primary.orders.placed). Changing an alias later renames the topics, so choose names that describe a place, not a role that can change after a failover.
  • sync.group.offsets.enabled = true is the line the incident lacked. MirrorMaker writes the translated offsets only while the group has no active consumers in the target, so keep the standby consumers stopped, or at least not subscribed to the mirrored topic, until failover.
  • The checkpoint and sync intervals of 30 seconds bound how stale the translated position can be. Shorter intervals shrink the replay window and add load; they never reduce it to zero.
  • --clusters dr tells this process that dr is the nearby cluster, so it only produces there. This is the consume-from-remote, produce-to-local placement.
  • Security settings are left out. Each cluster takes the usual client properties under its alias, for example dr.security.protocol=SASL_SSL, and MirrorMaker's principal needs ACLs on both clusters.
  • Exactly-once delivery into the target is available for dedicated MirrorMaker clusters since Kafka 3.5 through dr.exactly.once.source.support = enabled, which also requires dedicated.mode.enable.internal.rest = true. Without it, MirrorMaker itself can write a record twice after a retry.

Configuration

MirrorMaker 2 settings that decide what is mirrored, how topics are named and how consumer offsets travel. Defaults are from the Kafka 4.4 MirrorMaker configuration reference and the geo-replication chapter. In the dedicated-mode file, each can be set globally or per flow as source->target.name.

source->target.enabledDefault false
What it does
Switches a replication flow on. Flows are directional and none is enabled until you say so.
When to change it
Enable one direction for active-passive, both for active-active. Keep the same file on every MirrorMaker process that targets the same cluster: differing files conflict.
topicsDefault .*
What it does
Topics or regular expressions selecting what to mirror. topics.exclude removes internal and already-mirrored topics by default.
When to change it
Name the topics the other region really needs. Mirroring everything costs transfer and storage, and copies topics nobody will read.
groupsDefault .*
What it does
Consumer groups whose offsets are checkpointed. groups.exclude drops console consumer and Connect groups by default.
When to change it
Make sure every group that must fail over matches. A group that is filtered out has no checkpoints and starts from auto.offset.reset.
replication.policy.classDefault org.apache.kafka.connect.mirror.DefaultReplicationPolicy
What it does
How mirrored topics are named. The default prefixes the source alias, using replication.policy.separator (default .).
When to change it
Keep the default for active-active. IdentityReplicationPolicy keeps names unchanged and is only safe for one-way topologies, because cycles cannot be detected.
sync.group.offsets.enabledDefault false
What it does
Whether translated offsets are written to the target's __consumer_offsets, for groups with no active consumers there.
When to change it
Set to true for any disaster recovery pair, unless your failover tooling applies checkpoints itself.
emit.checkpoints.interval.secondsDefault 60
What it does
How often checkpoints are emitted. sync.group.offsets.interval.seconds (default 60) sets how often they are applied.
When to change it
Lower both to shorten the replay window after a failover. The window never reaches zero.
offset.lag.maxDefault 100
What it does
How far a partition's offset mapping may drift before the source connector writes a new offset sync.
When to change it
Lower it for low-volume topics where 100 records is a long time. 0 writes a sync for every record, at a cost in load and storage.
refresh.topics.interval.secondsDefault 600
What it does
How often MirrorMaker looks for new topics and partitions in the source.
When to change it
A topic created in the source is not mirrored for up to this long. Lower it, or create remote topics as part of topic provisioning.
replication.factorDefault 2
What it does
Replication factor of remote topics that MirrorMaker creates in the target.
When to change it
Set it to match your normal topics, usually 3. The default is lower than most production standards.
sync.topic.acls.enabledDefault true
What it does
Whether ACLs of source topics are propagated to the remote topics.
When to change it
Consumers also need group ACLs in the target cluster. Check that a failed-over service is authorised there before the day it matters.
tasks.maxDefault 1 (Kafka Connect default)
What it does
Maximum number of tasks per connector, which bounds the parallelism of mirroring.
When to change it
The Kafka documentation recommends at least 2. Raise it with the number of partitions and watch replication latency.

Hands-on exercise

Troubleshooting: the group that started again from zero

Use tool output from the target cluster after a failover to find out why a consumer group is reprocessing old records, and decide how to stop it and how to prevent it. All output below is illustrative: written for this exercise in the format the tools produce.

Steps · 0 of 6 done

What can go wrong

Consumers subscribed to a name the mirror does not fill
Symptom
After a failover the consumers start cleanly in the second region and process nothing. No errors, no lag reported on the topic they subscribe to.
Cause
With the default policy the mirrored data is in primary.orders.placed. The consumers subscribe to orders.placed, which in the second cluster is empty or does not exist.
Fix
Subscribe by a pattern that covers the local and the prefixed names, and run the standby consumers against the second cluster in drills to prove it.
Offsets never synced
Symptom
Failed-over consumers replay the whole retention of a topic, or skip everything that was pending, depending on auto.offset.reset.
Cause
sync.group.offsets.enabled is false by default, the group is excluded by the groups filter, or active consumers with the same group id in the target blocked the sync.
Fix
Enable the sync, check that the group matches the filter, keep the standby group inactive, and verify before any incident that the group shows committed offsets in the target cluster.
Assuming the mirror is complete
Symptom
After a failover, a small number of orders exist in the order database with no trace in the second cluster. They reappear hours later when the failed region returns.
Cause
Replication is asynchronous. Records acknowledged shortly before the failure were never copied. acks=all covers only the source cluster.
Fix
Measure replication latency and treat it as the RPO. Decide with the business how the unreplicated tail is reconciled, and build that reconciliation before the failover.
A consumer that cannot take a replay
Symptom
Duplicate emails, double stock movements or repeated charges after a failover or a drill.
Cause
Translated offsets are at or behind the true position by design, so a replay window always exists. The consumer has a side effect with no deduplication, or deduplicates on topic and offset, which change between clusters.
Fix
Deduplicate on a business or event id stored with the effect. This is application logic; producer idempotence does not help here.
Identity policy with flows in both directions
Symptom
Record counts and disk usage grow without matching traffic. The same events arrive again and again on both sides.
Cause
IdentityReplicationPolicy keeps topic names unchanged, so MirrorMaker cannot tell a mirrored topic from a local one and re-mirrors it in the opposite direction.
Fix
Use the default prefixing policy for any two-way topology. Keep the identity policy for strictly one-way replication and migrations.
Producing to a remote topic
Symptom
After a failover, new records are written into primary.orders.placed. When the primary returns, its topic and its mirror no longer correspond, and offset translation gives wrong results.
Cause
A producer was pointed at the mirrored topic to keep consumers working. Remote topics are meant to be written only by MirrorMaker.
Fix
Producers always write to the local topic name. Consumers read local and remote topics. Let ACL syncing keep remote topics read-only for applications.
Failback improvised on the day
Symptom
On returning to the original region, consumers replay from offsets committed before the outage, and records produced during the outage are missing there.
Cause
The old cluster's committed offsets are frozen at the failure, and nothing replicated the other cluster's new records back.
Fix
Write and rehearse a failback runbook with a reverse flow and explicit offset handling, or decide in advance to keep the surviving cluster as the new primary.

How to troubleshoot

Will this group resume correctly if we fail over now?

Check

kafka-consumer-groups.sh --bootstrap-server kafka-1.eu-central.example:9093 --describe --group notification-service
How to read it
Run it against the target cluster while the standby is idle. Committed offsets on primary.orders.placed that advance from one run to the next mean the sync works. No rows, or a group that does not exist, means a failover will fall back to auto.offset.reset.
Failed-over consumers started from the beginning or the end

Check

The consumer log lines `Found no committed offset for partition ...` and `Resetting offset for partition ...`
How to read it
They confirm the group had nothing committed in this cluster. Check sync.group.offsets.enabled, the groups filter and whether consumers of that group were already running in the target, which blocks the sync.
No checkpoints for a group

Check

kafka-console-consumer.sh --bootstrap-server kafka-1.eu-central.example:9093 --topic primary.checkpoints.internal --from-beginning --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter
How to read it
Each line names a group, a remote topic partition, the upstream offset and the translated downstream offset. A group that never appears is filtered out, has not committed in the source, or sits at a position for which no offset sync exists, for example one older than the start of replication.
Consumers in the second region receive nothing

Check

kafka-topics.sh --bootstrap-server kafka-1.eu-central.example:9093 --list
How to read it
Look for the prefixed name, such as primary.orders.placed. If it exists and has data while the consumers subscribe to orders.placed, the subscription is the problem. If it does not exist, the topic is not matched by topics or has not been picked up yet.
How far behind is the mirror?

Check

The `replication-latency-ms` and `record-age-ms` metrics of `kafka.connect.mirror:type=MirrorSourceConnector`, per target, topic and partition
How to read it
Replication latency is the time from a record's timestamp to its acknowledgement in the target. Its recent maximum is your realistic RPO. Do not compare log end offsets between the clusters: they are different numbering and their difference means nothing.
Is the flow alive at all?

Check

kafka-console-consumer.sh --bootstrap-server kafka-1.eu-central.example:9093 --topic primary.heartbeats --formatter org.apache.kafka.connect.mirror.formatters.HeartbeatFormatter
How to read it
Heartbeats are emitted every second by default and mirrored like data. Fresh heartbeats arriving in the target show that the source cluster is reachable and the connectors are running. Silence means the flow, the link or the source is down.
A new topic is not mirrored

Check

The `topics` and `topics.exclude` settings of the flow, and `refresh.topics.interval.seconds`
How to read it
New topics are discovered on a schedule, every 600 seconds by default, so a delay of minutes is normal. If it never appears, the name does not match the filter, or another MirrorMaker process with a different configuration for the same target is overriding yours.

Key takeaways

  • Stretch one cluster over nearby zones for zone failures. Use separate clusters with MirrorMaker 2 between distant regions.
  • MirrorMaker 2 is a consumer and a producer. Replication is asynchronous, so the mirror is always behind and the RPO is the replication delay.
  • Mirrored topics are prefixed with the source alias by default, and their offsets differ from the source. Consumers must subscribe to the remote name and need translated offsets.
  • Checkpoints are emitted by default, but writing them into the target's group offsets is off until you set sync.group.offsets.enabled.
  • Translated offsets are at or behind the real position. Every failover replays a window, so every consumer must be idempotent on a business key.
  • Order is kept within a mirrored partition, never between a local topic and a remote one.
  • Failback is harder than failover. Rehearse it, or plan not to do it.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A payments platform mirrors payments.captured from primary to dr. Producers use acks=all with min.insync.replicas=2 and replication factor 3. The topic takes about 1,200 records per second. The MirrorMaker metric replication-latency-ms-max has been between 600 and 900 over the last week. The disaster recovery document states that failover to dr loses no acknowledged payments.

What should the document say instead?

Scenario 2 of 3

An inventory consumer group fails over to dr with sync.group.offsets.enabled=true, checkpoints every 60 seconds and offset.lag.max at its default. It was consuming about 400 records per second with almost no lag. After the failover the team sees roughly 20,000 stock adjustments applied twice. The consumer applies each StockAdjusted event as quantity = quantity + delta.

What is the right fix?

Scenario 3 of 3

Two regions, west and east, each with local producers for orders.placed. A team wants consumers in both regions to see all orders under the single name orders.placed, so they configure replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy and enable west->east and east->west. Within an hour, disk usage on both clusters climbs steeply and consumers report every order many times.

What should they do?

References

References: Apache Kafka 4.4 documentation: Geo-replication (cross-cluster data mirroring) ↗ · Apache Kafka 4.4 documentation: MirrorMaker configs ↗ · Apache Kafka 4.4 documentation: Datacenters ↗ · KIP-382: MirrorMaker 2.0 ↗ · Apache Kafka source: IdentityReplicationPolicy (cycle warning) ↗ · Apache Kafka source: RemoteClusterUtils (translateOffsets) ↗ · Apache Kafka source: OffsetSyncStore (how offsets are translated) ↗ · Spring for Apache Kafka reference: @KafkaListener annotation (topicPattern) ↗ · Report a problem