Path 10 · lesson 5 of 6
Multi-region Kafka and MirrorMaker 2
Why regions get separate clusters, how MirrorMaker 2 copies topics and translates consumer offsets between them, and what a failover and a failback really involve.
What you will learn
- Explain why one cluster stretched across distant regions is discouraged, and how that differs from a cluster spread over nearby zones
- Choose between an active-passive and an active-active layout and name what each costs
- Describe what the three MirrorMaker 2 connectors do and which internal topics they write
- Predict the name of a mirrored topic and subscribe consumers so that they find it after a failover
- State the data that can be lost and the data that will be processed twice in a failover, and size both
- Write a failover and failback runbook that deals with consumer offsets explicitly
Before you start
The concept
Replication inside a Kafka cluster protects against the loss of a broker, a disk or, with rack awareness, a whole availability zone. It does not protect against the loss of the region the cluster lives in, and it does not put data near users on another continent. For that you need the data in a second cluster, and something that copies it there.
MirrorMaker 2 (MM2) is the tool that ships with Apache Kafka for this. It is built on Kafka Connect: it consumes from a source cluster and produces to a target cluster, continuously. The Kafka documentation calls each direction a replication flow, written source->target. This is a different mechanism from the replication inside a cluster. Followers copy a leader's log byte for byte under the control of the ISR. MM2 is an ordinary consumer and producer that happens to connect two clusters.
| Layout | Flows | Who writes where | Typical use |
|---|---|---|---|
| Active-passive | primary->dr | All producers write to the primary. The second cluster only receives copies until a failover | Disaster recovery |
| Active-active | west->east and east->west | Producers in each region write to their local cluster. Each cluster also holds the other's records, in separate topics | Users in several regions, each served locally |
| Aggregation | a->central, b->central | Regional clusters take local writes. A central cluster collects all of them | Analytics over all regions |
Three facts follow from MM2 being a consumer and a producer, and every multi-region design has to be built around them.
- Replication is asynchronous. A record is acknowledged to the producer by the source cluster before MM2 has copied it. If the source region is lost, the records not yet copied are not in the target. The recovery point objective (RPO) is the replication delay, never zero.
- Offsets differ between clusters. The copy of a record gets whatever offset is next in the target partition. Offset 181,150 in the source may be 181,100 in the target, or anything else. A committed offset from one cluster means nothing in the other.
- Topic names differ by default. Topic
orders.placedfrom the cluster with the aliasprimaryarrives in the target asprimary.orders.placed. A consumer that subscribes toorders.placedin the target does not see it.
Offsets across regions
Simulation, not a real clusterThe animated lesson shows the same record getting a different offset in each region, a region failing, and what a consumer does with an offset that belongs to the other cluster.
Step 1 · The pattern
A second copy, far away
Producers write to the log in region A and the consumer reads it there. A replication job copies every message to region B, so the data survives if region A is lost.
- message
- copy to region B
- processed
Why it exists
Why not one cluster across both regions?
It looks simpler: one cluster, brokers in two regions, replicas on both sides, and Kafka's own replication does the rest with no second tool. The Kafka documentation advises against it for distant locations, for two reasons that come straight from how replication works.
- Every write pays the distance. With
acks=allthe leader answers only when the in-sync followers have the record. If a follower is 80 ms away, every produce request waits for that round trip. A follower that cannot keep up withinreplica.lag.time.max.ms(30 seconds by default) is dropped from the ISR, and a wide-area link that stutters makes the ISR shrink and grow continuously. - A network split stops one side. The KRaft controller quorum needs a majority. With two locations, one of them holds the minority and loses the ability to elect leaders when the link breaks. The cluster does not stay available in both places.
Stretch cluster over nearby zones
- One cluster, brokers in three availability zones a few milliseconds apart
broker.rackspreads the replicas of each partition across zones- Replication is synchronous: with
acks=all, losing a zone loses no acknowledged record - Same topics, same offsets, no failover procedure for clients
- Needs low, stable latency and a third location for the controller majority
Separate clusters with MirrorMaker 2
- One independent cluster per region, each fully functional alone
- Replication is asynchronous: writes are fast locally, and the mirror lags by the replication delay
- Survives the loss of a region and of the link between regions
- Topic names and offsets differ, and consumers must be moved deliberately
- Failover loses the unreplicated tail and replays a window. Failback is a project
These are not competing answers to one question. A zone failure and a region failure are different events. Many production systems use both: a cluster spread over three zones in each region, and MirrorMaker between regions. The pattern the Kafka documentation recommends is a local cluster per data centre, with applications talking only to their local cluster and mirroring between clusters, so that each site keeps working when the link is down and the mirror catches up afterwards.
How it works inside
Three connectors
| Connector | What it does | What it writes |
|---|---|---|
MirrorSourceConnector | Consumes the selected topics from the source and produces them to the target, keeping the partition number. Also creates remote topics and new partitions, and syncs topic configuration and ACLs | Remote topics in the target. Offset syncs to mm2-offset-syncs.<target>.internal, by default in the source cluster |
MirrorCheckpointConnector | Reads the committed offsets of consumer groups in the source and translates them to target offsets | Checkpoints to <source>.checkpoints.internal in the target. Optionally the translated offsets straight into the target's __consumer_offsets |
MirrorHeartbeatConnector | Emits a heartbeat record at a fixed interval (every second by default), which is mirrored like any topic | The heartbeats topic, seen in the target as <source>.heartbeats |
In the dedicated mode described in the Kafka documentation, you write one properties file that names the clusters and enables flows, and start connect-mirror-maker.sh. The driver creates these connectors for each enabled flow. Several MirrorMaker processes with the same configuration find each other and share the work, like any Connect cluster. The recommendation is to run them next to the target cluster (consume from remote, produce to local), because a producer copes worse with a slow, unreliable link than a consumer does.
Topic naming and loops
The default DefaultReplicationPolicy names a mirrored topic <source alias>.<topic>. The prefix has a job: it keeps records from different clusters out of the same partition, and it lets MM2 detect cycles. In an active-active pair, west sends orders.placed to east as west.orders.placed. The flow back from east skips any topic whose name already carries the west prefix, so nothing bounces forever. A consumer that wants everything subscribes to the local topic and the remote ones; the merge happens in the consumer, never in the topic.
IdentityReplicationPolicy keeps the name unchanged: orders.placed stays orders.placed. That makes failover simpler for clients, and it suits one-way replication and migrations. Its own documentation carries the warning: MirrorMaker cannot prevent cycles with this policy. Enable flows in both directions with identical names and each cluster re-mirrors what it just received.
How offsets are translated
- While copying a partition, the source connector periodically records a pair: this source offset corresponds to this target offset. That is an offset sync.
offset.lag.max(default 100) bounds how far the mapping may drift before a new sync is written. - The checkpoint connector reads each group's committed offset in the source, finds the sync that most closely precedes it, and derives the matching target offset. It emits the result as a checkpoint every
emit.checkpoints.interval.seconds(default 60). - If
sync.group.offsets.enabledis true, it also writes the translated offset into the target cluster's__consumer_offsetsfor that group, as long as the group has no active consumers in the target. The default is false. - A consumer that starts in the target with the same group id then finds a committed offset for the remote topic and resumes from it.
Because translation uses a sync at or before the committed position, the translated offset points at or behind where the group really was. The checkpoint itself is up to an interval old. Both errors go the same way: after a failover the group resumes a little earlier than it stopped, and processes some records a second time. Translation is unavailable for a partition when no usable sync exists, for example when replication started after the group's position.
What a failover delivers
| Effect | Cause | Size | Who has to deal with it |
|---|---|---|---|
| Records missing in the target | Acknowledged in the source, not yet mirrored when the source was lost | The replication delay at that moment, times the produce rate | The business: accept the loss, or reconcile from the source when it returns |
| Records processed twice | Translated offsets lag the real position. MM2 may also write a record twice after its own retries unless exactly-once source support is enabled | Checkpoint interval plus the sync granularity, times the consume rate | Every consumer: it must be idempotent |
| Records in an unexpected order | The same key can appear in a local topic and in a remote topic, and nothing orders the two against each other | Any key written in both regions around the switch | Consumers that depend on per-key order: use a version or sequence in the event |
Within one mirrored partition the order of the source partition is kept, and the partition number is kept too. What is lost is any ordering between orders.placed and primary.orders.placed. During normal active-passive operation that does not matter, because only one of them receives writes. Around a failover both do.
Failback
Failing over is one decision under pressure. Coming back is harder, because by then the two clusters have diverged in both directions.
- The old primary still holds the tail that was never mirrored. When it returns and replication resumes, those records reach the other cluster late, after newer events for the same orders.
- The cluster you failed over to has taken new writes in its own local topics. The old primary has none of them. Getting them there needs a flow in the opposite direction, which produces
dr.orders.placedon the primary, a topic its consumers were never subscribed to. - The committed offsets of every group in the old primary are frozen at the moment of the outage. Work done since then happened against other topics with other offsets. A consumer that simply reconnects to the primary replays from the stale commit.
- Producers and consumers must move back in an order that does not leave a gap: stop producers on one side, let replication drain, move the consumers with translated offsets, then start producers on the other.
There is no single command that does this. Teams choose one of two paths: rehearse a real failback runbook regularly, or do not fail back at all and treat the surviving cluster as the new primary, rebuilding the other as its mirror. The second is often the safer plan.
Architecture
Drawing diagram…
A flowchart with two regions. In the primary region a producer writes to orders.placed and a consumer group commits offsets. In the second region the MirrorSourceConnector copies the topic to primary.orders.placed and writes offset pairs, and the MirrorCheckpointConnector combines committed offsets with those pairs to write checkpoints and translated offsets. A standby consumer group can read both the remote and the local topic.
flowchart LR
subgraph PRI["Region eu-west, cluster alias primary"]
P["Order Service producer"] --> T1["orders.placed, 12 partitions"]
T1 --> C1["notification-service group"]
C1 --> CO1["__consumer_offsets"]
OS["mm2-offset-syncs.dr.internal"]
end
subgraph DRR["Region eu-central, cluster alias dr"]
MSC["MirrorSourceConnector"]
MCC["MirrorCheckpointConnector"]
T2["primary.orders.placed, 12 partitions"]
CK["primary.checkpoints.internal"]
CO2["__consumer_offsets"]
T3["orders.placed, local, empty until failover"]
C2["notification-service group, standby"]
end
T1 -- "consume" --> MSC
MSC -- "produce" --> T2
MSC -- "offset pairs" --> OS
CO1 -- "committed offsets" --> MCC
OS -- "read syncs" --> MCC
MCC --> CK
MCC -- "translated offsets" --> CO2
T2 -.-> C2
T3 -.-> C2- MirrorMaker sits in the target region. It reaches across the link as a consumer and produces locally, which is the placement the Kafka documentation recommends.
- Two different things are copied by two different connectors: records by the source connector, and the consumer group's position by the checkpoint connector. The second only works because the first leaves offset pairs behind.
- The standby group has two dotted inputs. After a failover, the backlog is in
primary.orders.placedand new orders arrive in the localorders.placed. A consumer subscribed to only one of them misses half the picture.
Message flow
Drawing diagram…
A sequence diagram in which a producer sends an order to the primary cluster, MirrorMaker copies it to the second cluster at a different offset and records the offset pair, the consumer's committed offset is translated and written to the second cluster, a second order is acknowledged but not mirrored before the region fails, and the consumer resumes in the second cluster from the translated offset.
sequenceDiagram participant P as Producer participant A as Cluster primary participant M as MirrorMaker 2 participant D as Cluster dr participant C as notification-service P->>A: Send order 7001 to orders.placed p3 A-->>P: Ack at offset 181150 M->>A: Fetch orders.placed p3 M->>D: Produce to primary.orders.placed p3, lands at offset 181100 M->>A: Offset sync 181150 maps to 181100 C->>A: Commit offset 181151 for the group M->>A: Read committed offsets of the group M->>D: Checkpoint and group offset 181101 for primary.orders.placed p3 P->>A: Send order 7002, acked at offset 181151 Note over A: Region eu-west is lost before 7002 is mirrored C->>D: Start in dr, fetch committed offsets D-->>C: 181101 for primary.orders.placed p3 Note over C: Resumes at or slightly before its true position C->>D: Poll primary.orders.placed and orders.placed Note over C: Order 7002 never arrives from the mirror
- The ack to the producer comes before the mirror step. Order 7002 is the RPO made concrete: acknowledged, durable in the primary, and absent from the second cluster.
- The same record has offset 181,150 in one cluster and 181,100 in the other. The consumer's commit of 181,151 has to become 181,101, and only the offset sync makes that calculation possible.
- In this picture the translation is exact. In a real run the checkpoint is up to a minute old and the sync may be up to
offset.lag.maxrecords behind, so the group resumes earlier and repeats some work. - After the failover the consumer polls both the mirrored topic and the local one. New orders are produced to the local
orders.placedin the second cluster.
In a real system
Incident: the failover that re-sent a week of emails
A typical order platform, active-passive. The primary cluster is in eu-west, alias primary. The disaster recovery cluster is in eu-central, alias dr. MirrorMaker 2 runs in eu-central with the flow primary->dr and default settings otherwise. The topic is orders.placed, 12 partitions, keyed by order id, 7 days of retention, about 300,000 orders a day. The Payment Service (group payment-service) and the Notification Service (group notification-service) consume it, with auto.offset.reset=earliest. Standby copies of both services exist in eu-central, scaled to zero.
| Time | What happened | Why |
|---|---|---|
| 09:40 | A network failure isolates eu-west. Producers and consumers there stop | The region is unreachable. About 2 seconds of acknowledged orders had not been mirrored |
| 10:05 | The team declares a failover. The standby services in eu-central are scaled up against the dr cluster | The runbook says: start the services in the other region |
| 10:06 | The services run and consume nothing. Lag on primary.orders.placed is not even reported for their groups | They subscribe to orders.placed. In dr the data is in primary.orders.placed |
| 10:20 | A hotfix changes the subscription to a pattern covering both names. The services start consuming from offset 0 | sync.group.offsets.enabled was never set. Neither group has committed offsets in dr, so auto.offset.reset=earliest applies |
| 10:20 to 12:30 | 2.1 million orders from the last 7 days are reprocessed | Everything the mirrored topic retains is new to a group without offsets |
| 10:25 | Customers start receiving order confirmations for purchases made days ago | The Notification Service sends an email per event and keeps no record of what it has sent |
| 12:30 | The backlog is drained. Live orders, produced to the local orders.placed since 10:10, have waited behind the replay | One consumer group, one stream of work: the replay delayed current orders by over two hours |
The Payment Service came through it with nothing worse than wasted work: it stores each order id it has charged and skips repeats, so 2.1 million redeliveries produced no second charge. The Notification Service had no such check. The difference between the two services, on the same topic in the same incident, was consumer idempotency.
Three separate gaps lined up. Any one of them fixed would have reduced the damage.
- Topic names: the standby consumers had never been run against the
drcluster, so nobody noticed they were subscribed to a topic that the mirror does not fill. - Offsets: translated offsets were available as checkpoints the whole time. Nothing applied them. With
sync.group.offsets.enabled=truethe groups would have resumed within about a minute of their true position. - Idempotency: even with synced offsets a failover replays a window. A consumer with a side effect that cannot be repeated safely is broken for every failover, large or small.
The 2 seconds of orders stranded in eu-west surfaced four hours later, when the region returned. They were paid for by reconciliation between the order database and the payment records, not by Kafka. Real cases of the same family in the Break library: The offset that went backwards, where a group lost its committed position and reprocessed days of messages, and Forty-three seconds, a database failover across regions that left each side holding writes the other lacked.
Java and Spring Boot
A MirrorMaker 2 configuration for the active-passive pair above with offset syncing switched on, a Spring consumer that finds its topic under either name and is safe to replay, and a small tool that applies checkpoints by hand for clusters where automatic syncing is off. The MirrorMaker file uses the dedicated-mode syntax from the Kafka documentation.
# Cluster aliases. The alias becomes the prefix of mirrored topics.clusters = primary, drprimary.bootstrap.servers = kafka-1.eu-west.example:9093,kafka-2.eu-west.example:9093dr.bootstrap.servers = kafka-1.eu-central.example:9093,kafka-2.eu-central.example:9093# One direction only: active-passive.primary->dr.enabled = truedr->primary.enabled = false# What to mirror.primary->dr.topics = orders.*, payments.*primary->dr.groups = payment-service.*, notification-service.*# Remote topics are created with this replication factor (default 2).replication.factor = 3# Consumer group offsets: emit checkpoints and write translated offsets# into the target's __consumer_offsets (off by default).primary->dr.emit.checkpoints.interval.seconds = 30primary->dr.sync.group.offsets.enabled = trueprimary->dr.sync.group.offsets.interval.seconds = 30# Pick up new topics and partitions sooner than the default 600 seconds.primary->dr.refresh.topics.interval.seconds = 120# Do not mirror records of aborted transactions.primary.consumer.isolation.level = read_committed# Spread the work over several tasks (the Connect default is 1).tasks.max = 4# Start in eu-central, next to the target cluster:# bin/connect-mirror-maker.sh mm2.properties --clusters drWhy it is written this way
- The aliases
primaryanddrare part of every mirrored topic name (primary.orders.placed). Changing an alias later renames the topics, so choose names that describe a place, not a role that can change after a failover. sync.group.offsets.enabled = trueis the line the incident lacked. MirrorMaker writes the translated offsets only while the group has no active consumers in the target, so keep the standby consumers stopped, or at least not subscribed to the mirrored topic, until failover.- The checkpoint and sync intervals of 30 seconds bound how stale the translated position can be. Shorter intervals shrink the replay window and add load; they never reduce it to zero.
--clusters drtells this process thatdris the nearby cluster, so it only produces there. This is the consume-from-remote, produce-to-local placement.- Security settings are left out. Each cluster takes the usual client properties under its alias, for example
dr.security.protocol=SASL_SSL, and MirrorMaker's principal needs ACLs on both clusters. - Exactly-once delivery into the target is available for dedicated MirrorMaker clusters since Kafka 3.5 through
dr.exactly.once.source.support = enabled, which also requiresdedicated.mode.enable.internal.rest = true. Without it, MirrorMaker itself can write a record twice after a retry.
package com.example.notifications;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.jdbc.core.JdbcTemplate;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.messaging.handler.annotation.Header;import org.springframework.stereotype.Component;import org.springframework.transaction.annotation.Transactional;@Componentpublic class OrderPlacedListener { private static final Logger log = LoggerFactory.getLogger(OrderPlacedListener.class); private final JdbcTemplate jdbc; public OrderPlacedListener(JdbcTemplate jdbc) { this.jdbc = jdbc; } // Matches orders.placed, primary.orders.placed and dr.orders.placed. @KafkaListener( id = "order-placed", groupId = "notification-service", topicPattern = "(primary\\.|dr\\.)?orders\\.placed") @Transactional public void onOrderPlaced(OrderPlaced event, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic, @Header(KafkaHeaders.OFFSET) long offset) { // The event id travels with the record through MirrorMaker. Topic and offset do not survive: // they are different in every cluster, so they cannot be the deduplication key. int inserted = jdbc.update( "INSERT INTO processed_event (event_id) VALUES (?) ON CONFLICT (event_id) DO NOTHING", event.eventId()); if (inserted == 0) { log.info("Skipping duplicate {} from {} at offset {}", event.eventId(), topic, offset); return; } // Same database transaction as the marker above: either both are stored or neither. jdbc.update( "INSERT INTO outbound_email (order_id, recipient, template) VALUES (?, ?, 'order-confirmation')", event.orderId(), event.customerEmail()); }}record OrderPlaced(String eventId, String orderId, String customerEmail, long totalMinor, String currency) {}Why it is written this way
topicPatternsubscribes by regular expression, so the same build works in either region and picks up the mirrored topic as well as the local one. A consumer discovers new matching topics when it refreshes metadata, by default every 5 minutes (metadata.max.age.ms).- Deduplication uses a business identifier carried inside the event. After a failover the same record has another topic name and another offset, so a key built from topic, partition and offset would treat every replayed record as new.
- The marker row and the email request are written in one database transaction. A separate sender delivers rows from
outbound_email. A replayed event finds its marker and does nothing, which is what the Notification Service in the incident lacked. ON CONFLICT ... DO NOTHINGis PostgreSQL syntax and assumes a primary key onevent_id. Other databases have an equivalent, or you can catch the duplicate key exception.- The pattern does not restore ordering between the local and the remote topic. If handling depends on the order of events for one order, compare a version or timestamp in the event with what is stored.
package com.example.ops;import java.time.Duration;import java.util.HashMap;import java.util.Map;import org.apache.kafka.clients.admin.Admin;import org.apache.kafka.clients.consumer.OffsetAndMetadata;import org.apache.kafka.common.TopicPartition;import org.apache.kafka.connect.mirror.RemoteClusterUtils;/** * Failover helper: sets a group's offsets in the target cluster from MirrorMaker's checkpoints. * Run it while the group has no running consumers in the target cluster. * Usage: ApplyCheckpoints <target bootstrap servers> <source cluster alias> <group id> */public final class ApplyCheckpoints { public static void main(String[] args) throws Exception { String targetBootstrap = args[0]; String sourceAlias = args[1]; String groupId = args[2]; Map<String, Object> props = new HashMap<>(); props.put("bootstrap.servers", targetBootstrap); // Reads <source alias>.checkpoints.internal in the target cluster. Map<TopicPartition, OffsetAndMetadata> translated = RemoteClusterUtils.translateOffsets(props, sourceAlias, groupId, Duration.ofSeconds(30)); if (translated.isEmpty()) { System.err.println("No checkpoints for group " + groupId + ". Offsets were not changed."); System.exit(1); } translated.forEach((tp, offset) -> System.out.println(tp + " -> " + offset.offset())); try (Admin admin = Admin.create(props)) { admin.alterConsumerGroupOffsets(groupId, translated).all().get(); } }}Why it is written this way
RemoteClusterUtils.translateOffsetsis in theorg.apache.kafka:connect-mirror-clientartifact. It returns the latest checkpoint per partition, already expressed as target topic names and target offsets.- The tool refuses to continue when there are no checkpoints. Falling through to
auto.offset.resetis exactly the failure this tool exists to prevent, so it should be loud. - The broker rejects an offset change for a group that has active members. Stop the group's consumers in the target first, apply, then start them.
- Add the security properties of the target cluster to
propsin a real environment. They are left out to keep the example short.
Configuration
MirrorMaker 2 settings that decide what is mirrored, how topics are named and how consumer offsets travel. Defaults are from the Kafka 4.4 MirrorMaker configuration reference and the geo-replication chapter. In the dedicated-mode file, each can be set globally or per flow as source->target.name.
source->target.enabledDefault false
- What it does
- Switches a replication flow on. Flows are directional and none is enabled until you say so.
- When to change it
- Enable one direction for active-passive, both for active-active. Keep the same file on every MirrorMaker process that targets the same cluster: differing files conflict.
topicsDefault .*
- What it does
- Topics or regular expressions selecting what to mirror.
topics.excluderemoves internal and already-mirrored topics by default. - When to change it
- Name the topics the other region really needs. Mirroring everything costs transfer and storage, and copies topics nobody will read.
groupsDefault .*
- What it does
- Consumer groups whose offsets are checkpointed.
groups.excludedrops console consumer and Connect groups by default. - When to change it
- Make sure every group that must fail over matches. A group that is filtered out has no checkpoints and starts from
auto.offset.reset.
replication.policy.classDefault org.apache.kafka.connect.mirror.DefaultReplicationPolicy
- What it does
- How mirrored topics are named. The default prefixes the source alias, using
replication.policy.separator(default.). - When to change it
- Keep the default for active-active.
IdentityReplicationPolicykeeps names unchanged and is only safe for one-way topologies, because cycles cannot be detected.
sync.group.offsets.enabledDefault false
- What it does
- Whether translated offsets are written to the target's
__consumer_offsets, for groups with no active consumers there. - When to change it
- Set to
truefor any disaster recovery pair, unless your failover tooling applies checkpoints itself.
emit.checkpoints.interval.secondsDefault 60
- What it does
- How often checkpoints are emitted.
sync.group.offsets.interval.seconds(default 60) sets how often they are applied. - When to change it
- Lower both to shorten the replay window after a failover. The window never reaches zero.
offset.lag.maxDefault 100
- What it does
- How far a partition's offset mapping may drift before the source connector writes a new offset sync.
- When to change it
- Lower it for low-volume topics where 100 records is a long time.
0writes a sync for every record, at a cost in load and storage.
refresh.topics.interval.secondsDefault 600
- What it does
- How often MirrorMaker looks for new topics and partitions in the source.
- When to change it
- A topic created in the source is not mirrored for up to this long. Lower it, or create remote topics as part of topic provisioning.
replication.factorDefault 2
- What it does
- Replication factor of remote topics that MirrorMaker creates in the target.
- When to change it
- Set it to match your normal topics, usually 3. The default is lower than most production standards.
sync.topic.acls.enabledDefault true
- What it does
- Whether ACLs of source topics are propagated to the remote topics.
- When to change it
- Consumers also need group ACLs in the target cluster. Check that a failed-over service is authorised there before the day it matters.
tasks.maxDefault 1 (Kafka Connect default)
- What it does
- Maximum number of tasks per connector, which bounds the parallelism of mirroring.
- When to change it
- The Kafka documentation recommends at least 2. Raise it with the number of partitions and watch replication latency.
Hands-on exercise
Troubleshooting: the group that started again from zero
Use tool output from the target cluster after a failover to find out why a consumer group is reprocessing old records, and decide how to stop it and how to prevent it. All output below is illustrative: written for this exercise in the format the tools produce.
Steps · 0 of 6 done
What you should see
Cause: sync.group.offsets.enabled defaults to false, so MirrorMaker emitted checkpoints but never wrote them into the dr cluster's __consumer_offsets. The group found no committed offset ('Found no committed offset'), applied auto.offset.reset=earliest and began at 0, with 176,004 records of lag on one partition alone. Right now: stop the consumers (offsets of a group with active members cannot be changed), set the group's offsets on primary.orders.placed to the checkpoints' downstream offsets with kafka-consumer-groups.sh --reset-offsets or with RemoteClusterUtils.translateOffsets, then restart. Had the group resumed from 181,100, it would have repeated at most the records between the checkpoint and its true position: on the order of the sync granularity (up to 100 records by default) plus what it consumed since the last checkpoint, not 181,000. Permanently: set sync.group.offsets.enabled = true, keep standby consumers stopped so the sync is allowed to write, make the Notification Service idempotent by event id because some replay always remains, and rehearse the failover so the missing setting is found in a drill.
What can go wrong
Consumers subscribed to a name the mirror does not fill
- Symptom
- After a failover the consumers start cleanly in the second region and process nothing. No errors, no lag reported on the topic they subscribe to.
- Cause
- With the default policy the mirrored data is in
primary.orders.placed. The consumers subscribe toorders.placed, which in the second cluster is empty or does not exist. - Fix
- Subscribe by a pattern that covers the local and the prefixed names, and run the standby consumers against the second cluster in drills to prove it.
Offsets never synced
- Symptom
- Failed-over consumers replay the whole retention of a topic, or skip everything that was pending, depending on
auto.offset.reset. - Cause
sync.group.offsets.enabledisfalseby default, the group is excluded by thegroupsfilter, or active consumers with the same group id in the target blocked the sync.- Fix
- Enable the sync, check that the group matches the filter, keep the standby group inactive, and verify before any incident that the group shows committed offsets in the target cluster.
Assuming the mirror is complete
- Symptom
- After a failover, a small number of orders exist in the order database with no trace in the second cluster. They reappear hours later when the failed region returns.
- Cause
- Replication is asynchronous. Records acknowledged shortly before the failure were never copied.
acks=allcovers only the source cluster. - Fix
- Measure replication latency and treat it as the RPO. Decide with the business how the unreplicated tail is reconciled, and build that reconciliation before the failover.
A consumer that cannot take a replay
- Symptom
- Duplicate emails, double stock movements or repeated charges after a failover or a drill.
- Cause
- Translated offsets are at or behind the true position by design, so a replay window always exists. The consumer has a side effect with no deduplication, or deduplicates on topic and offset, which change between clusters.
- Fix
- Deduplicate on a business or event id stored with the effect. This is application logic; producer idempotence does not help here.
Identity policy with flows in both directions
- Symptom
- Record counts and disk usage grow without matching traffic. The same events arrive again and again on both sides.
- Cause
IdentityReplicationPolicykeeps topic names unchanged, so MirrorMaker cannot tell a mirrored topic from a local one and re-mirrors it in the opposite direction.- Fix
- Use the default prefixing policy for any two-way topology. Keep the identity policy for strictly one-way replication and migrations.
Producing to a remote topic
- Symptom
- After a failover, new records are written into
primary.orders.placed. When the primary returns, its topic and its mirror no longer correspond, and offset translation gives wrong results. - Cause
- A producer was pointed at the mirrored topic to keep consumers working. Remote topics are meant to be written only by MirrorMaker.
- Fix
- Producers always write to the local topic name. Consumers read local and remote topics. Let ACL syncing keep remote topics read-only for applications.
Failback improvised on the day
- Symptom
- On returning to the original region, consumers replay from offsets committed before the outage, and records produced during the outage are missing there.
- Cause
- The old cluster's committed offsets are frozen at the failure, and nothing replicated the other cluster's new records back.
- Fix
- Write and rehearse a failback runbook with a reverse flow and explicit offset handling, or decide in advance to keep the surviving cluster as the new primary.
How to troubleshoot
Will this group resume correctly if we fail over now?
Check
kafka-consumer-groups.sh --bootstrap-server kafka-1.eu-central.example:9093 --describe --group notification-service
- How to read it
- Run it against the target cluster while the standby is idle. Committed offsets on
primary.orders.placedthat advance from one run to the next mean the sync works. No rows, or a group that does not exist, means a failover will fall back toauto.offset.reset.
Failed-over consumers started from the beginning or the end
Check
The consumer log lines `Found no committed offset for partition ...` and `Resetting offset for partition ...`
- How to read it
- They confirm the group had nothing committed in this cluster. Check
sync.group.offsets.enabled, thegroupsfilter and whether consumers of that group were already running in the target, which blocks the sync.
No checkpoints for a group
Check
kafka-console-consumer.sh --bootstrap-server kafka-1.eu-central.example:9093 --topic primary.checkpoints.internal --from-beginning --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter
- How to read it
- Each line names a group, a remote topic partition, the upstream offset and the translated downstream offset. A group that never appears is filtered out, has not committed in the source, or sits at a position for which no offset sync exists, for example one older than the start of replication.
Consumers in the second region receive nothing
Check
kafka-topics.sh --bootstrap-server kafka-1.eu-central.example:9093 --list
- How to read it
- Look for the prefixed name, such as
primary.orders.placed. If it exists and has data while the consumers subscribe toorders.placed, the subscription is the problem. If it does not exist, the topic is not matched bytopicsor has not been picked up yet.
How far behind is the mirror?
Check
The `replication-latency-ms` and `record-age-ms` metrics of `kafka.connect.mirror:type=MirrorSourceConnector`, per target, topic and partition
- How to read it
- Replication latency is the time from a record's timestamp to its acknowledgement in the target. Its recent maximum is your realistic RPO. Do not compare log end offsets between the clusters: they are different numbering and their difference means nothing.
Is the flow alive at all?
Check
kafka-console-consumer.sh --bootstrap-server kafka-1.eu-central.example:9093 --topic primary.heartbeats --formatter org.apache.kafka.connect.mirror.formatters.HeartbeatFormatter
- How to read it
- Heartbeats are emitted every second by default and mirrored like data. Fresh heartbeats arriving in the target show that the source cluster is reachable and the connectors are running. Silence means the flow, the link or the source is down.
A new topic is not mirrored
Check
The `topics` and `topics.exclude` settings of the flow, and `refresh.topics.interval.seconds`
- How to read it
- New topics are discovered on a schedule, every 600 seconds by default, so a delay of minutes is normal. If it never appears, the name does not match the filter, or another MirrorMaker process with a different configuration for the same target is overriding yours.
Key takeaways
- Stretch one cluster over nearby zones for zone failures. Use separate clusters with MirrorMaker 2 between distant regions.
- MirrorMaker 2 is a consumer and a producer. Replication is asynchronous, so the mirror is always behind and the RPO is the replication delay.
- Mirrored topics are prefixed with the source alias by default, and their offsets differ from the source. Consumers must subscribe to the remote name and need translated offsets.
- Checkpoints are emitted by default, but writing them into the target's group offsets is off until you set
sync.group.offsets.enabled. - Translated offsets are at or behind the real position. Every failover replays a window, so every consumer must be idempotent on a business key.
- Order is kept within a mirrored partition, never between a local topic and a remote one.
- Failback is harder than failover. Rehearse it, or plan not to do it.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A payments platform mirrors payments.captured from primary to dr. Producers use acks=all with min.insync.replicas=2 and replication factor 3. The topic takes about 1,200 records per second. The MirrorMaker metric replication-latency-ms-max has been between 600 and 900 over the last week. The disaster recovery document states that failover to dr loses no acknowledged payments.
What should the document say instead?
Scenario 2 of 3
An inventory consumer group fails over to dr with sync.group.offsets.enabled=true, checkpoints every 60 seconds and offset.lag.max at its default. It was consuming about 400 records per second with almost no lag. After the failover the team sees roughly 20,000 stock adjustments applied twice. The consumer applies each StockAdjusted event as quantity = quantity + delta.
What is the right fix?
Scenario 3 of 3
Two regions, west and east, each with local producers for orders.placed. A team wants consumers in both regions to see all orders under the single name orders.placed, so they configure replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy and enable west->east and east->west. Within an hour, disk usage on both clusters climbs steeply and consumers report every order many times.
What should they do?
References
References: Apache Kafka 4.4 documentation: Geo-replication (cross-cluster data mirroring) ↗ · Apache Kafka 4.4 documentation: MirrorMaker configs ↗ · Apache Kafka 4.4 documentation: Datacenters ↗ · KIP-382: MirrorMaker 2.0 ↗ · Apache Kafka source: IdentityReplicationPolicy (cycle warning) ↗ · Apache Kafka source: RemoteClusterUtils (translateOffsets) ↗ · Apache Kafka source: OffsetSyncStore (how offsets are translated) ↗ · Spring for Apache Kafka reference: @KafkaListener annotation (topicPattern) ↗ · Report a problem