Path 1 · lesson 4 of 6
The Kafka ecosystem: what is Apache and what is not
The pieces around a Kafka cluster, which of them ship with Apache Kafka, which come from vendors or other projects, and where each sits in the order system.
What you will learn
- Name the components that ship in the Apache Kafka download and say what each one does
- Tell apart a server process, a client library and a tool, and say who has to run each
- Identify which common companions (schema registries, ksqlDB, Debezium, non-Java clients) are separate projects and under which licence
- Place each component on the order → payment → inventory → notification system
- Decide whether a job needs your own consumer code, Kafka Connect or Kafka Streams
Before you start
The concept
People say "we use Kafka" and mean very different things: three brokers and a Spring Boot service, or a platform with connectors, stream processors, a schema registry and a second cluster in another region. Before the next lessons go inside the broker, this one draws the map: every box you are likely to meet, what it does, and who publishes it.
Two questions sort every component. What kind of thing is it? A server process you must operate, a library that runs inside your own application, or a command-line tool. Where does it come from? The Apache Kafka project (Apache License 2.0, one release, one version number), or somebody else: another open-source project, or a vendor.
What ships with Apache Kafka
| Component | Kind | What it does |
|---|---|---|
| Broker | Server process | Stores topic partitions on disk, serves produce and fetch requests, replicates partitions to other brokers. |
| KRaft controller | Server process | Holds the cluster metadata (topics, partitions, leaders, configs) in a replicated log and decides partition leadership. A quorum of 3 or 5 controllers is typical. |
Java clients (kafka-clients) | Library | Producer, Consumer, Share Consumer and Admin APIs. Spring for Apache Kafka wraps this library. |
| Kafka Connect | Server process (workers) plus a connector API | Runs connectors that copy data from other systems into topics (source) or out of topics (sink), with offsets, scaling and a REST interface handled for you. |
Kafka Streams (kafka-streams) | Library | Stream processing inside your own Java application: filter, join, aggregate, window. Reads topics and writes topics. No separate processing cluster. |
| MirrorMaker 2 | Tool built on Connect | Replicates topics, their configuration, and consumer group offsets from one Kafka cluster to another. |
Command-line tools (bin/*.sh) | Tools | Create and describe topics, inspect consumer groups, change configs, produce and consume from a terminal, format storage, inspect the controller quorum. |
apache/kafka Docker image).What does not ship with Apache Kafka
| Component | Who publishes it | Licence | What it adds |
|---|---|---|---|
| Confluent Schema Registry | Confluent (vendor) | Confluent Community License for the server. Its client and serializer modules are Apache 2.0. | A service that stores Avro, Protobuf and JSON Schema definitions and checks compatibility when a schema changes. |
| Apicurio Registry | Apicurio project (CNCF Sandbox) | Apache 2.0 | The same job: a schema registry with Kafka serializers and Connect converters. |
| ksqlDB | Confluent (vendor) | Confluent Community License | SQL statements over topics, executed by its own servers. |
| Debezium | Debezium project | Apache 2.0 | Change data capture connectors (PostgreSQL, MySQL and others) that turn database changes into Kafka records. Usually deployed on Kafka Connect. |
| Clients for other languages | Independent projects, e.g. librdkafka (C/C++, the base of several Python, Go and .NET clients) | Varies per client (librdkafka is 2-clause BSD) | Producer and consumer APIs outside the JVM. They speak the same wire protocol but are released separately. |
| Spring for Apache Kafka | Spring project | Apache 2.0 | KafkaTemplate, @KafkaListener, error handlers and transactions on top of the Apache Java client. |
| Kubernetes operators, e.g. Strimzi | Independent project | Apache 2.0 | Deploy and upgrade Kafka on Kubernetes through custom resources. |
| Managed services | Cloud providers and vendors | Commercial service terms | Somebody else runs the brokers and controllers. You still own topics, clients, keys and consumer logic. |
Why it exists
Kafka started as one thing: a replicated log with a producer and a consumer. Teams that adopted it kept rebuilding the same three pieces around it, and each of those pieces eventually moved into the project or grew next to it.
- Getting data in and out. Every team wrote its own program to copy a database table into a topic, or a topic into a search index, and each one re-solved offset tracking, restarts and scaling. Kafka Connect exists so that only the system-specific part (the connector) has to be written, and often somebody already has.
- Processing a stream. Counting, joining and windowing on top of a plain consumer means managing local state and recovering it after a crash. Kafka Streams packages that as a library that keeps its state in Kafka topics.
- Copying between clusters. Replication inside a cluster does not help when a whole region is lost or when two organisations each run their own cluster. MirrorMaker 2 is the project's tool for cluster-to-cluster replication.
- Agreeing on the shape of the data. The broker deliberately does not parse records, so producers and consumers need another way to agree on a format and to change it safely. That gap is filled outside the project by schema registries.
Knowing the boundary matters for practical reasons. It tells you which documentation is authoritative (kafka.apache.org for the first table, the vendor or project site for the second), which version numbers move together, what a licence allows, and what breaks if a company changes its product. It also stops a common planning mistake: assuming that a managed Kafka service includes a registry, connectors or stream processing. Some do, some charge separately, some do not offer them.
How it works inside
Two roles, one binary
A Kafka server is one program started with bin/kafka-server-start.sh. The process.roles setting decides what it does: broker, controller, or broker,controller (called combined mode). Brokers handle client traffic and store partitions. Controllers form a quorum, elect one active controller, and replicate the metadata log among themselves. Brokers follow that log to learn which topics exist and who leads each partition. The documentation recommends separate controller nodes for critical deployments and treats combined mode as suitable for development and small setups.
Clients do the work you might expect a server to do
A producer does not send records to "the cluster". It asks any broker for metadata, learns which broker leads each partition, chooses a partition for each record, batches records per partition, and sends each batch straight to that leader. Consumers likewise fetch directly from partition leaders and track their own position. This is why the client library version and its settings matter so much, and why the next paths spend whole lessons on producers and consumers.
Everything a client does goes over a documented, language-independent TCP protocol. The project maintains only the Java clients. Clients for other languages are independent projects that implement the same protocol, so a new broker feature can reach them later than it reaches Java, or with different configuration names. Check the client's own documentation instead of assuming the Java behaviour.
Connect and Streams are built from producers and consumers
A Kafka Connect worker is a JVM process. In distributed mode, workers with the same group.id form a Connect cluster, share out connector tasks among themselves, and store connector configuration, source offsets and status in three Kafka topics (which should be compacted). You add a connector by sending JSON to the worker's REST interface, which listens on port 8083 by default. A source task produces to topics using a normal producer; a sink task is a normal consumer in a consumer group. Connector code is a plugin placed on the worker's plugin.path. That is where Debezium fits: its connectors are plugins for Connect, published by a different project.
A Kafka Streams application is your own Java service with the kafka-streams library in it. Instances with the same application.id share the input partitions like a consumer group. State for aggregations lives in a local store and is backed up to changelog topics in Kafka, so another instance can rebuild it. There is nothing extra to deploy on the server side.
MirrorMaker 2 is a set of Connect connectors plus a launcher (bin/connect-mirror-maker.sh). It consumes from the source cluster and produces to the target cluster, and by default it names the replicated topic with the source cluster's alias as a prefix (orders from cluster eu becomes eu.orders). It is asynchronous, so the target is always slightly behind. The older MirrorMaker 1 tool is no longer in the 4.x distribution.
Where a schema registry plugs in
With a registry, the producer's serializer registers or looks up the schema, and writes a short schema identifier in front of the encoded payload. The consumer's deserializer reads the identifier, fetches the schema from the registry (and caches it), and decodes. The broker is not involved. If the registry is unreachable and the schema is not cached, clients cannot serialize or deserialize, so the registry is a production dependency of every service that uses it. Schema Registry and schema evolution covers this in depth.
Share Consumer API (in Apache Kafka)
- Part of
kafka-clients; production-ready from Kafka 4.2 - Queue-style consumption: records are acknowledged one by one and several consumers can share a partition
- Needs brokers and clients that are new enough
ksqlDB (not in Apache Kafka)
- A separate server product from Confluent under the Confluent Community License
- SQL over topics, executed by ksqlDB servers you deploy
- Often mistaken for a Kafka feature because of its name
Architecture
Drawing diagram…
Flowchart with three groups. Your services: order, payment, inventory and notification services and a Kafka Streams application. Apache Kafka release: KRaft controllers, brokers, Kafka Connect workers, MirrorMaker 2 and command-line tools. Separate projects: a Debezium connector plugin and a schema registry. Arrows show records flowing from the orders database through Debezium on Connect into the brokers, out to the services, and through MirrorMaker 2 to a second cluster.
flowchart LR
subgraph APPS["Your services"]
OS["Order Service - Spring Kafka producer"]
PS["Payment Service - consumer and producer"]
IS["Inventory Service - consumer"]
NS["Notification Service - consumer"]
ST["Fraud stats - Kafka Streams app"]
end
subgraph AK["Apache Kafka release"]
CT["KRaft controllers x3"]
BR["Brokers x3"]
CW["Kafka Connect workers"]
MM["MirrorMaker 2"]
CLI["Command-line tools"]
end
subgraph EXT["Separate projects and vendors"]
DBZ["Debezium connector plugin"]
SR["Schema registry"]
end
ODB[("Orders database")]
DR["Second cluster in another region"]
CT -->|"metadata log"| BR
OS --> ODB
ODB -->|"change stream"| DBZ
DBZ -->|"runs inside"| CW
CW -->|"produce orders"| BR
BR --> PS
BR --> IS
BR --> NS
PS -->|"produce payments"| BR
BR --> ST
ST -->|"produce aggregates"| BR
PS -.->|"schema lookup"| SR
OS -.->|"schema lookup"| SR
BR --> MM
MM --> DR
CLI -.->|"admin requests"| BR- Your services contain client libraries only. The Order, Payment, Inventory and Notification services use the Apache Java client through Spring for Apache Kafka. The fraud statistics job uses Kafka Streams. None of them is a Kafka server.
- The Apache Kafka release provides every server process in the middle: controllers that own the metadata, brokers that own the data, Connect workers that host connectors, and MirrorMaker 2 feeding a second cluster.
- Separate projects plug in at defined points. Debezium is a connector plugin loaded by a Connect worker. The schema registry is called by serializers inside the services, never by the brokers (dotted arrows).
- This is the full picture, not the minimum. A first version of the system needs only the brokers, the controllers and four services. Add Connect, Streams, a registry or a second cluster when a concrete requirement asks for it.
Message flow
Drawing diagram…
Sequence diagram. The Order Service writes an order and an outbox row to its database. A Connect worker running a Debezium connector reads the change and produces it to the orders topic. The Payment Service fetches it and produces to payments. A Kafka Streams application fetches payments and produces an aggregate. MirrorMaker 2 fetches both topics and produces them to a cluster in another region with a prefixed topic name.
sequenceDiagram participant O as Order Service participant D as Orders DB participant C as Connect worker with Debezium participant B as Kafka brokers participant P as Payment Service participant S as Streams app participant M as MirrorMaker 2 participant R as DR cluster O->>D: One transaction - insert order and outbox row D-->>C: Change event from the database log C->>B: Produce to orders with key order id B-->>P: Fetch from orders P->>B: Produce to payments B-->>S: Fetch from payments S->>B: Produce failures per minute B-->>M: Fetch from orders and payments M->>R: Produce to eu.orders and eu.payments
- The Order Service never talks to Kafka in this variant. It commits the order and an outbox row in one database transaction, and a Debezium connector on a Connect worker publishes the change. This is the transactional outbox with change data capture.
- The Payment Service is a plain consumer and producer: it fetches from
orders, charges the card, and produces the result topayments. - The Streams application reads
paymentsand writes a derived topic. Its output is just another topic that any consumer can read. - MirrorMaker 2 is one more consumer on the source cluster and one more producer on the target. From the brokers' point of view, Connect, Streams and MirrorMaker are all clients. That is the main idea of the ecosystem: one log, many kinds of client.
In a real system
Take the example project of this academy, the Order → Payment → Inventory → Notification system, and suppose it runs for a typical mid-size online shop. Here is where each ecosystem piece would sit and who would run it.
| Need | Component | Concretely |
|---|---|---|
| Store and deliver order events | Brokers and KRaft controllers (Apache) | 3 brokers and 3 controllers. Topic orders, 6 partitions, replication factor 3, key = order id. |
| Publish and handle events in services | Java client through Spring for Apache Kafka | KafkaTemplate in the Order and Payment services, @KafkaListener in Payment, Inventory and Notification, each with its own consumer group. |
| Publish orders reliably from the database | Kafka Connect (Apache) with a Debezium PostgreSQL connector (separate project) | 2 Connect workers with one group.id. The connector reads the outbox table and writes to orders. |
| Load orders into the warehouse | Kafka Connect with a sink connector for that warehouse (third party) | A sink task in its own consumer group. No custom code to maintain. |
| Failed payments per minute, per payment method | Kafka Streams (Apache) | A small Spring Boot service with application.id payment-failure-stats, reading payments, writing payment-failures-per-minute. |
| Keep event formats compatible between teams | A schema registry (Confluent's or Apicurio) | Avro or Protobuf serializers in each service. The registry rejects an incompatible change before it reaches a topic. |
| Survive the loss of a region | MirrorMaker 2 (Apache) | Flow eu->us. Topics arrive as eu.orders and eu.payments. Checkpoints of translated consumer group offsets are emitted by default; writing them into the target cluster's groups is an opt-in setting (sync.group.offsets.enabled, default false). |
| Day-to-day operations | Command-line tools (Apache) | kafka-topics.sh, kafka-consumer-groups.sh, kafka-configs.sh, kafka-metadata-quorum.sh. |
On a managed service the first row is rented instead of operated. Everything below it still needs an owner. Before choosing a provider, ask which rows it covers, at which Kafka version, and whether Connect plugins of your choice can be installed.
Java and Spring Boot
Four short files show how the pieces appear in a Java project and on a Kafka installation: the dependencies, a Kafka Streams topology, a Debezium connector registered on Kafka Connect, and a tour of the command-line tools. They are for orientation: later paths build each one properly.
<!-- Spring Boot's dependency management chooses matching versions: no <version> elements here. --><dependencies> <!-- Spring for Apache Kafka (Spring project). Brings in the Apache Java client, org.apache.kafka:kafka-clients, transitively. --> <dependency> <groupId>org.springframework.kafka</groupId> <artifactId>spring-kafka</artifactId> </dependency> <!-- Kafka Streams (Apache Kafka project). Only for services that process streams. It is a library, not a server. --> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams</artifactId> </dependency></dependencies>Why it is written this way
spring-kafkais a wrapper. The producer and consumer underneath are the Apachekafka-clientsclasses, so every client setting in the Kafka documentation still applies.- Leave versions to Spring Boot's dependency management. Spring for Apache Kafka 3.3 is built against the 3.8 to 3.9 clients; Kafka clients and brokers of different versions interoperate within the limits of the compatibility matrix in the Kafka documentation.
- A schema registry serializer would be one more dependency from that registry's publisher. It is not an
org.apache.kafkaartifact.
package com.example.shop.stats;import java.time.Duration;import org.apache.kafka.common.serialization.Serdes;import org.apache.kafka.streams.KeyValue;import org.apache.kafka.streams.StreamsBuilder;import org.apache.kafka.streams.kstream.Consumed;import org.apache.kafka.streams.kstream.Grouped;import org.apache.kafka.streams.kstream.KStream;import org.apache.kafka.streams.kstream.Produced;import org.apache.kafka.streams.kstream.TimeWindows;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.annotation.EnableKafkaStreams;import org.springframework.kafka.support.serializer.JsonSerde;@Configuration@EnableKafkaStreamspublic class PaymentFailureStats { public record PaymentEvent(String orderId, String method, String status, long amountCents) {} @Bean public KStream<String, PaymentEvent> paymentFailures(StreamsBuilder builder) { JsonSerde<PaymentEvent> paymentSerde = new JsonSerde<>(PaymentEvent.class); // Input topic "payments": key = order id, value = PaymentEvent as JSON. KStream<String, PaymentEvent> payments = builder.stream("payments", Consumed.with(Serdes.String(), paymentSerde)); payments .filter((orderId, payment) -> "FAILED".equals(payment.status())) // Re-key by payment method so that all failures of one method meet in one partition. .groupBy((orderId, payment) -> payment.method(), Grouped.with(Serdes.String(), paymentSerde)) .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(1))) .count() .toStream() .map((window, count) -> KeyValue.pair(window.key(), count)) .to("payment-failures-per-minute", Produced.with(Serdes.String(), Serdes.Long())); return payments; }}Why it is written this way
JsonSerdeis the Spring for Apache Kafka 3.x class name (4.x renames the JSON classes with aJacksonprefix). This is an ordinary Spring Boot service.@EnableKafkaStreamsplusspring.kafka.streams.application-idandspring.kafka.bootstrap-serversinapplication.ymlare enough for Spring Boot to start the topology.application.idis also the consumer group name and the prefix of the internal topics. Two instances with the same id split the input partitions between them.groupBywith a new key makes Kafka Streams create an internal repartition topic, andcountcreates a changelog topic. They appear inkafka-topics.sh --list: expected, not a leak.- The default
processing.guaranteeisat_least_once, so after a crash a window can count a record twice.exactly_once_v2uses Kafka transactions and covers only the topics involved, not any external system.
{ "name": "orders-outbox", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "database.hostname": "orders-db", "database.port": "5432", "database.user": "debezium", "database.password": "change-me", "database.dbname": "orders", "topic.prefix": "shop", "table.include.list": "public.outbox", "transforms": "outbox", "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter" }}Why it is written this way
- Kafka Connect (Apache) runs this; the connector class and the
EventRoutertransformation come from Debezium (a separate Apache-2.0 project) and must be installed on every worker'splugin.path. - Register it with
curl -X POST -H "Content-Type: application/json" --data @orders-outbox-connector.json http://connect:8083/connectors. In distributed mode the configuration is then stored in Connect's config topic and survives worker restarts. - With its default settings the outbox router sends each row to a topic named
outbox.event.followed by the row'saggregatetypevalue, and usesaggregateidas the record key. The route can be changed to write toorders; see the Debezium outbox event router documentation. - Do not keep a real password in the connector JSON. Connect supports configuration providers that resolve secrets at runtime.
# All of these scripts are in bin/ of the Apache Kafka download.# Which servers are controllers, and who is the active one?bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status# Topics: create, then look at partitions, leaders and in-sync replicas.bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic orders \ --partitions 6 --replication-factor 3bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders# Consumer groups: who reads what, and how far behind they are.bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --listbin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service# Read a topic from a terminal.bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders --from-beginning# Kafka Connect and MirrorMaker 2 are started from the same directory.bin/connect-distributed.sh config/connect-distributed.propertiesbin/connect-mirror-maker.sh mm2.properties# Connect is managed over REST, not with a kafka-*.sh script.curl http://localhost:8083/connector-pluginscurl http://localhost:8083/connectors/orders-outbox/statusWhy it is written this way
- The tools take
--bootstrap-serverand talk to the brokers through the Admin API. There is no ZooKeeper address anywhere in 4.x. kafka-consumer-groups.sh --describeis the command you will use most: it shows the committed offset, the log end offset and the lag for every partition of a group./connector-pluginstells you which connector classes a worker can actually load. If Debezium is missing from that list, the plugin is not onplugin.path.
Configuration
A few settings decide what role a process plays and how the ecosystem pieces find each other. Defaults are from the Apache Kafka 4.4 configuration reference.
process.rolesDefault none (must be set)
- What it does
- What a Kafka server does:
broker,controller, orbroker,controllerfor a combined node. - When to change it
- Use combined mode for a laptop or a test environment. For production the documentation recommends dedicated controllers, so that they can be restarted and scaled separately from the brokers.
controller.quorum.bootstrap.serversDefault none (must be set)
- What it does
- The host and port list every broker and controller uses to find the controller quorum.
- When to change it
- List all controllers. Older KRaft clusters use the static
controller.quorum.votersinstead, which the docs now describe as the old way of defining the quorum.
plugin.path (Connect worker)Default null
- What it does
- Directories or archives from which a Connect worker loads connectors, converters and transformations.
- When to change it
- Set it on every worker and install third-party connectors such as Debezium there. Even the example file connectors shipped with Kafka are not loaded unless their jar is added here.
group.id (Connect worker, distributed mode)Default none (required)
- What it does
- Identifies the Connect cluster a worker belongs to. Workers with the same value share connectors and tasks.
- When to change it
- Give each Connect cluster its own value, and never reuse the name of an application's consumer group.
config.storage.topic,offset.storage.topicandstatus.storage.topicmust also be set per Connect cluster.
listeners (Connect worker)Default http://:8083
- What it does
- Where the Connect REST interface listens.
- When to change it
- Anyone who can reach this port can create and delete connectors, so restrict network access to it or secure it with HTTPS and authentication.
application.id (Kafka Streams)Default none (required)
- What it does
- Identifies a Streams application. Used as its consumer group name and as the prefix of its internal topics.
- When to change it
- Keep it stable across deployments. Changing it makes the application start again as a new group with empty state. In Spring Boot it is
spring.kafka.streams.application-id.
processing.guarantee (Kafka Streams)Default at_least_once
- What it does
- Whether Streams commits with plain offset commits or wraps input offsets, state changes and output records in Kafka transactions (
exactly_once_v2). - When to change it
- Switch to
exactly_once_v2when double counting in output topics is not acceptable. It requires at least three brokers by default and does not extend to calls a processor makes to other systems.
Hands-on exercise
Find the controllers, then draw your own ecosystem map
See that brokers and controllers are different roles of the same cluster, then classify every component of the order system by kind, publisher and owner.
Steps · 0 of 6 done
Brokers and the controller
Simulation · not a real clusterThree brokers and the `orders` and `payments` topics. Inspect the controller, then stop a broker and see who reacts.
What you should see
The controller, not the clients and not the failed broker, moves partition leadership to a surviving in-sync replica. On paper you should end with three server processes from Apache (broker, controller, Connect worker; MirrorMaker 2 runs as Connect connectors), two Apache libraries (kafka-clients, Kafka Streams), and three things from elsewhere (Spring for Apache Kafka, the Debezium connector, the schema registry). Only brokers, controllers and a client library are needed on day one.
What can go wrong
Assuming a vendor component is part of Apache Kafka
- Symptom
- A design relies on a schema registry, ksqlDB or a specific connector, and late in the project it turns out the managed service does not offer it or the licence does not fit the intended use.
- Cause
- Vendor distributions bundle Apache Kafka with their own components under one product name, so the boundary is easy to miss. The Confluent Community License, for example, is not an open-source licence in the Apache sense: it excludes offering the software as a competing hosted service.
- Fix
- For every component in the design, write down publisher, licence and who operates it. Read the licence file in the repository of the version you plan to run.
Treating the schema registry as optional infrastructure
- Symptom
- After a registry outage or a network change, freshly started service instances fail in the serializer or deserializer although the brokers are healthy.
- Cause
- Serializers fetch schemas from the registry. Instances that already cached the schemas keep working, new ones cannot start processing.
- Fix
- Run the registry with the same availability as the services that depend on it, monitor it, and include it in failover plans for a second region.
Writing a custom consumer where a connector exists
- Symptom
- A hand-written service copies a topic into a database or search index and keeps needing fixes for restarts, duplicates, retries and scaling.
- Cause
- Offset management, task distribution and error handling were re-implemented for a pure copy job.
- Fix
- Use Kafka Connect when the job is moving data between Kafka and another system without business logic. Keep your own consumers for code that makes business decisions.
Expecting non-Java clients to behave like the Java client
- Symptom
- A Python or Go service partitions the same keys differently from a Java service, or lacks a feature the Kafka documentation describes.
- Cause
- Other clients are independent implementations with their own defaults, including the default partitioner, and their own release schedule.
- Fix
- Read that client's documentation, set the partitioner explicitly when several languages write to one keyed topic, and test cross-language ordering before relying on it.
Using MirrorMaker 2 as if it were synchronous replication
- Symptom
- After failing over to the second region, the newest records are missing and consumers reprocess some older ones.
- Cause
- MirrorMaker 2 copies asynchronously, and offsets in the target cluster are not the same numbers as in the source. Translated consumer offsets are checkpointed periodically (every 60 seconds by default) and are applied to the target groups only when
sync.group.offsets.enabledis turned on. - Fix
- Plan for a small loss window and for duplicates on failover, make consumers idempotent, and rehearse the failover. See multi-region replication.
How to troubleshoot
You do not know whether a cluster runs dedicated controllers or which one is active
Check
bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
- How to read it
- The output shows the leader id of the metadata quorum and the current voters. If the voter ids are also broker ids, the cluster runs combined nodes.
A connector you POSTed to Connect is rejected because the class cannot be found
Check
curl http://connect:8083/connector-plugins
- How to read it
- Only classes in this list can be used. If the Debezium class is missing, the plugin is not on
plugin.pathof the worker that answered. In a multi-worker cluster, check every worker.
A connector exists but no records arrive in the topic
Check
curl http://connect:8083/connectors/orders-outbox/status
- How to read it
- The response gives the state of the connector and of each task, the worker each one runs on, and error information for anything that failed. Read the tasks, not only the connector: a connector can be running while one of its tasks has failed.
Unknown topics with long generated names appear in the cluster
Check
bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
- How to read it
- Names that start with a Streams
application.idand containchangelogorrepartitionare internal topics of that Kafka Streams application. Connect's three storage topics and__consumer_offsetsare internal too. Topics prefixed with a cluster alias come from MirrorMaker 2. Do not delete them.
It is unclear whether a Streams application or a sink connector is keeping up
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-failure-stats
- How to read it
- A Streams application on the classic protocol is a consumer group named after its
application.id, so the usual lag columns apply. Sink connectors are consumer groups too: find the group name with--list. Applications using the newer Streams rebalance protocol are inspected withkafka-streams-groups.shinstead.
Key takeaways
- The Apache Kafka release contains brokers, KRaft controllers, the Java clients (producer, consumer, share consumer, admin), Kafka Connect, Kafka Streams, MirrorMaker 2 and the command-line tools, all under the Apache License 2.0.
- Schema registries, ksqlDB, Debezium, non-Java clients, Spring for Apache Kafka, Kubernetes operators and managed services are separate. Each has its own publisher, licence, version and documentation.
- Brokers store and replicate bytes. Schemas, formats and compatibility checks happen in client serializers and an external registry.
- Connect, Streams and MirrorMaker 2 are built from ordinary producers and consumers. To the brokers they are just clients.
- Use your own consumers for business logic, Connect for copying data between Kafka and other systems, and Streams for computing new topics from existing ones.
- Start with brokers, controllers and client libraries. Add each further component for a stated requirement, with a named owner.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
The Order Service must load every record of the orders topic into the company's analytics warehouse. No business rule is applied: each record becomes one row. A colleague proposes a new Spring Boot service with a @KafkaListener and a JDBC batch insert, estimated at two weeks including retries, restarts and scaling to several instances.
What do you propose instead?
Scenario 2 of 3
Your company moves from a self-managed vendor platform to a managed service that advertises "100% Apache Kafka compatible". The current system uses Avro with a schema registry, two ksqlDB queries and a Debezium PostgreSQL connector. The migration plan lists only brokers, topics and client configuration.
What is the most important gap in the plan?
Scenario 3 of 3
A team adds a Go service that produces inventory events with the SKU as key into a topic that a Java service also writes with the same keys. A week later the Inventory Service reports that updates for one SKU sometimes arrive out of order. Both producers are healthy and each one's records for that SKU are in order within the partition it wrote to.
Where do you look first?
References
References: Apache Kafka documentation: Introduction (servers, clients, the core APIs) ↗ · Apache Kafka documentation: API (Producer, Consumer, Share Consumer, Streams, Connect, Admin; only the Java clients are maintained in the project) ↗ · Apache Kafka documentation: KRaft (process roles, controllers, quorum tools) ↗ · Apache Kafka documentation: Kafka Connect user guide (workers, storage topics, REST API, plugin.path) ↗ · Apache Kafka documentation: Geo-replication with MirrorMaker ↗ · Apache Kafka documentation: Kafka Streams introduction ↗ · Apache Kafka documentation: Upgrading (ZooKeeper removal in 4.0, share groups production-ready in 4.2) ↗ · Apache Kafka wiki: Clients (how the project handles non-Java clients) ↗ · Spring for Apache Kafka reference: Apache Kafka Streams support ↗ · Spring for Apache Kafka project page: client and Spring Boot compatibility ↗ · Confluent Schema Registry repository: licence section of the README ↗ · ksqlDB repository: licence ↗ · Apicurio Registry ↗ · Debezium documentation: PostgreSQL connector ↗ · Debezium documentation: Outbox event router ↗ · librdkafka repository (C/C++ client) ↗ · Strimzi: Apache Kafka on Kubernetes ↗ · Report a problem