Skip to content

Path 1 · lesson 5 of 6

Foundation16 minApache Kafka 4.4 documentation (KRaft only) · Java 21 · Spring Boot 3.4 with Spring for Apache Kafka 3.3 (kafka-clients 3.8 to 3.9)

What Kafka is used for: seven families of use

The seven uses named in the Kafka documentation, each turned into topics, keys, retention and consumers for a concrete online shop.

What you will learn

  • Recognise which of the seven documented families a requirement belongs to
  • Choose the topic, the record key and the consumers for each family
  • Pick retention or compaction for a topic from how its data will be read later
  • Decide how much durability each kind of data deserves, and configure the producer to match
  • Explain why one topic can serve several of these uses at once without the producer knowing

Before you start

The concept

The Kafka documentation has a short page called "Use cases" that lists seven families: messaging, website activity tracking, metrics, log aggregation, stream processing, event sourcing and commit log. The page describes them in a paragraph each. This lesson makes each one concrete: if you built it for an online shop, what would the topic be, what would the key be, how long would the data stay, and who would read it?

All seven use the same four mechanisms you met in What is Kafka: an append-only log per partition, a key that decides the partition, retention that is independent of reading, and consumer groups that each keep their own position. The families differ only in how they set those four dials.

FamilyTypical topicKeyKept forWho reads it
MessagingordersOrder idDaysOne consumer group per service that must react: payment, inventory, notification
Website activity trackingpage-views, searches (one topic per activity type)Session id or user idDaysRecommendations, fraud checks, a sink connector to the warehouse
Metricsservice-metricsService instanceHours to daysAlerting, a sink into a time-series database
Log aggregationapp-logsNone, or the service instanceHours to daysA sink into a log search system, ad hoc debugging consumers
Stream processingpayments in, payment-failures-per-minute outInput: order id. Output: whatever the result is grouped byAs the input and output each needA Kafka Streams application, then consumers of its output topic
Event sourcingorder-eventsOrder idForever, or as long as the entity mattersThe owning service when it rebuilds state, plus any new read model
Commit loginventory-levels (compacted)SKULatest value per keyInstances restoring their state, caches reloading after a restart
The same shop, seven ways of using a topic. Names and numbers are illustrative.

Why it exists

Before a shared log, each of these needs was met by a different tool. Services talked through a message broker. Click data was written to files and loaded into a warehouse overnight. Metrics went to a monitoring agent, logs to a log shipper, and database changes to a replication stream nobody else could read. Each pipe had one source and one destination, so every new use of existing data meant building another pipe.

Kafka's first job at LinkedIn was the second item in the list. Jay Kreps, one of its authors, describes on the LinkedIn engineering blog how Kafka was built to be a central pipeline first for activity data and then for other uses such as monitoring data, with several hundred event types covering page views, ad impressions, searches, service invocations and application exceptions. The point of that design was that a new consumer could subscribe to data that already flowed, without the producing team doing anything.

Three properties of the log make one system fit all seven families:

  • Reading does not remove. A record stays until retention or compaction removes it, so any number of consumer groups can read the same topic at their own pace, and a group created next month can still read last week's data if it is retained.
  • Order per key. Records with the same key go to the same partition and are read in the order the broker appended them, which is what messaging between services, event sourcing and a commit log all depend on.
  • Sequential disk writes and batching. High-volume, low-value data such as clicks and logs can share the infrastructure with low-volume, high-value data such as orders.

How it works inside

For each family, here is which dial matters most and why.

1. Messaging: a consumer group per reaction

One service publishes a fact ("order 1042 was placed") and others react. Each reacting service uses its own group.id, so each gets every record. Inside one group, the partitions are divided among the instances, so each record is handled by one instance of that service. The key is the business entity whose events must stay in order, here the order id. Volume is usually modest; what matters is that nothing is lost: acks=all, replication factor 3 and min.insync.replicas=2. Processing is at-least-once by default, so handlers must tolerate a repeated record.

2. Activity tracking: volume first

Every page view, search and click becomes a record. The documentation suggests one topic per activity type. Volume is high because one page view can produce many records, so these topics get more partitions, compression and batching. Keying by session id keeps one visitor's actions in order for sessionisation; keying by nothing spreads load evenly when order does not matter. Losing a handful of clicks during a broker failure is often acceptable, so some teams choose weaker acknowledgement here. That is a business decision to make explicitly per topic, never a default to copy onto orders.

3 and 4. Metrics and log aggregation: a buffer in front of slower systems

Applications or agents publish measurements and log lines; sink connectors move them into a time-series database and a log search system. Kafka's role is a buffer that absorbs bursts and lets several destinations read the same feed. Retention is short, because the destination is the long-term store. One caution: if the Kafka cluster is what you are debugging, logs and metrics that travel through that same cluster will not reach you. Keep an independent path for monitoring Kafka itself.

5. Stream processing: topics in, topics out

A processor reads one or more topics, transforms, joins or aggregates, and writes the result to a new topic. Stages chain into a pipeline, and every intermediate topic is itself readable by anyone. Kafka Streams is the library for this in the Apache project. Two details surprise newcomers. An aggregation needs all records of a group in one partition, so re-keying creates an internal repartition topic. And exactly-once processing in Streams covers reading, state and writing inside Kafka; it does not cover an email or a database write made from the processor.

6. Event sourcing: the log is the source of truth

Instead of storing only the current state of an order, the service stores every state change as an event and derives the current state by replaying them. The topic is keyed by entity id and retained without a time limit (retention.ms=-1). The benefit is a full history and the ability to build new read models from the beginning. The cost is real: event schemas must stay readable for years, replays get longer as history grows, and Kafka cannot fetch "all events of order 1042" without scanning a partition, so most designs keep a database view next to the log. Event sourcing covers the pattern.

7. Commit log: latest value per key

Here the topic backs up the state of some other system. Each change is written as key and new value, and a restarting node reads the topic to restore itself. Because only the latest value per key matters, the topic uses log compaction (cleanup.policy=compact) instead of time-based deletion. Kafka uses the same idea for itself: consumer group offsets are stored as records in an internal topic, Kafka Connect keeps connector configuration, offsets and status in compacted topics, and Kafka Streams backs its state stores with changelog topics.

Event sourcing topic

  • Keeps every event: OrderPlaced, OrderPaid, OrderShipped
  • cleanup.policy=delete with retention.ms=-1
  • Replay gives the full history and any state in between
  • Grows without bound; consider tiered storage for old segments

Commit log (compacted) topic

  • Keeps at least the latest value per key: order 1042 is SHIPPED
  • cleanup.policy=compact
  • Replay gives the current state only
  • Size is bounded by the number of distinct keys, not by time

Partitions & consumer groups

Simulation, not a real cluster

Animated lesson: how partitions are shared inside one consumer group, and why a second group gets its own copy of every record.

Step 1 · The pattern

A log and a reader

The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.

  • message
  • consumed
Open the full animated lesson

Architecture

Drawing diagram…

Flowchart of a Kafka cluster with seven topics. Frontends write page views, the Order Service writes orders and order events, agents write metrics and logs, the Payment Service writes payments. Orders are read by payment, inventory and notification services and by a warehouse sink. Page views go to the warehouse. Metrics and logs go to monitoring sinks. Payments feed a Kafka Streams application that writes failures per minute for an alerting consumer. Order events feed a read model. The Inventory Service writes and restores from a compacted inventory levels topic.

flowchart LR
  WEB["Web and mobile frontends"]
  OS["Order Service"]
  PAY["Payment Service"]
  AG["Metrics and log agents"]
  subgraph KAFKA["Kafka cluster"]
    T1["orders - key order id - 7 days"]
    T2["page-views - key session id - 3 days"]
    T3["service-metrics and app-logs - 1 day"]
    T4["payments - key order id - 7 days"]
    T5["payment-failures-per-minute"]
    T6["order-events - kept forever"]
    T7["inventory-levels - compacted"]
  end
  WEB --> T2
  OS --> T1
  OS --> T6
  AG --> T3
  PAY --> T4
  T1 -->|"messaging"| PAY
  T1 -->|"messaging"| INV["Inventory Service"]
  T1 -->|"messaging"| NOTIF["Notification Service"]
  T2 -->|"activity tracking"| DWH["Warehouse sink connector"]
  T1 --> DWH
  T3 -->|"metrics and logs"| OBS["Monitoring and log search sinks"]
  T4 -->|"stream processing"| KS["Kafka Streams app"]
  KS --> T5
  T5 --> ALERT["Alerting consumer"]
  T6 -->|"event sourcing"| RM["Order history read model"]
  INV --> T7
  T7 -->|"commit log"| INV
One cluster for a shop: seven topic shapes and the consumers each one feeds.
  • Producers are on the left and write to topics without knowing the readers. The label on each outgoing arrow names the family that this consumer represents.
  • orders has four readers in four consumer groups: three services (messaging) and the warehouse sink. Adding the fourth required no change to the Order Service.
  • The Kafka Streams application turns payments into payment-failures-per-minute. The alerting consumer reads the small derived topic instead of recomputing from the raw one.
  • inventory-levels is both written and read by the Inventory Service: it is that service's own changelog, used to restore stock levels after a restart. Compaction keeps it from growing with every stock movement.
  • Retention differs per topic because the purpose differs: days where consumers only need to catch up, forever where the topic is the record of truth, latest-per-key where only current state matters.

Message flow

Drawing diagram…

Sequence diagram. The Order Service produces an OrderPlaced record to the orders topic and gets an acknowledgement with partition and offset. The payment group, the warehouse sink group and a Streams application each fetch the same record and record their own progress. Three days later a new fraud detection group joins with auto.offset.reset set to earliest and reads the retained records from the oldest one.

sequenceDiagram
  participant O as Order Service
  participant K as orders topic
  participant P as Payment group
  participant W as Warehouse sink group
  participant S as Streams app
  participant N as New fraud group
  O->>K: Produce OrderPlaced with key 1042
  K-->>O: Acknowledged at partition 3 offset 880
  K-->>P: Fetch returns offset 880
  P->>K: Commit offset 881 for payment group
  K-->>W: Fetch returns offset 880
  W->>K: Commit offset 881 for warehouse group
  K-->>S: Fetch returns offset 880
  S->>K: Produce to orders-per-minute
  Note over K: Record stays for the 7 day retention
  N->>K: Join three days later with auto.offset.reset earliest
  K-->>N: Fetch returns offsets from the oldest retained
  N->>K: Commit own position for fraud group
One record, four independent readers. Each group commits its own position, and a group that did not exist when the record was written can still read it.
  1. The producer's part ends at the acknowledgement. Whether zero or ten groups read the record makes no difference to it.
  2. Each group commits the offset of the next record it will read (881 after handling 880). The commit records a position for that group and partition; it does not acknowledge or delete the record.
  3. The Streams application is one more reader, and its result is one more topic.
  4. The fraud group starts three days later. A group with no committed offset starts according to auto.offset.reset, whose default is latest: only new records. To process the retained history it must be set to earliest before the group's first start.

In a real system

Consider a typical online shop with about 40 orders per minute at peak and around 3,000 page views per second during a sale. The two streams differ by more than three orders of magnitude in volume and in the opposite direction in value per record, so they are configured differently on the same cluster.

TopicPartitionsKeyProducer settingsRetentionConsumer groups
orders6Order idacks=all, idempotence on (the defaults)7 dayspayment-service, inventory-service, notification-service, warehouse sink
page-views24Session idCompression and a longer linger.ms for bigger batches3 daysrecommendations, fraud-signals, warehouse sink
app-logs12NoneCompression; loss of a few lines tolerated1 dayLog search sink
order-events6Order idacks=all, idempotence onUnlimitedorder-history-view, any future read model
inventory-levels6SKUacks=all, idempotence onCompactedThe Inventory Service itself on restart
Illustrative sizing. Partition counts are derived from throughput and consumer parallelism in the performance path.

Follow one order. The Order Service writes OrderPlaced with key 1042 to orders. The Payment Service's group reads it and charges the card; the Inventory Service's group reserves stock; the Notification Service's group sends the confirmation email. That is messaging. The same record is copied to the warehouse by a sink connector, where it is joined with the customer's page views from the session: activity tracking. A Streams job counts orders per minute for a dashboard: stream processing. None of these readers slows the others down, and if the warehouse sink is stopped for a day, it resumes from its committed offsets because the records are still retained.

For a public example, the LinkedIn engineering post linked in the references describes the original deployment: Kafka as the central multi-subscriber event log for activity data, feeding both real-time systems and offline processing from the same topics.

Java and Spring Boot

Three files show how the differences between the families appear in a Spring Boot service: topics declared with different retention, a high-volume activity publisher, and a second consumer group added to an existing topic.

ShopTopics.javaJava
package com.example.shop.config;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class ShopTopics {    private static final String SEVEN_DAYS_MS = String.valueOf(7L * 24 * 60 * 60 * 1000);    private static final String THREE_DAYS_MS = String.valueOf(3L * 24 * 60 * 60 * 1000);    /** Messaging: modest volume, must not be lost, ordered per order id. */    @Bean    public NewTopic orders() {        return TopicBuilder.name("orders")                .partitions(6)                .replicas(3)                .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")                .config(TopicConfig.RETENTION_MS_CONFIG, SEVEN_DAYS_MS)                .build();    }    /** Activity tracking: high volume, more partitions, shorter retention. */    @Bean    public NewTopic pageViews() {        return TopicBuilder.name("page-views")                .partitions(24)                .replicas(3)                .config(TopicConfig.RETENTION_MS_CONFIG, THREE_DAYS_MS)                .build();    }    /** Event sourcing: the history is the data, so no time limit. */    @Bean    public NewTopic orderEvents() {        return TopicBuilder.name("order-events")                .partitions(6)                .replicas(3)                .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")                .config(TopicConfig.RETENTION_MS_CONFIG, "-1")                .build();    }    /** Commit log: only the latest stock level per SKU matters. */    @Bean    public NewTopic inventoryLevels() {        return TopicBuilder.name("inventory-levels")                .partitions(6)                .replicas(3)                .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2")                .compact()                .build();    }}

Why it is written this way

  • Spring Boot's auto-configured KafkaAdmin creates each NewTopic bean at startup if the topic does not exist. For a topic that already exists it adds partitions when the bean declares more than the topic has; do not rely on it to change retention or other settings of an existing topic.
  • Declaring topics in code keeps partition count, replication and retention reviewable. Relying on broker-side auto-creation gives every topic the broker defaults: one partition and replication factor 1 unless the cluster was configured otherwise.
  • min.insync.replicas=2 with replication factor 3 is what makes the producer's default acks=all a real durability guarantee while still tolerating one broker being down.
  • .compact() sets cleanup.policy=compact. Records in a compacted topic must have a key.

Configuration

The settings that turn one topic shape into another. Defaults are from the Apache Kafka 4.4 configuration reference; topic-level names are shown, each has a broker-level default.

cleanup.policyDefault delete
What it does
delete removes old segments by time or size. compact keeps at least the latest record per key. Both can be combined.
When to change it
Use compact for commit-log and current-state topics (always with keys). Keep delete for messaging, tracking, metrics, logs and event-sourced history.
retention.msDefault 604800000 (7 days)
What it does
How long a record is kept under the delete policy before its segment becomes eligible for removal. -1 means no time limit.
When to change it
Shorten it for high-volume data that has a long-term home elsewhere. Set -1 for event-sourced topics. Size it to cover the longest consumer outage you want to recover from without loss.
retention.bytesDefault -1 (no size limit)
What it does
Maximum size of one partition before old segments are removed, under the delete policy.
When to change it
Set it on log and metric topics to cap disk use during a burst. It applies per partition, so multiply by the partition count.
min.insync.replicasDefault 1
What it does
How many in-sync replicas, including the leader, must have a record before a producer using acks=all gets a success.
When to change it
Set 2 with replication factor 3 for orders, payments and event-sourced topics. With the default of 1, acks=all can succeed with a single copy.
acks (producer)Default all
What it does
How many acknowledgements the producer waits for: none (0), the leader only (1), or all in-sync replicas (all).
When to change it
Keep all for business events. Lowering it for clickstream or logs trades possible loss on leader failure for less waiting, and also disables idempotence.
compression.type (producer)Default none
What it does
Compresses whole batches on the producer: gzip, snappy, lz4 or zstd.
When to change it
Turn it on for activity, log and metric topics, where records are repetitive and batches are large. Measure CPU before enabling on latency-sensitive producers.
auto.offset.reset (consumer)Default latest
What it does
Where a group starts when it has no committed offset, or when its committed offset no longer exists on the broker.
When to change it
Use earliest for a new consumer that must process retained history, and for consumers where skipping data silently is worse than reprocessing.

Hands-on exercise

Shape three topics for three different uses

See on a simulated partition what time-based retention and compaction each keep, then choose key, retention and consumers for new requirements of the shop.

Steps · 0 of 6 done

Retention versus compaction on one partition

Simulation · not a real cluster

A partition of order status updates. Compact it, then compare with time-based retention.

What can go wrong

Copying the settings of one family onto another
Symptom
Orders occasionally go missing after a broker restart, or the clickstream producer is slow and expensive.
Cause
A producer configuration tuned for tracking data (acks=1 or 0) was reused for business events, or the reverse: small unbatched, uncompressed sends for high-volume data.
Fix
Classify each topic by family first, then set acknowledgements, replication, batching and compression for that topic's producers. Keep the two producer configurations separate in code.
Using a compacted topic as if it were a history
Symptom
A new consumer replays an order-status topic to rebuild an audit trail and finds only the latest status of older orders.
Cause
The topic has cleanup.policy=compact, so earlier values per key were removed once the cleaner ran.
Fix
Keep history in a topic with the delete policy and unlimited retention. Publish a compacted current-state topic next to it if consumers need fast reload.
Retention shorter than the longest consumer outage
Symptom
After a weekend outage of the warehouse sink, Monday's reports are missing Saturday's data, and the consumer's log shows its offset was reset.
Cause
The topic kept 24 hours of data. The group's committed offset pointed at segments that had been deleted, and auto.offset.reset moved it.
Fix
Set retention from the recovery time you need, alert on consumer lag measured in time, and decide deliberately between earliest and latest for each consumer.
One topic for all event types and all purposes
Symptom
Every consumer deserializes and discards most records, a schema change in one event type breaks unrelated consumers, and retention cannot suit everyone.
Cause
A single events topic mixes orders, page views and logs, which need different keys, volumes, durability and retention.
Fix
Group events that must be ordered relative to each other in one topic (all events of an order). Separate streams whose volume, retention or durability differ.
Event sourcing without a plan for reading
Symptom
Showing one order's history takes seconds, or a restart takes longer every month.
Cause
Kafka has no lookup by key: finding one entity's events means scanning a partition, and rebuilding state means replaying everything.
Fix
Maintain a read model in a database fed by a consumer, and snapshot state so a restart replays only recent events. Use Kafka as the log, not as the query engine.

How to troubleshoot

You are not sure which family an existing topic was set up for

Check

bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name inventory-levels --describe
How to read it
The output lists the settings that differ from the broker defaults. cleanup.policy=compact marks a commit-log topic, retention.ms=-1 a keep-forever topic. No overrides means broker defaults, usually 7 days of retention.
You need to know who reads a topic before changing it

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list   (then --describe --group <name> for each group)
How to read it
Each row of a description is a topic and partition with the committed offset, the log end offset and the lag. Every group listed for orders is an independent use of that topic. A group with no active members but a large lag is a stopped consumer racing against retention.
A newly deployed consumer processes nothing although the topic is full of records

Check

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-analytics
How to read it
If CURRENT-OFFSET equals LOG-END-OFFSET and the lag is 0, the group started at the end because auto.offset.reset is latest. Stop the application and reset the group with --reset-offsets --to-earliest --topic orders --execute, or use a new group id with earliest.
Records for one order appear in a different order to different consumers

Check

bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
How to read it
Check the partition count against what it was when the topic was created. Adding partitions changes which partition a key maps to, so old and new events of one order can sit in different partitions. Also confirm that every producer of that topic uses the same key.

Key takeaways

  • The documentation names seven families: messaging, website activity tracking, metrics, log aggregation, stream processing, event sourcing and commit log.
  • All seven are built from the same parts. They differ in key, partition count, retention or compaction, durability settings and who reads.
  • A consumer group is one independent use of a topic. Adding a use means adding a group, not changing the producer.
  • Retention is independent of consumption and is also the deadline for a stopped consumer to come back.
  • Use delete with unlimited retention when history matters, and compact when only the latest value per key matters. Compaction keeps at least the latest value, not exactly one.
  • Decide durability per topic: business events get acks=all, replication factor 3 and min.insync.replicas=2; lossy settings are a deliberate choice for low-value, high-volume data only.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A delivery company wants riders' phones to send a GPS position every 5 seconds: about 8,000 positions per second at peak. The dispatch screen needs each rider's latest position, and a data team wants 30 days of raw positions for route analysis. Per rider, positions must not be shown out of order.

Which topic design fits both needs?

Scenario 2 of 3

The team that owns orders (key = order id, 7-day retention, acks=all) is asked to feed a new loyalty-points service. The proposal on the table: the Order Service makes an extra REST call to the loyalty service after publishing each order event, "so we are sure they get it".

What is the better design?

Scenario 3 of 3

An app-logs topic receives about 20 MB per second and has retention.ms of 7 days because it was created with broker defaults. Brokers are at 85% disk. The log search system, fed by a sink connector, already keeps 30 days and the connector's lag is normally under one minute. The orders topic on the same brokers is 2 GB in total.

What do you change first?

References

References: Apache Kafka documentation: Use cases (the seven families) ↗ · Apache Kafka documentation: Design (log compaction, message delivery semantics) ↗ · Apache Kafka documentation: Topic configs (cleanup.policy, retention.ms, retention.bytes, min.insync.replicas) ↗ · Apache Kafka documentation: Consumer configs (auto.offset.reset) ↗ · Apache Kafka documentation: Kafka Connect user guide (compacted config, offset and status topics) ↗ · Spring for Apache Kafka reference: Configuring topics (NewTopic, TopicBuilder) ↗ · Spring for Apache Kafka reference: Sending messages (KafkaTemplate) ↗ · LinkedIn Engineering: The Log: What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, 2013) ↗ · Martin Fowler: Event Sourcing (linked from the Kafka use cases page) ↗ · Report a problem