Path 1 · lesson 5 of 6
What Kafka is used for: seven families of use
The seven uses named in the Kafka documentation, each turned into topics, keys, retention and consumers for a concrete online shop.
What you will learn
- Recognise which of the seven documented families a requirement belongs to
- Choose the topic, the record key and the consumers for each family
- Pick retention or compaction for a topic from how its data will be read later
- Decide how much durability each kind of data deserves, and configure the producer to match
- Explain why one topic can serve several of these uses at once without the producer knowing
Before you start
The concept
The Kafka documentation has a short page called "Use cases" that lists seven families: messaging, website activity tracking, metrics, log aggregation, stream processing, event sourcing and commit log. The page describes them in a paragraph each. This lesson makes each one concrete: if you built it for an online shop, what would the topic be, what would the key be, how long would the data stay, and who would read it?
All seven use the same four mechanisms you met in What is Kafka: an append-only log per partition, a key that decides the partition, retention that is independent of reading, and consumer groups that each keep their own position. The families differ only in how they set those four dials.
| Family | Typical topic | Key | Kept for | Who reads it |
|---|---|---|---|---|
| Messaging | orders | Order id | Days | One consumer group per service that must react: payment, inventory, notification |
| Website activity tracking | page-views, searches (one topic per activity type) | Session id or user id | Days | Recommendations, fraud checks, a sink connector to the warehouse |
| Metrics | service-metrics | Service instance | Hours to days | Alerting, a sink into a time-series database |
| Log aggregation | app-logs | None, or the service instance | Hours to days | A sink into a log search system, ad hoc debugging consumers |
| Stream processing | payments in, payment-failures-per-minute out | Input: order id. Output: whatever the result is grouped by | As the input and output each need | A Kafka Streams application, then consumers of its output topic |
| Event sourcing | order-events | Order id | Forever, or as long as the entity matters | The owning service when it rebuilds state, plus any new read model |
| Commit log | inventory-levels (compacted) | SKU | Latest value per key | Instances restoring their state, caches reloading after a restart |
Why it exists
Before a shared log, each of these needs was met by a different tool. Services talked through a message broker. Click data was written to files and loaded into a warehouse overnight. Metrics went to a monitoring agent, logs to a log shipper, and database changes to a replication stream nobody else could read. Each pipe had one source and one destination, so every new use of existing data meant building another pipe.
Kafka's first job at LinkedIn was the second item in the list. Jay Kreps, one of its authors, describes on the LinkedIn engineering blog how Kafka was built to be a central pipeline first for activity data and then for other uses such as monitoring data, with several hundred event types covering page views, ad impressions, searches, service invocations and application exceptions. The point of that design was that a new consumer could subscribe to data that already flowed, without the producing team doing anything.
Three properties of the log make one system fit all seven families:
- Reading does not remove. A record stays until retention or compaction removes it, so any number of consumer groups can read the same topic at their own pace, and a group created next month can still read last week's data if it is retained.
- Order per key. Records with the same key go to the same partition and are read in the order the broker appended them, which is what messaging between services, event sourcing and a commit log all depend on.
- Sequential disk writes and batching. High-volume, low-value data such as clicks and logs can share the infrastructure with low-volume, high-value data such as orders.
How it works inside
For each family, here is which dial matters most and why.
1. Messaging: a consumer group per reaction
One service publishes a fact ("order 1042 was placed") and others react. Each reacting service uses its own group.id, so each gets every record. Inside one group, the partitions are divided among the instances, so each record is handled by one instance of that service. The key is the business entity whose events must stay in order, here the order id. Volume is usually modest; what matters is that nothing is lost: acks=all, replication factor 3 and min.insync.replicas=2. Processing is at-least-once by default, so handlers must tolerate a repeated record.
2. Activity tracking: volume first
Every page view, search and click becomes a record. The documentation suggests one topic per activity type. Volume is high because one page view can produce many records, so these topics get more partitions, compression and batching. Keying by session id keeps one visitor's actions in order for sessionisation; keying by nothing spreads load evenly when order does not matter. Losing a handful of clicks during a broker failure is often acceptable, so some teams choose weaker acknowledgement here. That is a business decision to make explicitly per topic, never a default to copy onto orders.
3 and 4. Metrics and log aggregation: a buffer in front of slower systems
Applications or agents publish measurements and log lines; sink connectors move them into a time-series database and a log search system. Kafka's role is a buffer that absorbs bursts and lets several destinations read the same feed. Retention is short, because the destination is the long-term store. One caution: if the Kafka cluster is what you are debugging, logs and metrics that travel through that same cluster will not reach you. Keep an independent path for monitoring Kafka itself.
5. Stream processing: topics in, topics out
A processor reads one or more topics, transforms, joins or aggregates, and writes the result to a new topic. Stages chain into a pipeline, and every intermediate topic is itself readable by anyone. Kafka Streams is the library for this in the Apache project. Two details surprise newcomers. An aggregation needs all records of a group in one partition, so re-keying creates an internal repartition topic. And exactly-once processing in Streams covers reading, state and writing inside Kafka; it does not cover an email or a database write made from the processor.
6. Event sourcing: the log is the source of truth
Instead of storing only the current state of an order, the service stores every state change as an event and derives the current state by replaying them. The topic is keyed by entity id and retained without a time limit (retention.ms=-1). The benefit is a full history and the ability to build new read models from the beginning. The cost is real: event schemas must stay readable for years, replays get longer as history grows, and Kafka cannot fetch "all events of order 1042" without scanning a partition, so most designs keep a database view next to the log. Event sourcing covers the pattern.
7. Commit log: latest value per key
Here the topic backs up the state of some other system. Each change is written as key and new value, and a restarting node reads the topic to restore itself. Because only the latest value per key matters, the topic uses log compaction (cleanup.policy=compact) instead of time-based deletion. Kafka uses the same idea for itself: consumer group offsets are stored as records in an internal topic, Kafka Connect keeps connector configuration, offsets and status in compacted topics, and Kafka Streams backs its state stores with changelog topics.
Event sourcing topic
- Keeps every event: OrderPlaced, OrderPaid, OrderShipped
cleanup.policy=deletewithretention.ms=-1- Replay gives the full history and any state in between
- Grows without bound; consider tiered storage for old segments
Commit log (compacted) topic
- Keeps at least the latest value per key: order 1042 is SHIPPED
cleanup.policy=compact- Replay gives the current state only
- Size is bounded by the number of distinct keys, not by time
Partitions & consumer groups
Simulation, not a real clusterAnimated lesson: how partitions are shared inside one consumer group, and why a second group gets its own copy of every record.
Step 1 · The pattern
A log and a reader
The producer appends messages to a partition: an ordered log. A consumer reads them one after another, in the order they were written.
- message
- consumed
Architecture
Drawing diagram…
Flowchart of a Kafka cluster with seven topics. Frontends write page views, the Order Service writes orders and order events, agents write metrics and logs, the Payment Service writes payments. Orders are read by payment, inventory and notification services and by a warehouse sink. Page views go to the warehouse. Metrics and logs go to monitoring sinks. Payments feed a Kafka Streams application that writes failures per minute for an alerting consumer. Order events feed a read model. The Inventory Service writes and restores from a compacted inventory levels topic.
flowchart LR
WEB["Web and mobile frontends"]
OS["Order Service"]
PAY["Payment Service"]
AG["Metrics and log agents"]
subgraph KAFKA["Kafka cluster"]
T1["orders - key order id - 7 days"]
T2["page-views - key session id - 3 days"]
T3["service-metrics and app-logs - 1 day"]
T4["payments - key order id - 7 days"]
T5["payment-failures-per-minute"]
T6["order-events - kept forever"]
T7["inventory-levels - compacted"]
end
WEB --> T2
OS --> T1
OS --> T6
AG --> T3
PAY --> T4
T1 -->|"messaging"| PAY
T1 -->|"messaging"| INV["Inventory Service"]
T1 -->|"messaging"| NOTIF["Notification Service"]
T2 -->|"activity tracking"| DWH["Warehouse sink connector"]
T1 --> DWH
T3 -->|"metrics and logs"| OBS["Monitoring and log search sinks"]
T4 -->|"stream processing"| KS["Kafka Streams app"]
KS --> T5
T5 --> ALERT["Alerting consumer"]
T6 -->|"event sourcing"| RM["Order history read model"]
INV --> T7
T7 -->|"commit log"| INV- Producers are on the left and write to topics without knowing the readers. The label on each outgoing arrow names the family that this consumer represents.
ordershas four readers in four consumer groups: three services (messaging) and the warehouse sink. Adding the fourth required no change to the Order Service.- The Kafka Streams application turns
paymentsintopayment-failures-per-minute. The alerting consumer reads the small derived topic instead of recomputing from the raw one. inventory-levelsis both written and read by the Inventory Service: it is that service's own changelog, used to restore stock levels after a restart. Compaction keeps it from growing with every stock movement.- Retention differs per topic because the purpose differs: days where consumers only need to catch up, forever where the topic is the record of truth, latest-per-key where only current state matters.
Message flow
Drawing diagram…
Sequence diagram. The Order Service produces an OrderPlaced record to the orders topic and gets an acknowledgement with partition and offset. The payment group, the warehouse sink group and a Streams application each fetch the same record and record their own progress. Three days later a new fraud detection group joins with auto.offset.reset set to earliest and reads the retained records from the oldest one.
sequenceDiagram participant O as Order Service participant K as orders topic participant P as Payment group participant W as Warehouse sink group participant S as Streams app participant N as New fraud group O->>K: Produce OrderPlaced with key 1042 K-->>O: Acknowledged at partition 3 offset 880 K-->>P: Fetch returns offset 880 P->>K: Commit offset 881 for payment group K-->>W: Fetch returns offset 880 W->>K: Commit offset 881 for warehouse group K-->>S: Fetch returns offset 880 S->>K: Produce to orders-per-minute Note over K: Record stays for the 7 day retention N->>K: Join three days later with auto.offset.reset earliest K-->>N: Fetch returns offsets from the oldest retained N->>K: Commit own position for fraud group
- The producer's part ends at the acknowledgement. Whether zero or ten groups read the record makes no difference to it.
- Each group commits the offset of the next record it will read (881 after handling 880). The commit records a position for that group and partition; it does not acknowledge or delete the record.
- The Streams application is one more reader, and its result is one more topic.
- The fraud group starts three days later. A group with no committed offset starts according to
auto.offset.reset, whose default islatest: only new records. To process the retained history it must be set toearliestbefore the group's first start.
In a real system
Consider a typical online shop with about 40 orders per minute at peak and around 3,000 page views per second during a sale. The two streams differ by more than three orders of magnitude in volume and in the opposite direction in value per record, so they are configured differently on the same cluster.
| Topic | Partitions | Key | Producer settings | Retention | Consumer groups |
|---|---|---|---|---|---|
orders | 6 | Order id | acks=all, idempotence on (the defaults) | 7 days | payment-service, inventory-service, notification-service, warehouse sink |
page-views | 24 | Session id | Compression and a longer linger.ms for bigger batches | 3 days | recommendations, fraud-signals, warehouse sink |
app-logs | 12 | None | Compression; loss of a few lines tolerated | 1 day | Log search sink |
order-events | 6 | Order id | acks=all, idempotence on | Unlimited | order-history-view, any future read model |
inventory-levels | 6 | SKU | acks=all, idempotence on | Compacted | The Inventory Service itself on restart |
Follow one order. The Order Service writes OrderPlaced with key 1042 to orders. The Payment Service's group reads it and charges the card; the Inventory Service's group reserves stock; the Notification Service's group sends the confirmation email. That is messaging. The same record is copied to the warehouse by a sink connector, where it is joined with the customer's page views from the session: activity tracking. A Streams job counts orders per minute for a dashboard: stream processing. None of these readers slows the others down, and if the warehouse sink is stopped for a day, it resumes from its committed offsets because the records are still retained.
For a public example, the LinkedIn engineering post linked in the references describes the original deployment: Kafka as the central multi-subscriber event log for activity data, feeding both real-time systems and offline processing from the same topics.
Java and Spring Boot
Three files show how the differences between the families appear in a Spring Boot service: topics declared with different retention, a high-volume activity publisher, and a second consumer group added to an existing topic.
package com.example.shop.config;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class ShopTopics { private static final String SEVEN_DAYS_MS = String.valueOf(7L * 24 * 60 * 60 * 1000); private static final String THREE_DAYS_MS = String.valueOf(3L * 24 * 60 * 60 * 1000); /** Messaging: modest volume, must not be lost, ordered per order id. */ @Bean public NewTopic orders() { return TopicBuilder.name("orders") .partitions(6) .replicas(3) .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") .config(TopicConfig.RETENTION_MS_CONFIG, SEVEN_DAYS_MS) .build(); } /** Activity tracking: high volume, more partitions, shorter retention. */ @Bean public NewTopic pageViews() { return TopicBuilder.name("page-views") .partitions(24) .replicas(3) .config(TopicConfig.RETENTION_MS_CONFIG, THREE_DAYS_MS) .build(); } /** Event sourcing: the history is the data, so no time limit. */ @Bean public NewTopic orderEvents() { return TopicBuilder.name("order-events") .partitions(6) .replicas(3) .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") .config(TopicConfig.RETENTION_MS_CONFIG, "-1") .build(); } /** Commit log: only the latest stock level per SKU matters. */ @Bean public NewTopic inventoryLevels() { return TopicBuilder.name("inventory-levels") .partitions(6) .replicas(3) .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") .compact() .build(); }}Why it is written this way
- Spring Boot's auto-configured
KafkaAdmincreates eachNewTopicbean at startup if the topic does not exist. For a topic that already exists it adds partitions when the bean declares more than the topic has; do not rely on it to change retention or other settings of an existing topic. - Declaring topics in code keeps partition count, replication and retention reviewable. Relying on broker-side auto-creation gives every topic the broker defaults: one partition and replication factor 1 unless the cluster was configured otherwise.
min.insync.replicas=2with replication factor 3 is what makes the producer's defaultacks=alla real durability guarantee while still tolerating one broker being down..compact()setscleanup.policy=compact. Records in a compacted topic must have a key.
package com.example.shop.tracking;import java.time.Instant;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Component;@Componentpublic class PageViewPublisher { public record PageViewed(String sessionId, String userId, String path, Instant at) {} private static final Logger log = LoggerFactory.getLogger(PageViewPublisher.class); private static final String TOPIC = "page-views"; private final KafkaTemplate<String, Object> kafkaTemplate; public PageViewPublisher(KafkaTemplate<String, Object> kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; } public void publish(PageViewed event) { // Key = session id: one visitor's page views stay in order in one partition. kafkaTemplate.send(TOPIC, event.sessionId(), event) .whenComplete((result, failure) -> { if (failure != null) { // Tracking must never fail the page. Count it, do not rethrow. log.warn("page view not published for session {}", event.sessionId(), failure); } }); }}Why it is written this way
sendreturns aCompletableFutureand does not block the web request. The callback runs when the broker acknowledges or the send finally fails.- For tracking data a failed send is logged and dropped. For
ordersthe same failure must stop the business operation or be retried from an outbox; the code shape is the same, the decision is different. - The key choice is a use-case decision: session id for per-visitor order, user id if logged-in history matters more, no key if only even spread matters.
package com.example.shop.analytics;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.stereotype.Component;@Componentpublic class OrderAnalyticsListener { public record OrderPlaced(String orderId, String customerId, long totalCents) {} private static final Logger log = LoggerFactory.getLogger(OrderAnalyticsListener.class); private final RevenueCounter revenueCounter; public OrderAnalyticsListener(RevenueCounter revenueCounter) { this.revenueCounter = revenueCounter; } // A new group id means a new, independent reader of the same topic. // The Payment, Inventory and Notification services are not affected. @KafkaListener(id = "order-analytics", groupId = "order-analytics", topics = "orders") public void onOrderPlaced(ConsumerRecord<String, OrderPlaced> record) { OrderPlaced order = record.value(); revenueCounter.add(order.orderId(), order.totalCents()); log.debug("counted order {} from partition {} offset {}", order.orderId(), record.partition(), record.offset()); } /** Must ignore an order id it has already counted: delivery is at-least-once. */ public interface RevenueCounter { void add(String orderId, long totalCents); }}Why it is written this way
- Adding this consumer needs no change in the Order Service and no new topic. That is the difference from a queue, where a second reader would compete for the same messages.
- With JSON values, configure
JsonDeserializerwithspring.json.value.default.typeandspring.json.trusted.packagesfor this service (Spring for Apache Kafka 3.x class names; 4.x renames them toJacksonJsonDeserializer). See Spring serialization. - Set
spring.kafka.consumer.auto-offset-reset=earliestbefore the first start if the new group should process retained history. The Kafka default islatest. - After a rebalance or a restart, the listener can receive a record again, so
RevenueCounterdeduplicates by order id. That is consumer-side idempotency: application logic, unrelated to the producer'senable.idempotence.
Configuration
The settings that turn one topic shape into another. Defaults are from the Apache Kafka 4.4 configuration reference; topic-level names are shown, each has a broker-level default.
cleanup.policyDefault delete
- What it does
deleteremoves old segments by time or size.compactkeeps at least the latest record per key. Both can be combined.- When to change it
- Use
compactfor commit-log and current-state topics (always with keys). Keepdeletefor messaging, tracking, metrics, logs and event-sourced history.
retention.msDefault 604800000 (7 days)
- What it does
- How long a record is kept under the
deletepolicy before its segment becomes eligible for removal.-1means no time limit. - When to change it
- Shorten it for high-volume data that has a long-term home elsewhere. Set
-1for event-sourced topics. Size it to cover the longest consumer outage you want to recover from without loss.
retention.bytesDefault -1 (no size limit)
- What it does
- Maximum size of one partition before old segments are removed, under the
deletepolicy. - When to change it
- Set it on log and metric topics to cap disk use during a burst. It applies per partition, so multiply by the partition count.
min.insync.replicasDefault 1
- What it does
- How many in-sync replicas, including the leader, must have a record before a producer using
acks=allgets a success. - When to change it
- Set 2 with replication factor 3 for orders, payments and event-sourced topics. With the default of 1,
acks=allcan succeed with a single copy.
acks (producer)Default all
- What it does
- How many acknowledgements the producer waits for: none (
0), the leader only (1), or all in-sync replicas (all). - When to change it
- Keep
allfor business events. Lowering it for clickstream or logs trades possible loss on leader failure for less waiting, and also disables idempotence.
compression.type (producer)Default none
- What it does
- Compresses whole batches on the producer:
gzip,snappy,lz4orzstd. - When to change it
- Turn it on for activity, log and metric topics, where records are repetitive and batches are large. Measure CPU before enabling on latency-sensitive producers.
auto.offset.reset (consumer)Default latest
- What it does
- Where a group starts when it has no committed offset, or when its committed offset no longer exists on the broker.
- When to change it
- Use
earliestfor a new consumer that must process retained history, and for consumers where skipping data silently is worse than reprocessing.
Hands-on exercise
Shape three topics for three different uses
See on a simulated partition what time-based retention and compaction each keep, then choose key, retention and consumers for new requirements of the shop.
Steps · 0 of 6 done
Retention versus compaction on one partition
Simulation · not a real clusterA partition of order status updates. Compact it, then compare with time-based retention.
What you should see
Compaction leaves at least the newest record of each key and gaps in the offset sequence; retention removes whole old segments regardless of key. A: an event-sourcing shape, key order id, cleanup.policy=delete, retention.ms=-1. B: a commit-log shape, key product id, cleanup.policy=compact. C: an activity-tracking shape, a searches topic keyed by session id or unkeyed, a few days of retention, and a sink connector on Kafka Connect.
What can go wrong
Copying the settings of one family onto another
- Symptom
- Orders occasionally go missing after a broker restart, or the clickstream producer is slow and expensive.
- Cause
- A producer configuration tuned for tracking data (
acks=1or0) was reused for business events, or the reverse: small unbatched, uncompressed sends for high-volume data. - Fix
- Classify each topic by family first, then set acknowledgements, replication, batching and compression for that topic's producers. Keep the two producer configurations separate in code.
Using a compacted topic as if it were a history
- Symptom
- A new consumer replays an
order-statustopic to rebuild an audit trail and finds only the latest status of older orders. - Cause
- The topic has
cleanup.policy=compact, so earlier values per key were removed once the cleaner ran. - Fix
- Keep history in a topic with the
deletepolicy and unlimited retention. Publish a compacted current-state topic next to it if consumers need fast reload.
Retention shorter than the longest consumer outage
- Symptom
- After a weekend outage of the warehouse sink, Monday's reports are missing Saturday's data, and the consumer's log shows its offset was reset.
- Cause
- The topic kept 24 hours of data. The group's committed offset pointed at segments that had been deleted, and
auto.offset.resetmoved it. - Fix
- Set retention from the recovery time you need, alert on consumer lag measured in time, and decide deliberately between
earliestandlatestfor each consumer.
One topic for all event types and all purposes
- Symptom
- Every consumer deserializes and discards most records, a schema change in one event type breaks unrelated consumers, and retention cannot suit everyone.
- Cause
- A single
eventstopic mixes orders, page views and logs, which need different keys, volumes, durability and retention. - Fix
- Group events that must be ordered relative to each other in one topic (all events of an order). Separate streams whose volume, retention or durability differ.
Event sourcing without a plan for reading
- Symptom
- Showing one order's history takes seconds, or a restart takes longer every month.
- Cause
- Kafka has no lookup by key: finding one entity's events means scanning a partition, and rebuilding state means replaying everything.
- Fix
- Maintain a read model in a database fed by a consumer, and snapshot state so a restart replays only recent events. Use Kafka as the log, not as the query engine.
How to troubleshoot
You are not sure which family an existing topic was set up for
Check
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name inventory-levels --describe
- How to read it
- The output lists the settings that differ from the broker defaults.
cleanup.policy=compactmarks a commit-log topic,retention.ms=-1a keep-forever topic. No overrides means broker defaults, usually 7 days of retention.
You need to know who reads a topic before changing it
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list (then --describe --group <name> for each group)
- How to read it
- Each row of a description is a topic and partition with the committed offset, the log end offset and the lag. Every group listed for
ordersis an independent use of that topic. A group with no active members but a large lag is a stopped consumer racing against retention.
A newly deployed consumer processes nothing although the topic is full of records
Check
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group order-analytics
- How to read it
- If CURRENT-OFFSET equals LOG-END-OFFSET and the lag is 0, the group started at the end because
auto.offset.resetislatest. Stop the application and reset the group with--reset-offsets --to-earliest --topic orders --execute, or use a new group id withearliest.
Records for one order appear in a different order to different consumers
Check
bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
- How to read it
- Check the partition count against what it was when the topic was created. Adding partitions changes which partition a key maps to, so old and new events of one order can sit in different partitions. Also confirm that every producer of that topic uses the same key.
Key takeaways
- The documentation names seven families: messaging, website activity tracking, metrics, log aggregation, stream processing, event sourcing and commit log.
- All seven are built from the same parts. They differ in key, partition count, retention or compaction, durability settings and who reads.
- A consumer group is one independent use of a topic. Adding a use means adding a group, not changing the producer.
- Retention is independent of consumption and is also the deadline for a stopped consumer to come back.
- Use
deletewith unlimited retention when history matters, andcompactwhen only the latest value per key matters. Compaction keeps at least the latest value, not exactly one. - Decide durability per topic: business events get
acks=all, replication factor 3 andmin.insync.replicas=2; lossy settings are a deliberate choice for low-value, high-volume data only.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A delivery company wants riders' phones to send a GPS position every 5 seconds: about 8,000 positions per second at peak. The dispatch screen needs each rider's latest position, and a data team wants 30 days of raw positions for route analysis. Per rider, positions must not be shown out of order.
Which topic design fits both needs?
Scenario 2 of 3
The team that owns orders (key = order id, 7-day retention, acks=all) is asked to feed a new loyalty-points service. The proposal on the table: the Order Service makes an extra REST call to the loyalty service after publishing each order event, "so we are sure they get it".
What is the better design?
Scenario 3 of 3
An app-logs topic receives about 20 MB per second and has retention.ms of 7 days because it was created with broker defaults. Brokers are at 85% disk. The log search system, fed by a sink connector, already keeps 30 days and the connector's lag is normally under one minute. The orders topic on the same brokers is 2 GB in total.
What do you change first?
References
References: Apache Kafka documentation: Use cases (the seven families) ↗ · Apache Kafka documentation: Design (log compaction, message delivery semantics) ↗ · Apache Kafka documentation: Topic configs (cleanup.policy, retention.ms, retention.bytes, min.insync.replicas) ↗ · Apache Kafka documentation: Consumer configs (auto.offset.reset) ↗ · Apache Kafka documentation: Kafka Connect user guide (compacted config, offset and status topics) ↗ · Spring for Apache Kafka reference: Configuring topics (NewTopic, TopicBuilder) ↗ · Spring for Apache Kafka reference: Sending messages (KafkaTemplate) ↗ · LinkedIn Engineering: The Log: What every software engineer should know about real-time data's unifying abstraction (Jay Kreps, 2013) ↗ · Martin Fowler: Event Sourcing (linked from the Kafka use cases page) ↗ · Report a problem