Path 2 · lesson 6 of 8
Log compaction
How a compacted topic keeps at least the latest record per key, how the log cleaner does it, what tombstones are, and what compaction does not promise.
What you will learn
- Decide whether a topic should use
delete,compactor both from what its records mean - State exactly what compaction guarantees to a reader and what it does not
- Explain how the log cleaner picks a log, builds its offset map and rewrites segments
- Delete a key with a tombstone and reason about
delete.retention.ms - Tune when compaction happens with the dirty ratio, compaction lag and segment settings
- Publish to and rebuild state from a compacted topic in Spring Boot, including tombstones
Before you start
The concept
Time and size retention treat every record alike: old ones go. That suits events, where each record stands alone (an order was placed, a page was viewed). It does not suit records that mean the current value of something: a customer's profile, a product's price, a consumer group's position. For those, the newest record per key is all that matters, however old it is, and every earlier record for the same key is obsolete the moment a newer one is written.
Log compaction is the retention policy for that kind of data. With cleanup.policy=compact, the broker keeps at least the latest record for every key in each partition and, in the background, removes older records whose key appears again later in the log. A record with a key and a null value, called a tombstone, means the key is deleted: it removes the earlier records for that key and is itself removed some time later.
cleanup.policy=delete
- Retention by age or size, a whole segment at a time
- A record's survival depends on when it was written
- After 7 days a key that has not changed is gone
- Reading from the start gives recent history
- Fits events: orders placed, payments captured, clicks
cleanup.policy=compact
- Retention by key: older records for a key are removed
- A record's survival depends on whether a newer one has the same key
- A key that has not changed for a year is still there
- Reading from the start gives the current state of every key
- Fits state: profiles, prices, settings, offsets, changelogs
Compaction on a small partition
Simulation · not a real clusterWrite several values for the same keys, then run the cleaner. Look at which records survive, at the gaps left in the offsets, and at the part of the log the cleaner does not touch.
Why it exists
Take a topic customers.profile, keyed by customer id, where each record is the customer's full current profile. A new service wants a local copy of every profile. There are two obvious ways to give it one, and both are bad.
- Keep everything forever. Reading from offset 0 does rebuild the current state, but a customer who edited their address 40 times costs 40 records, the topic grows without limit even when the number of customers is stable, and every new reader replays years of obsolete updates.
- Use time retention. Disk stays bounded, but a customer who has not changed anything for longer than the retention period has no record left. Reading from the start no longer gives the full state, so you need a second mechanism (a bulk export, a snapshot API) plus the logic to stitch the snapshot to the live stream without gaps or duplicates.
Compaction removes that dilemma. The topic's size is bounded by the number of distinct keys rather than by the number of updates, and the same topic serves both purposes: read it from the start to load the full current state, keep reading to follow changes live. The Kafka documentation names the cases it was built for: subscribing to database changes, event-sourced state, and journaling the local state of a stream processor so another instance can take over.
Kafka depends on it internally. __consumer_offsets is a compacted topic: every offset commit is a record keyed by group, topic and partition, and only the latest commit per key is needed. Kafka Streams state store changelogs and Kafka Connect's own configuration, offset and status topics work the same way.
How it works inside
Head, tail and the active segment
A compacted partition has three regions. The tail is the older part that the cleaner has already processed: offsets there have gaps and, apart from records written since, each key appears once. The head is everything written after the last cleaning: it looks like a normal log, with dense offsets and every update present. The active segment is the end of the head and is never cleaned, whatever its age.
What the cleaner does
The log cleaner is a pool of background threads on each broker (log.cleaner.threads, 1 by default), cleaning each broker's own replicas independently. One pass works like this:
- Pick a log. Among compacted partitions, choose the one with the highest dirty ratio: the size of the uncleaned head divided by the total size. A log is only eligible once the ratio reaches
min.cleanable.dirty.ratio(0.5 by default). - Build an offset map. Scan the head and record, for each key, the offset of its last occurrence. The map is a compact hash table of 24 bytes per entry, held in the cleaner's buffer (
log.cleaner.dedupe.buffer.size, 128 MiB in total by default). - Recopy. Read the closed segments from the beginning and write each record to a new segment only if no later offset for its key is in the map.
- Swap. Replace the old segment files with the cleaned ones. Each swap happens as soon as a segment is ready, so the extra disk needed is about one segment, not a copy of the log.
Cleaning does not block producers or consumers, and its I/O can be throttled with log.cleaner.io.max.bytes.per.second. With a dirty ratio of 0.5, a log is cleaned when the head is as large as the tail, so at most about half of a compacted log is ever obsolete data. A lower ratio wastes less space but rewrites the log more often.
What a reader is guaranteed
| Guaranteed | Not guaranteed |
|---|---|
| The latest record for every key is retained (until a tombstone for it expires). | That only one record per key exists. The head and the active segment still hold older values. |
| Order is preserved. Compaction removes records, it never reorders them. | That a slow reader sees every intermediate value. Older values may be gone before it gets there. |
| Offsets never change. A record keeps the offset it was written with. | That offsets are consecutive. Gaps are normal, so never compute counts from offset arithmetic. |
| A consumer that keeps up with the head sees every record, in order. | That compaction happens within any fixed time, unless you set max.compaction.lag.ms, and even then it depends on cleaner capacity. |
| A reader starting from the beginning ends with the final value of every key. | That such a reader sees every tombstone, if it takes longer than delete.retention.ms to reach the head. |
A fetch at an offset that was compacted away is not an error. The broker returns the next record that does exist, so a consumer that committed offset 37 and returns after cleaning simply continues with 38 or whatever comes next.
Tombstones
To delete a key, produce a record with that key and a null value. The cleaner treats it as the latest value, so it removes every earlier record for the key. The tombstone itself must stay for a while, because a consumer that has already read the old value needs to read the tombstone to learn about the deletion. It is kept for delete.retention.ms (24 hours by default) and removed by a later cleaning pass.
Controlling when compaction happens
min.compaction.lag.ms(default 0) is a lower bound: a record stays uncompacted for at least this long, which gives live consumers time to see every update.max.compaction.lag.ms(default: no limit) is an upper bound on how long a record can stay ineligible. It makes a log cleanable even when its dirty ratio is below the threshold, and it also rolls the active segment: for compacted topics the effective roll time is the smaller ofsegment.msandmax.compaction.lag.ms.segment.msandsegment.bytesmatter because the active segment is never cleaned. On a low-volume compacted topic with defaults, a superseded record can sit in the active segment for 7 days before the cleaner is even allowed to look at it. This is why the internal offsets topic uses 100 MiB segments (offsets.topic.segment.bytes).
Keys, partitions and compact,delete
Compaction works within one partition and compares keys byte for byte. Every record needs a key (the broker rejects records without one on a compacted topic), and all records for a key must land in the same partition. If the partition count changes, a key can start hashing to a different partition: its old value then stays in the old partition for ever, and a tombstone sent later goes to the new partition and deletes nothing in the old one.
cleanup.policy=compact,delete applies both rules: segments older than retention.ms (or beyond retention.bytes) are deleted, and the segments that remain are compacted. Use it for keyed state that expires on its own, such as session state or windowed aggregates that are meaningless after a few days. Understand what you give up: a key that has not been updated within the retention period disappears, so the topic is no longer a complete snapshot. Also note that tiered storage does not support compacted topics.
Architecture
Drawing diagram…
A flowchart of a compacted partition split into a cleaned tail, a dirty head and an active segment. A cleaner thread picks the log with the highest dirty ratio, builds a map from key to last offset from the head, recopies the closed segments keeping only records without a later offset for their key, and swaps the cleaned segments in. Tombstones are dropped by a later pass after delete.retention.ms.
flowchart TB
PROD["Producer writes key and value"] --> ACT
subgraph LOG["Partition customers.profile-0"]
direction LR
TAIL["Tail, already cleaned, offsets have gaps"]
HEAD["Head, closed dirty segments"]
ACT["Active segment, never cleaned"]
TAIL --- HEAD
HEAD --- ACT
end
PICK["Cleaner thread picks the log with the highest dirty ratio"] --> MAP
HEAD --> MAP["Offset map, key to last offset, 24 bytes per entry"]
TAIL --> COPY["Recopy closed segments, keep a record only if no later offset for its key"]
HEAD --> COPY
MAP --> COPY
COPY --> SWAP["Swap cleaned segments in, one at a time"]
SWAP --> NEWTAIL["New tail, at least the latest value per key"]
TOMB["Tombstone, key with null value"] --> EXP["Dropped by a later pass after delete.retention.ms"]- Producers and consumers are unaffected while this runs. Appends keep going to the active segment, which the cleaner leaves alone.
- The offset map is the only memory the cleaner needs: one small entry per distinct key in the head, not the records themselves.
- The recopy covers the old tail as well as the head, because a key last cleaned a month ago may have a newer value in the head now.
- After the swap, what was head has become tail and the dirty ratio is back near zero. The log becomes eligible again when new writes push the ratio over the threshold.
Message flow
Drawing diagram…
A sequence diagram. A producer writes two values for customer c-17 and a value then a tombstone for c-42. The cleaner builds an offset map and drops offsets 0 and 1. A new consumer fetching from offset 0 receives c-17 version 2 at offset 2 and the tombstone at offset 3. After delete.retention.ms a later pass removes the tombstone.
sequenceDiagram participant P as Customer service participant L as Partition leader participant K as Log cleaner participant N as New consumer P->>L: c-17 with profile v1, offset 0 P->>L: c-42 with profile v1, offset 1 P->>L: c-17 with profile v2, offset 2 P->>L: c-42 with null value, offset 3 Note over L: Segment rolls and dirty ratio passes 0.5 K->>L: Build offset map, c-17 at 2 and c-42 at 3 K->>L: Recopy, drop offsets 0 and 1 Note over L: Log now holds offsets 2 and 3 N->>L: Fetch from offset 0 L-->>N: c-17 v2 at offset 2, c-42 tombstone at offset 3 Note over N: State is c-17 only Note over L: delete.retention.ms passes K->>L: Next pass drops the tombstone at offset 3 Note over L: Log now holds offset 2 only
- The cleaner drops offset 0 because the map says c-17 appears again at offset 2, and offset 1 because c-42 appears again at offset 3. Offsets 2 and 3 keep their numbers.
- The new consumer asked for offset 0, which no longer exists. It gets the next record that does, without an error.
- It receives the tombstone and can discard c-42. A consumer that had read offset 1 earlier needs that tombstone just as much, which is why it is retained for
delete.retention.ms. - Before the first pass, a consumer reading from 0 would have received all four records. Both readers end with the same state, which is the point of the guarantee.
In a real system
A typical setup: the customer service owns customer data in its database and publishes each customer's current contact preferences to customers.profile, so that other services do not call it synchronously on every request.
| Aspect | Choice | Reason |
|---|---|---|
| Key | Customer id, as a string | Compaction is per key. The key must be stable and identical bytes every time. |
| Value | The full current profile | Each record must be complete on its own, because earlier records for the key will be removed. Never publish partial updates to a compacted topic. |
| Partitions | 12, fixed | Chosen once with headroom. Changing the count breaks the key-to-partition mapping that compaction relies on. |
| Policy | cleanup.policy=compact | A customer who never edits their profile must still be present for a new reader. |
| Deletion | Tombstone when an account is erased | Removes the profile from the topic and tells every consumer to remove its copy. |
delete.retention.ms | 7 days | Consumers may be down for a weekend. They must still see the tombstone when they return. |
max.compaction.lag.ms | 1 day | Erased data should leave the log within about a day, even on a quiet partition. |
The notification service consumes the topic from the beginning at startup to fill an in-memory map and then keeps it current. The search indexer does the same into its index. Neither needs a bulk export from the customer service, and a new consumer added next year gets the full state the same way. With 2 million customers at about 500 bytes each, the compacted topic settles around 1 GB per replica plus whatever is in the uncleaned head, regardless of how many edits have ever been made.
Java and Spring Boot
Three pieces: the topic declared as compacted, a publisher that sends full state and tombstones, and a consumer that rebuilds its state from the beginning and handles null values. Code follows the Spring for Apache Kafka reference sections on configuring topics, tombstones and seeking.
package com.example.customers;import java.time.Instant;/** The complete current state for one customer. Never a partial update. */public record CustomerProfile( String customerId, String email, boolean marketingOptIn, String locale, Instant updatedAt) {}Why it is written this way
- On a compacted topic every record must stand alone. If a record carried only the changed field, compaction would throw away the fields sent in earlier records.
package com.example.customers;import java.time.Duration;import org.apache.kafka.clients.admin.NewTopic;import org.apache.kafka.common.config.TopicConfig;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.TopicBuilder;@Configurationpublic class CustomerProfileTopicConfig { public static final String TOPIC = "customers.profile"; @Bean NewTopic customerProfiles() { return TopicBuilder.name(TOPIC) .partitions(12) .replicas(3) .compact() .config(TopicConfig.MIN_IN_SYNC_REPLICAS_CONFIG, "2") // Consumers may be down for a weekend and must still see deletions. .config(TopicConfig.DELETE_RETENTION_MS_CONFIG, millis(Duration.ofDays(7))) // Superseded and erased data leaves the log within about a day. .config(TopicConfig.MAX_COMPACTION_LAG_MS_CONFIG, millis(Duration.ofDays(1))) // Live consumers get at least a minute to see every update. .config(TopicConfig.MIN_COMPACTION_LAG_MS_CONFIG, millis(Duration.ofMinutes(1))) .build(); } private static String millis(Duration duration) { return String.valueOf(duration.toMillis()); }}Why it is written this way
compact()setscleanup.policy=compact. There is deliberately noretention.ms: it has no effect unless the policy also includesdelete.max.compaction.lag.msalso rolls the active segment after a day, so a quiet partition does not hold obsolete records for the default 7 days ofsegment.ms.- The partition count is a permanent decision for this topic. Pick it with growth in mind.
package com.example.customers;import java.util.concurrent.CompletableFuture;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.support.SendResult;import org.springframework.stereotype.Service;@Servicepublic class CustomerProfilePublisher { private final KafkaTemplate<String, CustomerProfile> kafkaTemplate; public CustomerProfilePublisher(KafkaTemplate<String, CustomerProfile> kafkaTemplate) { this.kafkaTemplate = kafkaTemplate; } /** Publishes the full current profile. The key decides what this record replaces. */ public CompletableFuture<SendResult<String, CustomerProfile>> publish(CustomerProfile profile) { return kafkaTemplate.send(CustomerProfileTopicConfig.TOPIC, profile.customerId(), profile); } /** Publishes a tombstone: same key, null value. */ public CompletableFuture<SendResult<String, CustomerProfile>> erase(String customerId) { return kafkaTemplate.send(CustomerProfileTopicConfig.TOPIC, customerId, null); }}Why it is written this way
- The key is always the customer id as a
String. Mixing key formats (a number in one service, a string in another) produces different bytes, different partitions and records that never compact against each other. - A tombstone is an ordinary send with a null value. The serializer passes null through, and the broker stores a record whose value is null.
- Callers should handle the returned future: a failed tombstone means the customer's data is still in the topic.
package com.example.notifications;import java.util.Map;import java.util.Optional;import java.util.concurrent.ConcurrentHashMap;import org.apache.kafka.common.TopicPartition;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.listener.ConsumerSeekAware;import org.springframework.kafka.support.KafkaHeaders;import org.springframework.messaging.handler.annotation.Header;import org.springframework.messaging.handler.annotation.Payload;import org.springframework.stereotype.Component;import com.example.customers.CustomerProfile;@Componentpublic class CustomerProfileCache implements ConsumerSeekAware { private final Map<String, CustomerProfile> profiles = new ConcurrentHashMap<>(); @KafkaListener( id = "customer-profile-cache", topics = "customers.profile", groupId = "notification-profile-cache-${random.uuid}") public void onRecord( @Payload(required = false) CustomerProfile profile, @Header(KafkaHeaders.RECEIVED_KEY) String customerId) { if (profile == null) { profiles.remove(customerId); // tombstone: the customer was erased } else { profiles.put(customerId, profile); // upsert: the last record for a key wins } } /** Rebuild the whole map every time partitions are assigned to this instance. */ @Override public void onPartitionsAssigned(Map<TopicPartition, Long> assignments, ConsumerSeekCallback callback) { callback.seekToBeginning(assignments.keySet()); } public Optional<CustomerProfile> find(String customerId) { return Optional.ofNullable(profiles.get(customerId)); }}Why it is written this way
@Payload(required = false)is what lets a null value reach the method. Without it Spring cannot deliver a tombstone to a listener with a typed payload.- The key comes from the
KafkaHeaders.RECEIVED_KEYheader, since a tombstone has no value to read the id from. - Each instance needs every partition, so each uses its own group id (the
random.uuidplaceholder gives a new one per start). Members of one shared group would split the partitions and each hold part of the data. - The handler is an upsert, so receiving several values for a key, or the same record twice, ends in the same state.
- The map is not complete until the listener has caught up to the end of the topic. A production version should not answer lookups before that. Kafka Streams offers this pattern ready-made as a table backed by a compacted topic.
spring: kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092} producer: key-serializer: org.apache.kafka.common.serialization.StringSerializer value-serializer: org.springframework.kafka.support.serializer.JsonSerializer acks: all consumer: key-deserializer: org.apache.kafka.common.serialization.StringDeserializer value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer auto-offset-reset: earliest properties: spring.json.value.default.type: com.example.customers.CustomerProfile spring.json.trusted.packages: com.example.customersWhy it is written this way
auto-offset-reset: earliestmatters here because every start uses a new group with no committed offsets. The explicit seek in the listener covers later reassignments.- Both JSON classes pass a null value through unchanged, which is what makes tombstones work with typed payloads.
Configuration
Topic-level settings first, then the broker-level cleaner settings. Defaults are from the Kafka 4.4 reference. Each topic setting has a broker-wide default, named in the entry where it is not obvious.
cleanup.policyDefault delete
- What it does
compactkeeps at least the latest record per key.compact,deletealso deletes segments by age or size. Broker default:log.cleanup.policy.- When to change it
- Use
compactwhen a record means the current state of its key and readers need all keys. Adddeleteonly when old keys should expire by themselves.
min.cleanable.dirty.ratioDefault 0.5
- What it does
- A log becomes eligible for cleaning when its uncleaned part is at least this fraction of its total size. Bounds the space wasted on obsolete records.
- When to change it
- Lower it (for example 0.1) when a topic has few keys and many updates and you want quick, small logs. Lower values mean more frequent rewriting and more I/O.
delete.retention.msDefault 86400000 (1 day)
- What it does
- How long tombstones are retained. Also the time a reader starting from the beginning has to reach the head and still be sure of a consistent result.
- When to change it
- Set it longer than the longest time any consumer of the topic may be offline or lagging.
min.compaction.lag.msDefault 0
- What it does
- Minimum time a record stays uncompacted after it is written.
- When to change it
- Raise it when live consumers need a guaranteed window to observe every intermediate update.
max.compaction.lag.msDefault 9223372036854775807 (no limit)
- What it does
- Maximum time a record may remain ineligible for compaction. Rolls the active segment (the smaller of this and
segment.msapplies) and makes the log cleanable below the dirty ratio. - When to change it
- Set it when superseded or erased data must leave the log within a known time. It is a target, not a hard deadline: it still depends on cleaner threads being available.
segment.msDefault 604800000 (7 days)
- What it does
- Age at which the active segment rolls. Only rolled segments can be compacted.
- When to change it
- On low-volume compacted topics lower this, or set
max.compaction.lag.ms, or obsolete records wait up to a week in the active segment.
log.cleaner.threadsDefault 1
- What it does
- Broker setting: number of background threads that clean logs.
- When to change it
- Increase when many compacted partitions fall behind (rising compaction delay), watching broker disk I/O.
log.cleaner.dedupe.buffer.sizeDefault 134217728 (128 MiB)
- What it does
- Broker setting: total memory for the cleaner's offset maps across all cleaner threads, at 24 bytes per key.
- When to change it
- Increase for topics with very many distinct keys in the head, so one pass can cover more of the log.
log.cleaner.io.max.bytes.per.secondDefault 1.7976931348623157E308 (unthrottled)
- What it does
- Broker setting: average read plus write I/O the cleaner may use.
- When to change it
- Set a limit if cleaning competes with produce and fetch traffic for disk throughput.
log.cleaner.enableDefault true
- What it does
- Broker setting that runs the cleaner. Deprecated, to be removed in Kafka 5.0.
- When to change it
- Never set it to false:
__consumer_offsetsand every other compacted topic would grow without limit.
Hands-on exercise
Compact a log by hand, then check yourself
Predict which records survive a cleaning pass, including a tombstone, and confirm with the log explorer that compaction keeps the latest value per key, leaves offset gaps and skips the active segment.
Steps · 0 of 6 done
Run the cleaner yourself
Simulation · not a real clusterWrite several values per key, roll a segment and run the cleaner. Then send a tombstone and run it again.
What you should see
After the first pass each key in the closed segments appears once, with its newest value from those segments, and the surviving offsets have gaps. A key written again in the active segment still shows two records in the log, which is the difference between at least the latest and exactly one. After the tombstone pass, user-2 has only its tombstone left, and a reader building a map from offset 0 ends with user-1, user-3 and user-4 at their newest values.
What can go wrong
Assuming one record per key
- Symptom
- A job counts the records in a compacted topic to count customers and gets a number that is too high and changes from run to run.
- Cause
- The head and the active segment are not compacted, so superseded values are still present. Compaction keeps at least the latest value, with no promise about the others.
- Fix
- Always build a map: apply records in order, let the last value per key win, remove keys on tombstones, and count the map.
A deleted key stays in a consumer's local state
- Symptom
- A customer was erased last week. One service still sends them email. Other services removed them correctly.
- Cause
- That service was down or lagging for longer than
delete.retention.ms. The tombstone was removed before the service read it, so nothing told it to delete its copy. - Fix
- Set
delete.retention.msabove the longest tolerated outage, and make consumers rebuild from the beginning after a long outage instead of resuming from their committed offset.
Compaction never seems to run on a quiet topic
- Symptom
- A small configuration topic with 40 keys holds thousands of records, and reading it from the start takes far longer than it should.
- Cause
- All data is in the active segment, which rolls only after
segment.ms(7 days) or 1 GiB, and the active segment is never cleaned. Even after a roll, the dirty ratio must reach 0.5. - Fix
- Set
max.compaction.lag.ms(or a lowersegment.ms) on the topic, and lowermin.cleanable.dirty.ratioif needed.
Changing the partition count of a compacted topic
- Symptom
- After adding partitions, services that rebuild state see stale values for some keys, and keys that were deleted come back.
- Cause
- Keys now hash to different partitions. The old partition keeps the key's last value before the change for ever, and new updates and tombstones go elsewhere. A reader that merges partitions has no order between them.
- Fix
- Do not resize a compacted topic. Create a new topic with the right partition count, republish the current state for every key from the source of truth, and switch consumers over.
Publishing partial updates or events to a compacted topic
- Symptom
- After a while, state rebuilt from the topic is missing fields, or history that a team expected to replay is gone.
- Cause
- Compaction keeps only the newest record per key. A record holding just the changed field, or one event among many for the same key, replaces everything before it.
- Fix
- Publish the full current state per key to compacted topics. Keep events that must all be retained in a topic with
cleanup.policy=deleteand a suitable retention.
Records without a key
- Symptom
- Producing to a newly compacted topic fails with an invalid record error, or a
kafka-console-producer.shtest is rejected. - Cause
- Compaction is defined per key, so the broker refuses records with a null key on a topic whose policy includes
compact. - Fix
- Send a key with every record. For console tests use the producer's
parse.keyandkey.separatorproperties.
How to troubleshoot
A compacted topic keeps growing on disk
Check
`bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe --topic-list customers.profile`, then the cleaner metrics `uncleanable-partitions-count`, `max-clean-time-secs` and `max-compaction-delay-secs`, and the broker's log cleaner log file
- How to read it
- Growing size with a healthy cleaner usually means the number of distinct keys is growing (ids used as keys that are never tombstoned). A non-zero uncleanable count means the cleaner hit an error on a partition and skipped it: the cleaner log names the partition and the cause.
You are not sure a topic is compacted, or with which settings
Check
`bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics --entity-name customers.profile --describe`
- How to read it
- Look for
cleanup.policy=compact. Settings not listed use the broker defaults, including the 1 daydelete.retention.msand 7 daysegment.ms.
You want to see what is actually stored for a key
Check
`bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic customers.profile --from-beginning --formatter-property print.key=true --formatter-property print.offset=true` (before Kafka 4.2 the option is `--property`)
- How to read it
- Offset gaps show where the cleaner has been. Several records for one key near the end are the uncleaned head. A value printed as
nullis a tombstone that has not expired yet.
Erased data is still readable days later
Check
The topic's `max.compaction.lag.ms`, `segment.ms` and `min.cleanable.dirty.ratio`, and `ls -l` in the partition directory on a broker
- How to read it
- A single large active segment means nothing is eligible for cleaning yet. Set
max.compaction.lag.msso the segment rolls and is cleaned within the time you need.
A consumer of a compacted topic reports gaps in offsets
Check
The consumer's own logic and any monitoring that subtracts offsets to count records
- How to read it
- Gaps are expected on a compacted topic. Lag computed from offsets overstates the number of records left to read. Nothing was lost.
Key takeaways
- Compaction is retention by key: the broker keeps at least the latest record for every key in a partition, however old it is.
- It is not exactly one record per key. The head and the active segment still contain superseded values, so consumers must upsert and let the last record win.
- Offsets and order never change. Removed records leave gaps, and fetching a missing offset returns the next existing record.
- A tombstone (key with null value) deletes a key and is itself removed after
delete.retention.ms, so consumers that are away longer than that can miss deletions. - The cleaner picks the dirtiest log, maps key to last offset for the head, recopies closed segments and swaps them in. The active segment is never cleaned.
- Compaction is per partition and per exact key bytes: every record needs a key, values must be full state, and the partition count must not change.
compact,deleteadds expiry by age or size. The topic is then no longer a complete snapshot of all keys.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
pricing.current is compacted, keyed by product id, with about 5,000 products and default topic settings. Prices change a few hundred times a day. A new service reads the topic from the start to load prices and receives 31,000 records, with up to nine records for some products. The team suspects the log cleaner is broken.
What should the team do?
Scenario 2 of 3
customers.profile is compacted with default settings. On Friday 18:00 the email service is stopped for a data centre move. On Saturday 09:00 customer c-88 is erased and a tombstone is published. The email service restarts on Monday 08:00 and resumes from its committed offsets, updating its local database of profiles as records arrive.
What is the risk, and how do you remove it?
Scenario 3 of 3
inventory.levels is compacted, keyed by SKU, with 6 partitions. Consumers lag at peak, so an engineer proposes increasing it to 12 partitions tomorrow. Several services rebuild their stock view by reading the topic from the beginning on startup.
What do you decide?
References
References: Apache Kafka documentation: Design, Log Compaction ↗ · Apache Kafka documentation: Topic configuration reference ↗ · Apache Kafka documentation: Broker configuration reference (log.cleaner.*) ↗ · Apache Kafka documentation: Implementation, Distribution (the compacted __consumer_offsets topic) ↗ · Apache Kafka documentation: Tiered Storage (limitations) ↗ · Spring for Apache Kafka reference: Null Payloads and Log Compaction of Tombstone Records ↗ · Spring for Apache Kafka reference: Seeking to a Specific Offset ↗ · Report a problem