Skip to content

Path 5 · lesson 3 of 5

Advanced30 minApache Kafka 4.x (checked against the 4.4 documentation; transaction protocol v2 from KIP-890 is used by 4.0+ clients and brokers) · Java 21 · Spring Boot 3.x with Spring for Apache Kafka 3.3

Kafka transactions and the read-process-write loop

How a transactional producer makes writes to several partitions and the consumer's offset commit atomic, and exactly where that guarantee stops.

What you will learn

  • State what a Kafka transaction makes atomic, and name three things it never includes
  • Trace a read-process-write transaction through the producer, the transaction coordinator, the partition leaders and the group coordinator
  • Explain how a transactional.id and its epoch fence a zombie producer, and what consumer group metadata adds
  • Predict what read_committed and read_uncommitted consumers see while a transaction is open, committed or aborted
  • Configure a transactional KafkaTemplate and listener container in Spring so outputs and offsets commit together
  • Decide whether a pipeline stage should use a Kafka transaction or an idempotent consumer

Before you start

The concept

A Kafka transaction groups writes made by one producer so that they all become visible together or none of them does. The writes can go to any number of partitions in any number of topics. One of those writes can be the committed offsets of a consumer group, because offsets are themselves records in the internal __consumer_offsets topic.

That second point is what turns transactions into exactly-once processing. A service that reads a record, computes something and writes the result to another topic can put the output records and the new input offset in one transaction. After any crash, either both exist, or neither does and the input is read again with no visible trace of the first attempt. This is the read-process-write loop, also called consume-transform-produce. Kafka Streams uses the same mechanism for its exactly-once mode.

Inside the transactionNot inside, ever
Records this producer sends to Kafka topics between begin and commitA row written to PostgreSQL, MySQL, MongoDB or any other database
Offsets passed to sendOffsetsToTransaction for one or more consumer groupsAn HTTP or gRPC call, including a call to a payment provider
The commit or abort decision, applied to every partition involvedAn email, SMS or push notification
In-memory state, local files and caches of the processing service
What a read_uncommitted consumer downstream chooses to read
The boundary of a Kafka transaction is Kafka's own log.

In Kafka only the producer is transactional. The consumer takes part in two passive ways: its group's offsets are written by the producer on its behalf, and downstream consumers choose with isolation.level whether to see uncommitted data.

Why it exists

Take a stage that reads payments.requested, screens each payment against risk rules and writes it to payments.approved or payments.review. Without transactions it makes two separate writes for every input: the output record, and later the offset commit. A crash between them is the at-least-once window from the first lesson of this path. The input is read again and a second approved record is written.

Producer idempotence does not help, because the second output is a new send from a restarted process, not a retry of the same batch. Every downstream consumer would have to deduplicate. Transactions remove the window instead of asking everyone downstream to cope with it.

A second problem is the zombie: an instance that the group has given up on, because of a long pause or a network partition, but that is still running. Its partitions now belong to another instance. When the zombie wakes up it finishes what it was doing and writes output for records the new owner is also processing. A transaction system has to be able to refuse that write.

Use a Kafka transaction when

  • The stage reads from Kafka and its only outputs are Kafka records
  • One input fans out to several topics that must agree, such as a decision and its audit record
  • Downstream consumers cannot reasonably deduplicate, for example aggregations
  • You control the readers and can set read_committed on them

Use an idempotent consumer instead when

  • The main effect is a database write or a call to an external API
  • The producer is a request handler with no input offsets to commit
  • The extra commit latency is not acceptable for the readers
  • You need non-blocking retry topics around the listener

How it works inside

The moving parts

  • Transaction coordinator. A component running on brokers. Each transactional.id is assigned to one coordinator, which keeps the state of its transactions in the internal, replicated __transaction_state topic.
  • Producer id and epoch. initTransactions() registers the transactional.id and returns a producer id and an epoch. The same transactional.id gets the same identity back after a restart, with a higher epoch.
  • Markers. Committing or aborting writes a small control record, a commit or abort marker, into every partition that took part. Markers occupy an offset in the log and are never handed to your code.
  • Last stable offset (LSO). The offset just below the first record of the earliest transaction still open on a partition. read_committed consumers are served only up to it.

One transaction, step by step

  1. initTransactions() runs once per producer. The coordinator bumps the epoch, which fences any older instance with the same transactional.id, and finishes any transaction that instance left behind: rolled forward if it had begun to commit, aborted otherwise.
  2. beginTransaction() only changes local state. Nothing is sent yet.
  3. send() writes records to the partition leaders as usual. They are appended to the log at once, tagged with the producer id and epoch and flagged as transactional. Each partition is registered with the coordinator the first time it is written to in the transaction, and the transaction timer starts when the first partition is added.
  4. sendOffsetsToTransaction(offsets, consumer.groupMetadata()) adds the group's __consumer_offsets partition to the transaction and sends the offsets to the group coordinator. They stay pending and do not move the group's position yet.
  5. commitTransaction() flushes unsent records and asks the coordinator to end the transaction. The coordinator first writes its decision to __transaction_state. From that moment the outcome is fixed even if brokers fail. It then writes a commit marker to every partition involved, including the offsets partition, and records the transaction as complete.
  6. abortTransaction() follows the same path with abort markers. The records stay physically in the log, and read_committed consumers skip them.

What readers see

Consumer isolation.levelOpen transactionAborted transactionCommitted transaction
read_uncommitted (default)Records are returned as soon as they pass the high watermarkRecords are returned as if they were normalRecords are returned
read_committedNothing at or beyond the LSO is returned, including non-transactional records that sit behind the open transactionRecords are filtered out by the consumerRecords are returned once the commit marker is written

Two consequences follow. A read_committed consumer's latency includes the duration of the transactions in front of it, and a transaction that hangs holds back every read_committed reader of the partitions it touched until it times out. And since the default is read_uncommitted, a downstream service that forgets the setting reads aborted output, and the exactly-once guarantee ends at that consumer.

Fencing zombies

Every transactional request carries the producer id and epoch. When a new instance calls initTransactions() with the same transactional.id, the epoch increases and the old instance's next send or commit fails with ProducerFencedException. That error is fatal: the only correct action is to close that producer.

Epoch fencing alone required each transactional.id to map to a fixed set of input partitions, which in practice meant one producer per input partition. KIP-447 (opens in a new tab) (brokers 2.5 and later) removed that. sendOffsetsToTransaction now sends the consumer group metadata: member id, generation and group instance id. The group coordinator rejects the offset commit if that member no longer holds a current place in the group. A zombie that lost its partitions in a rebalance therefore cannot commit, and one producer per application instance is enough, with a transactional.id that is unique per instance.

Aborts, timeouts, cost and limits

  • An abort does not rewind the consumer. Its in-memory position is already past the records. After abortTransaction() you must seek back to the last committed offsets so the records are fetched again. Spring's DefaultAfterRollbackProcessor does this for you.
  • The coordinator aborts slow transactions. A transaction open for longer than transaction.timeout.ms (1 minute by default) is aborted by the coordinator, and the producer's next transactional call fails. Processing a batch must fit well inside that time.
  • What to do with errors. ProducerFencedException, OutOfOrderSequenceException and AuthorizationException are fatal: close the producer. For TimeoutException, TransactionAbortableException and other KafkaExceptions, abort and reprocess.
  • Cost. Each transaction adds coordinator round trips and one marker write per partition, and the commit is synchronous. One transaction per record is slow. One per polled batch spreads the cost and lengthens the time downstream readers wait.
  • Processing is still at-least-once. After an abort the same input is processed again. Only the Kafka writes of the failed attempt are hidden. Any other side effect of the handler happens once per attempt.
  • Atomic write, not atomic read. A consumer may read only some of the partitions a transaction wrote to, and compaction or retention can remove some of its records. Nothing guarantees a reader sees a transaction as a unit.
  • Durability still comes from the topics. The KafkaProducer documentation recommends replication factor 3 and min.insync.replicas=2 for topics used in transactions.

Architecture

Drawing diagram…

Flowchart. A screening service reads payments.requested and, within one transaction, sends to payments.approved, payments.review and payments.audit and sends its offsets to the consumer offsets topic. The transaction coordinator stores its decision in the transaction state topic and writes markers to all four partitions, which are grouped as atomic together. A downstream capture service reads payments.approved with read_committed and calls a card provider API and a payments database, both marked as not in the transaction.

flowchart TB
  IN["payments.requested"] -->|"poll, read_committed"| APP["Screening service"]
  APP -->|"transactional send"| OUT1["payments.approved"]
  APP -->|"transactional send"| OUT2["payments.review"]
  APP -->|"transactional send"| AUD["payments.audit"]
  APP -->|"sendOffsetsToTransaction"| OFF["Consumer offsets topic"]
  APP -->|"init, commit or abort"| TC["Transaction coordinator"]
  TC -->|"decision"| TL["Transaction state topic"]
  TC -->|"commit or abort markers"| OUT1
  TC -->|"markers"| OUT2
  TC -->|"markers"| AUD
  TC -->|"markers"| OFF
  OUT1 -->|"read_committed"| CAP["Capture service"]
  CAP -.->|"NOT in the transaction"| PSP["Card provider API"]
  CAP -.->|"NOT in the transaction"| DB["Payments database"]
  subgraph ATOMIC["Atomic together"]
    OUT1
    OUT2
    AUD
    OFF
  end
One transaction spans three output topics and the group's offsets. The capture service's API call and database write are outside it.
  • The screening service only reads from Kafka and writes to Kafka, so its whole effect fits inside the transaction: outputs, audit record and input offset.
  • The coordinator does not carry the data. The service writes records straight to the partition leaders. The coordinator records which partitions took part and writes the markers that settle them.
  • The decision is durable once it is in the transaction state topic. If a broker fails while markers are being written, the coordinator, or its replacement, finishes the job.
  • The capture service reads committed records exactly as it would any other record. Its card charge and database write get no protection from the upstream transaction and need their own idempotency.

Message flow

Drawing diagram…

Sequence diagram. The producer initialises with its transactional id and receives a producer id and bumped epoch. The consumer polls offsets 40 to 44. The producer begins a transaction, sends output records that are appended but not yet visible, registers the consumer offsets partition, and sends offset 45 with group metadata to the group coordinator, where it stays pending. On commitTransaction the coordinator writes its decision, writes commit markers to the output partitions and the offsets partition, and the downstream read_committed consumer then receives the records.

sequenceDiagram
  participant C as Consumer in the service
  participant P as Transactional producer
  participant TC as Transaction coordinator
  participant PL as Output partition leaders
  participant GC as Group coordinator
  participant D as Downstream read_committed
  P->>TC: initTransactions with transactional.id
  TC-->>P: producer id and bumped epoch
  C->>C: poll payments.requested offsets 40 to 44
  P->>P: beginTransaction
  P->>PL: send approved, review and audit records
  Note over PL: appended but beyond the last stable offset
  P->>TC: add consumer offsets partition
  P->>GC: offset 45 with group metadata, kept pending
  P->>TC: commitTransaction
  TC->>TC: write decision to transaction state
  TC->>PL: commit markers
  TC->>GC: commit marker, offset 45 takes effect
  TC-->>P: committed
  PL-->>D: records now returned by poll
The read-process-write loop: outputs and the offset for the next input become visible together when the markers are written.
  1. The output records are in the log before the commit. What changes at commit time is their visibility to read_committed readers, through the marker.
  2. The offset 45 is the position of the next record to read, sent by the producer, not by the consumer. The consumer must not commit on its own, and enable.auto.commit must be false.
  3. If the service dies anywhere before the coordinator writes its decision, the transaction is aborted, at the latest when it times out or when the restarted instance calls initTransactions(). Offset 45 never takes effect and offsets 40 to 44 are read again.
  4. If it dies after the decision is written, the coordinator still completes the markers. The restarted instance finds offset 45 committed and does not reprocess.

In a real system

A typical payments platform puts a screening stage between accepting a payment request and capturing the money. The Screening Service consumes payments.requested (keyed by paymentId, 24 partitions) in consumer group payment-screening. It applies risk rules held in memory and, for every request, writes one record to payments.approved or payments.review and one to payments.audit. Compliance requires that a decision and its audit record can never exist without each other.

  • Transaction contents: the decision record, the audit record, and the group's next offsets on payments.requested.
  • transactional.id: a prefix made unique per instance, for example payment-screening- followed by the pod name. Spring appends a counter for each producer it creates.
  • Readers: the Capture Service on payments.approved, the case-management tool on payments.review and the audit archiver on payments.audit all set isolation.level=read_committed.
  • Topics: replication factor 3 and min.insync.replicas=2, as the producer documentation recommends for topics used in transactions.

Now look at the next stage. The Capture Service reads payments.approved, calls the card provider and updates its payments table. Wrapping its listener in a Kafka transaction would make its outgoing payments.captured event atomic with its input offset, but it would not make the card charge atomic with anything. If that transaction aborts after the charge went through, the record is processed again and the customer is charged again, unless the request to the provider carries an idempotency key and the table has a unique constraint on the payment id.

StageEffectsRight tool
Screening ServiceKafka in, Kafka outKafka transaction
Capture ServiceKafka in, card provider call, database row, Kafka outIdempotent handler with an outbox for the outgoing event
Order API publishing OrderPlacedDatabase row, Kafka out, no input offsetsTransactional outbox

Java and Spring Boot

The Screening Service with Spring for Apache Kafka: a producer factory with a transaction id prefix, a KafkaTransactionManager set on the listener container, and a listener whose sends and offsets commit together. The last file is the same loop on the plain client, to show the calls Spring makes for you.

src/main/java/com/example/screening/KafkaTransactionConfig.javaJava
package com.example.screening;import java.util.HashMap;import java.util.Map;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.common.serialization.StringDeserializer;import org.apache.kafka.common.serialization.StringSerializer;import org.springframework.beans.factory.annotation.Value;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;import org.springframework.kafka.core.ConsumerFactory;import org.springframework.kafka.core.DefaultKafkaConsumerFactory;import org.springframework.kafka.core.DefaultKafkaProducerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.core.ProducerFactory;import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;import org.springframework.kafka.listener.DefaultAfterRollbackProcessor;import org.springframework.kafka.support.serializer.JsonDeserializer;import org.springframework.kafka.support.serializer.JsonSerializer;import org.springframework.kafka.transaction.KafkaTransactionManager;import org.springframework.util.backoff.FixedBackOff;@Configurationpublic class KafkaTransactionConfig {    @Bean    public ProducerFactory<String, Object> producerFactory(            @Value("${spring.kafka.bootstrap-servers}") String bootstrapServers,            @Value("${app.instance-id}") String instanceId) {        Map<String, Object> props = new HashMap<>();        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);        props.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 30_000);        DefaultKafkaProducerFactory<String, Object> factory = new DefaultKafkaProducerFactory<>(props);        // Setting a prefix is what makes the factory transactional.        // It must be different on every running instance of the service.        factory.setTransactionIdPrefix("payment-screening-" + instanceId + "-");        return factory;    }    @Bean    public KafkaTemplate<String, Object> kafkaTemplate(ProducerFactory<String, Object> producerFactory) {        return new KafkaTemplate<>(producerFactory);    }    @Bean    public KafkaTransactionManager<String, Object> kafkaTransactionManager(            ProducerFactory<String, Object> producerFactory) {        return new KafkaTransactionManager<>(producerFactory);    }    @Bean    public ConsumerFactory<String, PaymentRequested> consumerFactory(            @Value("${spring.kafka.bootstrap-servers}") String bootstrapServers) {        Map<String, Object> props = new HashMap<>();        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);        props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-screening");        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);        props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");        return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(),                new JsonDeserializer<>(PaymentRequested.class, false));    }    @Bean    public ConcurrentKafkaListenerContainerFactory<String, PaymentRequested> kafkaListenerContainerFactory(            ConsumerFactory<String, PaymentRequested> consumerFactory,            KafkaTransactionManager<String, Object> kafkaTransactionManager,            KafkaTemplate<String, Object> kafkaTemplate) {        ConcurrentKafkaListenerContainerFactory<String, PaymentRequested> factory =                new ConcurrentKafkaListenerContainerFactory<>();        factory.setConsumerFactory(consumerFactory);        factory.setConcurrency(3);        // The container starts a Kafka transaction before calling the listener and        // sends the offsets to it before committing.        factory.getContainerProperties().setKafkaAwareTransactionManager(kafkaTransactionManager);        // After a rollback: redeliver twice, one second apart, then publish the record        // to the dead letter topic and commit its offset in a new transaction.        factory.setAfterRollbackProcessor(new DefaultAfterRollbackProcessor<>(                new DeadLetterPublishingRecoverer(kafkaTemplate),                new FixedBackOff(1000L, 2L), kafkaTemplate, true));        return factory;    }}

Why it is written this way

  • setTransactionIdPrefix switches the factory to transactional producers. Each producer's transactional.id is the prefix plus a counter. The reference states that the prefix must be unique per application instance, because two instances with the same id would fence each other.
  • setKafkaAwareTransactionManager is the property to use since Spring for Apache Kafka 3.2. The older setTransactionManager is deprecated.
  • With Spring Boot you can get the same wiring by setting only spring.kafka.producer.transaction-id-prefix: Boot then creates the KafkaTransactionManager and wires it into the listener container. The explicit version is shown so every part is visible.
  • isolation.level=read_committed is on the consumer because payments.requested may itself be written transactionally by the stage before this one.
  • transaction.timeout.ms is lowered to 30 seconds so a hung instance blocks downstream read_committed readers for less time. It must stay above the longest time one listener call can take.
  • The after-rollback processor takes the template and commitRecovered=true so that publishing to the dead letter topic and moving the offset past the failed record happen in one new transaction.

Configuration

Producer, consumer and broker settings that govern transactions, with the defaults documented for Apache Kafka 4.4, and the two Spring Boot properties that map onto them.

transactional.idDefault null
What it does
Makes the producer transactional and gives it an identity that survives restarts, so a new instance can finish or abort the old one's transaction and fence it. Setting it implies enable.idempotence.
When to change it
Must be stable across restarts of one instance and different between instances that run at the same time. In Spring set transactionIdPrefix and let the factory add the suffix.
transaction.timeout.msDefault 60000 (1 minute)
What it does
How long a transaction may stay open before the coordinator aborts it. The clock starts when the first partition is added to the transaction.
When to change it
Keep it above the worst-case time to process one batch and as low as that allows, since an abandoned transaction blocks read_committed readers for this long. Values above the broker's transaction.max.timeout.ms are rejected.
isolation.levelDefault read_uncommitted
What it does
read_committed makes poll() return only committed transactional records, and only up to the last stable offset. Non-transactional records are returned in both modes.
When to change it
Set read_committed on every consumer of a topic written by a transactional producer. The default lets consumers read aborted records.
enable.auto.commitDefault true (Spring for Apache Kafka sets false unless you configure it)
What it does
Background offset commits by the consumer itself.
When to change it
Must be false in a read-process-write loop. Offsets have to reach Kafka only through sendOffsetsToTransaction.
transaction.max.timeout.ms (broker)Default 900000 (15 minutes)
What it does
Upper limit the broker accepts for a producer's transaction.timeout.ms.
When to change it
Leave as is unless you want to stop clients from asking for long transactions that could stall readers.
transaction.state.log.replication.factor (broker)Default 3
What it does
Replication factor of the internal transaction state topic. The topic cannot be created until the cluster has this many brokers.
When to change it
Keep 3 in production. On a one-broker development cluster set it to 1 together with transaction.state.log.min.isr=1, or initTransactions() will time out.
transaction.state.log.min.isr (broker)Default 2
What it does
Minimum in-sync replicas required for a write to the transaction state topic to succeed.
When to change it
Keep 2 with replication factor 3. If fewer replicas are in sync, transactions cannot begin or complete until they return.
transactional.id.expiration.ms (broker)Default 604800000 (7 days)
What it does
How long the coordinator keeps a transactional.id that has shown no transaction activity.
When to change it
Matters for producers that sit idle for days. Spring's producer factory has a maxAge property to replace cached producers before this expires.
spring.kafka.producer.transaction-id-prefixDefault none
What it does
When set, Spring Boot makes the producer factory transactional, creates a KafkaTransactionManager and wires it into the listener container.
When to change it
Include something unique to the instance, such as the pod name. Every use of the template then needs a transaction.
spring.kafka.consumer.isolation-levelDefault read-uncommitted
What it does
Spring Boot's property for the consumer isolation.level.
When to change it
Set read-committed in every service that consumes transactional topics.

Hands-on exercise

Crash a transactional stage and look for the duplicate

See that a crash between processing and commit leaves no visible output and no moved offset when both are in one transaction, and identify what would still be repeated outside Kafka.

Steps · 0 of 5 done

The same crash, inside a transaction

Simulation · not a real cluster

The stage crashes after it has produced its output and before the commit. Compare at-least-once with transactional mode.

Take it further in the lab: Kafka transaction

What can go wrong

A database write or API call inside the transactional listener
Symptom
Duplicate rows or double charges after a rollback, although the service "uses exactly-once".
Cause
The Kafka transaction aborted after the external effect had already happened, and the redelivery repeated it. Kafka cannot roll back a database or a card provider.
Fix
Make the external effect idempotent with a unique key or an idempotency key, or write to the database only and publish through an outbox. See Transactions and databases.
Downstream consumer left on the default isolation level
Symptom
A downstream service acts on records that the producing stage rolled back, and totals do not match.
Cause
isolation.level defaults to read_uncommitted, which returns aborted and still-open transactional records.
Fix
Set isolation.level=read_committed (spring.kafka.consumer.isolation-level: read-committed) on every consumer of the transactional topics.
The same transactional id on two live instances
Symptom
Instances fail in turn with ProducerFencedException, and throughput collapses as soon as the service scales to two pods.
Cause
Both pods use the same transaction-id-prefix, so each initTransactions() bumps the epoch and fences the other pod's producer.
Fix
Make the prefix unique per instance, for example by including the pod name, as the Spring reference requires.
Processing takes longer than the transaction timeout
Symptom
Commits fail after slow batches and the same records are processed again and again.
Cause
The transaction stayed open beyond transaction.timeout.ms (1 minute by default), so the coordinator aborted it before the commit arrived.
Fix
Shorten the work per transaction, for example with a lower max.poll.records, move slow calls out of the transactional stage, or raise the timeout within transaction.max.timeout.ms.
Readers stalled behind an open transaction
Symptom
Lag grows for read_committed consumers of a partition while producers are healthy and other partitions are fine.
Cause
A transaction on that partition is still open, for example because its producer crashed, so the last stable offset cannot advance until the coordinator times it out.
Fix
Keep transaction.timeout.ms modest so abandoned transactions clear quickly, and inspect open transactions with kafka-transactions.sh.
Combining container transactions with non-blocking retries
Symptom
A failed record goes to a retry topic, and the listener's other output for that failed attempt is committed instead of rolled back.
Cause
The Spring reference states that non-blocking retries cannot be combined with container transactions: when the listener throws, the record is sent to the retry topic and the container transaction commits.
Fix
With transactions, use the DefaultAfterRollbackProcessor with a back-off and a dead letter recoverer instead of @RetryableTopic.

How to troubleshoot

initTransactions() fails with a timeout on a local or test cluster

Check

Broker settings transaction.state.log.replication.factor and transaction.state.log.min.isr, and the number of brokers in the cluster
How to read it
By default the transaction state topic needs three brokers. On a smaller cluster it cannot be created, so the coordinator never becomes available. Set both values to 1 for single-broker development.
ProducerFencedException in the application log

Check

kafka-transactions.sh --bootstrap-server localhost:9092 describe --transactional-id payment-screening-pod-0-0
How to read it
The output shows the coordinator, producer id, epoch and state for that id. An epoch that keeps climbing means another live producer shares the id. A single fencing right after a restart or a rebalance is the mechanism doing its job.
A read_committed consumer group is stuck while the topic keeps growing

Check

kafka-transactions.sh --bootstrap-server localhost:9092 list, then kafka-transactions.sh --bootstrap-server localhost:9092 describe-producers --topic payments.approved --partition 7
How to read it
list shows transactional ids and their state. An id stuck in Ongoing is holding the last stable offset back. describe-producers shows the active producers on the partition and the start offset of any open transaction.
A transaction stays open far beyond its timeout

Check

kafka-transactions.sh --bootstrap-server localhost:9092 find-hanging --broker-id 1
How to read it
Lists transactions the broker considers hanging. They can be ended with the tool's abort command after confirming that no producer still owns them. Treat that as a last resort.
Lag never reaches zero on a topic written transactionally

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group capture-service
How to read it
Commit and abort markers occupy offsets, so the log end offset can sit ahead of the last record the consumer was given. A small constant lag on an idle partition is this artefact. Spring's container property fixTxOffsets corrects the reported position.
Throughput dropped after enabling transactions

Check

Producer metrics txn-commit-time-ns-total, txn-send-offsets-time-ns-total and txn-abort-time-ns-total
How to read it
Commit time that dominates means too many small transactions: process more records per transaction. A growing abort time means transactions are failing and being redone, so look for timeouts and rebalances.

Key takeaways

  • A Kafka transaction makes one producer's writes to Kafka partitions, plus a consumer group's offset commit, atomic. That is all it covers.
  • It never includes a database write, an HTTP call or an email. Those happen once per processing attempt and need their own idempotency or an outbox.
  • The coordinator stores the decision in the transaction state topic and settles each partition with a commit or abort marker.
  • transactional.id plus an epoch fences zombies. Since KIP-447, consumer group metadata sent with the offsets lets one producer per instance serve a whole group. The id must be unique per instance.
  • Only read_committed consumers are protected, and they read only up to the last stable offset. The default is read_uncommitted.
  • An abort hides the output and leaves the offsets unchanged. The consumer still has to be rewound, which Spring's after-rollback processor does.
  • In Spring, set a transaction id prefix on the producer factory and a KafkaTransactionManager on the listener container. The container sends the offsets to the transaction before it commits.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A Capture Service listener runs in a container-managed Kafka transaction. For each PaymentApproved record it calls the card provider's charge API, then sends PaymentCaptured with the transactional KafkaTemplate. One night the brokers are slow and several transactions are aborted on timeout. Next morning 9 customers have been charged twice, but payments.captured holds one committed record for each.

What does this show about the design?

Scenario 2 of 3

A fraud-scoring stage writes scores transactionally to payments.scored. Its integration tests include a forced rollback. A reporting service that consumes payments.scored with default consumer settings shows scores for payments that the scoring stage's logs say were rolled back.

What is the smallest correct fix?

Scenario 3 of 3

You scale a transactional Spring service from 1 pod to 3. All pods have spring.kafka.producer.transaction-id-prefix=tx-screening-. Within seconds the logs fill with ProducerFencedException, containers restart their consumers, and output drops to a trickle.

What is happening?

References

References: Apache Kafka documentation: Design, Message Delivery Semantics and Using Transactions ↗ · KafkaProducer Javadoc: initTransactions, beginTransaction, sendOffsetsToTransaction, commitTransaction, abortTransaction ↗ · Apache Kafka documentation: Producer configs (transactional.id, transaction.timeout.ms) ↗ · Apache Kafka documentation: Consumer configs (isolation.level and the last stable offset) ↗ · Apache Kafka documentation: Broker configs (transaction.max.timeout.ms, transaction.state.log.*) ↗ · Apache Kafka documentation: Transaction protocol (KIP-890, transaction.version) ↗ · KIP-98: Exactly Once Delivery and Transactional Messaging ↗ · KIP-447: Producer scalability for exactly once semantics ↗ · KIP-664: Provide tooling to detect and abort hanging transactions (kafka-transactions.sh) ↗ · Spring for Apache Kafka reference: Transactions ↗ · Spring for Apache Kafka reference: Exactly Once Semantics ↗ · Spring for Apache Kafka reference: After Rollback Processor ↗ · Report a problem