Path 5 · lesson 3 of 5
Kafka transactions and the read-process-write loop
How a transactional producer makes writes to several partitions and the consumer's offset commit atomic, and exactly where that guarantee stops.
What you will learn
- State what a Kafka transaction makes atomic, and name three things it never includes
- Trace a read-process-write transaction through the producer, the transaction coordinator, the partition leaders and the group coordinator
- Explain how a
transactional.idand its epoch fence a zombie producer, and what consumer group metadata adds - Predict what
read_committedandread_uncommittedconsumers see while a transaction is open, committed or aborted - Configure a transactional
KafkaTemplateand listener container in Spring so outputs and offsets commit together - Decide whether a pipeline stage should use a Kafka transaction or an idempotent consumer
Before you start
The concept
A Kafka transaction groups writes made by one producer so that they all become visible together or none of them does. The writes can go to any number of partitions in any number of topics. One of those writes can be the committed offsets of a consumer group, because offsets are themselves records in the internal __consumer_offsets topic.
That second point is what turns transactions into exactly-once processing. A service that reads a record, computes something and writes the result to another topic can put the output records and the new input offset in one transaction. After any crash, either both exist, or neither does and the input is read again with no visible trace of the first attempt. This is the read-process-write loop, also called consume-transform-produce. Kafka Streams uses the same mechanism for its exactly-once mode.
| Inside the transaction | Not inside, ever |
|---|---|
| Records this producer sends to Kafka topics between begin and commit | A row written to PostgreSQL, MySQL, MongoDB or any other database |
Offsets passed to sendOffsetsToTransaction for one or more consumer groups | An HTTP or gRPC call, including a call to a payment provider |
| The commit or abort decision, applied to every partition involved | An email, SMS or push notification |
| In-memory state, local files and caches of the processing service | |
What a read_uncommitted consumer downstream chooses to read |
In Kafka only the producer is transactional. The consumer takes part in two passive ways: its group's offsets are written by the producer on its behalf, and downstream consumers choose with isolation.level whether to see uncommitted data.
Why it exists
Take a stage that reads payments.requested, screens each payment against risk rules and writes it to payments.approved or payments.review. Without transactions it makes two separate writes for every input: the output record, and later the offset commit. A crash between them is the at-least-once window from the first lesson of this path. The input is read again and a second approved record is written.
Producer idempotence does not help, because the second output is a new send from a restarted process, not a retry of the same batch. Every downstream consumer would have to deduplicate. Transactions remove the window instead of asking everyone downstream to cope with it.
A second problem is the zombie: an instance that the group has given up on, because of a long pause or a network partition, but that is still running. Its partitions now belong to another instance. When the zombie wakes up it finishes what it was doing and writes output for records the new owner is also processing. A transaction system has to be able to refuse that write.
Use a Kafka transaction when
- The stage reads from Kafka and its only outputs are Kafka records
- One input fans out to several topics that must agree, such as a decision and its audit record
- Downstream consumers cannot reasonably deduplicate, for example aggregations
- You control the readers and can set
read_committedon them
Use an idempotent consumer instead when
- The main effect is a database write or a call to an external API
- The producer is a request handler with no input offsets to commit
- The extra commit latency is not acceptable for the readers
- You need non-blocking retry topics around the listener
How it works inside
The moving parts
- Transaction coordinator. A component running on brokers. Each
transactional.idis assigned to one coordinator, which keeps the state of its transactions in the internal, replicated__transaction_statetopic. - Producer id and epoch.
initTransactions()registers thetransactional.idand returns a producer id and an epoch. The sametransactional.idgets the same identity back after a restart, with a higher epoch. - Markers. Committing or aborting writes a small control record, a commit or abort marker, into every partition that took part. Markers occupy an offset in the log and are never handed to your code.
- Last stable offset (LSO). The offset just below the first record of the earliest transaction still open on a partition.
read_committedconsumers are served only up to it.
One transaction, step by step
initTransactions()runs once per producer. The coordinator bumps the epoch, which fences any older instance with the sametransactional.id, and finishes any transaction that instance left behind: rolled forward if it had begun to commit, aborted otherwise.beginTransaction()only changes local state. Nothing is sent yet.send()writes records to the partition leaders as usual. They are appended to the log at once, tagged with the producer id and epoch and flagged as transactional. Each partition is registered with the coordinator the first time it is written to in the transaction, and the transaction timer starts when the first partition is added.sendOffsetsToTransaction(offsets, consumer.groupMetadata())adds the group's__consumer_offsetspartition to the transaction and sends the offsets to the group coordinator. They stay pending and do not move the group's position yet.commitTransaction()flushes unsent records and asks the coordinator to end the transaction. The coordinator first writes its decision to__transaction_state. From that moment the outcome is fixed even if brokers fail. It then writes a commit marker to every partition involved, including the offsets partition, and records the transaction as complete.abortTransaction()follows the same path with abort markers. The records stay physically in the log, andread_committedconsumers skip them.
What readers see
Consumer isolation.level | Open transaction | Aborted transaction | Committed transaction |
|---|---|---|---|
read_uncommitted (default) | Records are returned as soon as they pass the high watermark | Records are returned as if they were normal | Records are returned |
read_committed | Nothing at or beyond the LSO is returned, including non-transactional records that sit behind the open transaction | Records are filtered out by the consumer | Records are returned once the commit marker is written |
Two consequences follow. A read_committed consumer's latency includes the duration of the transactions in front of it, and a transaction that hangs holds back every read_committed reader of the partitions it touched until it times out. And since the default is read_uncommitted, a downstream service that forgets the setting reads aborted output, and the exactly-once guarantee ends at that consumer.
Fencing zombies
Every transactional request carries the producer id and epoch. When a new instance calls initTransactions() with the same transactional.id, the epoch increases and the old instance's next send or commit fails with ProducerFencedException. That error is fatal: the only correct action is to close that producer.
Epoch fencing alone required each transactional.id to map to a fixed set of input partitions, which in practice meant one producer per input partition. KIP-447 (opens in a new tab) (brokers 2.5 and later) removed that. sendOffsetsToTransaction now sends the consumer group metadata: member id, generation and group instance id. The group coordinator rejects the offset commit if that member no longer holds a current place in the group. A zombie that lost its partitions in a rebalance therefore cannot commit, and one producer per application instance is enough, with a transactional.id that is unique per instance.
Aborts, timeouts, cost and limits
- An abort does not rewind the consumer. Its in-memory position is already past the records. After
abortTransaction()you must seek back to the last committed offsets so the records are fetched again. Spring'sDefaultAfterRollbackProcessordoes this for you. - The coordinator aborts slow transactions. A transaction open for longer than
transaction.timeout.ms(1 minute by default) is aborted by the coordinator, and the producer's next transactional call fails. Processing a batch must fit well inside that time. - What to do with errors.
ProducerFencedException,OutOfOrderSequenceExceptionandAuthorizationExceptionare fatal: close the producer. ForTimeoutException,TransactionAbortableExceptionand otherKafkaExceptions, abort and reprocess. - Cost. Each transaction adds coordinator round trips and one marker write per partition, and the commit is synchronous. One transaction per record is slow. One per polled batch spreads the cost and lengthens the time downstream readers wait.
- Processing is still at-least-once. After an abort the same input is processed again. Only the Kafka writes of the failed attempt are hidden. Any other side effect of the handler happens once per attempt.
- Atomic write, not atomic read. A consumer may read only some of the partitions a transaction wrote to, and compaction or retention can remove some of its records. Nothing guarantees a reader sees a transaction as a unit.
- Durability still comes from the topics. The KafkaProducer documentation recommends replication factor 3 and
min.insync.replicas=2for topics used in transactions.
Architecture
Drawing diagram…
Flowchart. A screening service reads payments.requested and, within one transaction, sends to payments.approved, payments.review and payments.audit and sends its offsets to the consumer offsets topic. The transaction coordinator stores its decision in the transaction state topic and writes markers to all four partitions, which are grouped as atomic together. A downstream capture service reads payments.approved with read_committed and calls a card provider API and a payments database, both marked as not in the transaction.
flowchart TB
IN["payments.requested"] -->|"poll, read_committed"| APP["Screening service"]
APP -->|"transactional send"| OUT1["payments.approved"]
APP -->|"transactional send"| OUT2["payments.review"]
APP -->|"transactional send"| AUD["payments.audit"]
APP -->|"sendOffsetsToTransaction"| OFF["Consumer offsets topic"]
APP -->|"init, commit or abort"| TC["Transaction coordinator"]
TC -->|"decision"| TL["Transaction state topic"]
TC -->|"commit or abort markers"| OUT1
TC -->|"markers"| OUT2
TC -->|"markers"| AUD
TC -->|"markers"| OFF
OUT1 -->|"read_committed"| CAP["Capture service"]
CAP -.->|"NOT in the transaction"| PSP["Card provider API"]
CAP -.->|"NOT in the transaction"| DB["Payments database"]
subgraph ATOMIC["Atomic together"]
OUT1
OUT2
AUD
OFF
end- The screening service only reads from Kafka and writes to Kafka, so its whole effect fits inside the transaction: outputs, audit record and input offset.
- The coordinator does not carry the data. The service writes records straight to the partition leaders. The coordinator records which partitions took part and writes the markers that settle them.
- The decision is durable once it is in the transaction state topic. If a broker fails while markers are being written, the coordinator, or its replacement, finishes the job.
- The capture service reads committed records exactly as it would any other record. Its card charge and database write get no protection from the upstream transaction and need their own idempotency.
Message flow
Drawing diagram…
Sequence diagram. The producer initialises with its transactional id and receives a producer id and bumped epoch. The consumer polls offsets 40 to 44. The producer begins a transaction, sends output records that are appended but not yet visible, registers the consumer offsets partition, and sends offset 45 with group metadata to the group coordinator, where it stays pending. On commitTransaction the coordinator writes its decision, writes commit markers to the output partitions and the offsets partition, and the downstream read_committed consumer then receives the records.
sequenceDiagram participant C as Consumer in the service participant P as Transactional producer participant TC as Transaction coordinator participant PL as Output partition leaders participant GC as Group coordinator participant D as Downstream read_committed P->>TC: initTransactions with transactional.id TC-->>P: producer id and bumped epoch C->>C: poll payments.requested offsets 40 to 44 P->>P: beginTransaction P->>PL: send approved, review and audit records Note over PL: appended but beyond the last stable offset P->>TC: add consumer offsets partition P->>GC: offset 45 with group metadata, kept pending P->>TC: commitTransaction TC->>TC: write decision to transaction state TC->>PL: commit markers TC->>GC: commit marker, offset 45 takes effect TC-->>P: committed PL-->>D: records now returned by poll
- The output records are in the log before the commit. What changes at commit time is their visibility to
read_committedreaders, through the marker. - The offset 45 is the position of the next record to read, sent by the producer, not by the consumer. The consumer must not commit on its own, and
enable.auto.commitmust befalse. - If the service dies anywhere before the coordinator writes its decision, the transaction is aborted, at the latest when it times out or when the restarted instance calls
initTransactions(). Offset 45 never takes effect and offsets 40 to 44 are read again. - If it dies after the decision is written, the coordinator still completes the markers. The restarted instance finds offset 45 committed and does not reprocess.
In a real system
A typical payments platform puts a screening stage between accepting a payment request and capturing the money. The Screening Service consumes payments.requested (keyed by paymentId, 24 partitions) in consumer group payment-screening. It applies risk rules held in memory and, for every request, writes one record to payments.approved or payments.review and one to payments.audit. Compliance requires that a decision and its audit record can never exist without each other.
- Transaction contents: the decision record, the audit record, and the group's next offsets on
payments.requested. transactional.id: a prefix made unique per instance, for examplepayment-screening-followed by the pod name. Spring appends a counter for each producer it creates.- Readers: the Capture Service on
payments.approved, the case-management tool onpayments.reviewand the audit archiver onpayments.auditall setisolation.level=read_committed. - Topics: replication factor 3 and
min.insync.replicas=2, as the producer documentation recommends for topics used in transactions.
Now look at the next stage. The Capture Service reads payments.approved, calls the card provider and updates its payments table. Wrapping its listener in a Kafka transaction would make its outgoing payments.captured event atomic with its input offset, but it would not make the card charge atomic with anything. If that transaction aborts after the charge went through, the record is processed again and the customer is charged again, unless the request to the provider carries an idempotency key and the table has a unique constraint on the payment id.
| Stage | Effects | Right tool |
|---|---|---|
| Screening Service | Kafka in, Kafka out | Kafka transaction |
| Capture Service | Kafka in, card provider call, database row, Kafka out | Idempotent handler with an outbox for the outgoing event |
Order API publishing OrderPlaced | Database row, Kafka out, no input offsets | Transactional outbox |
Java and Spring Boot
The Screening Service with Spring for Apache Kafka: a producer factory with a transaction id prefix, a KafkaTransactionManager set on the listener container, and a listener whose sends and offsets commit together. The last file is the same loop on the plain client, to show the calls Spring makes for you.
package com.example.screening;import java.util.HashMap;import java.util.Map;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.producer.ProducerConfig;import org.apache.kafka.common.serialization.StringDeserializer;import org.apache.kafka.common.serialization.StringSerializer;import org.springframework.beans.factory.annotation.Value;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;import org.springframework.kafka.core.ConsumerFactory;import org.springframework.kafka.core.DefaultKafkaConsumerFactory;import org.springframework.kafka.core.DefaultKafkaProducerFactory;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.kafka.core.ProducerFactory;import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;import org.springframework.kafka.listener.DefaultAfterRollbackProcessor;import org.springframework.kafka.support.serializer.JsonDeserializer;import org.springframework.kafka.support.serializer.JsonSerializer;import org.springframework.kafka.transaction.KafkaTransactionManager;import org.springframework.util.backoff.FixedBackOff;@Configurationpublic class KafkaTransactionConfig { @Bean public ProducerFactory<String, Object> producerFactory( @Value("${spring.kafka.bootstrap-servers}") String bootstrapServers, @Value("${app.instance-id}") String instanceId) { Map<String, Object> props = new HashMap<>(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class); props.put(ProducerConfig.TRANSACTION_TIMEOUT_CONFIG, 30_000); DefaultKafkaProducerFactory<String, Object> factory = new DefaultKafkaProducerFactory<>(props); // Setting a prefix is what makes the factory transactional. // It must be different on every running instance of the service. factory.setTransactionIdPrefix("payment-screening-" + instanceId + "-"); return factory; } @Bean public KafkaTemplate<String, Object> kafkaTemplate(ProducerFactory<String, Object> producerFactory) { return new KafkaTemplate<>(producerFactory); } @Bean public KafkaTransactionManager<String, Object> kafkaTransactionManager( ProducerFactory<String, Object> producerFactory) { return new KafkaTransactionManager<>(producerFactory); } @Bean public ConsumerFactory<String, PaymentRequested> consumerFactory( @Value("${spring.kafka.bootstrap-servers}") String bootstrapServers) { Map<String, Object> props = new HashMap<>(); props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-screening"); props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false); props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed"); props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new JsonDeserializer<>(PaymentRequested.class, false)); } @Bean public ConcurrentKafkaListenerContainerFactory<String, PaymentRequested> kafkaListenerContainerFactory( ConsumerFactory<String, PaymentRequested> consumerFactory, KafkaTransactionManager<String, Object> kafkaTransactionManager, KafkaTemplate<String, Object> kafkaTemplate) { ConcurrentKafkaListenerContainerFactory<String, PaymentRequested> factory = new ConcurrentKafkaListenerContainerFactory<>(); factory.setConsumerFactory(consumerFactory); factory.setConcurrency(3); // The container starts a Kafka transaction before calling the listener and // sends the offsets to it before committing. factory.getContainerProperties().setKafkaAwareTransactionManager(kafkaTransactionManager); // After a rollback: redeliver twice, one second apart, then publish the record // to the dead letter topic and commit its offset in a new transaction. factory.setAfterRollbackProcessor(new DefaultAfterRollbackProcessor<>( new DeadLetterPublishingRecoverer(kafkaTemplate), new FixedBackOff(1000L, 2L), kafkaTemplate, true)); return factory; }}Why it is written this way
setTransactionIdPrefixswitches the factory to transactional producers. Each producer'stransactional.idis the prefix plus a counter. The reference states that the prefix must be unique per application instance, because two instances with the same id would fence each other.setKafkaAwareTransactionManageris the property to use since Spring for Apache Kafka 3.2. The oldersetTransactionManageris deprecated.- With Spring Boot you can get the same wiring by setting only
spring.kafka.producer.transaction-id-prefix: Boot then creates theKafkaTransactionManagerand wires it into the listener container. The explicit version is shown so every part is visible. isolation.level=read_committedis on the consumer becausepayments.requestedmay itself be written transactionally by the stage before this one.transaction.timeout.msis lowered to 30 seconds so a hung instance blocks downstreamread_committedreaders for less time. It must stay above the longest time one listener call can take.- The after-rollback processor takes the template and
commitRecovered=trueso that publishing to the dead letter topic and moving the offset past the failed record happen in one new transaction.
package com.example.screening;import java.time.Instant;import org.springframework.kafka.annotation.KafkaListener;import org.springframework.kafka.core.KafkaTemplate;import org.springframework.stereotype.Component;@Componentpublic class PaymentScreeningListener { private final KafkaTemplate<String, Object> kafkaTemplate; private final RiskRules riskRules; public PaymentScreeningListener(KafkaTemplate<String, Object> kafkaTemplate, RiskRules riskRules) { this.kafkaTemplate = kafkaTemplate; this.riskRules = riskRules; } @KafkaListener(topics = "payments.requested", groupId = "payment-screening") public void onPaymentRequested(PaymentRequested request) { // Pure computation: no database, no HTTP. Running it twice is harmless. RiskDecision decision = riskRules.evaluate(request); String target = decision.approved() ? "payments.approved" : "payments.review"; kafkaTemplate.send(target, request.paymentId(), new PaymentScreened(request.paymentId(), request.merchantId(), request.amountMinor(), request.currency(), decision.reason())); kafkaTemplate.send("payments.audit", request.paymentId(), new ScreeningAudit(request.paymentId(), decision.approved(), decision.reason(), Instant.now())); // Returning normally: the container sends the offsets to the transaction and commits. // Throwing: the transaction is rolled back, both sends stay invisible to // read_committed consumers, and the record is delivered again. }}record PaymentRequested(String paymentId, String merchantId, long amountMinor, String currency) {}record PaymentScreened(String paymentId, String merchantId, long amountMinor, String currency, String reason) {}record ScreeningAudit(String paymentId, boolean approved, String reason, Instant decidedAt) {}record RiskDecision(boolean approved, String reason) {}interface RiskRules { RiskDecision evaluate(PaymentRequested request);}Why it is written this way
- There is no transaction code in the listener. The container began the transaction, and
KafkaTemplateoperations performed by the listener take part in it. - The send futures are not awaited. A failed send makes the commit fail, which rolls the whole transaction back.
- The listener does nothing outside Kafka, and that is why a transaction is enough here. A JDBC insert or a call to a card provider in this method would run again after every rollback.
- A transactional
KafkaTemplaterequires a transaction. Using it from a thread with no transaction throwsIllegalStateExceptionunlessallowNonTransactionalis set, so producer-only code paths useexecuteInTransactionor@Transactional.
package com.example.screening;import java.time.Duration;import java.util.HashMap;import java.util.List;import java.util.Map;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.KafkaConsumer;import org.apache.kafka.clients.consumer.OffsetAndMetadata;import org.apache.kafka.clients.producer.KafkaProducer;import org.apache.kafka.clients.producer.ProducerRecord;import org.apache.kafka.common.KafkaException;import org.apache.kafka.common.TopicPartition;import org.apache.kafka.common.errors.AuthorizationException;import org.apache.kafka.common.errors.OutOfOrderSequenceException;import org.apache.kafka.common.errors.ProducerFencedException;/** * The read-process-write loop on the plain clients. * Consumer: enable.auto.commit=false, isolation.level=read_committed. * Producer: transactional.id set and unique to this instance. */public final class PlainClientLoop { private final KafkaConsumer<String, String> consumer; private final KafkaProducer<String, String> producer; private volatile boolean running = true; public PlainClientLoop(KafkaConsumer<String, String> consumer, KafkaProducer<String, String> producer) { this.consumer = consumer; this.producer = producer; } public void run() { producer.initTransactions(); consumer.subscribe(List.of("payments.requested")); try { while (running) { ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500)); if (records.isEmpty()) { continue; } try { producer.beginTransaction(); Map<TopicPartition, OffsetAndMetadata> nextOffsets = new HashMap<>(); for (ConsumerRecord<String, String> record : records) { producer.send(new ProducerRecord<>(route(record.value()), record.key(), record.value())); nextOffsets.put(new TopicPartition(record.topic(), record.partition()), new OffsetAndMetadata(record.offset() + 1)); } producer.sendOffsetsToTransaction(nextOffsets, consumer.groupMetadata()); producer.commitTransaction(); } catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException fatal) { throw fatal; // this producer can no longer be used } catch (KafkaException abortable) { producer.abortTransaction(); rewindToLastCommitted(); } } } finally { producer.close(); consumer.close(); } } /** An abort does not move the consumer back. Do it explicitly. */ private void rewindToLastCommitted() { Map<TopicPartition, OffsetAndMetadata> committed = consumer.committed(consumer.assignment()); committed.forEach((partition, offset) -> { if (offset != null) { consumer.seek(partition, offset.offset()); } else { consumer.seekToBeginning(List.of(partition)); } }); } private String route(String paymentJson) { return paymentJson.contains("\"highRisk\":true") ? "payments.review" : "payments.approved"; } public void stop() { running = false; }}Why it is written this way
- The committed value is
record.offset() + 1: the next record to read, not the last one processed. consumer.groupMetadata()is what lets the group coordinator reject a commit from an instance that has lost its partitions. The overload that took only a group id was removed in Kafka 4.0.- The consumer never calls
commitSyncorcommitAsync. Its position is committed only through the producer. - The two catch blocks follow the KafkaProducer documentation: fenced, sequence and authorization errors are fatal and the producer is closed. Any other
KafkaExceptionis answered withabortTransaction(). - The rewind after an abort is the step most often forgotten. Without it the loop carries on with the next batch and the aborted records are skipped until the next restart.
Configuration
Producer, consumer and broker settings that govern transactions, with the defaults documented for Apache Kafka 4.4, and the two Spring Boot properties that map onto them.
transactional.idDefault null
- What it does
- Makes the producer transactional and gives it an identity that survives restarts, so a new instance can finish or abort the old one's transaction and fence it. Setting it implies
enable.idempotence. - When to change it
- Must be stable across restarts of one instance and different between instances that run at the same time. In Spring set
transactionIdPrefixand let the factory add the suffix.
transaction.timeout.msDefault 60000 (1 minute)
- What it does
- How long a transaction may stay open before the coordinator aborts it. The clock starts when the first partition is added to the transaction.
- When to change it
- Keep it above the worst-case time to process one batch and as low as that allows, since an abandoned transaction blocks
read_committedreaders for this long. Values above the broker'stransaction.max.timeout.msare rejected.
isolation.levelDefault read_uncommitted
- What it does
read_committedmakespoll()return only committed transactional records, and only up to the last stable offset. Non-transactional records are returned in both modes.- When to change it
- Set
read_committedon every consumer of a topic written by a transactional producer. The default lets consumers read aborted records.
enable.auto.commitDefault true (Spring for Apache Kafka sets false unless you configure it)
- What it does
- Background offset commits by the consumer itself.
- When to change it
- Must be
falsein a read-process-write loop. Offsets have to reach Kafka only throughsendOffsetsToTransaction.
transaction.max.timeout.ms (broker)Default 900000 (15 minutes)
- What it does
- Upper limit the broker accepts for a producer's
transaction.timeout.ms. - When to change it
- Leave as is unless you want to stop clients from asking for long transactions that could stall readers.
transaction.state.log.replication.factor (broker)Default 3
- What it does
- Replication factor of the internal transaction state topic. The topic cannot be created until the cluster has this many brokers.
- When to change it
- Keep 3 in production. On a one-broker development cluster set it to 1 together with
transaction.state.log.min.isr=1, orinitTransactions()will time out.
transaction.state.log.min.isr (broker)Default 2
- What it does
- Minimum in-sync replicas required for a write to the transaction state topic to succeed.
- When to change it
- Keep 2 with replication factor 3. If fewer replicas are in sync, transactions cannot begin or complete until they return.
transactional.id.expiration.ms (broker)Default 604800000 (7 days)
- What it does
- How long the coordinator keeps a
transactional.idthat has shown no transaction activity. - When to change it
- Matters for producers that sit idle for days. Spring's producer factory has a
maxAgeproperty to replace cached producers before this expires.
spring.kafka.producer.transaction-id-prefixDefault none
- What it does
- When set, Spring Boot makes the producer factory transactional, creates a
KafkaTransactionManagerand wires it into the listener container. - When to change it
- Include something unique to the instance, such as the pod name. Every use of the template then needs a transaction.
spring.kafka.consumer.isolation-levelDefault read-uncommitted
- What it does
- Spring Boot's property for the consumer
isolation.level. - When to change it
- Set
read-committedin every service that consumes transactional topics.
Hands-on exercise
Crash a transactional stage and look for the duplicate
See that a crash between processing and commit leaves no visible output and no moved offset when both are in one transaction, and identify what would still be repeated outside Kafka.
Steps · 0 of 5 done
The same crash, inside a transaction
Simulation · not a real clusterThe stage crashes after it has produced its output and before the commit. Compare at-least-once with transactional mode.
What you should see
In at-least-once mode the crash leaves an output record and an unmoved offset, so the restart writes a second output. In transactional mode the first attempt's output is aborted and never shown to a read_committed reader, the offset does not move, and the retry produces the only visible output. The handler body still ran twice, so anything it did outside Kafka happened twice.
What can go wrong
A database write or API call inside the transactional listener
- Symptom
- Duplicate rows or double charges after a rollback, although the service "uses exactly-once".
- Cause
- The Kafka transaction aborted after the external effect had already happened, and the redelivery repeated it. Kafka cannot roll back a database or a card provider.
- Fix
- Make the external effect idempotent with a unique key or an idempotency key, or write to the database only and publish through an outbox. See Transactions and databases.
Downstream consumer left on the default isolation level
- Symptom
- A downstream service acts on records that the producing stage rolled back, and totals do not match.
- Cause
isolation.leveldefaults toread_uncommitted, which returns aborted and still-open transactional records.- Fix
- Set
isolation.level=read_committed(spring.kafka.consumer.isolation-level: read-committed) on every consumer of the transactional topics.
The same transactional id on two live instances
- Symptom
- Instances fail in turn with
ProducerFencedException, and throughput collapses as soon as the service scales to two pods. - Cause
- Both pods use the same
transaction-id-prefix, so eachinitTransactions()bumps the epoch and fences the other pod's producer. - Fix
- Make the prefix unique per instance, for example by including the pod name, as the Spring reference requires.
Processing takes longer than the transaction timeout
- Symptom
- Commits fail after slow batches and the same records are processed again and again.
- Cause
- The transaction stayed open beyond
transaction.timeout.ms(1 minute by default), so the coordinator aborted it before the commit arrived. - Fix
- Shorten the work per transaction, for example with a lower
max.poll.records, move slow calls out of the transactional stage, or raise the timeout withintransaction.max.timeout.ms.
Readers stalled behind an open transaction
- Symptom
- Lag grows for
read_committedconsumers of a partition while producers are healthy and other partitions are fine. - Cause
- A transaction on that partition is still open, for example because its producer crashed, so the last stable offset cannot advance until the coordinator times it out.
- Fix
- Keep
transaction.timeout.msmodest so abandoned transactions clear quickly, and inspect open transactions withkafka-transactions.sh.
Combining container transactions with non-blocking retries
- Symptom
- A failed record goes to a retry topic, and the listener's other output for that failed attempt is committed instead of rolled back.
- Cause
- The Spring reference states that non-blocking retries cannot be combined with container transactions: when the listener throws, the record is sent to the retry topic and the container transaction commits.
- Fix
- With transactions, use the
DefaultAfterRollbackProcessorwith a back-off and a dead letter recoverer instead of@RetryableTopic.
How to troubleshoot
initTransactions() fails with a timeout on a local or test cluster
Check
Broker settings transaction.state.log.replication.factor and transaction.state.log.min.isr, and the number of brokers in the cluster
- How to read it
- By default the transaction state topic needs three brokers. On a smaller cluster it cannot be created, so the coordinator never becomes available. Set both values to 1 for single-broker development.
ProducerFencedException in the application log
Check
kafka-transactions.sh --bootstrap-server localhost:9092 describe --transactional-id payment-screening-pod-0-0
- How to read it
- The output shows the coordinator, producer id, epoch and state for that id. An epoch that keeps climbing means another live producer shares the id. A single fencing right after a restart or a rebalance is the mechanism doing its job.
A read_committed consumer group is stuck while the topic keeps growing
Check
kafka-transactions.sh --bootstrap-server localhost:9092 list, then kafka-transactions.sh --bootstrap-server localhost:9092 describe-producers --topic payments.approved --partition 7
- How to read it
listshows transactional ids and their state. An id stuck inOngoingis holding the last stable offset back.describe-producersshows the active producers on the partition and the start offset of any open transaction.
A transaction stays open far beyond its timeout
Check
kafka-transactions.sh --bootstrap-server localhost:9092 find-hanging --broker-id 1
- How to read it
- Lists transactions the broker considers hanging. They can be ended with the tool's
abortcommand after confirming that no producer still owns them. Treat that as a last resort.
Lag never reaches zero on a topic written transactionally
Check
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group capture-service
- How to read it
- Commit and abort markers occupy offsets, so the log end offset can sit ahead of the last record the consumer was given. A small constant lag on an idle partition is this artefact. Spring's container property
fixTxOffsetscorrects the reported position.
Throughput dropped after enabling transactions
Check
Producer metrics txn-commit-time-ns-total, txn-send-offsets-time-ns-total and txn-abort-time-ns-total
- How to read it
- Commit time that dominates means too many small transactions: process more records per transaction. A growing abort time means transactions are failing and being redone, so look for timeouts and rebalances.
Key takeaways
- A Kafka transaction makes one producer's writes to Kafka partitions, plus a consumer group's offset commit, atomic. That is all it covers.
- It never includes a database write, an HTTP call or an email. Those happen once per processing attempt and need their own idempotency or an outbox.
- The coordinator stores the decision in the transaction state topic and settles each partition with a commit or abort marker.
transactional.idplus an epoch fences zombies. Since KIP-447, consumer group metadata sent with the offsets lets one producer per instance serve a whole group. The id must be unique per instance.- Only
read_committedconsumers are protected, and they read only up to the last stable offset. The default isread_uncommitted. - An abort hides the output and leaves the offsets unchanged. The consumer still has to be rewound, which Spring's after-rollback processor does.
- In Spring, set a transaction id prefix on the producer factory and a
KafkaTransactionManageron the listener container. The container sends the offsets to the transaction before it commits.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A Capture Service listener runs in a container-managed Kafka transaction. For each PaymentApproved record it calls the card provider's charge API, then sends PaymentCaptured with the transactional KafkaTemplate. One night the brokers are slow and several transactions are aborted on timeout. Next morning 9 customers have been charged twice, but payments.captured holds one committed record for each.
What does this show about the design?
Scenario 2 of 3
A fraud-scoring stage writes scores transactionally to payments.scored. Its integration tests include a forced rollback. A reporting service that consumes payments.scored with default consumer settings shows scores for payments that the scoring stage's logs say were rolled back.
What is the smallest correct fix?
Scenario 3 of 3
You scale a transactional Spring service from 1 pod to 3. All pods have spring.kafka.producer.transaction-id-prefix=tx-screening-. Within seconds the logs fill with ProducerFencedException, containers restart their consumers, and output drops to a trickle.
What is happening?
References
References: Apache Kafka documentation: Design, Message Delivery Semantics and Using Transactions ↗ · KafkaProducer Javadoc: initTransactions, beginTransaction, sendOffsetsToTransaction, commitTransaction, abortTransaction ↗ · Apache Kafka documentation: Producer configs (transactional.id, transaction.timeout.ms) ↗ · Apache Kafka documentation: Consumer configs (isolation.level and the last stable offset) ↗ · Apache Kafka documentation: Broker configs (transaction.max.timeout.ms, transaction.state.log.*) ↗ · Apache Kafka documentation: Transaction protocol (KIP-890, transaction.version) ↗ · KIP-98: Exactly Once Delivery and Transactional Messaging ↗ · KIP-447: Producer scalability for exactly once semantics ↗ · KIP-664: Provide tooling to detect and abort hanging transactions (kafka-transactions.sh) ↗ · Spring for Apache Kafka reference: Transactions ↗ · Spring for Apache Kafka reference: Exactly Once Semantics ↗ · Spring for Apache Kafka reference: After Rollback Processor ↗ · Report a problem