Kafka Transactions and Exactly-Once Semantics with Spring

The idempotent producer removes duplicates from producer retries, but many services do something more complex: they consume a record, transform it, and produce a result, then commit the input offset. If that read-process-write spans a crash, you can produce the output twice or commit an offset for work you did not finish. Kafka transactions make the whole cycle atomic.

This is the mechanism behind true exactly-once semantics, but only for Kafka-to-Kafka work. This module shows how it works, how to use it in Spring, and exactly where it stops helping.

What you’ll be able to do after this module

Explain the read-process-write pattern and why it needs atomicity.
Configure a transactional producer with a transactional.id.
Commit consumer offsets inside a producer transaction.
Explain isolation.level=read_committed and transaction markers.
Explain zombie fencing and the real limits of exactly-once.

1. The read-process-write problem

Consider a processor that reads from payments, computes a result, and writes to notifications, then commits its offset on payments. Three separate actions must all happen or none happen:

Produce the output record.
Commit the input offset.
Do both atomically.

Without a transaction, a crash between steps produces wrong outcomes. Produce then crash before commit: the input is reprocessed and the output is written twice. Commit then crash before produce: the output is lost but the input is marked done. Neither is acceptable for money-adjacent work.

2. Transactions make the cycle atomic

A Kafka transaction groups the output records and the offset commit into one atomic unit. Either all of it becomes visible to downstream consumers, or none of it does.

sequenceDiagram
    participant C as Consumer (payments)
    participant A as Processor
    participant P as Transactional producer
    participant K as Kafka

    A->>P: beginTransaction()
    C->>A: record from payments (offset N)
    A->>P: send result to notifications
    A->>P: sendOffsetsToTransaction(offset N+1)
    A->>P: commitTransaction()
    P->>K: output + offset commit, atomically

The key move is that the input offset is committed through the producer’s transaction, not by the consumer separately. The offset commit and the output records share one fate.

3. Configuring transactions in Spring

Set a transactional-id-prefix on the producer. Spring uses it to create a transactional KafkaTemplate and a KafkaTransactionManager. The consumer that feeds the transaction must not auto-commit.

spring:
  kafka:
    producer:
      transaction-id-prefix: tx-payments-processor-
      acks: all
    consumer:
      isolation-level: read_committed
      enable-auto-commit: false

With a transactional producer configured, annotate the processing method with @Transactional. Spring begins a Kafka transaction, and the listener container sends the consumed offsets into it automatically.

@KafkaListener(topics = "payments", groupId = "notification-builder")
@Transactional
public void process(PaymentSucceeded payment) {
    Notification note = buildNotification(payment);
    kafkaTemplate.send("notifications", payment.orderId(), note);
    // On normal return, Spring commits the transaction:
    // the output record and the payments offset commit together.
}

4. read_committed and transaction markers

A transaction is not visible to consumers until it commits. Kafka writes control records called transaction markers into the partitions to signal commit or abort.

The consumer setting isolation.level decides what a downstream consumer sees:

read_uncommitted (default for plain consumers): sees all records, including those from transactions that later abort.
read_committed: sees only records from committed transactions, and never reads past an open transaction on that partition.

For an exactly-once pipeline, every downstream consumer must use read_committed, or it will read aborted output and the guarantee is lost.

flowchart LR
    subgraph log [notifications partition]
        r1["record (tx1)"]
        m1["commit marker (tx1)"]
        r2["record (tx2)"]
        m2["abort marker (tx2)"]
    end
    log --> rc["read_committed consumer<br/>sees tx1 record only"]

5. Zombie fencing

A dangerous failure is a zombie: a processor that hangs, is presumed dead and replaced, then wakes up and tries to write. Two producers writing the same logical output would break exactly-once.

Kafka fences zombies using the transactional.id. When a new producer registers with the same transactional.id, it bumps an epoch number. The broker then rejects any write from the older epoch, so the zombie’s writes are refused. This is why the transactional.id must be stable across restarts of the same logical processor.

6. The honest limits of EOS

Exactly-once is real, but narrow. It guarantees exactly-once for records and offsets that stay inside Kafka. It cannot make an external side effect exactly-once.

Covered: consume from payments, produce to notifications, commit the offset, all atomically.
Not covered: consume from orders and charge a credit card. The card API is outside the transaction, so a retry can charge twice.

For external side effects, use at-least-once with an idempotent consumer, covered next in Idempotent Consumers, Ordering, and Duplicates. Transactions also cost throughput and add operational surface, so reserve them for genuine Kafka-to-Kafka pipelines.

7. Guided practical

Run this against the local lab.

Create payments and notifications topics.
Configure a processor service with transaction-id-prefix and a read_committed consumer.
Annotate the read-process-write method with @Transactional and run it end to end.
Throw an exception inside the method after the send, and confirm the downstream read_committed consumer never sees the output and the offset is not advanced.
Remove the exception, rerun, and confirm the output appears and the offset advances together.

Next:Retries, Error Handling, and Dead Letter Topics, where you handle the failures that transactions do not, on the consumer side.