Kafka Transactional Outbox and Change Data Capture with Debezium

The Order service has a hidden bug waiting to happen. When it handles a request, it writes an order to its database and publishes OrderCreated to Kafka. Those are two separate systems, and there is no way to make both happen atomically with a plain publish. This module fixes that dual-write problem with the transactional outbox pattern, delivers it to Kafka with change data capture, and shows how failures are undone across services with a saga.

What you’ll be able to do after this module

Explain the dual-write problem and why it causes lost or phantom events.
Apply the transactional outbox pattern to publish atomically with a DB commit.
Explain how Debezium change data capture moves outbox rows into Kafka.
Design a saga with compensating actions for cross-service failures.

1. The dual-write problem

A dual write is any operation that must update two systems that do not share a transaction. The Order service writes its database and publishes to Kafka. Four things can happen, and two are broken:

DB commit succeeds, publish succeeds: correct.
DB commit fails, publish never happens: correct (nothing occurred).
DB commit succeeds, publish fails: an order exists that no one was told about, a lost event.
Publish succeeds, DB commit fails: a phantom event for an order that does not exist.

You cannot fix this by reordering the two steps or by retrying, because a crash can land between them either way. The fix is to make the event part of the same database transaction as the data.

flowchart TD
    req["handle request"]
    db["commit order to DB"]
    pub["publish to Kafka"]
    req --> db --> pub
    db -.crash before publish.-> lost["order saved, event lost"]

2. The transactional outbox pattern

The outbox pattern writes the event into an outbox table in the same local transaction as the business data. Since it is one database transaction, the order and its event commit together or not at all. A separate process then relays outbox rows to Kafka.

@Transactional
public void createOrder(CreateOrderRequest request) {
    Order order = orderRepository.save(new Order(request));
    OutboxEvent event = OutboxEvent.of(
        "orders",                       // destination topic
        String.valueOf(order.getId()),  // key
        new OrderCreated(order.getId(), order.getItem(), order.getAmount()));
    outboxRepository.save(event);
    // one transaction: order row and outbox row commit atomically
}

Now the order and the intent to publish are inseparable. What remains is to get the outbox rows into Kafka reliably, which is what change data capture does.

3. Change data capture with Debezium

Rather than polling the outbox table, change data capture reads the database’s own transaction log (the write-ahead log) and streams every committed change. Debezium is the common Kafka-native CDC tool. It tails the log, turns each new outbox row into a Kafka record, and does so with at-least-once delivery.

flowchart LR
    app["Order service"] -->|"one tx"| db[("Orders DB<br/>orders + outbox tables")]
    db -->|WAL| dbz["Debezium connector"]
    dbz -->|publish| k["Kafka orders topic"]
    k --> consumers["Payment, Notification"]

Debezium runs as a Kafka Connect connector, covered in Kafka Connect and the Wider Ecosystem. Its outbox event router can read the destination topic and key from columns in the outbox row, so each event lands on the right topic with the right key. Because delivery is at-least-once, consumers must be idempotent, exactly as built in Idempotent Consumers, Ordering, and Duplicates.

4. Sagas and compensation

The outbox guarantees an event is published, but a business flow can still fail partway. Payment succeeds, then Inventory finds the item out of stock. There is no distributed transaction across services, so you undo work with a compensating action: a new event that reverses a prior step.

sequenceDiagram
    participant O as Order
    participant P as Payment
    participant I as Inventory

    O->>P: OrderCreated
    P->>P: charge card
    P->>I: PaymentSucceeded
    I->>I: reserve stock: OUT OF STOCK
    I->>P: StockRejected
    P->>P: refund (compensating action)
    P->>O: OrderFailed

This is a choreographed saga: each service reacts and, on failure, emits a compensating event that upstream services handle to roll their state back. Compensation is a business-level undo, not a database rollback, so design each step to be reversible or safely retryable.

5. Guided practical

Run this against the local lab with a database for the Order service.

Add an outbox table and write to it in the same transaction as the order.
Verify that rolling back the order transaction also removes the outbox row (no phantom event).
Run a Debezium connector (or, to keep it simple, a small relay that reads unsent outbox rows and publishes them) and confirm OrderCreated reaches the orders topic.
Simulate a downstream failure and emit a compensating OrderFailed, confirming the Order service reverts state.
Deliver a duplicate compensating event and confirm the idempotent handler does not double-refund.

Next:Kafka Streams: Stateless and Stateful Processing, where you process and aggregate event streams instead of consuming record by record.