Reliable Kafka Producer: Idempotence, Acks, and Retries

At-least-once delivery starts at the producer. If a write is lost before enough replicas hold it, no consumer retry can recover it. This module hardens the Order service’s write path so an acknowledged event is genuinely durable, and so retries do not silently create duplicates.

You already met acks in Producing Deeper. Here you combine it with min.insync.replicas, turn on the idempotent producer, and set retry and timeout values that behave correctly under failure.

What you’ll be able to do after this module

Combine acks=all with min.insync.replicas for durable writes.
Explain how the idempotent producer prevents duplicates on retry.
Configure retries and delivery.timeout.ms sensibly.
Explain the max.in.flight.requests.per.connection ordering caveat.
Configure a reliable producer in Spring.

1. Durable writes: acks=all plus min.insync.replicas

acks=all tells the leader to wait until all in-sync replicas have the record before acknowledging. On its own that is not enough, because if the in-sync set has shrunk to just the leader, “all” means one replica, and a leader failure still loses data.

The broker setting min.insync.replicas closes that gap. It sets the minimum number of in-sync replicas that must hold a record for an acks=all write to succeed. With replication factor 3 and min.insync.replicas=2, a write needs at least two replicas, so the cluster can lose one broker without losing data or blocking writes.

flowchart TD
    p["Producer: acks=all"]
    l["Leader"]
    f1["Follower 1 (in sync)"]
    f2["Follower 2 (in sync)"]
    p --> l
    l --> f1
    l --> f2
    f1 -->|ack| l
    f2 -->|ack| l
    l -->|"ack once min.insync.replicas met"| p

2. The idempotent producer

Durability creates a new problem: retries. If the producer sends a record, the broker writes it, but the acknowledgement is lost on the way back, the producer retries and the broker writes a second copy. That is a duplicate born entirely on the write side.

The idempotent producer eliminates it. Each producer gets a producer id, and each record carries a monotonic sequence number per partition. The broker tracks the last sequence number it accepted per producer and partition, so a retried record with an already-seen sequence number is acknowledged but not written again.

sequenceDiagram
    participant P as Producer (idempotent)
    participant B as Broker

    P->>B: record seq=5
    B->>B: append seq=5
    B--xP: ack lost in transit
    P->>B: retry record seq=5
    B->>B: seq=5 already seen, skip append
    B-->>P: ack (no duplicate written)

In modern Kafka the idempotent producer is on by default, and acks=all is implied. It is worth enabling explicitly so the intent is clear:

spring:
  kafka:
    producer:
      acks: all
      properties:
        enable.idempotence: true

3. Retries and timeouts

You rarely set a retry count directly anymore. The controlling setting is delivery.timeout.ms, the total time the client will spend trying to deliver a record, including all retries and backoffs. The producer retries until either the record is acknowledged or this deadline passes.

Setting	Meaning	Guidance
`delivery.timeout.ms`	Total time to deliver, across all retries	The real deadline; default 120000 (2 minutes)
`retries`	Max retry attempts	Leave high; `delivery.timeout.ms` bounds it
`retry.backoff.ms`	Wait between retries	Small default is usually fine
`request.timeout.ms`	Per-request wait for a broker response	Must be less than `delivery.timeout.ms`

The practical rule: tune delivery.timeout.ms to how long the Order service can acceptably keep trying, and let retries stay high so transient blips are absorbed automatically.

4. The ordering caveat: max.in.flight

max.in.flight.requests.per.connection sets how many unacknowledged batches the producer sends to a broker at once. If it is greater than 1 and a batch fails and is retried after a later batch already succeeded, records can be reordered.

The idempotent producer removes this risk. With idempotence enabled, Kafka safely allows up to 5 in-flight requests while still preserving order, because the broker uses the sequence numbers to reject out-of-order batches. This is one more reason to keep idempotence on: you get pipelining and ordering together.

5. A reliable producer in Spring

Putting it together, the Order service’s application.yml for durable, duplicate-free, ordered writes:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      acks: all
      properties:
        enable.idempotence: true
        delivery.timeout.ms: 120000
        max.in.flight.requests.per.connection: 5

Handle the send result so a genuine failure, one that outlived all retries, is never silent:

kafkaTemplate.send("orders", String.valueOf(event.orderId()), event)
    .whenComplete((result, ex) -> {
        if (ex != null) {
            log.error("Order {} not delivered within delivery.timeout.ms", event.orderId(), ex);
            // surface to the caller or an outbox; do not swallow
        }
    });

Guaranteeing the database write and the Kafka publish happen together is a separate problem, the dual-write problem, solved with the outbox pattern in Transactional Outbox and CDC.

6. Guided practical

Run this against the three-broker cluster from Local Lab.

Create orders with replication factor 3, and set min.insync.replicas=2 on it with kafka-configs.sh.
Configure the Order service with acks=all and enable.idempotence=true, and publish a few records.
Stop one broker and confirm writes still succeed with two in-sync replicas.
Stop a second broker and confirm acks=all writes now fail with NotEnoughReplicasException.
Restart the brokers, confirm the in-sync set recovers, and confirm writes succeed again.

Next:Transactions and Exactly-Once Semantics, where the producer and consumer join one atomic read-process-write.