Kafka Delivery Guarantees: At-Most-Once, At-Least-Once, Exactly-Once

You have a working producer and consumer, but “working” hides an important question: if something fails, is a record delivered once, more than once, or not at all? The answer is not automatic. It is the sum of choices on both sides, and getting it wrong causes either lost orders or double charges.

This module defines the three delivery semantics precisely and shows where each one comes from. It sets up the two modules that follow: Reliable Producing hardens the write side, and Transactions and Exactly-Once Semantics covers the strongest guarantee.

What you’ll be able to do after this module

Define at-most-once, at-least-once, and exactly-once precisely.
Explain where each guarantee comes from on the producer and consumer.
State the honest trade-off each one makes.
Choose the right guarantee for a given service in the running scenario.

1. The three semantics

Delivery semantics describe how many times a consumer effectively processes a record when failures and retries are in play.

At-most-once: each record is processed zero or one times. Records can be lost, never duplicated.
At-least-once: each record is processed one or more times. Records are never lost, but can be duplicated.
Exactly-once: each record is effectively processed once. No loss, no duplicates.

Kafka’s default, with sensible settings, is at-least-once. That is the right starting point for most services, because losing an order is worse than occasionally handling one twice, and duplicates can be neutralized by an idempotent consumer.

2. Where the guarantee comes from

No single setting picks a semantic. It emerges from three decisions: how the producer acknowledges writes, whether the producer is idempotent, and when the consumer commits its offset.

flowchart TD
    subgraph producer [Producer side]
        acks["acks: 0 / 1 / all"]
        idem["idempotence on/off"]
    end
    subgraph consumer [Consumer side]
        commit["commit before or after processing"]
    end
    acks --> outcome["Effective delivery semantic"]
    idem --> outcome
    commit --> outcome

The consumer commit timing is the sharpest lever:

Commit before processing: if the handler crashes after the commit, the record is never reprocessed. That is at-most-once, and it can lose data.
Commit after processing: if the handler crashes before the commit, the record is redelivered and processed again. That is at-least-once, and it can duplicate work.

There is no commit timing that gives exactly-once by itself. That guarantee needs either Kafka transactions or an idempotent consumer, covered in the next modules.

3. The honest trade-offs

Each semantic buys something and costs something. Choose deliberately.

Semantic	You get	You pay	Typical use
At-most-once	Lowest latency, simplest	Silent data loss on failure	High-volume metrics where a gap is acceptable
At-least-once	No data loss, simple producer	Duplicates the consumer must tolerate	Most business events (the default)
Exactly-once	No loss, no duplicates	More config, some throughput cost, Kafka-to-Kafka only	Money movement, read-process-write pipelines

The most important honesty about exactly-once: it applies to Kafka-to-Kafka processing. The moment your handler has an external side effect, such as charging a card or calling a REST API, Kafka transactions cannot make that side effect exactly-once. There you still need an idempotent consumer, as covered in Idempotent Consumers, Ordering, and Duplicates.

4. Choosing for the running scenario

Map the requirement to the guarantee, then to the configuration.

flowchart TD
    q1{"Can you lose a record?"}
    q1 -->|yes| amo["At-most-once<br/>acks=0/1, commit early"]
    q1 -->|no| q2{"External side effect<br/>in the handler?"}
    q2 -->|yes| alo["At-least-once<br/>+ idempotent consumer"]
    q2 -->|no| q3{"Pure Kafka-to-Kafka<br/>transform?"}
    q3 -->|yes| eos["Exactly-once<br/>transactions"]
    q3 -->|no| alo

In the scenario:

Notification service: at-least-once is fine; a rare duplicate email is tolerable, and a lost one is not.
Payment service: it charges a card, an external side effect, so it uses at-least-once and an idempotent consumer keyed on the order id.
A pure aggregation reading payments and writing a running total back to Kafka is a genuine exactly-once candidate, which you build in Transactions and Exactly-Once Semantics.

Checkpoint

You should now be able to:

Define at-most-once, at-least-once, and exactly-once precisely.
Explain how acks, idempotence, and commit timing combine to produce a semantic.
State the trade-off each guarantee makes.
Explain why exactly-once does not cover external side effects.
Choose the right guarantee for a service and justify it.

Next:Reliable Producing, where you harden the write side with acks=all, the idempotent producer, and correct retry settings.