Read time: ~

Idempotency, Ordering and Duplicate Handling

Why at-least-once delivery forces idempotent consumers, practical deduplication strategies, and how to reason about ordering.

Prerequisite:Consumer Acknowledgements and PrefetchYou’ll need: A Spring AMQP consumer and a datastore you can query.

RabbitMQ delivers at least once, so your consumers will occasionally see the same message twice, and under parallelism, out of order. This is not a bug to fix in the broker; it is a property to design around in your code. This module shows how.

What you’ll be able to do after this module

  • Explain why at-least-once delivery makes duplicates inevitable.
  • Make a consumer idempotent with a deduplication strategy.
  • Reason about ordering and when it is and is not preserved.
  • Use single active consumer when strict per-queue order matters.

1. Why duplicates happen

As introduced in Core Concepts, a message is redelivered if the broker never receives the consumer’s ack, even when the work actually succeeded.

sequenceDiagram
    participant Broker as RabbitMQ
    participant Consumer as Inventory Service
    participant DB as Database

    Broker->>Consumer: deliver OrderCreated (id=ord-1)
    Consumer->>DB: reserve stock for ord-1
    DB-->>Consumer: ok
    Note over Consumer: crashes before ack
    Broker->>Consumer: redeliver OrderCreated (id=ord-1)
    Consumer->>DB: reserve stock for ord-1 again?
    Note over DB: must not double-reserve

Retries (previous modules) and connection recovery (next module) both increase the chance of a duplicate. Assume every handler can run more than once for the same message.

2. Make the consumer idempotent

Idempotent means processing the same message twice has the same effect as processing it once. There are two broad approaches.

Natural idempotency

Design the operation so repeating it is harmless. “Set order status to SHIPPED” is naturally idempotent; “increment shipped count by 1” is not. Prefer absolute state changes over relative ones where you can.

Deduplication with a processed-id store

Give every message a stable business key (an order id, or a producer-set message-id) and record which keys you have handled. Skip anything you have seen.

@RabbitListener(queues = "inventory.queue")
public void onOrderCreated(OrderCreated event) {
    if (!processedRepo.markIfNew(event.orderId())) {
        return; // already handled, ignore duplicate
    }
    inventoryService.reserve(event);
}

markIfNew must be atomic. Back it with a unique constraint (insert the id, catch the duplicate-key violation) or an atomic operation in Redis. Ideally do the dedup insert and the business write in the same database transaction, so a crash cannot leave one without the other.

Note: A producer-set message-id (a MessageProperties field) is a convenient dedup key, but a business key like the order id is more robust because it survives republishing and retries that mint a new message.

3. Ordering caveats

RabbitMQ preserves order within a single queue delivered to a single consumer. That guarantee breaks in ordinary configurations:

  • Multiple consumers on one queue (a work queue) process in parallel, so overall order is not preserved.
  • Concurrency greater than one on a single listener does the same within one service.
  • Requeues and retries move a message behind others.

Design so you do not depend on global ordering. If message B depends on message A, prefer one of:

  • Carry enough context in each message to process independently.
  • Key related messages to the same queue and consumer.
  • Handle out-of-order arrival explicitly (for example, ignore an update older than the current state’s version).

4. Single active consumer

When you genuinely need strict order for a queue, enable single active consumer: only one consumer processes the queue at a time, and others stand by to take over on failure. You keep failover without parallel processing.

@Bean
Queue orderedQueue() {
    return QueueBuilder.durable("orders.sequenced.queue")
            .singleActiveConsumer()
            .build();
}

This trades throughput for order. Use it only for the specific queues that require it, not as a global default.

5. A practical checklist

  • Every consumer has a dedup key and an atomic “have I seen this?” check.
  • Dedup and business writes share a transaction where possible.
  • Handlers prefer absolute state changes over relative ones.
  • Only order-sensitive queues use single active consumer.

The operational symptoms of duplicates and reordering, and how to spot them on rotation, are in the Latency, Ordering and Duplicates playbook.

Checkpoint

You should now be able to:

  • Explain why at-least-once delivery makes duplicates inevitable.
  • Make a consumer idempotent with natural idempotency or a dedup store.
  • Say when ordering is preserved and when it is not.
  • Use single active consumer for strict per-queue order.