Kafka Producer Internals: Keys, Partitioning, Acks, and Batching

Your producer works, but so far you have let Spring pick the defaults. This module opens up the send path so you can make deliberate choices: which partition a record lands on, how durable a write is, and how the producer trades a little latency for a lot of throughput. These choices decide whether the Order service preserves ordering and never silently loses events.

What you’ll be able to do after this module

Explain how the record key maps to a partition, and why keys drive ordering.
Write a custom partitioner and know when it is worth it.
Choose an acks setting and state its durability trade-off.
Tune batching with linger.ms and batch.size, and pick a compression type.
Attach headers to a record for metadata like a correlation id.
Send asynchronously with a callback instead of blocking on every send.

1. The producer send path

A send does not go straight to the broker. The record passes through several stages inside the client first, which is what makes the producer efficient.

flowchart TD
    app["send(record)"]
    ser["Serializer<br/>key and value to bytes"]
    part["Partitioner<br/>choose partition"]
    acc["Record accumulator<br/>group by partition into batches"]
    send["Sender thread<br/>batch to partition leader"]
    broker["Broker<br/>append to partition"]
    app --> ser --> part --> acc --> send --> broker

The key insight is that send() returns before the record reaches the broker. The record is serialized, assigned a partition, and placed into an in-memory batch. A background sender thread ships whole batches to each partition leader. This is why batching and async sends matter so much for throughput.

2. Keys and the default partitioner

The record key decides the partition. With the default partitioner, Kafka hashes the key and takes the result modulo the partition count, so the same key always maps to the same partition (as long as the partition count does not change).

// Same key, always the same partition, so ordering is preserved
kafkaTemplate.send("orders", String.valueOf(order.orderId()), event);

Same key: all OrderCreated events for orderId=42 land on one partition, in order.
Null key: records are spread across partitions (recent clients use a sticky strategy that fills one batch at a time), so there is no per-key ordering.

This is the concrete version of the ordering rule from Core Concepts: Kafka guarantees order within a partition only. Choosing the key is choosing your ordering unit.

3. A custom partitioner

The default partitioner is right most of the time. Write a custom one only when key hashing does not express your routing need, for example to send high-priority customers to a dedicated partition.

public class PriorityPartitioner implements Partitioner {

    @Override
    public int partition(String topic, Object key, byte[] keyBytes,
                         Object value, byte[] valueBytes, Cluster cluster) {
        int partitions = cluster.partitionCountForTopic(topic);
        if (key instanceof String k && k.startsWith("vip-")) {
            return 0; // reserve partition 0 for VIP orders
        }
        return Math.floorMod(Utils.murmur2(keyBytes), partitions - 1) + 1;
    }

    @Override
    public void close() { }

    @Override
    public void configure(Map<String, ?> configs) { }
}

spring:
  kafka:
    producer:
      properties:
        partitioner.class: com.example.orders.PriorityPartitioner

4. Acks: how durable is a write

The acks setting decides how many replicas must confirm a write before the producer considers it successful. It is the single most important durability knob.

`acks`	Waits for	Trade-off
`0`	Nothing; fire and forget	Fastest, can lose records silently
`1`	Leader only	Loses records if the leader fails before followers copy them
`all`	Leader plus all in-sync replicas	Safest, slightly higher latency

For the Order service, use acks=all with min.insync.replicas on the broker so a write is only acknowledged once enough replicas hold it. That combination is what makes an acknowledged event genuinely durable, building on the ISR set from Cluster Anatomy.

spring:
  kafka:
    producer:
      acks: all

5. Batching and compression

The producer groups records headed for the same partition into batches. Two settings control the batch, and they trade latency for throughput.

batch.size: the maximum size in bytes of one batch. A larger batch sends more per request.
linger.ms: how long the sender waits to fill a batch before sending it. A small linger lets more records accumulate, improving throughput at the cost of a few milliseconds.

spring:
  kafka:
    producer:
      batch-size: 16384      # 16 KB
      linger-ms: 5           # wait up to 5 ms to fill a batch
      compression-type: lz4  # compress each batch

Compression is applied per batch, so larger batches compress better. lz4 and snappy are fast with modest ratios; zstd and gzip compress harder at more CPU cost. Compression cuts network and disk use and often improves throughput end to end.

flowchart LR
    r1["records for orders-0"] --> b["batch<br/>(up to batch.size,<br/>held up to linger.ms)"]
    b --> c["compress batch<br/>(lz4)"]
    c --> leader["partition leader"]

6. Record headers

A record carries a key, a value, and optional headers. Headers hold metadata that is not part of the business payload, such as a correlation id for tracing across services.

ProducerRecord<String, OrderCreated> record =
        new ProducerRecord<>("orders", String.valueOf(event.orderId()), event);
record.headers().add("correlation-id", correlationId.getBytes(StandardCharsets.UTF_8));
kafkaTemplate.send(record);

The consumer can read the header alongside the payload. Full tracing across producer and consumer is covered in Observability; here it is enough to know headers exist and travel with the record.

7. Async sends and callbacks

Because send() is asynchronous and returns a future, you should not block on it for every record. Blocking with .get() after each send serializes your throughput to one round trip at a time. Instead, attach a callback and let the sender thread work in the background.

kafkaTemplate.send("orders", key, event)
    .whenComplete((result, ex) -> {
        if (ex != null) {
            log.error("Failed to publish order {}", key, ex);
        } else {
            log.debug("Published to partition {} at offset {}",
                result.getRecordMetadata().partition(),
                result.getRecordMetadata().offset());
        }
    });

Block only when you genuinely need the result before continuing, for example inside a request that must confirm the write before responding. Even then, blocking is a deliberate choice, not the default.

8. Guided practical

Run this against the single-broker lab, using the Order service from the previous module.

Set acks=all, linger-ms: 5, and compression-type: lz4 in the producer config and restart the app.
Publish 10 orders that all share one key (for example order-42). In the Kafka UI, confirm every record landed on the same partition.
Publish 10 orders with different keys and confirm they spread across partitions.
Add a correlation-id header and confirm it appears on the record in the Kafka UI.
Switch a send to a callback and log the partition and offset it was written to.

Next:Consuming Deeper, where you take control of consumer groups, offsets, commit modes, and concurrency on the reading side.