Read time: ~

Local Lab: Run Kafka Yourself (Docker Compose, KRaft)

Stand up a single-broker and a three-broker KRaft cluster with Docker Compose, understand listeners and advertised.listeners, add a Kafka UI, and manage topics and messages from the console.

You now understand brokers, partitions, replicas, and the KRaft control plane in theory. This module makes it real: a Kafka cluster running on your machine that you can create topics on, produce to, and break on purpose. Everything from here on assumes you can run this lab, so keep it handy.

You will build two setups. A single-broker cluster gets you started in seconds. A three-broker cluster lets you see replication, in-sync replicas, and leader failover for yourself, which matters for the reliability modules later. Both run in KRaft mode, with no ZooKeeper, matching a modern cluster.

---

What you’ll be able to do after this module

• Run a single-broker KRaft cluster locally with Docker Compose.
• Explain what listeners and advertised.listeners do, and why a wrong value is the most common local Kafka failure.
• Add a Kafka UI and browse brokers, topics, and messages.
• Create, describe, alter, and delete topics from the command line.
• Produce and consume records from the console, without any application code.
• Run a three-broker cluster and observe replication and in-sync replicas.

---

1. The local development topology

For local development you run the broker in a container. Your future Spring Boot services and the command-line tools connect to it, and a Kafka UI container gives you a browser view of the cluster.

flowchart LR
    App["Your Spring Boot app<br/>(later sections)"]
    CLI["Kafka CLI tools"]
    Browser["Browser"]
    subgraph net [Docker network]
        Broker["Kafka broker<br/>(broker + controller,<br/>KRaft mode)"]
        UI["Kafka UI"]
    end
    App -->|"localhost:9092"| Broker
    CLI -->|"localhost:9092"| Broker
    UI -->|"kafka:29092"| Broker
    Browser -->|"http :8080"| UI

Notice that clients on your host reach the broker at localhost:9092, while the Kafka UI container reaches the same broker at kafka:29092. Those two addresses point at one broker through two different listeners. Understanding why is the single most useful thing in this module, so section 3 covers it in full.

---

2. Quick start: a single-broker KRaft cluster

Create a file named docker-compose.yml with a single broker running in KRaft combined mode, meaning one process acts as both broker and controller.

services:
  kafka:
    image: apache/kafka:3.9.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_LISTENERS: HOST://0.0.0.0:9092,DOCKER://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: HOST://localhost:9092,DOCKER://kafka:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: HOST:PLAINTEXT,DOCKER:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

Start it and confirm it is healthy:

docker compose up -d
docker compose ps
docker compose logs kafka | tail -20

You should see the broker reach a running state in the logs. The three replication settings are all set to 1 because a single broker cannot replicate the internal topics across more brokers. On a real cluster these default to 3, which you will use in section 8.

---

3. Listeners and advertised.listeners: the footgun

This is where most people get stuck, so it is worth slowing down. A Kafka client does not just connect once. It first contacts a bootstrap address, then the broker replies with the addresses each client should actually use to reach every broker. Those returned addresses come from advertised.listeners. If they are wrong, the first connection succeeds and every following one fails with a confusing timeout.

Two settings work together:

Setting	Meaning
listeners	The addresses and ports the broker binds to and accepts connections on.
advertised.listeners	The addresses the broker tells clients to use when they reconnect.

A single broker often needs more than one listener because different clients reach it by different names. In the lab, a process on your host uses localhost, but a container on the Docker network uses the service name kafka. So the broker advertises both, one per named listener.

sequenceDiagram
    participant C as Client
    participant B as Broker
    C->>B: connect to bootstrap (localhost:9092)
    B-->>C: metadata: reach me at advertised address
    Note over C: client now uses the advertised address
    C->>B: produce / consume via advertised address

The listener map in the compose file wires this up:

• HOST://localhost:9092: advertised to clients on your machine (CLI, Spring app).
• DOCKER://kafka:29092: advertised to other containers, like the Kafka UI.
• CONTROLLER://...:9093: the internal KRaft control-plane listener, used by the controller quorum, never by clients.

---

4. Add a Kafka UI

A browser view makes the cluster tangible. Add a Kafka UI service to the same docker-compose.yml, under services, alongside the kafka service.

  kafka-ui:
    image: provectuslabs/kafka-ui:latest
    container_name: kafka-ui
    depends_on:
      - kafka
    ports:
      - "8080:8080"
    environment:
      KAFKA_CLUSTERS_0_NAME: local
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
      DYNAMIC_CONFIG_ENABLED: "true"

Apply the change and open the UI:

docker compose up -d

Browse to http://localhost:8080. The UI connects to the broker over the DOCKER listener at kafka:29092, which is exactly why that second advertised listener exists. Explore the Brokers, Topics, and Consumers views. They are empty until you create a topic next.

---

5. Manage topics from the command line

The apache/kafka image ships the standard CLI scripts under /opt/kafka/bin/. Run them with docker exec, pointing at the host listener from inside the container.

Create the orders topic from the running scenario, with three partitions:

docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic orders --partitions 3 --replication-factor 1

Describe it to see partitions, the leader, and the replica set:

docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe --topic orders

List all topics, add partitions, and delete a topic:

# List every topic
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --list

# Increase orders to 6 partitions (you can grow, never shrink)
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --alter --topic orders --partitions 6

# Delete a throwaway topic
docker exec kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --delete --topic scratch

---

6. Produce and consume from the console

You can exercise the topic with no application code, which proves the round trip before Spring is involved. Kafka records have an optional key and a value, and the key decides the partition, so the console producer is configured to parse key:value.

Start a consumer in one terminal, reading from the beginning:

docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders --from-beginning \
  --property print.key=true --property key.separator=:

In a second terminal, produce a couple of OrderCreated records keyed by order id:

docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders \
  --property parse.key=true --property key.separator=:

At the > prompt, type records and press Enter after each:

order-1:{"orderId":1,"item":"widget","amount":49.90}
order-2:{"orderId":2,"item":"gadget","amount":19.00}

The consumer terminal prints each record as order-1:{...}. Records with the same key always land on the same partition, which is how Kafka preserves per-key ordering. Open the Kafka UI Topics view, click orders, and watch the messages appear per partition alongside the console output.

When your Spring services connect in section 3, they will point at the same broker with one line of configuration:

spring:
  kafka:
    bootstrap-servers: localhost:9092

---

7. A three-broker cluster for realistic experiments

A single broker cannot show replication, because there is nowhere to place the copies. Swap in a three-broker cluster when you want to see in-sync replicas and leader failover. Use a separate file, docker-compose-cluster.yml, so it does not clash with the single-broker lab.

x-broker-env: &broker-env
  KAFKA_PROCESS_ROLES: broker,controller
  KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: HOST:PLAINTEXT,DOCKER:PLAINTEXT,CONTROLLER:PLAINTEXT
  KAFKA_INTER_BROKER_LISTENER_NAME: DOCKER
  KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
  KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
  KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
  KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0

services:
  kafka-1:
    image: apache/kafka:3.9.0
    container_name: kafka-1
    ports:
      - "9092:9092"
    environment:
      <<: *broker-env
      KAFKA_NODE_ID: 1
      KAFKA_LISTENERS: HOST://0.0.0.0:9092,DOCKER://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: HOST://localhost:9092,DOCKER://kafka-1:29092

  kafka-2:
    image: apache/kafka:3.9.0
    container_name: kafka-2
    ports:
      - "9093:9092"
    environment:
      <<: *broker-env
      KAFKA_NODE_ID: 2
      KAFKA_LISTENERS: HOST://0.0.0.0:9092,DOCKER://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: HOST://localhost:9093,DOCKER://kafka-2:29092

  kafka-3:
    image: apache/kafka:3.9.0
    container_name: kafka-3
    ports:
      - "9094:9092"
    environment:
      <<: *broker-env
      KAFKA_NODE_ID: 3
      KAFKA_LISTENERS: HOST://0.0.0.0:9092,DOCKER://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: HOST://localhost:9094,DOCKER://kafka-3:29092

Each broker publishes its own host port (9092, 9093, 9094) so you can reach any of them from your machine, while brokers talk to each other over the DOCKER listener. Start the cluster and create a replicated topic:

docker compose -f docker-compose-cluster.yml up -d

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create --topic payments --partitions 3 --replication-factor 3

Describe it and read the Leader, Replicas, and Isr columns. With replication factor 3, each partition lists three replicas and, when healthy, three in-sync replicas. This is the concrete version of the ISR set from Cluster Anatomy.

docker exec kafka-1 /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --describe --topic payments

---

8. Guided practical

Run this end to end in the single-broker lab, then repeat the topic steps against the three-broker cluster.

1. Start the single-broker lab with docker compose up -d and confirm both kafka and kafka-ui are running.
2. Open the Kafka UI at http://localhost:8080 and confirm the local cluster shows one broker.
3. Create the orders topic with three partitions from the CLI, then find it in the UI.
4. Start a console consumer, produce two keyed OrderCreated records, and confirm the consumer prints them.
5. Alter orders to six partitions and confirm the new count in the UI.
6. Start the three-broker cluster, create the payments topic with replication factor 3, and read its ISR set.
7. Stop one broker and describe payments again to watch the leader move and the ISR shrink.
8. Tear down with docker compose down -v when finished.

Keep the single-broker lab handy. The next section connects real Spring Boot code to it, starting with First Producer and Consumer.