Why Apache Kafka: REST vs Messaging vs Event Streaming

This is where the course begins. You already build services that call each other over HTTP; here you add the mental model for asynchronous, event-driven communication that the rest of the course, from architecture internals to exactly-once semantics, rests on.

What you’ll be able to do after this module

Explain the difference between synchronous request/response and asynchronous messaging, and the trade-offs of each.
Describe how a broker decouples services and improves availability, load handling, and fan-out.
Explain what makes Kafka a durable, replayable log rather than a transient queue.
Name the core moving parts an event flows through in Kafka.
Decide when Kafka is a better fit than RabbitMQ or Amazon SQS.

The running scenario

The whole course uses one small e-commerce system so patterns build on each other instead of living in isolation:

Order Service: accepts a new order from the customer and records it.
Payment Service: charges the customer for the order.
Inventory Service: reserves stock for the order.
Notification Service: emails the customer a confirmation.

When an order is placed, Payment, Inventory, and Notification all need to react. Hold that in mind as you compare the two styles below.

1. The synchronous REST approach

You already know this style. The Order Service calls each downstream service in turn, waiting for each response before it continues.

sequenceDiagram
    participant Client
    participant OrderSvc as Order Service
    participant PaymentSvc as Payment Service
    participant InventorySvc as Inventory Service
    participant NotificationSvc as Notification Service

    Client->>OrderSvc: POST /orders
    OrderSvc->>PaymentSvc: POST /payments
    PaymentSvc-->>OrderSvc: 200 OK
    OrderSvc->>InventorySvc: POST /reservations
    InventorySvc-->>OrderSvc: 200 OK
    OrderSvc->>NotificationSvc: POST /notifications
    NotificationSvc-->>OrderSvc: 200 OK
    OrderSvc-->>Client: 201 Created
    Note over OrderSvc: Order Service blocks on every downstream call

This works, but it has structural weaknesses:

Tight coupling: the Order Service must know the address of every downstream service and call each one. Adding a fifth reaction, say analytics, means changing and redeploying the Order Service.
Cascading failure: if the Notification Service is down or slow, the customer’s order request is delayed or fails, even though the order itself was fine.
Latency stacks up: the client waits for the sum of every downstream call.
No natural fan-out: each new consumer is another explicit call the caller has to make.
No memory: once a REST call returns, the interaction is gone. There is no built-in way to replay what happened yesterday.

2. The asynchronous event-streaming approach

With Kafka, the Order Service publishes one OrderCreated event to a topic and returns immediately. Kafka stores the event durably, and each interested service reads it independently, at its own pace.

sequenceDiagram
    participant Client
    participant OrderSvc as Order Service
    participant Kafka as Kafka (orders topic)
    participant PaymentSvc as Payment Service
    participant NotificationSvc as Notification Service

    Client->>OrderSvc: POST /orders
    OrderSvc->>Kafka: publish OrderCreated
    Kafka-->>OrderSvc: acknowledged (stored)
    OrderSvc-->>Client: 201 Created
    Note over OrderSvc: Returns without waiting for consumers
    Kafka->>PaymentSvc: read OrderCreated
    Kafka->>NotificationSvc: read OrderCreated

The producer never calls the consumers. It appends an event to a topic, which stores it, and each consumer reads it whenever it is ready. This buys properties that are hard to get with direct REST:

Decoupling: the Order Service knows only Kafka. Adding an analytics consumer means deploying a new service that reads the topic, with no change to the Order Service.
Load leveling: a traffic spike fills the log instead of overloading downstream services. Consumers drain the backlog at their own pace.
Fan-out: one published event can be read by many independent consumers, each tracking its own position.
Resilience: if the Notification Service is down, the events wait safely in the log and are read when it recovers, without affecting the order flow.

3. What makes Kafka different: the durable, replayable log

Kafka is not just an asynchronous queue. The key idea, which you explore fully in The Kafka Mental Model, is that a topic is a durable, append-only log.

In a traditional queue, a message is deleted once a consumer reads it. In Kafka, reading does not delete anything. Events stay in the log for a configured retention period, and each consumer tracks its own position (its offset).

flowchart LR
    Producer["Order Service<br/>producer"]
    subgraph topic [orders topic - append-only log]
        e0["offset 0<br/>OrderCreated"]
        e1["offset 1<br/>OrderCreated"]
        e2["offset 2<br/>OrderCreated"]
        e0 --> e1 --> e2
    end
    Producer -->|append| e2
    topic -->|read, does not delete| Payment["Payment Service"]
    topic -->|read independently| Notification["Notification Service"]

This unlocks two things a queue cannot easily do:

Replay: a brand-new service can start reading from the beginning of the log and rebuild its state from history.
Independent readers: many consumers read the same events without competing, because each tracks its own offset rather than removing the event.

4. The core moving parts

Every Kafka event passes through a small set of components. A producer appends an event to a topic; the topic is split into partitions for scale and ordering; consumers read from partitions and are organized into consumer groups.

flowchart LR
    Producer["Order Service<br/>producer"]
    subgraph cluster [Kafka Cluster]
        topic["Topic: orders<br/>(partitioned log)"]
    end
    subgraph group [Payment consumer group]
        c1["consumer instance"]
    end
    Producer -->|publish OrderCreated| topic
    topic -->|read| c1

Producer: the service that publishes an event, such as OrderCreated.
Topic: a named, partitioned, append-only log that stores events.
Partition: an ordered slice of a topic; the unit of parallelism and ordering.
Consumer: the service that reads and processes events.
Consumer group: a set of consumer instances that share the work of reading a topic.

You go deeper on each of these in Core Concepts and the Architecture section. For now, remember: producers append to topics, topics are split into partitions, and consumer groups read from those partitions.

5. When event streaming is the wrong tool

Kafka is not a replacement for every REST call. Prefer synchronous request/response when:

The caller genuinely needs the result before it can continue, such as validating a payment inline before confirming an order in the same request.
You need a strong read-after-write guarantee for the same user request.
The interaction is a simple query with no downstream side effects.

A healthy system mixes both: synchronous calls where an immediate answer is required, event streaming where work can happen independently of the caller.

Appendix: Kafka vs RabbitMQ vs SQS

All three move data between services, but they optimize for different things.

Tool	Model	Best fit	Watch out for
Apache Kafka	Distributed, append-only log	High-throughput event streaming, replay, many independent readers of the same events, stream processing	More concepts to learn (partitions, offsets, consumer groups); routing is consumer-side
RabbitMQ	Smart broker, flexible routing (AMQP)	Task queues, complex routing, request/reply, per-message acknowledgement	Not built for long-term event replay by default
Amazon SQS	Managed cloud queue	Simple, fully managed queues on AWS with minimal ops	Fewer routing features, no replay, no rich streaming model

Rule of thumb

Choose Kafka when you need a durable, replayable event log at high throughput, or you want stream processing and many independent consumers.
Choose RabbitMQ when you need flexible routing and per-message handling with modest operational overhead.
Choose SQS when you want a simple managed queue on AWS and do not need replay or advanced routing.

If you already know RabbitMQ, the biggest mental shift is that Kafka keeps events after they are read, and consumers track their own position in the log.

Checkpoint

You should now be able to:

Explain in one sentence why a team would publish an event instead of making a direct REST call.
List the benefits a log-based broker adds: decoupling, load leveling, fan-out, and replay.
Explain why reading from Kafka is non-destructive and what an offset is.
Name the core moving parts an event flows through: producer, topic, partition, consumer, consumer group.
Say when Kafka is a better fit than RabbitMQ or SQS.

Next:The Kafka Mental Model, where the log idea becomes concrete with partitions, offsets, and retention.