Apache Kafka Basics

Apache Kafka is a distributed platform for moving events between services. Instead of the Order service phoning the Inventory, Analytics, and Notification services one by one, it publishes a single "order placed" event to Kafka, and each of those services reads it on its own schedule. The producer does not know or care who is listening.

That decoupling is why Kafka shows up in almost every serious microservices system. It turns brittle chains of direct calls into a durable stream of events that services can consume, replay, and add onto without the producer ever changing.

The mental model: an append-only log

Forget the word "queue" for a moment. Kafka is best pictured as an append-only log — a file you can only add to the end of. Producers append events; consumers read forward through the log at their own position.

The crucial difference from a traditional queue is that reading an event does not delete it. Kafka keeps events for a configured retention period (hours, days, or forever). So five different services can each read the same events independently, and a service that was down for an hour catches up when it restarts. This durability is what makes Kafka safe as the backbone for the saga pattern and other event-driven flows.

Topics and partitions

Events are organized into topics — named streams like order-events or payment-events. A topic is split into partitions, and this split is the source of Kafka's scale.

• Each partition is an ordered, independent log.
• Kafka guarantees order within a partition, not across the whole topic.
• More partitions means more consumers can read in parallel.

Which partition an event lands in is decided by its key. All events with the same key — say, the same orderId — go to the same partition, so they stay in order relative to each other. Pick the key carefully: it is how you control both ordering and how evenly load spreads.

Pro tip: Choose a partition key that gives you the ordering you need and spreads load evenly. Keying by orderId keeps each order's events ordered while distributing different orders across partitions. Keying by something low-cardinality like country can jam most traffic onto one partition and destroy your parallelism.

Producing events from Spring Boot

In the Java world, spring-kafka makes producing straightforward. You configure the broker and serializers in application.yml:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
    consumer:
      group-id: inventory-service          # this service's consumer group
      auto-offset-reset: earliest          # new group reads from the start
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer

Then a producer publishes an event, using the order id as the key so all events for one order stay ordered:

@Service
public class OrderEventPublisher {

    private final KafkaTemplate<String, OrderPlaced> kafka;

    public OrderEventPublisher(KafkaTemplate<String, OrderPlaced> kafka) {
        this.kafka = kafka;
    }

    public void publish(OrderPlaced event) {
        // Key = orderId, so every event for this order lands in one partition
        kafka.send("order-events", event.orderId(), event);
    }
}

The Order service publishes once and is done. It has no idea that three other services will react — and that ignorance is exactly what keeps the services decoupled.

Consuming events and consumer groups

A consumer reads events from a topic. The important concept is the consumer group: a set of consumers that share the work, where each partition is handled by exactly one consumer in the group.

If order-events has four partitions and your Inventory service runs four instances in one group, each instance handles one partition and you get four-way parallelism. Add a fifth instance and it sits idle — you cannot have more active consumers than partitions in a group.

Meanwhile, a completely separate service reads the same topic in its own group and gets its own full copy of every event. Here Inventory consumes order events:

@Component
public class InventoryConsumer {

    private final InventoryService inventory;

    @KafkaListener(topics = "order-events", groupId = "inventory-service")
    public void handle(OrderPlaced event) {
        // Idempotent: safe if Kafka redelivers the same event
        inventory.reserveIfNotAlready(event.orderId(), event.items());
    }
}

Notice the method is idempotent. Kafka guarantees at-least-once delivery by default, which means an event can be delivered more than once after a restart or rebalance. If reserving stock twice would be a bug, you must guard against duplicates yourself. This is one of the most common real-world Kafka pitfalls.

Common mistake: Assuming each event is delivered exactly once and writing consumers that break on a duplicate. Design consumers to be idempotent — check whether you already processed this event id — because redelivery is normal Kafka behavior, not an error.

Offsets: where each group left off

Kafka tracks a per-group offset — the position of the next event to read in each partition. When a consumer processes an event, it commits its offset so that after a restart it resumes where it stopped rather than reprocessing everything.

This is also why you can rewind. Reset a group's offset to the start of a topic and it replays history — invaluable for rebuilding a broken read model or onboarding a brand-new service that needs the full backlog. Traditional queues, which delete on consume, cannot do this.

How Kafka stays durable: replication

Kafka is trusted as a system of record because it does not keep your events in one place. Each partition is replicated across several brokers. One replica is the leader that handles reads and writes; the others are followers that copy the leader's log. If the broker holding the leader dies, a follower that is caught up is promoted, and no committed events are lost.

The setting that controls this is the replication factor, usually three in production. You also tune how many replicas must acknowledge a write before the producer considers it done via the acks setting. Using acks=all means the write is confirmed only after all in-sync replicas have it — slower, but you do not lose data if a broker fails right after. This durability, not raw speed, is the reason teams put Kafka at the center of their event flows.

Serialization and the schema problem

Events outlive the code that produced them. An event written today might be read by a service that gets updated next month, so the shape of your events — their schema — matters a lot. If the Order service adds a field and the Inventory consumer chokes on it, you have an outage.

The common answer is a schema registry with a format like Avro or Protobuf, which enforces backward-compatible changes: you can add optional fields but not remove or rename existing ones without a versioning plan. Even with plain JSON, treat your event schemas as a public contract between services. Casual changes to an event's shape break every consumer downstream, and unlike a REST API you cannot see all the callers at once.

Pro tip: Design events to describe facts that happened — OrderPlaced, PaymentCaptured — in the past tense, not commands telling another service what to do. Fact-style events keep producers ignorant of consumers, which is the whole point of decoupling. Command-style events quietly recreate the tight coupling you used Kafka to escape.

When Kafka is the wrong choice

Be honest about the cost. Kafka is a distributed system you have to run, monitor, and reason about, and it adds eventual consistency to your architecture. If your services only need plain request-response — "give me this user's profile" — then a REST API in Spring Boot is simpler, synchronous, and easier to debug.

Kafka earns its place when you need event-driven communication, high throughput, multiple independent consumers, or replay. If your throughput is modest and you want simpler operations, a broker like RabbitMQ may fit better — the Kafka vs RabbitMQ comparison lays out that choice. Reach for Kafka because you have an event-streaming need, not because it is on every architecture diagram.

Interview relevance

Common questions include "what is the difference between a topic and a partition?" and "how does Kafka differ from a message queue?" The answers that stand out mention that ordering is per-partition, that consumer groups scale up to the partition count, and that Kafka is a durable log rather than transient delivery. Adding that delivery is at-least-once and consumers must be idempotent signals you have actually built with it.

Related concepts

Kafka is the plumbing under most event-driven microservices. Combine it with the saga pattern to coordinate workflows across services, and revisit microservices architecture explained to see where the event backbone sits in the overall design.