Event-Driven Architecture with Apache Kafka

Event-driven architecture (EDA) has become the backbone of modern distributed systems. When services communicate through events rather than direct API calls, they gain loose coupling, independent scalability, and natural audit trails. Apache Kafka has emerged as the de-facto event streaming platform for these workloads, handling trillions of events per day across industries.

Why Events Instead of APIs

Synchronous REST or gRPC calls create temporal coupling — the caller blocks until the receiver responds. If the receiver is slow or unavailable, the caller suffers. Events invert this: the producer publishes to a durable log and moves on immediately. Consumers process at their own pace.

This model unlocks three architectural properties that are hard to achieve with synchronous communication:

Decoupled scalability. A payment processor and a fraud detection service can scale completely independently because neither calls the other directly. The event log is the interface.

Replayability. Kafka retains events for a configurable retention period. A new consumer — a reporting service, a new ML model — can replay the entire history and derive its own materialized view without burdening other services.

Event sourcing compatibility. When every state change is expressed as an immutable event, your log becomes the system of record. CRUD databases become mere projections.

Core Kafka Concepts

Topics and Partitions

A topic is a named, durable, append-only log. Topics are divided into partitions — the unit of parallelism. Events with the same key are always written to the same partition, preserving order for that key.

Topic: order-events
  Partition 0: [order-placed, order-paid, order-shipped]   # key = order-123
  Partition 1: [order-placed, order-cancelled]              # key = order-456
  Partition 2: [order-placed, order-paid]                   # key = order-789

Choose your partition key carefully. For order events, order_id is the natural key — you want all events for a single order to land in the same partition and be processed in sequence.

Consumer Groups

A consumer group is a set of consumers that collectively read a topic. Each partition is assigned to exactly one consumer in the group at a time. Add more consumers (up to the number of partitions) to increase throughput linearly.

from confluent_kafka import Consumer

consumer = Consumer({
    'bootstrap.servers': 'kafka:9092',
    'group.id': 'fraud-detection-v2',
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': False,   # manual commit for at-least-once
})

consumer.subscribe(['order-events'])

while True:
    msg = consumer.poll(timeout=1.0)
    if msg is None:
        continue
    if msg.error():
        handle_error(msg.error())
        continue

    process_event(msg.value())
    consumer.commit(asynchronous=False)  # commit after successful processing

Setting enable.auto.commit = False and committing only after successful processing gives you at-least-once delivery — events may be reprocessed after a crash, but they will never be lost.

Exactly-Once Semantics

Kafka 0.11+ supports idempotent producers and transactional APIs. Combine them for exactly-once semantics across produce and consume:

producer = Producer({
    'bootstrap.servers': 'kafka:9092',
    'enable.idempotence': True,
    'transactional.id': 'fraud-detector-txn-1',
    'acks': 'all',
})

producer.init_transactions()

try:
    producer.begin_transaction()
    result = detect_fraud(event)
    producer.produce('fraud-decisions', value=result)
    producer.send_offsets_to_transaction(offsets, consumer_group)
    producer.commit_transaction()
except Exception as e:
    producer.abort_transaction()
    raise

This ensures the output event is written and the input offset is committed atomically — either both happen or neither does.

Designing Topics for Scale

Event Schema Strategy

Use Apache Avro with a Schema Registry. Avro enforces schema evolution rules: adding optional fields is safe (backward compatible), removing required fields breaks consumers. The Schema Registry stores schema versions and validates compatibility before a producer publishes a new version.

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "order_id", "type": "string"},
    {"name": "customer_id", "type": "string"},
    {"name": "total_amount_cents", "type": "long"},
    {"name": "placed_at", "type": {"type": "long", "logicalType": "timestamp-millis"}},
    {"name": "shipping_address", "type": "string", "default": ""}
  ]
}

Topic Naming Conventions

A consistent naming convention becomes load-bearing at scale. One pattern that works well:

{domain}.{entity}.{event-type}.{version}

examples:
  orders.order.placed.v1
  payments.payment.processed.v2
  inventory.product.stock-updated.v1

This makes topic purpose immediately clear and allows wildcard subscriptions (orders.*) for cross-cutting consumers like audit loggers.

Stream Processing with Kafka Streams

Kafka Streams is a client library (not a separate cluster) for stateful stream processing. It reads from input topics, maintains local state stores backed by changelog topics, and writes results to output topics.

StreamsBuilder builder = new StreamsBuilder();

KStream<String, OrderEvent> orders = builder.stream("orders.order.placed.v1");

// Aggregate order counts per customer in a 1-hour window
KTable<Windowed<String>, Long> orderCounts = orders
    .groupBy((key, event) -> event.getCustomerId())
    .windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofHours(1)))
    .count(Materialized.as("order-count-store"));

// Flag customers exceeding 10 orders per hour (velocity check)
orderCounts
    .toStream()
    .filter((windowedKey, count) -> count > 10)
    .mapValues((windowedKey, count) -> new VelocityAlert(windowedKey.key(), count))
    .to("fraud.velocity-alerts.v1");

The state store (order-count-store) is backed by a Kafka topic and replicated across standby replicas — if the processing node fails, another node restores the store from the changelog and resumes processing with minimal lag.

Operational Considerations

Monitoring What Matters

Three metrics deserve dashboards and alerts above all others:

• Consumer lag — the difference between the latest offset and the committed offset per partition. Rising lag means consumers are falling behind. Alert when lag exceeds your acceptable processing delay budget.
• Producer error rate — failed produce calls indicate network issues, topic authorization problems, or broker overload.
• Partition leader skew — if most partitions' leaders are on a single broker, that broker becomes a bottleneck. Use kafka-preferred-replica-election.sh to rebalance.

Retention and Storage

Kafka's default 7-day retention is often too short for event sourcing use cases. Consider topic-level overrides:

kafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type topics \
  --entity-name orders.order.placed.v1 \
  --alter \
  --add-config retention.ms=2592000000   # 30 days

For truly infinite retention (event sourcing), use log compaction: Kafka keeps the latest event per key indefinitely, discarding older versions. This gives you a compact, current-state snapshot at no additional storage cost.

When Kafka Is the Right Choice

Kafka excels when: you need durable, replayable event streams; throughput exceeds what a message queue can handle; multiple independent consumers need the same events; or you're building event-sourced systems with complex projections.

Kafka is overkill when: you have simple point-to-point messaging with a single consumer; message volumes are modest; or operational complexity is a primary constraint. In those cases, a managed queue (SQS, Cloud Tasks, RabbitMQ) is the pragmatic choice.

The payoff of event-driven architecture with Kafka is systems that scale independently, recover gracefully, and accumulate institutional memory in their event logs — but that payoff is proportional to the discipline applied to schema design, topic structure, and consumer group management.