Retention, Replication & When to Choose Kafka

Retention, Not Delete-on-Consume

Unlike traditional queues that delete a message once consumed, Kafka retains messages for a configured time/size window regardless of consumption — multiple independent consumer groups can each read/reread the same stream at their own pace, and consumers can replay historical events. Log compaction is an alternative policy: keep only the latest message per key, suited to "current state" changelog-style topics.

Replication & Fault Tolerance

Each partition can be replicated across multiple brokers — one leader (handles reads/writes), others as followers. If the leader fails, a follower is promoted automatically, avoiding data loss/downtime from a single broker failure.

At-Least-Once Delivery

A message can occasionally be redelivered (e.g. a consumer crashes after processing but before committing its offset) — design consumers to be idempotent, handling potential duplicate processing safely, rather than assuming exactly-once by default (Kafka does offer exactly-once semantics with extra configuration for cases that specifically need it).

Kafka Streams & Kafka Connect

Kafka Streams provides higher-level stream-processing abstractions (filter/join/aggregate) beyond raw producer/consumer APIs. Kafka Connect offers pre-built connectors for streaming data in/out of external systems (databases, warehouses) without hand-writing integration code.

Kafka vs. RabbitMQ, and the Event Backbone Pattern

Kafka optimizes for high throughput, replay, and multiple independent consumer groups on the same stream — event sourcing, log aggregation, analytics. RabbitMQ suits flexible general-purpose task/message routing. A central Kafka event backbone also reduces service-to-service coupling in microservices architectures — new consumers can subscribe to an existing event stream without the publisher needing to know about them.