Topics, Partitions, Ordering & Consumer Groups

The Core Model

A distributed event-streaming platform for high-throughput, durable, ordered event logs. Producers write records to a TOPIC; consumers read from topics they're interested in. A topic splits into PARTITIONS distributed across BROKERS — enabling parallelism and higher throughput than one unpartitioned log.

Ordering: Per-Partition, Not Global

producer.send(topic="user-events", key=userId, value=event)
// same key -> same partition (via hash) -> preserves relative order
// for THAT user's events specifically

Ordering is guaranteed only within a single partition, never across partitions of the same topic. When order matters for related events (e.g. all events for one user), key them consistently so they land in the same partition.

Consumer Groups — Parallel Processing

Within one consumer group, each partition is actively consumed by exactly one member — adding consumers (up to the partition count) increases parallelism. A rebalance redistributes partitions when a consumer joins/leaves the group. Offsets track each consumer's read position per partition, letting it resume exactly where it left off after a restart.

Partition Count Tradeoff

More partitions = more consumer parallelism, but more per-partition broker overhead. Increasing count later doesn't retroactively reorganize existing data — worth planning upfront based on expected throughput needs.