Coordination & Reliability

Choreography vs. Orchestration

Choreography: each service reacts to events and decides its own next step, with no central coordinator — maximizes decoupling, but the overall process becomes implicit and harder to trace without good observability. Orchestration: a central process explicitly tells each service what to do and when — easier to see and control, at the cost of a more central dependency.

Sagas

Distributed ACID transactions across services aren't practical, so a saga coordinates a multi-step business process as a sequence of local transactions, each with a defined compensating action to undo its effect if a later step in the sequence fails.

// Orchestrated saga for order fulfillment
async function fulfillOrderSaga(orderId: string) {
  await reserveInventory(orderId);
  try {
    await chargePayment(orderId);
  } catch (err) {
    await releaseInventory(orderId); // compensating action
    throw err;
  }
  try {
    await scheduleShipment(orderId);
  } catch (err) {
    await refundPayment(orderId);      // compensate step 2
    await releaseInventory(orderId);   // compensate step 1
    throw err;
  }
}

Idempotency & At-Least-Once Delivery

Most brokers guarantee at-least-once delivery, not exactly-once — a network blip during acknowledgment can cause the same message to be redelivered. Handlers must be idempotent: processing an event twice must have the same effect as processing it once, typically by checking a processed-event-ID log before applying a side effect.

async function handleOrderPlaced(event: OrderPlacedEvent) {
  const alreadyProcessed = await processedEvents.exists(event.id);
  if (alreadyProcessed) return; // safe no-op on redelivery

  await chargeCard(event.customerId, event.total, {
    idempotencyKey: event.id, // belt-and-suspenders at the payment provider too
  });
  await processedEvents.markDone(event.id);
}

Outbox Pattern & Dead-Letter Queues

The outbox pattern writes a business change and its event to the same local database transaction, then a separate relay process reliably publishes it — avoiding a state where the DB write succeeds but the broker publish fails. A dead-letter queue catches messages that repeatedly fail processing so one poison-pill message can't block an entire queue while awaiting manual review.

-- Outbox table, written in the same transaction as the business change
BEGIN;
  INSERT INTO orders (id, customer_id, total) VALUES ('o1', 'c1', 49.99);
  INSERT INTO outbox (id, event_type, payload, published)
    VALUES ('e1', 'OrderPlaced', '{"orderId":"o1"}', false);
COMMIT;
-- A separate relay process polls `outbox WHERE published = false`,
-- publishes to the broker, then marks the row published.

Ordering & Change Data Capture

Ordering is usually only guaranteed per-partition/per-key, not globally across a topic — partitioning by entity ID (e.g. all events for one order go to the same partition) preserves relative order for that entity while still scaling out. Change Data Capture (CDC) tools like Debezium tail a database's write-ahead log to emit an event per row change, a common way to bridge an existing database into an event-driven system without touching application code.