Message Queues and At-Least-Once Delivery

#Context

// producer: fire and forget, returns in a millisecond
await queue.send({type: 'order.paid', orderId: 'o_42', messageId: 'm_9f3a'})

// consumer: runs later, on another machine, possibly more than once
for await (const msg of queue.receive()) {
  await sendInvoiceEmail(msg.body.orderId)   // the side effect
  await msg.ack()                             // "delete it, I'm done"
}

Broker

The server that stores messages between producer and consumer: RabbitMQ, Kafka, SQS, Redis.

Ack

The consumer telling the broker a message is fully processed. Until then the broker considers it in flight.

Redelivery / visibility timeout

If no ack arrives within a window (SQS: visibility timeout, RabbitMQ: consumer disconnect), the broker gives the message to a consumer again.

Dead-letter queue (DLQ)

Where a message goes after failing N times, so one poison message does not block everything behind it.

Offset

In a log (Kafka), a consumer’s position: the index of the last message it has committed. Acking is moving the offset.

Consumer group

A set of consumer instances that share the work of one subscription; each partition is read by exactly one member at a time.

Lag / backlog

How far consumers are behind producers: messages waiting in a queue, or offsets behind the log head.

#Why it matters

#Why exactly-once delivery is impossible

#Queue versus log

#Building effectively-once processing

Consumer side: idempotent handler with a dedup record

async function handle(msg: Message) {
  await db.transaction(async tx => {
    // 1. claim the message id; a duplicate hits the unique index and stops
    const claimed = await tx.execute(
      `INSERT INTO processed_messages (id) VALUES ($1)
       ON CONFLICT (id) DO NOTHING`, [msg.id],
    )
    if (claimed.rowCount === 0) return          // already done → just ack

    // 2. the business write, in the SAME transaction as the claim
    await tx.execute(
      `UPDATE orders SET status = 'invoiced' WHERE id = $1`, [msg.body.orderId],
    )
  })
  // 3. side effects that live outside the DB need their own idempotency key
  await email.send({idempotencyKey: msg.id, template: 'invoice', ...})
  await msg.ack()
}
// crash anywhere → redelivery → step 1 short-circuits or the tx rolled back;
// either way the order is invoiced once and the email provider dedups on msg.id

Producer side: the transactional outbox

#Operating a queue

1. 1
   Receive with a lease long enough for the p99 processing time, and a heartbeat that extends it for outliers.
2. 2
   Check the dedup store first (or rely on the unique constraint inside the transaction) so a redelivery is cheap.
3. 3
   Do the work. Any external call gets an idempotency key derived from the message id.
4. 4
   Ack only after the work is durably committed. On failure, do not ack: let the backoff schedule redeliver, and after N attempts let the broker move it to the DLQ.
5. 5
   Export lag, oldest-message age, retry rate and DLQ depth as metrics; those four graphs describe the health of the whole pipeline.

#Pitfalls

• Acking before the work is done

  Often accidental: a framework auto-acks on receive, or the handler returns a promise the runner does not await. A crash or a thrown error after that point loses the message silently. Turn off auto-ack, ack in a finally only on success, and test by killing the consumer mid-handler.

• A handler that is "usually" idempotent

  Setting status = paid is safe to repeat; incrementing a counter, appending a row or sending an email is not. Every handler will be called twice eventually. Give each a dedup key stored in the same transaction as its write, and pass that key to any external API that supports one.

• Visibility timeout shorter than the job

  The broker assumes the first consumer died and hands the message to a second one; now two workers run the same job concurrently and both may succeed. Size the timeout from measured p99 duration, extend it with a heartbeat for long work, and keep the dedup check so the loser exits early.

• No dead-letter queue

  One message whose payload crashes the handler is retried forever, and in a FIFO or single-partition setup everything behind it waits. Cap attempts, route failures to a DLQ with the exception attached, alert on it, and have a way to fix and replay.

• Dual write: save to the database, then publish

  Two systems, no shared transaction. A crash after the commit means downstream services never hear about the order; a broker outage means the same. Write the event to an outbox table in the same transaction and let a relay publish it, or use change data capture on the table itself.

• Assuming global order

  Competing consumers, retries and multiple partitions all reorder messages. Design handlers to tolerate "shipped" arriving before "paid" (store and reconcile, or use version numbers), and use a partition key when one entity's events must stay in sequence.

#Interview questions