Message Queues & async processing

A **message queue** (or broker) sits between two systems. The producer pushes messages; consumers pull them at their pace. If the consumer is slow, messages buffer up — the producer doesn't have to wait or fail.

When to reach for one: (a) smooth a traffic spike (10k signups/minute during a launch → buffer → process at 1k/min steadily), (b) async work (send a welcome email after signup — don't block the signup response), (c) isolate failures (email server is down → messages wait in queue, system recovers when it's back), (d) fan-out (one event → notify 5 downstream systems).

**Delivery guarantees**: **at-most-once** (send and forget — may lose messages, never duplicate), **at-least-once** (retry until acked — never lose but may duplicate; most common), **exactly-once** (very hard in distributed — usually approximated via idempotent consumers).

**Tools**: **RabbitMQ** (traditional queue, easy to use, solid for most needs), **Kafka** (high-throughput event log, retain history, multi-subscriber), **AWS SQS / Google Pub/Sub / Azure Service Bus** (managed cloud queues, zero ops), **Redis Streams** (lightweight queue-like), **NATS** (low-latency, cloud-native). Choose by scale + features needed.

Idempotency is the consumer's duty: if you might receive the same message twice (at-least-once), processing it twice must not double-charge / double-email. Solve via a dedup key + a 'seen' table, or by making operations naturally idempotent (SET balance = 100 vs ADD 10 to balance).

**Dead-letter queue (DLQ)**: a separate queue for messages the consumer failed to process after N retries. Not silently dropped — accumulated for inspection. Critical for debugging and not losing data.