High Level Design
Message Queues
How message queues decouple producers and consumers, core components, edge cases like poison messages and duplicates, and popular implementations.
A Message Queue is an asynchronous communication mechanism where producers send messages to a queue and consumers process them independently, enabling decoupled and resilient systems.
Pain Points Solved#
- Synchronous calls cause high latency and cascading failures
- Traffic spikes overwhelm downstream services
- Tight coupling blocks independent scaling
- Retries and failure handling are complex
Use Cases#
- Background job processing (emails, notifications)
- Order processing pipelines
- Payment retries
- Event-driven microservices
Architecture#
Core Components#
| Component | Role |
|---|---|
| Producer | Publishes messages |
| Queue | Buffer holding messages |
| Consumer | Processes messages |
| Broker | Manages queues and delivery |
| Acknowledgement (Ack) | Confirms successful processing |
| Dead Letter Queue (DLQ) | Stores messages that failed after max retries |
Flow#
- Producer sends message to Queue
- Queue persists the message
- Consumer polls or subscribes
- Consumer processes the message
- Message is acknowledged and removed
Key idea: Work is buffered and processed asynchronously.
Trade-offs#
| Pros | Cons |
|---|---|
| Loose coupling | Limited message replay |
| Better fault tolerance | Ordering guarantees can be weak |
| Load leveling | Debugging async flows is harder |
| Easy retries | Throughput lower than log-based systems (Kafka) |
Popular Implementations#
| Tool | Best For |
|---|---|
| RabbitMQ | Routing, exchanges, low latency |
| Amazon SQS | Fully managed, simple scaling |
| ActiveMQ | JMS-based enterprise use |
Edge Cases#
1. Poison Messages Blocking Consumers#
A malformed/invalid message that always fails processing, no matter how many retries.
Problem: Consumer keeps retrying → queue head is blocked → throughput drops to zero.
Solutions:
- Retry limit (max retry count)
- Exponential backoff to avoid tight retry loops
- Dead Letter Queue (DLQ) after N failures
2. Duplicate Message Processing#
The same message is processed more than once.
Happens due to: Consumer crashes after processing but before ack; network timeout → broker resends.
Problem: Double charges, duplicate emails.
Solutions:
- Idempotent consumers — track processed message IDs
- Upserts instead of inserts
3. Message Ordering Violations#
Messages processed out of the order they were produced.
Happens due to: Multiple parallel consumers; retries reorder; unordered queues.
Solutions:
- Partition by key (orderId, userId)
- Single consumer per key
- FIFO queues (with reduced throughput)
- Sequence numbers + reordering logic
4. DLQ Growth Monitoring#
A Dead Letter Queue stores permanently failed messages. DLQ growth = silent data loss.
Failure modes: DLQ fills up unnoticed; same bug keeps producing bad messages; no alerting.
Solutions:
- Monitor DLQ size and rate
- Alert on abnormal growth
- Build reprocessing pipelines
- Store failure reason and stack trace in DLQ message
Use a message queue when you need asynchronous task processing, retries, and buffering — but do not need event replay.
Queues are for tasks, not event history. Use Kafka if you need replay.