System Design

Message Queues

Asynchronous communication pattern enabling decoupled services to exchange messages via queues.

Popular Queue Systems

🌊

Apache Kafka

Distributed event streaming platform. Partitioned topics, persistent logs, high throughput for real-time data pipelines.

Event StreamingPersistentPartitions

Use case:Activity tracking, log aggregation, real-time analytics, event sourcing

Throughput:Millions of messages/sec, designed for massive scale

🐰

RabbitMQ

Traditional message broker. Flexible routing via exchanges, supports multiple protocols (AMQP, MQTT), reliable delivery.

Message BrokerAMQPRouting

Use case:Task queues, microservices communication, background jobs, RPC

Throughput:Thousands to hundreds of thousands msgs/sec

☁️

AWS SQS

Fully managed queue service. Simple API, auto-scaling, no server management. Standard & FIFO queues available.

ManagedServerlessFIFO Option

Use case:Decoupling AWS services, buffering requests, asynchronous workflows

Throughput:Standard: unlimited, FIFO: 3,000 msgs/sec (batched)

⚡

Redis Streams

Lightweight append-only log in Redis. Consumer groups, persistence, fast in-memory operations with replication.

In-MemoryLow LatencyStreams

Use case:Real-time leaderboards, notifications, chat systems, simple event logs

Throughput:Hundreds of thousands msgs/sec, sub-millisecond latency

Message Flow Patterns

Explore different messaging patterns with interactive animations

Waiting...

Simple Queue: Producer → Queue → Consumer

📤PRODUCER

API Server

📦QUEUE

RabbitMQ / SQS

📥CONSUMER

Worker Process

1. Producer publishes message to queue

2. Message stored in queue (persistent)

3. Consumer polls queue for messages

4. Consumer processes message

5. Consumer sends ACK to queue

6. Queue removes message after ACK

Producer

Queue/Topic

Consumer

Dead Letter Queue

Delivery Guarantees

Understand the trade-offs between reliability and performance

At-Most-Once

Messages delivered once or not at all. Fastest but may lose messages. No retry on failure.

Use: Metrics, non-critical logs, telemetry where loss is acceptable

At-Least-Once

Messages guaranteed delivery but may duplicate. Retry on failure. Requires idempotent consumers.

Use: Order processing, email notifications (with deduplication)

Exactly-Once

Each message delivered exactly once. Most expensive, requires distributed transactions or careful design.

Use: Financial transactions, payment processing, critical state changes

System Comparison

Compare popular message queue systems by key metrics

System	Persistence	Ordering	Best For
Apache Kafka	✓	✓	Event streaming, real-time analytics, log aggregation
RabbitMQ	✓	✓	Task queues, microservices, traditional messaging
AWS SQS	✓	✓	Decoupling AWS services, serverless workflows
Redis Streams	✓	✓	Real-time apps, low latency, simple use cases

Resources

Title	Type	Link
System Design Primer - Message Queues	youtube	Open
RabbitMQ Tutorials	web	Open
Apache Kafka Documentation	web	Open
AWS SQS Documentation	web	Open

Guide Contributors

ObydulSenior Software Engineer

Edit this page on GitHub↗

Init guide environment...

Popular Queue Systems

🌊

Apache Kafka

Distributed event streaming platform. Partitioned topics, persistent logs, high throughput for real-time data pipelines.

Event StreamingPersistentPartitions

Use case:Activity tracking, log aggregation, real-time analytics, event sourcing

Throughput:Millions of messages/sec, designed for massive scale

🐰

RabbitMQ

Traditional message broker. Flexible routing via exchanges, supports multiple protocols (AMQP, MQTT), reliable delivery.

Message BrokerAMQPRouting

Use case:Task queues, microservices communication, background jobs, RPC

Throughput:Thousands to hundreds of thousands msgs/sec

☁️

AWS SQS

Fully managed queue service. Simple API, auto-scaling, no server management. Standard & FIFO queues available.

ManagedServerlessFIFO Option

Use case:Decoupling AWS services, buffering requests, asynchronous workflows

Throughput:Standard: unlimited, FIFO: 3,000 msgs/sec (batched)

⚡

Redis Streams

Lightweight append-only log in Redis. Consumer groups, persistence, fast in-memory operations with replication.

In-MemoryLow LatencyStreams

Use case:Real-time leaderboards, notifications, chat systems, simple event logs

Throughput:Hundreds of thousands msgs/sec, sub-millisecond latency

Message Flow Patterns

Explore different messaging patterns with interactive animations

Waiting...

Simple Queue: Producer → Queue → Consumer

📤PRODUCER

API Server

📦QUEUE

RabbitMQ / SQS

📥CONSUMER

Worker Process

1. Producer publishes message to queue

2. Message stored in queue (persistent)

3. Consumer polls queue for messages

4. Consumer processes message

5. Consumer sends ACK to queue

6. Queue removes message after ACK

Producer

Queue/Topic

Consumer

Dead Letter Queue

Delivery Guarantees

Understand the trade-offs between reliability and performance

At-Most-Once

Messages delivered once or not at all. Fastest but may lose messages. No retry on failure.

Use: Metrics, non-critical logs, telemetry where loss is acceptable

At-Least-Once

Messages guaranteed delivery but may duplicate. Retry on failure. Requires idempotent consumers.

Use: Order processing, email notifications (with deduplication)

Exactly-Once

Each message delivered exactly once. Most expensive, requires distributed transactions or careful design.

Use: Financial transactions, payment processing, critical state changes

System Comparison

Compare popular message queue systems by key metrics

System	Persistence	Ordering	Best For
Apache Kafka	✓	✓	Event streaming, real-time analytics, log aggregation
RabbitMQ	✓	✓	Task queues, microservices, traditional messaging
AWS SQS	✓	✓	Decoupling AWS services, serverless workflows
Redis Streams	✓	✓	Real-time apps, low latency, simple use cases

Resources

Title	Type	Link
System Design Primer - Message Queues	youtube	Open
RabbitMQ Tutorials	web	Open
Apache Kafka Documentation	web	Open
AWS SQS Documentation	web	Open

Guide Contributors

ObydulSenior Software Engineer