Kafka vs RabbitMQ

Kafka vs RabbitMQ

Event streaming platform vs traditional message broker - understanding the fundamental differences

TL;DR

Kafka = Event streaming. High throughput, message replay, data pipelines.
RabbitMQ = Message broker. Complex routing, guaranteed delivery, task queues.

Architecture Difference

The fundamental architectural difference explains most of their behavior differences.

{architectureDiff.kafka.title}

{architectureDiff.kafka.description}

{#each architectureDiff.kafka.flow as step, i}
{i + 1}
{step.label}
{step.desc}
{#if i < architectureDiff.kafka.flow.length - 1}
{/if}
{/each}
{architectureDiff.kafka.keyPoint}

{architectureDiff.rabbitmq.title}

{architectureDiff.rabbitmq.description}

{#each architectureDiff.rabbitmq.flow as step, i}
{i + 1}
{step.label}
{step.desc}
{#if i < architectureDiff.rabbitmq.flow.length - 1}
{/if}
{/each}
{architectureDiff.rabbitmq.keyPoint}

Message Lifecycle

Kafka: Log-Based

1.
Message written to partition
Appended to end of log with offset
2.
Consumers read at their offset
Multiple consumers can read same message
3.
Retention policy deletes old messages
After 7 days or 1GB (configurable)

RabbitMQ: Queue-Based

1.
Message routed to queue(s)
Exchange determines destination
2.
Broker pushes to consumer
One consumer receives each message
3.
Consumer acknowledges
Message deleted from queue

Feature Comparison

{#each features as feature, i} {/each}
Feature Kafka RabbitMQ
{feature.name} {feature.kafka} {feature.rabbitmq}

When to Use Each

RabbitMQ Routing Patterns

RabbitMQ's flexible exchange types enable complex routing scenarios that Kafka cannot do natively.

Direct
Route by exact routing key match
order.created to orders queue
Topic
Route by pattern matching (*.error)
app.*.error to error-handler
Fanout
Broadcast to all bound queues
notification to all subscribers
Headers
Route by message headers
x-priority: high to fast queue

Kafka Partitioning

Kafka achieves scalability through partitions - each partition is an ordered, immutable log.

Partition 0
{#each [0, 1, 2, 3, 4] as offset}
{offset}
{/each}
Partition 1
{#each [0, 1, 2, 3] as offset}
{offset}
{/each}
Partition 2
{#each [0, 1, 2, 3, 4, 5] as offset}
{offset}
{/each}
Ordering
Guaranteed within partition only
Parallelism
Max consumers = number of partitions
Key-based routing
Same key always goes to same partition

Real World Examples

Kafka Use Cases

Real-time Analytics
Stream clickstream data to analytics pipeline
Event Sourcing
Store all state changes as immutable events
Log Aggregation
Centralize logs from thousands of services
CDC Pipelines
Stream database changes to data warehouse

RabbitMQ Use Cases

Task Queues
Background job processing with retries
Microservices
Async communication between services
IoT Messaging
MQTT protocol for device communication
Request/Reply
RPC-style communication patterns