When you press «pay» on a website, your request does not travel straight from the browser to the bank’s database. In almost every modern system, what happens in between is a long queue of messages: a message broker, an intermediate server that collects each event, stores it safely, and hands it to whoever has to process it. That is why, even if a server goes down at that exact instant, your order is almost never lost.
The idea is decoupling. If the checkout interface called the processor directly and it was down or overloaded, the request would simply fail. With a queue in between, the sender deposits the message and moves on; the receiver consumes it whenever it can. This absorbs traffic spikes and lets different parts of the system evolve independently. Tools like Apache Kafka, RabbitMQ or Amazon SQS are the best-known examples of this architecture.
The minimal unit: the message
Everything revolves around the message: a structure holding data (usually JSON or binary), headers with metadata, and often a key. That key is not just an identifier: it decides which partition the message ends up in. In systems like Kafka, a topic is split into several partitions, each one an ordered, append-only log —data can only be added at the end. A message with the same key always lands in the same partition, and that guarantees that messages from the same client arrive in strict order.
This distinction is subtle but crucial. Kafka only guarantees ordering within a partition, not across partitions. If global order mattered, you would need a single partition — but then you would lose parallelization. The trade-off between ordering and throughput is one of the central dilemmas of distributed messaging.
How processing works: consumers and groups
On the other side are the consumers. Each reads from one or more partitions and keeps track of which message it has processed using an offset: the numeric position inside the log. To scale, consumers are grouped into a consumer group: the system distributes the partitions among the group’s members, so that each partition is read by a single member at a time. If a consumer dies, its partition is reassigned (rebalanced) to another, and the work continues.
Here lies the most important design decision: what to do when a consumer reads a message but dies before finishing it. This is the guaranteed-delivery dilemma, and it boils down to three models.
At-most-once, at-least-once, exactly-once
In the at-most-once model, the consumer acknowledges the message before processing it. If it dies afterwards, the message is lost: no duplicates, but data can disappear. In at-least-once, it acknowledges after processing. If the process dies halfway, the message is redelivered, which can produce duplicates. It is the default model of most brokers, and the way to tolerate it is to make processing idempotent: running it twice has the same effect as running it once, for example by checking whether the record already exists before inserting it.
Exactly-once is the holy grail: each message is processed exactly once. It is achieved with distributed transaction mechanisms (like Kafka’s idempotent transactions, based on a transaction coordinator) or by combining the offset with the result write in the same atomic operation. It is more expensive in latency and coordination, and it is not always worth it.
Persistence: the log as source of truth
Reliability does not come out of nowhere. The broker writes every message to disk sequentially and replicates it across several nodes (in Kafka, typical replication factors are 3). Because writes are append-only and sequential, they take advantage of disk speed and the operating system’s page cache, reaching throughputs of hundreds of thousands of messages per second on a single node. If a node fails, another holding a copy takes over as leader.
And the log is not deleted once the message is consumed: it is kept for a retention period (days or weeks). That enables something very valuable: reprocessing. A consumer can go back to an old offset and re-read the full history, for example to rebuild an index or fix a bug. The message does not “disappear” when read; the position marker simply moves forward.
Why all this matters
Understanding message queues is understanding how systems that cannot afford to lose a single piece of data are built: payments, bookings, telemetry, coordination between microservices. The next time an order survives a server crash, remember it was not luck: it was a well-guarded offset, a well-ordered partition, and a backup waiting on another node.
