Message queues and event-driven architecture explained

A message queue stores messages from a producer until a consumer is ready to process them. The producer hands off work and moves on, and consumers work through the backlog at their own pace. Event-driven architecture builds on this: services announce facts such as OrderPlaced, and other services react to them.

The benefit is decoupling in time. A slow email provider no longer slows down checkout, and a traffic spike fills the queue instead of overloading workers. A consumer that is down catches up when it comes back.

The cost is that work now happens later, somewhere else. Failures no longer show up in the original request, so you need retries, monitoring and consumers that tolerate duplicates.

In a point-to-point queue, each message goes to one consumer. Several workers can read from the same queue as competing consumers, which spreads the load. This fits jobs that must be done once, such as resizing an uploaded image.

In publish/subscribe (pub/sub), a publisher sends a message to a topic, and every subscriber gets its own copy. This fits events that several parts of the system care about. One OrderPlaced event might drive the confirmation email, stock reservation and analytics independently.

Products mix the two. RabbitMQ routes messages through exchanges into queues, and Amazon SNS fans out to subscribers that are often SQS queues. Kafka keeps events in a log that each consumer group reads in full, while consumers within a group share its partitions.

Message names show intent. A command such as SendInvoiceEmail asks one handler to do something. An event such as InvoiceIssued states something that already happened, and the publisher doesn't know who listens.

Most brokers deliver at least once. If a consumer processes a message and crashes before acknowledging it, the broker delivers it again, so every consumer must expect duplicates.

Some systems offer exactly-once processing inside their own boundaries, such as Kafka transactions between topics. Side effects outside the broker, such as an email or a card charge, still need protection against running twice.

Producers need care too. Saving an order and then publishing an event are two separate writes, and a crash between them loses the event. The outbox pattern writes the event to a table in the same transaction as the order, and a separate process publishes it from there.

An idempotent consumer produces the same outcome however many times it sees a message. Common techniques:

Don't assume messages arrive in the order they were sent. Retries, redelivery and competing consumers all reorder them. Most brokers only keep order within a narrower scope, such as a Kafka partition or a message group in an Amazon SQS FIFO queue.

Use that scope deliberately. Route every message about one entity, such as one order ID, to the same partition or group. Order then holds where it matters, while unrelated orders process in parallel.

Make consumers tolerate the rest. Include a version or sequence number in each event, and ignore any event older than the state you already hold. That way an OrderPlaced event delivered after OrderCancelled doesn't bring a cancelled order back to life.

When a consumer fails, the message comes back for another attempt. Retry with exponential backoff and some random jitter, so a struggling downstream service isn't hit by every retry at once.

Separate transient failures from permanent ones. A timeout may succeed on the next try, but a malformed payload never will, and retrying it forever wastes capacity or blocks the queue. A message that fails every time is often called a poison message.

After a set number of attempts, move the message to a dead-letter queue (DLQ). Amazon SQS does this with a redrive policy and a maximum receive count, and RabbitMQ with a dead-letter exchange. Alert on DLQ depth and build a way to inspect, fix and replay messages, because a DLQ nobody reads is silent data loss.

Match processing timeouts to the work. In SQS, if the visibility timeout is shorter than the time a consumer needs, the message reappears and a second worker processes it in parallel.

A queue fits work that can happen later, where the caller doesn't need the result. When the user is waiting for the answer, such as a price quote or a sign-in, a direct call is simpler and easier to debug.

Events also make a flow harder to follow. The steps of one business process spread across consumers, so you need correlation IDs and distributed tracing to reconstruct what happened. Event schemas become contracts with consumers you may not know about.

For background jobs inside one application, a jobs table in your existing database is often enough. Workers can claim rows with SELECT ... FOR UPDATE SKIP LOCKED in PostgreSQL or MySQL 8. Creating a job can then share a transaction with the data change that caused it, which removes the two-write problem the outbox pattern solves.