Message Broker

Hi everyone 👋. I'm Hung Anh.

In an era where applications must communicate constantly and process massive amounts of data in real time, how do we keep our systems running smoothly? Have you ever wondered what's really happening behind the scenes of the services we use every day, like payment systems, social networks, or delivery apps?

One of the key answers is the Message Broker - the 'invisible connector' that ensures information is delivered safely, quickly, and reliably between systems.

Today, let's explore what a Message Broker is, why it matters, and what its strengths, weaknesses, and best applications are in building distributed systems.

Let's get started.

1. The Request - Response Model

First, let's take a look at the characteristics of the traditional request - response communication model.

Consider an example where Service A sends a request to Service B, then waits for Service B to finish processing and return a response to Service A. This style of communication is synchronous. Because Service A has to wait for Service B's response before it can carry on with subsequent work, we say that Service A is coupled to Service B.

2. What is a Message Broker?

A Message Broker introduces a new way of communicating that overcomes the limitations of the request - response model. It is a piece of middleware that receives messages from sending services (producers) and routes them to the appropriate target services (consumers). With a Message Broker, services no longer need to communicate directly with one another; instead, they send and receive messages through the broker, which reduces the interdependency between services.

Here, we call Service A the Producer because its role is to send messages to the Message Broker. Services B, C, and D are called Consumers because their role is to receive or pull messages from the Message Broker.

Next, let's explore some of the advantages of the Message Broker by working through a few concrete problems.

3. Advantages of a Message Broker

3.1. Asynchronous Communication

Consider the following problem: Service A needs to send a message to Services B, C, and D.

With the traditional request - response model, Service A has to send three separate requests to Services B, C, and D.

There are two ways to do this:

• Send synchronously.
• Send asynchronously.

With the synchronous approach, our flow looks like this:

Service A sends a message to Service B, then waits for Service B to respond. Only after receiving the response from Service B does Service A move on to send a message to Service C, and so on.

From this approach, we can identify a few pros and cons:

• Pros: Easy to implement, and the order of responses from the services is guaranteed.
• Cons: Service A is "blocked" while waiting for responses from the other services, so it doesn't make full use of Service A's resources.

With the asynchronous approach, our flow looks like this:

Service A sends messages to Services B and C at the same time without waiting for their responses. Once Services B and C finish processing, they respond back to Service A.

Let's look at the pros and cons of this approach:

• Pros: Service A doesn't have to wait for responses from the other services, making full use of Service A's resources.
• Cons: The order of responses from the receiving services may not match the order they were sent (Service B's response might arrive after Service C's).

With a Message Broker, Service A only needs to publish the message once to the Broker, after which the Broker pushes these messages to the corresponding Services B, C, and D, or Services B, C, and D pull the messages themselves when needed. As a result, Services A, B, C, and D communicate fully asynchronously with one another.

3.2. Decoupling

Let's keep working with the same problem: Service A needs to send a message to Services B, C, and D.

With the request - response model, Service A needs to know the exact addresses of Services B, C, and D in order to send messages. If sending a message fails, it can affect other logic within Service A.

With a Message Broker, Service A (acting as the producer) only needs to know the address of the Message Broker, and Services B, C, and D (acting as Consumers) likewise only need to know the address of the Message Broker.

The producer and consumers don't need to be aware of each other's existence, and if one consumer goes down, it doesn't affect the producer or the other consumers. In other words, Services A, B, C, and D are independent of one another.

3.3. Scalability

Let's stay with the same problem, but now suppose Service A also needs to send messages to a new Service E to support some new features.

With the request - response model, Service A would need to create a new connection to Service E. This can cause downtime for Service A.

With a Message Broker, we can easily add a Consumer E to receive messages from Producer A without affecting the other services. In other words, adding or removing services in the system becomes incredibly easy.

3.4. Reliability

Consider this problem: suppose we need to collect behavioral events from users interacting with IoT devices and store them in a database. But the volume of user events is enormous, so the traffic hitting the database is huge, which can bring the DB down.

Solution: Here we can introduce a Message Broker that acts as a Message Queue to throttle the traffic flowing into the DB. A Message Queue works like a queue of requests: requests are stored in the queue and gradually pulled out to be processed and written to the DB. This makes the system more stable and more robust.

4. Conclusion

A Message Broker is a piece of middleware that receives messages from sending services (producers) and routes them to the appropriate target services (consumers). A Message Broker has many strong advantages, but it also comes with a few drawbacks we need to keep in mind:

• Advantages:
  • Asynchronous Communication
  • Decoupling
  • Scalability
  • Reliability
  • ...
• Drawbacks
  • Complexity
  • High cost
  • Increased latency, since messages have to pass through an additional intermediary component
  • ...

Thank you for sticking with me to the end of this article! I hope this post has helped you better understand the Message Broker along with its pros and cons, so you can apply it in real-world projects. If you have any questions, opinions, or anything you'd like to share, feel free to leave a comment below.

I wish you many enjoyable learning experiences, and I'll see you in upcoming articles, where we'll continue exploring plenty more interesting topics.

Happy reading! 🍻

References

1. Message Broker - The Backbone of Large-Scale Systems | Ronin Engineer
2. Message Brokers: An Introduction | Confluent