Introduction to Message Brokers

Source: https://commons.wikimedia.org/

What is a message broker?

“A message broker is an intermediary computer program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver”

— Wikipedia.org

Anyway, now you know what the term means let’s look into what message broker is.

A message broker is an intermediary program that applications and services use to communicate with each other to exchange information. Message brokers can be used to validate, store, route and deliver messages to the required destinations. Not only applications can communicate information, even if they are implemented in different programming languages. Since message brokers act as an intermediary program, the sender has no idea how many recipients there are if they are online.

Most importantly brokers ensure that recipients receive the message even if they are not online or active. (It’s kinda like email. You don’t have to be online but you still receive the message).

Do you know: Message brokers are also known as integration brokers or interface engines.

So how do message brokers ensure the delivery of the messages?

Message queue (Source: https://www.cloudamqp.com/)

Message brokers use a message queue for this and it’s saved in memory or a hard disk. They are used to store messages and deliver them. (I hope you are familiar with the data structure “Queue”. This is the same as this, First In First Out).

Message queue stores messages in the exact order they are received and sends to the receiver in the same order. If somehow a message was unable to be delivered (problem with the network) the message will be rescheduled for later in the queue. Also in a message queue, messages are stored in the exact order in which they were transmitted and remain in the queue until receipt is confirmed.

With message queues applications can work asynchronously it prevents data loss and enables systems to continue to function if processes or connections fail. This allows developers to keep processes and applications separate,

The message queue is a data structure and is a part of the Message broker.

Components of a message broker

Now let’s look into the basic components of a message broker:

Producer –This component is responsible for sending messages. It’s connected to the message broker. In publish/subscribe pattern (We have discussed in the below section) they are called publishers.

Consumer — This component consumes messages in the message broker. In publish/subscribe pattern they are called subscribers.

Queue/topic — Message broker store messages here.

I know there are a lot of things we can speak about when it comes to message brokers, but nothing beats a real-world example to help us understand them better.

Real-world use cases of message brokers

Integrating microservices

When microservice architecture is used to build a system, there could be multiple microservices at the end of development. The problem is communication with each other.

A message broker can be used as a central broker and microservices can communicate with each other. Also, if you need to add another service to this you only have to connect it with the message broker. Easy right?

Integrating microservices with a message broker

Rest API’s

At present often we implement web frontends or mobile application with a REST API backend. Client and Server architecture uses HTTP protocol for communication (Request and response).

But what if the data from the request needs to be processed for a long time. Like 10 minutes. What if a time-consuming calculation has to be done?

Also, what if the user had a connection issue during that time. The response might never receive the user. A message broker can be used here so that delivery of the message can be guaranteed.

Mobile Applications

Think that you want to send notifications to a mobile app. But what happens if some users are not online. With the use of a message broker, you can ensure they receive the message when they come online.

Using message broker for Mobile Applications (Source:https://www.petermorlion.com/)

Handling data of IoT (Internet of Things) devices

IoT devices generate a lot of data, and there could be thousands of devices in a system. Even though you can use the HTTP protocol to handle these data it's not the best practice and could not be the best solution due to a large amount of data and the rate they are generated.

(During my last year of study, I tried to use HTTP for an IoT device that generated over 300 records per second, but I ended up using the MQTT message broker instead.)

Website activity tracking

Imagine that you have deployed a web application and you want to know about user interactions of the web app. In a situation like this web app activities (button clicks, page views, searches etc.) can be published to central topics. Topics can be used per activity and the data can be used for real-time processing, real-time monitoring etc.

Summary: Using a message broker is an optimal solution where when there are time and resource consuming tasks and when the response of a request is not required immediately.

Different messaging models/distribution patterns

Even though there are several message broker models we will discuss the main 02 messaging patterns here.

  1. Point-to-Point.
  2. Publish-Subscribe.

Point-to-Point.

point-to-point messaging (Source: https://tsh.io/)
  • message queues have a one-to-one relationship with the message’s sender (Producer) and receiver (Consumer).
  • Each message in the queue is only sent to one recipient and is received only once.
  • Optimal for financial transaction processing since payment needs to be done only once

Publish-Subscribe.

Pub sub-model (Source: https://tsh.io/)
  • This message model is also known as “pub/sub”.
  • In this model, message queues have a one-to-many relationship with the message sender and receiver. Also, the sender is known as the publisher and the receiver is known as the subscribe.
  • According to this model, a sender publishes messages on a topic. These messages are distributed among all the consumers that have subscribed to the topic.
  • This model can be used for a system where data should be shared among several parties constantly.
  • Ex: An airline system should continuously share updates about its landing time and location. These updates should be shared among several parties such as ground crews (to prepare for landing), the control centre (to manage flights), the public etc.

What are the advantages of using message brokers?

Message delivery is guaranteed — Even though the consumer was not active or offline the message will deliver when active.

System performance is increased with asynchronous processing — Processes that take a long time can be processed separately and therefore the main thread of the application will not be affected.

Applications to communicate with each other, even if they are implemented in different programming languages — (Ex: SOAP, REST API, IoT devices etc).

What are some drawbacks of using message brokers?

Learning curve — There are several messaging brokers and design patterns you can use them with. You need to know the differences and which one to use. Not to mention the configuration process of setting up message brokers.

Debugging — With system complexity, increased debugging can be hard.

Complexity — Having a message broker definitely adds a whole new component to the system architecture.

Most popular message brokers

Some of the most famous message brokers are Apache Kafka, RabbitMQ and Amazon SNS.

Kafka — Kafka was created by LinkedIn in 2011. Since its capable of working with a wide range of systems like web and desktop applications, Microservices, Monitoring and analytical systems, NoSQL, Oracle and Hadoop it has been popular in the software engineering world.

ActiveMQ — Active MQ (Stands for Active Message Queuing) was developed by Apache Software Foundation in 2004.

ActiveMQ and Kafka, both are Apache and both written in Java

RabbitMQ — This was message broker was originally developed by “ Rabbit Technologies Ltd” and since 2013 its been under “Pivotal Software”. Unlike both above this is written with Erlang.

I hope now you have an idea about what are message brokers and their use in software development. In our upcoming article let's discuss more about message brokers.

References

Associate Software Engineer at Virtusa