Introduction to Message Brokers
What is a message broker?
āA message broker is an intermediary computer program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiverā
ā Wikipedia.org
Didn't understand much about message brokers from the definition ?š¤ . That's totally fineš. Read this article to the end, and come back to the definition, and read again.
Anyway, now you know what the term means letās look into what message broker is.
A message broker is an intermediary program that applications and services use to communicate with each other to exchange information. Message brokers can be used to validate, store, route and deliver messages to the required destinations. Not only applications can communicate information, even if they are implemented in different programming languages. Since message brokers act as an intermediary program, the sender has no idea how many recipients there are if they are online.
Most importantly brokers ensure that recipients receive the message even if they are not online or active. (Itās kinda like email. You donāt have to be online but you still receive the message).
Do you know: Message brokers are also known as integration brokers or interface engines.
So how do message brokers ensure the delivery of the messages?
Message brokers use a message queue for this and itās saved in memory or a hard disk. They are used to store messages and deliver them. (I hope you are familiar with the data structure āQueueā. This is the same as this, First In First Out).
Message queue stores messages in the exact order they are received and sends to the receiver in the same order. If somehow a message was unable to be delivered (problem with the network) the message will be rescheduled for later in the queue. Also in a message queue, messages are stored in the exact order in which they were transmitted and remain in the queue until receipt is confirmed.
With message queues applications can work asynchronously it prevents data loss and enables systems to continue to function if processes or connections fail. This allows developers to keep processes and applications separate,
The message queue is a data structure and is a part of the Message broker.
Components of a message broker
Now letās look into the basic components of a message broker:
Producer āThis component is responsible for sending messages. Itās connected to the message broker. In publish/subscribe pattern (We have discussed in the below section) they are called publishers.
Consumer ā This component consumes messages in the message broker. In publish/subscribe pattern they are called subscribers.
Queue/topic ā Message broker store messages here.
I know there are a lot of things we can speak about when it comes to message brokers, but nothing beats a real-world example to help us understand them better.
Real-world use cases of message brokers
Integrating microservices
When microservice architecture is used to build a system, there could be multiple microservices at the end of development. The problem is communication with each other.
A message broker can be used as a central broker and microservices can communicate with each other. Also, if you need to add another service to this you only have to connect it with the message broker. Easy right?
Rest APIās
At present often we implement web frontends or mobile application with a REST API backend. Client and Server architecture uses HTTP protocol for communication (Request and response).
But what if the data from the request needs to be processed for a long time. Like 10 minutes. What if a time-consuming calculation has to be done?
Also, what if the user had a connection issue during that time. The response might never receive the user. A message broker can be used here so that delivery of the message can be guaranteed.
Mobile Applications
Think that you want to send notifications to a mobile app. But what happens if some users are not online. With the use of a message broker, you can ensure they receive the message when they come online.
Handling data of IoT (Internet of Things) devices
IoT devices generate a lot of data, and there could be thousands of devices in a system. Even though you can use the HTTP protocol to handle these data it's not the best practice and could not be the best solution due to a large amount of data and the rate they are generated.
(During my last year of study, I tried to use HTTP for an IoT device that generated over 300 records per second, but I ended up using the MQTT message broker instead.)
Website activity tracking
Imagine that you have deployed a web application and you want to know about user interactions of the web app. In a situation like this web app activities (button clicks, page views, searches etc.) can be published to central topics. Topics can be used per activity and the data can be used for real-time processing, real-time monitoring etc.
Summary: Using a message broker is an optimal solution where when there are time and resource consuming tasks and when the response of a request is not required immediately.
Different messaging models/distribution patterns
Even though there are several message broker models we will discuss the main 02 messaging patterns here.
- Point-to-Point.
- Publish-Subscribe.
Point-to-Point.
- message queues have a one-to-one relationship with the messageās sender (Producer) and receiver (Consumer).
- Each message in the queue is only sent to one recipient and is received only once.
- Optimal for financial transaction processing since payment needs to be done only once
Publish-Subscribe.
- This message model is also known as āpub/subā.
- In this model, message queues have a one-to-many relationship with the message sender and receiver. Also, the sender is known as the publisher and the receiver is known as the subscribe.
- According to this model, a sender publishes messages on a topic. These messages are distributed among all the consumers that have subscribed to the topic.
- This model can be used for a system where data should be shared among several parties constantly.
- Ex: An airline system should continuously share updates about its landing time and location. These updates should be shared among several parties such as ground crews (to prepare for landing), the control centre (to manage flights), the public etc.
What are the advantages of using message brokers?
Message delivery is guaranteed ā Even though the consumer was not active or offline the message will deliver when active.
System performance is increased with asynchronous processing ā Processes that take a long time can be processed separately and therefore the main thread of the application will not be affected.
Applications to communicate with each other, even if they are implemented in different programming languages ā (Ex: SOAP, REST API, IoT devices etc).
What are some drawbacks of using message brokers?
Learning curve ā There are several messaging brokers and design patterns you can use them with. You need to know the differences and which one to use. Not to mention the configuration process of setting up message brokers.
Debugging ā With system complexity, increased debugging can be hard.
Complexity ā Having a message broker definitely adds a whole new component to the system architecture.
Most popular message brokers
Some of the most famous message brokers are Apache Kafka, RabbitMQ and Amazon SNS.
Kafka ā Kafka was created by LinkedIn in 2011. Since its capable of working with a wide range of systems like web and desktop applications, Microservices, Monitoring and analytical systems, NoSQL, Oracle and Hadoop it has been popular in the software engineering world.
ActiveMQ ā Active MQ (Stands for Active Message Queuing) was developed by Apache Software Foundation in 2004.
ActiveMQ and Kafka, both are Apache and both written in Java
RabbitMQ ā This was message broker was originally developed by ā Rabbit Technologies Ltdā and since 2013 its been under āPivotal Softwareā. Unlike both above this is written with Erlang.
I hope now you have an idea about what are message brokers and their use in software development. In our upcoming article let's discuss more about message brokers.
References
