Message queues provide a means of fast asynchronous communication between software components. They come handy in solving some common problems that arise while building software, e.g.
At Better.com, we tried Beanstalk and Amazon SWF and realized that they weren't a fit for our stack. For example, Beanstalk doesn't have message persistence or authentication. SWF doesn't scale well at high frequencies and it is built for complex asynchronous multi-step workflows. So it isn't the right product for high frequency simple async communications.
A Message queue simply enqueues and dequeues messages
A Message broker can perform additional operations on messages before dequeueing them. You can find more information regarding these transforms here
Our criteria for a message broker were thus:
Must have: Delivery retry, CI/CD friendly, Fast, High Availability Nice to have: Admin console, PubSub
Here are the contenders
ActiveMQ
: Open source Apache project written in Java. 15+ years old and very mature.Amazon SQS
: AWS product for message queueing.RabbitMQ
: Open source message broker written in Erlang.IronMQ
: Closed source message broker.Redis
: In memory cache that can be used as a queue using redis operations.Pros:
Cons:
Pros:
Cons:
Pros:
link
Cons:
Pros:
Cons:
None of the queue implementations check all the boxes. We went with ActiveMQ since it was the best fit for our stack and traffic volume. in production, ActiveMQ on AWS was very easy to setup. The ability to spin up ActiveMQ instances in Docker ties in neatly into our CI/CD pipelines and lets us run proper integration tests without the need for mocking. It was significant work to implement and test client libraries in Javascript and Python, but this was a good tradeoff for the ease of use and speed provided by ActiveMQ.
We've been running ActiveMQ for over a year in production. It's proven to be lightweight, fast and requires low maintenence. On the other hand, writing client libraries in Javascript and Python has not been as easy as we thought. It required a lot of effort to write these libraries and get them production ready. If this is not something you are up for, RabbitMQ might be a very good alternative.
Tags
ActiveMQ
Redis
If you have any questions on best message queue. We will give the professional answers to your questions.