RabbitMQ Shovel
“Such a shovel, it seemed a waste not to use it.”
― Daniel Kraus, Rotters
In the message broker RabbitMQ, a typical setup is to deploy a single RMQ instance that resides in a box. The job of RMQ is to deal with incoming messages and make sure these are forwarded to the correct destination.
The publisher sends the messages to an exchange which are the message routing agents. The exchange routes the messages based on routing keys to different queues. Each queue has consumers attached who will consume the messages and process them.
Now, what if we need a queue-based system where we have multiple queues in a single broker, and each RMQ instance can send and receive messages to one another. A second use case could be offloading messages from one queue to another in a different RMQ broker. That's where Shovel comes into the play.
R️abbitMQ Shovel continually moves messages from a source to a destination. The source and destination can be situated in the same cluster with different virtual hosts or in different geographic and administrative domains. It can move messages between queues and exchanges within the same broker or it can act as a bridge between two different brokers.
A shovel can be used to balance the load of a queue or when we need to take messages out of one RabbitMQ broker and insert them into another. In essence, the shovel connects to the source and destination broker, consumes messages from the queue, and re-publishes each message to the destination broker.
Why do we need Shovel
- Loose coupling
A shovel can move messages between different RMQ clusters that may have different users, RMQ version, vhosts, and messaging protocols (AMQP 0.9.1 and AMQP 1.0). - WAN Friendly
Shovel uses AMQP client connection under the hood. It is designed to tolerate intermittent connectivity without message loss by performing Consumer Acknowledgements. - Configurable parameters
Shovels are highly tailorable and can be configured to perform any number of explicit methods.
Types of Shovels
There are two types of shovels- Static shovel and Dynamic shovel.
Static Shovel is defined in the configuration file and the node restart is required to apply any update. This is more general in nature as queues, exchanges, and bindings need to be declared manually at startup. If multiple RMQ nodes are present in the cluster, then a static shovel should be defined for all the nodes in the config file. Each shovel will only start on one node, and if the node goes down, it will move to another healthy node.
Dynamic Shovels are defined in runtime parameters. They can be created and deleted at any time and don't require a node restart. These are more automation-friendly as the queues and exchanges used by shovel are declared automatically. In a dynamic shovel, we don't need to manually define it for all the nodes. This is done automatically and each shovel will only start on one node and failover to another when needed.
Set up a shovel
Enable Shovel plugin
Installing a shovel is plain sailing. This can be done by using RMQ CLI.
rabbitmq-plugins enable rabbitmq_shovel
rabbitmq-plugins enable rabbitmq_shovel_managementShovel mapping
Shovel mapping can be created from the RMQ Management dashboard. Go to the Admin section and click on Shovel Management.
Syntax of the RMQ URI is amqp://username:password@host:port/vhost
Shovel params:
- Virtual Host: Select vhost from the dropdown.
- Name of the shovel.
- Source: Enter the RMQ URI and source queue/exchange.
- Destination: RMQ URI of destination instance and queue/exchange.
Once all the details are in place, click on Add shovel button.
The status of the shovel can be visible from the Admin > Shovel Status tab. Status value can be:
- Starting: Shovel is starting and creating resources and trying to connect to endpoints.
- Running: Shovel is successfully connected to both endpoints and consuming from source and republishing to destination.
- Terminating: Shovel has terminated or it ran into some exception. The reason is visible in the logs.
