Portal:Cloud VPS/Admin/Rabbitmq

Many OpenStack services communicate with one another via rabbitmq. For example:

- Nova services relay messages to nova-conductor via rabbitmq, and nova-conductor marshals database reads and writes - When an instance is created via nova-api, nova-api passes a rabbitmq message to nova-scheduler, who then schedules a VM (again via a rabbit message) on a nova-compute node - nova-scheduler assesses capacity of compute nodes via rabbitmq messages - designate-sink subscribes to rabbitmq notifications in order to detect and respond to VM creation/deletion - etc.

When VM creation is failing, very often the issue is with rabbitmq. Typically rabbit can be restarted with minimal harm, which will prompt all clients to reconnect:

root@cloudcontrol1003:~# service rabbitmq-server restart
root@cloudcontrol1004:~# service rabbitmq-server restart

HA setup

For redundancy we use a cluster of two rabbitmq servers in a primary/secondary relationship. Some documentation about how this is set up can be found [| here]. Most of the pieces of this are puppetized, but when standing up a new pair a couple of manual steps are needed.

On the secondary host (where the primary host is cloudcontrol1003):

 root@cloudcontrol1004:~# rabbitmqctl stop_app
 root@cloudcontrol1004:~# rabbitmqctl join_cluster --ram rabbit@cloudcontrol1003
 root@cloudcontrol1004:~# rabbitmqctl start_app
 root@cloudcontrol1004:~# rabbitmqctl set_policy ha-all '^(?!amq\.).*' '{"ha-mode": "all"}'