You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org
Portal:Cloud VPS/Admin/Rabbitmq
Many OpenStack services communicate with one another via rabbitmq. For example:
- Nova services relay messages to nova-conductor via rabbitmq, and nova-conductor marshals database reads and writes - When an instance is created via nova-api, nova-api passes a rabbitmq message to nova-scheduler, who then schedules a VM (again via a rabbit message) on a nova-compute node - nova-scheduler assesses capacity of compute nodes via rabbitmq messages - designate-sink subscribes to rabbitmq notifications in order to detect and respond to VM creation/deletion - etc.
When VM creation is failing, very often the issue is with rabbitmq. Typically rabbit can be restarted with minimal harm, which will prompt all clients to reconnect:
root@cloudcontrol1003:~# service rabbitmq-server restart root@cloudcontrol1004:~# service rabbitmq-server restart
HA setup
For redundancy we use a cluster of two rabbitmq servers in a primary/secondary relationship. Some documentation about how this is set up can be found [| here]. Most of the pieces of this are puppetized, but when standing up a new pair a couple of manual steps are needed.
On the secondary host (where the primary host is cloudcontrol1003):
root@cloudcontrol1004:~# rabbitmqctl stop_app root@cloudcontrol1004:~# rabbitmqctl join_cluster --ram rabbit@cloudcontrol1003 root@cloudcontrol1004:~# rabbitmqctl start_app root@cloudcontrol1004:~# rabbitmqctl set_policy ha-all '^(?!amq\.).*' '{"ha-mode": "all"}'