Switch Datacenter/db read-only

List of current masters, per datacenter:

root@neodymium:~$ cat eqiad-masters.txt
db1052.eqiad.wmnet
db1018.eqiad.wmnet
db1038.eqiad.wmnet
db1040.eqiad.wmnet
db1058.eqiad.wmnet
db1023.eqiad.wmnet
db1033.eqiad.wmnet
es1015.eqiad.wmnet
es1019.eqiad.wmnet
db1029.eqiad.wmnet
pc1004.eqiad.wmnet
pc1005.eqiad.wmnet
pc1006.eqiad.wmnet

root@neodymium:~$ cat codfw-masters.txt
db2016.codfw.wmnet
db2017.codfw.wmnet
db2018.codfw.wmnet
db2019.codfw.wmnet
db2023.codfw.wmnet
db2028.codfw.wmnet
db2029.codfw.wmnet
es2015.codfw.wmnet
es2018.codfw.wmnet
db2009.codfw.wmnet
pc2004.codfw.wmnet
pc2005.codfw.wmnet
pc2006.codfw.wmnet

• Check the current state of the masters (e.g. for codfw):

while read master; do salt $master cmd.run "mysql --defaults-file=/root/.my.cnf -A -BN -e \"SELECT @@global.read_only\""; done < codfw-masters.txt

• Set eqiad masters as read-only:

while read master; do salt $master cmd.run "mysql --defaults-file=/root/.my.cnf -A -BN -e \"SET GLOBAL read_only = 1\""; done < eqiad-masters.txt

• Set codfw masters as read-write:

while read master; do salt $master cmd.run "mysql --defaults-file=/root/.my.cnf -A -BN -e \"SET GLOBAL read_only = 0\""; done < codfw-masters.txt

The above commands will be greatly simplified when older servers will be decommissioned (which have a different and heterogeneous puppet class, and different mysql grants), so that we can just use salt or mysql to send the commands. Until then, explicitly indicate the masters.