Switch Datacenter/db read-only

List of current masters excluding parsercache ones, per datacenter:

$ cat eqiad-nopc-masters.txt
db1075.eqiad.wmnet
db1031.eqiad.wmnet
db1057.eqiad.wmnet
db1018.eqiad.wmnet
db1040.eqiad.wmnet
db1041.eqiad.wmnet
es1015.eqiad.wmnet
es1019.eqiad.wmnet
db1049.eqiad.wmnet
db1050.eqiad.wmnet

$ cat codfw-nopc-masters.txt
db2016.codfw.wmnet
db2029.codfw.wmnet
db2019.codfw.wmnet
db2023.codfw.wmnet
db2018.codfw.wmnet
db2028.codfw.wmnet
db2017.codfw.wmnet
db2033.codfw.wmnet
es2018.codfw.wmnet
es2015.codfw.wmnet

• Check the current state of the masters (e.g. for codfw):

while read master; do salt $master cmd.run "mysql --defaults-file=/root/.my.cnf -A -BN -e \"SELECT @@global.read_only\""; done < codfw-nopc-masters.txt

• Set eqiad masters as read-only:

while read master; do salt $master cmd.run "mysql --defaults-file=/root/.my.cnf -A -BN -e \"SET GLOBAL read_only = 1\""; done < eqiad-nopc-masters.txt

• Set codfw masters as read-write:

while read master; do salt $master cmd.run "mysql --defaults-file=/root/.my.cnf -A -BN -e \"SET GLOBAL read_only = 0\""; done < codfw-nopc-masters.txt

The above commands will be greatly simplified when older servers will be decommissioned (which have a different and heterogeneous puppet class, and different mysql grants), so that we can just use salt or mysql to send the commands. Until then, explicitly indicate the masters.