Nova Resource:Clouddb-services/SAL

2022-05-17

• 14:12 taavi: enable gtid on toolsdb T301993

2022-02-17

• 12:03 taavi: add myself as a project member so I don't need to ssh in as root@

2021-09-02

• 18:52 bstorm: removed strange old duplicate cron for osmupdater T285668

2021-08-31

• 20:19 bstorm: attempting to resync OSMDB back to Feb 21st 2019 T285668

2021-08-30

• 22:07 bstorm: restarting osmdb on clouddb1003 to try to capture enough connections T285668
• 21:53 bstorm: disable puppet and osm updater script on clouddb1003 T285668

2021-05-26

• 16:53 bstorm: restarting postgresql since T220164 was closed. Hoping all connections don't get used up again.

2021-05-25

• 14:35 dcaro: taking down clouddb1002 replica for reboot of cloudvirt1020 (T275893)

2021-05-04

• 22:57 bstorm: manually added cnames for toolsdb, osmdb and wikilabelsdb in db.svc.wikimedia.cloud zone T278252

2021-04-05

• 09:56 arturo: make jhernandez (IRC joakino) projectadmin (T278975)

2021-03-10

• 12:44 arturo: briefly stopping VM clouddb-wikireplicas-proxy-2 to migrate hypervisor
• 10:57 arturo: briefly stopped VM clouddb-wikireplicas-proxy-1 to disable VMX cpu flag

2021-03-02

• 23:29 bstorm: bringing toolsdb back up 😟

2021-02-26

• 23:20 bstorm: rebooting clouddb-wikilabels-02 for patches
• 22:55 bstorm: rebooting clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 before (hopefully) many people are using them

2021-01-29

• 18:21 bstorm: deleting clouddb-toolsdb-03 as it isn't used

2020-12-23

• 19:20 bstorm: created clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 as well as the 16 neutron ports for wikireplicas proxying

2020-12-17

• 02:14 bstorm: toolsdb is back and so is the replica T266587
• 01:10 bstorm: the sync is done and we have a good copy of the toolsdb data, proceeding with the upgrades and stuff to that hypervisor while configuring replication to work again T266587

2020-12-16

• 18:34 bstorm: restarted sync from toolsdb to its replica server after cleanup to prevent disk filling T266587
• 17:31 bstorm: sync started from toolsdb to its replica server T266587
• 17:29 bstorm: stopped mariadb on the replica T266587
• 17:28 bstorm: shutdown toolsdb T266587
• 17:24 bstorm: settings toolsdb to readonly to prepare for shutdown T266587
• 17:06 bstorm: switching the secondary config back to clouddb1002 in order to minimize concerns about affecting ceph performance T266587

2020-11-15

• 19:45 bstorm: restarting the import to clouddb-toolsdb-03 with --max-allowed-packet=1G to rule out that as a problem entirely T266587
• 19:36 bstorm: set max_allowed_package to 64MB on clouddb-toolsdb-03 T266587

2020-11-11

• 08:20 bstorm: loading dump on the replica T266587
• 06:06 bstorm: dump completed, transferring to replica to start things up again T266587

2020-11-10

• 16:24 bstorm: set toolsdb to read-only T266587

2020-10-29

• 23:49 bstorm: launching clouddb-toolsdb-03 (trying to make a better naming convention) to replace failed replica T266587

2020-10-27

• 18:04 bstorm: set clouddb1002 to read-only since it was not supposed to be writeable anyway T266587

2020-10-20

• 18:36 bstorm: brought up mariadb and replication on clouddb1002 T263677
• 17:14 bstorm: shutting down clouddb1003 T263677
• 17:13 bstorm: stopping postgresql on clouddb1003 T263677
• 17:08 bstorm: poweroff clouddb1002 T263677
• 17:08 bstorm: stopping mariadb on clouddb1002 T263677
• 17:07 bstorm: shut down replication on clouddb1002 (now with task) T263677
• 17:05 bstorm: shut down replication on clouddb1002

2020-09-29

• 23:48 bstorm: restarting postgresql on clouddb1003 to clear the number of connections

2020-09-08

• 18:21 bstorm: copied the profile::mariadb::section_ports key into prefix puppet to fix puppet after the refactor for wmfmariadbpy

2020-07-16

• 17:56 bstorm: Significantly lifted traffic shaping limits on clouddb1001/toolsdb to improve network performance T257884

2020-06-09

• 22:55 bstorm_: Reset the passwords for T254931

2020-06-04

• 17:41 bd808: Restarting mariadb after hand applying {{gerrit|602433}} (T253738)

2020-06-01

• 18:51 bstorm_: restarting mariadb on clouddb1002 T253738

2020-05-29

• 23:42 bstorm_: stopped puppet and restarting mariadb on clouddb1002 after filtering out a table T253738

2020-03-27

• 12:26 arturo: add myself as project admin

2020-03-20

• 13:43 jeh: upgrade puppetmaster to v5 T241719

2020-03-18

• 17:49 bstorm_: updated the tools-prometheus security group with the current IP addresses for the prometheus servers

2019-10-25

• 10:49 arturo: (jynus) clouddb1002 mariadb (toolsdb secondary) being upgraded from 10.1.38 to 10.1.39 is done !(T236384 , T236420)
• 10:46 arturo: (jynus) clouddb1002 mariadb (toolsdb secondary) being upgraded from 10.1.38 to 10.1.39 (T236384 , T236420)
• 10:45 arturo: icinga downtime toolschecker for 1 to upgrade clouddb1002 mariadb (toolsdb secondary) (T236384 , T236420)
• 10:37 arturo: clouddb1002 downgrading wmf-mariadb101 from 10.1.41-1 to 10.1.39-1 (T236384 , T236420)
• 08:13 arturo: enable puppet in clouddb1001/clouddb1002 to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/546102 (T236384 , T236420)

2019-10-24

• 18:34 bstorm_: downgraded clouddb1001 to 10.1.39 T236420 T236384
• 18:19 bstorm_: stopped puppet on clouddb1001 and removed unattended-upgrades for now T236384
• 16:03 bstorm_: stopped puppet on clouddb1002 and removed unattended-upgrades for now T236384
• 13:27 phamhi: add phamhi as user and projectadmin
• 12:34 arturo: start both clouddb1001 and clouddb1004 (T227540)
• 11:15 arturo: poweroff VM clouddb1001, hypervisor will be powered off (T227540)
• 11:13 arturo: poweroff VM clouddb1004 (T227540)
• 11:09 phamhi: stopped postgresl in clouddb1004 (T227540)
• 11:04 arturo: stopped mariadb in clouddb1001 (T227540)

2019-09-30

• 17:15 bstorm_: restart wikilabels postgresql
• 17:15 bstorm_: restart osmdb on clouddb1003

2019-09-16

• 19:41 jeh: add openstack security group rules for Tools prometheus scrapes to mariadb exporter
• 19:28 jeh: add openstack security group rules for labmon prometheus scraps to mariadb exporter

2019-05-24

• 23:38 bstorm_: T224062 clouddb1002 is now free of postgresql services and the volume is reclaimed for toolsdb use
• 23:06 bstorm_: T224318 cloudb-wikilabels-02 is now a replica of clouddb-wikilabels-01!
• 21:00 bstorm_: T224062 Moved wikilabels postgres db to clouddb-wikilabels-01

2019-05-22

• 00:14 bstorm_: T224062 wikilabels postgres is now replicating to clouddb-wikilabels-01

2019-04-05

• 00:35 bstorm_: added the rsyncd ports to security groups for the osmdb cluster

2019-04-04

• 23:58 bstorm_: pg_basebackup finished on clouddb1004 and streaming replication is flowing
• 18:46 bstorm_: T219652 pg_basebackup started on clouddb1004
• 18:35 bstorm_: T219652 postgresql on clouddb1003 had to be restarted to get it to accept WAL connections
• 18:00 bstorm_: T219652 clouddb1003 is now the OSMdb primary

2019-03-28

• 23:17 bstorm_: clouddb1003 is now a full osmdb replica T193264
• 19:32 bstorm_: pg_basebackup started on clouddb1003 for osmdb T193264

2019-02-28

• 02:09 bstorm_: clouddb1002 is now the replica for clouddb1001/toolsdb and is catching up slowly

2019-02-18

• 19:01 bstorm_: labsdb1004 is now a replica of clouddb1001, which is toolsdb now.
• 18:26 arturo: (jaime T193264) setting clouddb1001 in read_write mode
• 18:12 arturo: T193264 pointing tools.db.svc.eqiad.wmflabs to clouddb1001
• 17:55 arturo: (jaime T193264) set clouddb1001 in read_only=1
• 16:11 andrewbogott: deleting clouddb-utils-01 VM and associated puppet prefix; we aren't going to run maintain_dbusers here after all
• 15:51 andrewbogott: removing dns and public IP from clouddb1001

2019-02-17

• 21:21 bstorm_: The slave of labsdb1005.eqiad.wmnet is now clouddb1001.clouddb-services.eqiad.wmflabs
• 19:16 arturo: T193264 delete VM clouddb-services-01
• 18:54 arturo: T193264 create VM clouddb-services-01 for PoC of running maintain-dbusers from here

2019-02-16

• 13:59 arturo: T193264 switched clouddb1001/1004 to the new project local puppetmaster
• 13:54 arturo: T193264 create 'clouddb10' puppet prefix to store puppet/hiera config for database servers in this project
• 13:47 arturo: T193264 create 'clouddb-services-puppetmaster' puppet prefix to store puppet/hiera config for this project puppetmaster
• 13:43 arturo: T193264 create 'clouddb-services-puppetmaster-01' instance
• 13:33 arturo: add myself as user and projectadmin

2019-02-15

• 23:42 bd808: Added BryanDavis (self), Arturo Borrero Gonzalez, Marostegui, and Jcrespo as admins in project
• 22:49 bstorm_: created mariadb security group and lvs for a new database T193264