You are browsing a read-only backup copy of Wikitech. The live site can be found at

Nova Resource:Clouddb-services/SAL

From Wikitech-static
Jump to navigation Jump to search


  • 05:56 taavi: toolsdb: add s54518__mw to list of replication ignored databases, data mismatch between primary and replica


  • 16:30 taavi: stopped mariadb on clouddb1002, starting rsync clouddb1002->tools-db-1 on a root screen session on tools-db-1 T301949


  • 14:12 taavi: enable gtid on toolsdb T301993


  • 12:03 taavi: add myself as a project member so I don't need to ssh in as root@


  • 18:52 bstorm: removed strange old duplicate cron for osmupdater T285668


  • 20:19 bstorm: attempting to resync OSMDB back to Feb 21st 2019 T285668


  • 22:07 bstorm: restarting osmdb on clouddb1003 to try to capture enough connections T285668
  • 21:53 bstorm: disable puppet and osm updater script on clouddb1003 T285668


  • 16:53 bstorm: restarting postgresql since T220164 was closed. Hoping all connections don't get used up again.


  • 14:35 dcaro: taking down clouddb1002 replica for reboot of cloudvirt1020 (T275893)


  • 22:57 bstorm: manually added cnames for toolsdb, osmdb and wikilabelsdb in zone T278252


  • 09:56 arturo: make jhernandez (IRC joakino) projectadmin (T278975)


  • 12:44 arturo: briefly stopping VM clouddb-wikireplicas-proxy-2 to migrate hypervisor
  • 10:57 arturo: briefly stopped VM clouddb-wikireplicas-proxy-1 to disable VMX cpu flag


  • 23:29 bstorm: bringing toolsdb back up 😟


  • 23:20 bstorm: rebooting clouddb-wikilabels-02 for patches
  • 22:55 bstorm: rebooting clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 before (hopefully) many people are using them


  • 18:21 bstorm: deleting clouddb-toolsdb-03 as it isn't used


  • 19:20 bstorm: created clouddb-wikireplicas-proxy-1 and clouddb-wikireplicas-proxy-2 as well as the 16 neutron ports for wikireplicas proxying


  • 02:14 bstorm: toolsdb is back and so is the replica T266587
  • 01:10 bstorm: the sync is done and we have a good copy of the toolsdb data, proceeding with the upgrades and stuff to that hypervisor while configuring replication to work again T266587


  • 18:34 bstorm: restarted sync from toolsdb to its replica server after cleanup to prevent disk filling T266587
  • 17:31 bstorm: sync started from toolsdb to its replica server T266587
  • 17:29 bstorm: stopped mariadb on the replica T266587
  • 17:28 bstorm: shutdown toolsdb T266587
  • 17:24 bstorm: settings toolsdb to readonly to prepare for shutdown T266587
  • 17:06 bstorm: switching the secondary config back to clouddb1002 in order to minimize concerns about affecting ceph performance T266587


  • 19:45 bstorm: restarting the import to clouddb-toolsdb-03 with --max-allowed-packet=1G to rule out that as a problem entirely T266587
  • 19:36 bstorm: set max_allowed_package to 64MB on clouddb-toolsdb-03 T266587


  • 08:20 bstorm: loading dump on the replica T266587
  • 06:06 bstorm: dump completed, transferring to replica to start things up again T266587


  • 16:24 bstorm: set toolsdb to read-only T266587


  • 23:49 bstorm: launching clouddb-toolsdb-03 (trying to make a better naming convention) to replace failed replica T266587


  • 18:04 bstorm: set clouddb1002 to read-only since it was not supposed to be writeable anyway T266587


  • 18:36 bstorm: brought up mariadb and replication on clouddb1002 T263677
  • 17:14 bstorm: shutting down clouddb1003 T263677
  • 17:13 bstorm: stopping postgresql on clouddb1003 T263677
  • 17:08 bstorm: poweroff clouddb1002 T263677
  • 17:08 bstorm: stopping mariadb on clouddb1002 T263677
  • 17:07 bstorm: shut down replication on clouddb1002 (now with task) T263677
  • 17:05 bstorm: shut down replication on clouddb1002


  • 23:48 bstorm: restarting postgresql on clouddb1003 to clear the number of connections


  • 18:21 bstorm: copied the profile::mariadb::section_ports key into prefix puppet to fix puppet after the refactor for wmfmariadbpy


  • 17:56 bstorm: Significantly lifted traffic shaping limits on clouddb1001/toolsdb to improve network performance T257884


  • 22:55 bstorm_: Reset the passwords for T254931


  • 17:41 bd808: Restarting mariadb after hand applying {{gerrit|602433}} (T253738)


  • 18:51 bstorm_: restarting mariadb on clouddb1002 T253738


  • 23:42 bstorm_: stopped puppet and restarting mariadb on clouddb1002 after filtering out a table T253738


  • 12:26 arturo: add myself as project admin


  • 13:43 jeh: upgrade puppetmaster to v5 T241719


  • 17:49 bstorm_: updated the tools-prometheus security group with the current IP addresses for the prometheus servers



  • 18:34 bstorm_: downgraded clouddb1001 to 10.1.39 T236420 T236384
  • 18:19 bstorm_: stopped puppet on clouddb1001 and removed unattended-upgrades for now T236384
  • 16:03 bstorm_: stopped puppet on clouddb1002 and removed unattended-upgrades for now T236384
  • 13:27 phamhi: add phamhi as user and projectadmin
  • 12:34 arturo: start both clouddb1001 and clouddb1004 (T227540)
  • 11:15 arturo: poweroff VM clouddb1001, hypervisor will be powered off (T227540)
  • 11:13 arturo: poweroff VM clouddb1004 (T227540)
  • 11:09 phamhi: stopped postgresl in clouddb1004 (T227540)
  • 11:04 arturo: stopped mariadb in clouddb1001 (T227540)


  • 17:15 bstorm_: restart wikilabels postgresql
  • 17:15 bstorm_: restart osmdb on clouddb1003


  • 19:41 jeh: add openstack security group rules for Tools prometheus scrapes to mariadb exporter
  • 19:28 jeh: add openstack security group rules for labmon prometheus scraps to mariadb exporter


  • 23:38 bstorm_: T224062 clouddb1002 is now free of postgresql services and the volume is reclaimed for toolsdb use
  • 23:06 bstorm_: T224318 cloudb-wikilabels-02 is now a replica of clouddb-wikilabels-01!
  • 21:00 bstorm_: T224062 Moved wikilabels postgres db to clouddb-wikilabels-01


  • 00:14 bstorm_: T224062 wikilabels postgres is now replicating to clouddb-wikilabels-01


  • 00:35 bstorm_: added the rsyncd ports to security groups for the osmdb cluster


  • 23:58 bstorm_: pg_basebackup finished on clouddb1004 and streaming replication is flowing
  • 18:46 bstorm_: T219652 pg_basebackup started on clouddb1004
  • 18:35 bstorm_: T219652 postgresql on clouddb1003 had to be restarted to get it to accept WAL connections
  • 18:00 bstorm_: T219652 clouddb1003 is now the OSMdb primary


  • 23:17 bstorm_: clouddb1003 is now a full osmdb replica T193264
  • 19:32 bstorm_: pg_basebackup started on clouddb1003 for osmdb T193264


  • 02:09 bstorm_: clouddb1002 is now the replica for clouddb1001/toolsdb and is catching up slowly


  • 19:01 bstorm_: labsdb1004 is now a replica of clouddb1001, which is toolsdb now.
  • 18:26 arturo: (jaime T193264) setting clouddb1001 in read_write mode
  • 18:12 arturo: T193264 pointing tools.db.svc.eqiad.wmflabs to clouddb1001
  • 17:55 arturo: (jaime T193264) set clouddb1001 in read_only=1
  • 16:11 andrewbogott: deleting clouddb-utils-01 VM and associated puppet prefix; we aren't going to run maintain_dbusers here after all
  • 15:51 andrewbogott: removing dns and public IP from clouddb1001


  • 21:21 bstorm_: The slave of labsdb1005.eqiad.wmnet is now clouddb1001.clouddb-services.eqiad.wmflabs
  • 19:16 arturo: T193264 delete VM clouddb-services-01
  • 18:54 arturo: T193264 create VM clouddb-services-01 for PoC of running maintain-dbusers from here


  • 13:59 arturo: T193264 switched clouddb1001/1004 to the new project local puppetmaster
  • 13:54 arturo: T193264 create 'clouddb10' puppet prefix to store puppet/hiera config for database servers in this project
  • 13:47 arturo: T193264 create 'clouddb-services-puppetmaster' puppet prefix to store puppet/hiera config for this project puppetmaster
  • 13:43 arturo: T193264 create 'clouddb-services-puppetmaster-01' instance
  • 13:33 arturo: add myself as user and projectadmin


  • 23:42 bd808: Added BryanDavis (self), Arturo Borrero Gonzalez, Marostegui, and Jcrespo as admins in project
  • 22:49 bstorm_: created mariadb security group and lvs for a new database T193264