You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Revision history of "Nova Resource:Admin/SAL"

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 250 | ) (20 | 50 | 100 | 250 | 500)
  • curprev 18:56, 3 December 2021imported>Stashbot 151,474 bytes +176 andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020
  • curprev 01:17, 2 December 2021imported>Stashbot 151,298 bytes +1,832 wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster
  • curprev 17:48, 28 November 2021imported>Stashbot 149,466 bytes +209 andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for T296592. It will need to be moved back after the raid is rebuilt.
  • curprev 07:19, 21 November 2021imported>Stashbot 149,257 bytes +120 dcaro_away: restarting designate-sink with some extra logs in it (T296144)
  • curprev 15:48, 17 November 2021imported>Stashbot 149,137 bytes +280 andrewbogott: upgrading mariadb packages on eqiad1 cloudcontrols
  • curprev 13:31, 12 November 2021imported>Stashbot 148,857 bytes +142 arturo: restarting glance-api services to make sure they work with new ceph auth creds (T293752)
  • curprev 21:50, 8 November 2021imported>Stashbot 148,715 bytes +620 andrewbogott: returned clouddb pools back to normal after maintain_views run: https://gerrit.wikimedia.org/r/c/operations/puppet/+/737505 T216481
  • curprev 11:18, 5 November 2021imported>Stashbot 148,095 bytes +742 wm-bot: Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance
  • curprev 16:39, 4 November 2021imported>Stashbot 147,353 bytes +2,597 wm-bot: Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance
  • curprev 17:22, 3 November 2021imported>Stashbot 144,756 bytes +279 arturo: [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev (T294956)
  • curprev 10:54, 2 November 2021imported>Stashbot 144,477 bytes +179 arturo: rebooting cloudnet1004/1003 for T291813
  • curprev 00:47, 24 October 2021imported>Stashbot 144,298 bytes +166 andrewbogott: deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738
  • curprev 10:19, 21 October 2021imported>Stashbot 144,132 bytes +227 arturo: drop firewall exception on core routers for wiki replicas legacy setup (T293897)
  • curprev 21:06, 20 October 2021imported>Stashbot 143,905 bytes +99 andrewbogott: creating cloudinfra-nfs project T293936
  • curprev 19:21, 18 October 2021imported>Stashbot 143,806 bytes +252 andrewbogott: also ticked the 'admin' box on wikitech for majavah T292827
  • curprev 12:28, 14 October 2021imported>Stashbot 143,554 bytes +149 arturo: [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB (T292546)
  • curprev 10:46, 13 October 2021imported>Stashbot 143,405 bytes +105 arturo: updating python3-neutron across the fleet (T292936)
  • curprev 09:06, 12 October 2021imported>Stashbot 143,300 bytes +200 dcaro: upgrading eqiad cloudnet hosts neutron packages (T292936)
  • curprev 09:39, 5 October 2021imported>Stashbot 143,100 bytes +152 arturo: [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) T291257
  • curprev 14:50, 30 September 2021imported>Stashbot 142,948 bytes +391 andrewbogott: sudo cumin "cloud*" "ps -ef | grep nslcd && service nslcd restart" and sudo cumin "lab*" "ps -ef | grep nslcd && service nslcd restart" T292202
  • curprev 09:41, 29 September 2021imported>Stashbot 142,557 bytes +196 arturo: [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured (T291257)
  • curprev 16:23, 28 September 2021imported>Stashbot 142,361 bytes +531 bstorm: downtime for clouddb1020 to reduce re-pages in case this goes badly T291963
  • curprev 10:07, 27 September 2021imported>Stashbot 141,830 bytes +169 arturo: cloudcontrol1004 apparently healthy T291446
  • curprev 13:02, 24 September 2021imported>Stashbot 141,661 bytes +211 arturo: [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev
  • curprev 12:13, 21 September 2021imported>Stashbot 141,450 bytes +677 arturo: [codfw1dev] trying to create a manila service image (T291257)
  • curprev 23:08, 20 September 2021imported>Stashbot 140,773 bytes +408 bstorm: ran `echo check > /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid
  • curprev 11:35, 17 September 2021imported>Stashbot 140,365 bytes +114 arturo: [codfw1dev] install manila on cloudcontrol2001-dev (T291257)
  • curprev 15:56, 16 September 2021imported>Stashbot 140,251 bytes +134 bstorm: removing downtime for labstore1005 so we'll know if it has another issue T290318
  • curprev 22:03, 9 September 2021imported>Stashbot 140,117 bytes +315 bstorm: restarted the prometheus-mysqld-exporter@s1 service as it was not working T290630
  • curprev 15:34, 3 September 2021imported>Stashbot 139,802 bytes +365 bstorm: rebooting labstore1005 to disconnect the drives from labstore1004 T290318
  • curprev 16:16, 30 August 2021imported>Stashbot 139,437 bytes +825 wm-bot: Added 1 new OSDs ['cloudcephosd1018.eqiad.wmnet'] - cookbook ran by andrew@buster
  • curprev 18:57, 27 August 2021imported>Stashbot 138,612 bytes +126 andrewbogott: raising toolsbeta ram/core/instances quotas so majavah can experiment with bullseye
  • curprev 14:45, 25 August 2021imported>Stashbot 138,486 bytes +534 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
  • curprev 17:39, 19 August 2021imported>Stashbot 137,952 bytes +93 bstorm: restarting glance image backup to try and clear the page
  • curprev 16:21, 18 August 2021imported>Stashbot 137,859 bytes +899 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster
  • curprev 15:11, 17 August 2021imported>Stashbot 136,960 bytes +119 andrewbogott: rebooting cloudcephosd1008 to force raid rebuild -- T287838
  • curprev 13:51, 11 August 2021imported>Stashbot 136,841 bytes +480 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus
  • curprev 15:15, 10 August 2021imported>Stashbot 136,361 bytes +214 andrewbogott: restarting all designate services in eqiad1
  • curprev 09:37, 5 August 2021imported>Stashbot 136,147 bytes +106 dcaro: Taking one osd daemon down ot codfw cluster (T288203)
  • curprev 19:20, 4 August 2021imported>Stashbot 136,041 bytes +126 bd808: Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki
  • curprev 17:40, 3 August 2021imported>Stashbot 135,915 bytes +85 bstorm: rerunning the glance backup script after failure
  • curprev 00:10, 31 July 2021imported>Stashbot 135,830 bytes +233 andrewbogott: "systemctl reset-failed cloud-init.service" on all VMs for T287309
  • curprev 21:32, 27 July 2021imported>Stashbot 135,597 bytes +313 andrewbogott: putting cloudvirt1012 back into service T286748
  • curprev 15:22, 23 July 2021imported>Stashbot 135,284 bytes +88 bstorm: update wikireplicas-dns for s7 fix for web replicas
  • curprev 17:07, 20 July 2021imported>Stashbot 135,196 bytes +215 andrewbogott: reloading haproxy on dbproxy1018 for T286598
  • curprev 00:10, 20 July 2021imported>Stashbot 134,981 bytes +465 bstorm: restarting nova-api on cloudcontrol1003 to try and recover whatever it's doing with designate_floating_ip_ptr_records_updater
  • curprev 09:55, 16 July 2021imported>Stashbot 134,516 bytes +103 dcaro: checking HP raid issues on coludvirt1012 (T286766)
  • curprev 21:08, 14 July 2021imported>Stashbot 134,413 bytes +316 andrewbogott: restarting lots of openstack services while trying to resolve T286675
  • curprev 10:12, 2 July 2021imported>Stashbot 134,097 bytes +1,731 wm-bot: The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus
  • curprev 16:27, 1 July 2021imported>Stashbot 132,366 bytes +2,402 bstorm: failed over cloudstore1009 to cloudstore1008 T224747
  • curprev 21:48, 30 June 2021imported>Stashbot 129,964 bytes +115 bstorm: downtimed space alerts for scratch on cloudstore1008 until after the migration
  • curprev 15:28, 25 June 2021imported>Stashbot 129,849 bytes +238 andrewbogott: restarting openstack services on cloudcontrol1005
  • curprev 13:54, 21 June 2021imported>Stashbot 129,611 bytes +228 dcaro: puppet fix merged and deployed, servers are back to normal
  • curprev 22:21, 20 June 2021imported>Stashbot 129,383 bytes +144 andrewbogott: clearing admin-monitoring VMs; puppet has been failing lately due to a full drive on the puppetmaster
  • curprev 01:18, 15 June 2021imported>Stashbot 129,239 bytes +130 bstorm: running a modified version of the prometheus dir size cron in screen T284964
  • curprev 10:13, 14 June 2021imported>Stashbot 129,109 bytes +110 dcaro: setting ssd to debug mode on tools-sgeexec-0917 (T284130)
  • curprev 10:58, 10 June 2021imported>Stashbot 128,999 bytes +3,910 wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] (T281248) - cookbook ran by dcaro@vulcanus
  • curprev 17:33, 9 June 2021imported>Stashbot 125,089 bytes +1,815 arturo: removed icinga downtime for cloudmetrics1002 -- to see if hardware is healthy (T281881)
  • curprev 23:19, 8 June 2021imported>Stashbot 123,274 bytes +2,253 bd808: Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 (T281881)
  • curprev 14:27, 7 June 2021imported>Stashbot 121,021 bytes +138 andrewbogott: moving cloudvirt1040 from 'maintenance' aggregate to 'ceph' aggregate T281399
  • curprev 13:12, 1 June 2021imported>Stashbot 120,883 bytes +293 dcaro: Changed the ceph osd_memory_target on eqiad pool to 6Gi (we were reaching the limit, swapping at some points)
  • curprev 14:58, 27 May 2021imported>Stashbot 120,590 bytes +77 wm-bot: Testing - cookbook ran by dcaro@vulcanus
  • curprev 19:10, 26 May 2021imported>Stashbot 120,513 bytes +688 andrewbogott: reimaging cloudvirt1018 to support local VM storage
  • curprev 16:14, 25 May 2021imported>Stashbot 119,825 bytes +412 bd808: Closed #wikimedia-cloud-admin on f***node
  • curprev 22:32, 24 May 2021imported>Stashbot 119,413 bytes +302 andrewbogott: changing the default ttl for eqiad1.wikimedia.cloud. from 3600 to 60; this should help us avoid madness when re-using hostnames.
  • curprev 02:14, 22 May 2021imported>Stashbot 119,111 bytes +159 bstorm: downtiming SMART alerts on dumps server labstore1007 for the weekend because it has been flapping T281045
  • curprev 21:25, 13 May 2021imported>Stashbot 118,952 bytes +245 bstorm: converted the maps and scratch volumes on cloudstore1008 (standby) to drbd T224747
  • curprev 14:23, 12 May 2021imported>Stashbot 118,707 bytes +189 arturo: [codfw1dev] cleanup old unused agents (bgp, ovs)
  • curprev 18:00, 11 May 2021imported>Stashbot 118,518 bytes +198 andrewbogott: adding 'trove' service project in advance of deploying trove in eqiad1
  • curprev 10:53, 9 May 2021imported>Stashbot 118,320 bytes +109 arturo: icinga-downtime cloudmetrics1002 for 3 months (T275605)
  • curprev 13:51, 7 May 2021imported>Stashbot 118,211 bytes +252 andrewbogott: add inherited 'admin' right to novaadmin user throughout eqiad1. I was trying to narrow down the rights here but lack of admin breaks some workflows, e.g. T281894 and T282235
  • curprev 15:31, 6 May 2021imported>Stashbot 117,959 bytes +249 arturo: about to migrating CloudVPS network to the cloudgw architecture T270704
  • curprev 16:07, 5 May 2021imported>Stashbot 117,710 bytes +4,552 dcaro: disallowing insecure global ids on the eqiad ceph cluster (T280641)
  • curprev 16:05, 4 May 2021imported>Stashbot 113,158 bytes +1,656 wm-bot: Safe reboot of 'cloudvirt1028.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus
  • curprev 23:53, 3 May 2021imported>Stashbot 111,502 bytes +1,153 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004 T281287
  • curprev 11:16, 30 April 2021imported>Stashbot 110,349 bytes +267 dcaro: draining and rebooting coludvirt1017, last one today (T280641)
  • curprev 15:11, 29 April 2021imported>Stashbot 110,082 bytes +404 dcaro: hard rebooting cloudmetrics1002, got hung again (T275605)
  • curprev 21:11, 28 April 2021imported>Stashbot 109,678 bytes +2,619 andrewbogott: cleaning up more references to deleted hypervisors with delete from services where topic='compute' and version != 53;
  • curprev 14:10, 27 April 2021imported>Stashbot 107,059 bytes +1,057 dcaro: codfw.openstack upgraded ceph libraries to 15.2.11 (T280641)
  • curprev 20:56, 26 April 2021imported>Stashbot 106,002 bytes +265 andrewbogott: deleting spurious 'codfw1dev' and 'codw1dev-4' regions in the dallas deployment; regions without endpoints break a bunch of things
  • curprev 13:49, 23 April 2021imported>Stashbot 105,737 bytes +569 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster, draining cloudvirt2001 (T280641)
  • curprev 17:59, 21 April 2021imported>Stashbot 105,168 bytes +439 dcaro: all monitors upgraded on codfw1 with one cookbook `cookbook --verbose -c ~/.config/spicerack/cookbook.yaml wmcs.ceph.upgrade_mons --monitor-node-fqdn cloudcephmon2002-dev.codfw.wmnet` (T280641)
  • curprev 20:21, 20 April 2021imported>Stashbot 104,729 bytes +114 andrewbogott: reboot cloudservices1003
  • curprev 08:40, 19 April 2021imported>Stashbot 104,615 bytes +218 dcaro: enabling puppet on labstore1004 after mysql restart (T279657)
  • curprev 10:48, 14 April 2021imported>Stashbot 104,397 bytes +588 dcaro: Upgrade of codfw ceph to octopus 15.2.20 done, will run some performance tests now (T274566)
  • curprev 16:42, 13 April 2021imported>Stashbot 103,809 bytes +989 dcaro: Ceph balancer got the cluster to eval 0.014916, that is 88-77% usage for compute pool, and 28-19% usage for the cinder one \o/ (T274573)
  • curprev 21:33, 7 April 2021imported>Stashbot 102,820 bytes +84 andrewbogott: upgrading codfw1dev designate to Victoria
  • curprev 17:36, 4 April 2021imported>Stashbot 102,736 bytes +79 andrewbogott: upgrading eqiad1 designate to Ussuri
  • curprev 14:12, 2 April 2021imported>Stashbot 102,657 bytes +90 andrewbogott: upgrading codfw1dev to OpenStack version Ussuri
  • curprev 12:15, 1 April 2021imported>Stashbot 102,567 bytes +431 dcaro: Restoring the 4.9 kernel on cloudcephosd2003-dev and upgrading (T274565)
  • curprev 08:47, 31 March 2021imported>Stashbot 102,136 bytes +109 dcaro: upgrading cinder on codfw cloudcontrol2* nodes (T278845)
  • curprev 09:53, 30 March 2021imported>Stashbot 102,027 bytes +119 arturo: rebooting cloudnet1003 to cleanup conntrack table, it wouldn't cleanup by hand ...
  • curprev 15:42, 28 March 2021imported>Stashbot 101,908 bytes +80 andrewbogott: updated debian-10.0-buster base image
  • curprev 09:54, 27 March 2021imported>Stashbot 101,828 bytes +102 arturo: cleanup conntrack table in qrouter nents in cloudnet1003 (backup)
  • curprev 19:03, 25 March 2021imported>Stashbot 101,726 bytes +576 andrewbogott: deleting all unused (per wmcs-imageusage) Jessie base images from Glance
  • curprev 09:19, 24 March 2021imported>Stashbot 101,150 bytes +158 dcaro: restarted wmcs-backup on cloudvirt1024 as it failed due to an image being removed while running (T276892)
  • curprev 11:33, 23 March 2021imported>Stashbot 100,992 bytes +94 arturo: root@cloudcontrol1005:~# wmcs-novastats-dnsleaks --delete
  • curprev 10:10, 22 March 2021imported>Stashbot 100,898 bytes +191 arturo: cleanup conntrack table in standby node: aborrero@cloudnet1003:~ $ sudo ip netns exec qrouter-d93771ba-2711-4f88-804a-8df6fd03978a conntrack -F
  • curprev 17:18, 19 March 2021imported>Stashbot 100,707 bytes +293 bstorm: running `ALTER TABLE account MODIFY COLUMN type ENUM('user','tool','paws');` against the labsdbaccounts database on m5 T276284
  • curprev 00:30, 19 March 2021imported>Stashbot 100,414 bytes +94 bstorm: downtimed labstore1004 to check some things in debug mode
  • curprev 17:28, 17 March 2021imported>Stashbot 100,320 bytes +432 bstorm: restarted the backup-glance-images job to clear errors in systemd T271782
  • curprev 16:51, 10 March 2021imported>Stashbot 99,888 bytes +1,162 arturo: rebooting cloudvirt1030 for T275753
  • curprev 16:27, 9 March 2021imported>Stashbot 98,726 bytes +871 arturo: rebooting cloudvirt1027 (T275753)
  • curprev 21:40, 5 March 2021imported>Stashbot 97,855 bytes +748 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1 T276018
  • curprev 18:36, 4 March 2021imported>Stashbot 97,107 bytes +1,044 andrewbogott: rebooting cloudmetrics1002; the console is hanging
  • curprev 17:16, 3 March 2021imported>Stashbot 96,063 bytes +1,804 andrewbogott: restarting rabbitmq-server on cloudcontrol1003,1004,1005; trying to explain amqp errors in scheduler logs
  • curprev 17:16, 2 March 2021imported>Stashbot 94,259 bytes +717 andrewbogott: rebooting cloudvirt1039 to see if I can trigger T276208
  • curprev 20:12, 1 March 2021imported>Stashbot 93,542 bytes +347 andrewbogott: removing novaadmin from all projects save 'admin' for T274385
  • curprev 04:54, 28 February 2021imported>Stashbot 93,195 bytes +162 andrewbogott: restarted redis-server on tools-redis-1003 and tools-redis-1004 in an attempt to reduce replag, no real change detected
  • curprev 00:33, 27 February 2021imported>Stashbot 93,033 bytes +4,713 andrewbogott: sudo cumin --timeout 500 "A:all and not O{project:clouddb-services}" 'lsb_release -c | grep -i buster && uname -r | grep -v 4.19.0-14-amd64 && reboot'
  • curprev 14:56, 25 February 2021imported>Stashbot 88,320 bytes +121 arturo: deployed wmcs-netns-events daemon to all cloudnet servers (T275483)
  • curprev 11:07, 24 February 2021imported>Stashbot 88,199 bytes +112 arturo: force-reboot cloudmetrics1002, add icinga downtime for 2 hours. Investigating some server issue
  • curprev 00:17, 24 February 2021imported>Stashbot 88,087 bytes +717 bstorm: set --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main stretch image in glance on eqiad1 T275430
  • curprev 17:15, 22 February 2021imported>Stashbot 87,370 bytes +368 bstorm: restarting nova-compute on cloudvirt1016 and cloudvirt1036 in case it helps T275411
  • curprev 14:50, 18 February 2021imported>Stashbot 87,002 bytes +426 arturo: rebooting cloudnet1004 for T271058
  • curprev 15:58, 17 February 2021imported>Stashbot 86,576 bytes +153 arturo: deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/664845 to cloudnet servers (T268335)
  • curprev 16:25, 15 February 2021imported>Stashbot 86,423 bytes +395 arturo: [codfw1dev] rebooting all cloudgw200x-dev / cloudnet200x-dev servers (T272963)
  • curprev 12:01, 11 February 2021imported>Stashbot 86,028 bytes +692 arturo: [codfw1dev] drop instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` (was buster, cannot use it yet)
  • curprev 15:23, 9 February 2021imported>Stashbot 85,336 bytes +224 arturo: icinga-downtime for 2h everything *labs *cloud for openstack upgrades
  • curprev 18:50, 8 February 2021imported>Stashbot 85,112 bytes +253 bstorm: enabled puppet on cloudvirt1023 for now T274144
  • curprev 10:59, 5 February 2021imported>Stashbot 84,859 bytes +334 arturo: icinga-downtime labstore1004 tools share space check for 1 week (T272247)
  • curprev 10:12, 4 February 2021imported>Stashbot 84,525 bytes +147 dcaro: Increasing the memory limit of osds in eqiad from 8589934592(8G) to 12884901888(12G) (T273851)
  • curprev 09:59, 3 February 2021imported>Stashbot 84,378 bytes +203 dcaro: Doing a full vm backup on cloudvirt1024 with the new script (T260692)
  • curprev 17:14, 2 February 2021imported>Stashbot 84,175 bytes +346 dcaro: Changed osd memory limit from 4G to 8G (T273649)
  • curprev 15:36, 29 January 2021imported>Stashbot 83,829 bytes +155 andrewbogott: disabling puppet and some services on eqiad1 cloudcontrol nodes; replacing nova-placement-api with placement-api
  • curprev 19:44, 28 January 2021imported>Stashbot 83,674 bytes +158 andrewbogott: shutting down cloudcontrol2001-dev because it's in a partially upgraded state; will revive when it's time for Train
  • curprev 00:50, 27 January 2021imported>Stashbot 83,516 bytes +101 bstorm: icinga-downtime cloudnet1004 for a week T271058
  • curprev 16:44, 22 January 2021imported>Stashbot 83,415 bytes +191 andrewbogott: upgrading designate on cloudvirt1003/1004 to OpenStack 'train'
  • curprev 11:35, 21 January 2021imported>Stashbot 83,224 bytes +338 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657439 (T209082)
  • curprev 10:49, 20 January 2021imported>Stashbot 82,886 bytes +1,118 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657302 (T209082)
  • curprev 10:17, 19 January 2021imported>Stashbot 81,768 bytes +103 arturo: icinga-downtime cloudnet1004 for 1 week (T271058)
  • curprev 16:00, 18 January 2021imported>Stashbot 81,665 bytes +865 dcaro: Codfw1 ceph cluster uprgaded, will wait until tomorrow to see if there's any instability, but everything looks fine (T272303)
  • curprev 16:53, 17 January 2021imported>Stashbot 80,800 bytes +126 arturo: icinga downtime labstore1004 /srv/tools space check for 3 days (T272247)
  • curprev 13:41, 15 January 2021imported>Stashbot 80,674 bytes +405 arturo: icinga downtime labstore1004 maintain-dbuser alert until 2021-01-19 (T272125)
  • curprev 17:03, 13 January 2021imported>Stashbot 80,269 bytes +927 arturo: remove cloudvirt1013 cloudvirt1032 cloudvirt1037 to the 'toobusy' host aggregate to prevent further CPU oversubscribing
  • curprev 10:33, 12 January 2021imported>Stashbot 79,342 bytes +175 arturo: reboot cloudnet1004
  • curprev 10:22, 11 January 2021imported>Stashbot 79,167 bytes +573 arturo: doubling size of conntrack table in cloudnet servers https://gerrit.wikimedia.org/r/c/operations/puppet/+/655407 (T271058)
  • curprev 16:02, 10 January 2021imported>Stashbot 78,594 bytes +198 andrewbogott: restarting rabbitmq-server on all eqiad1 cloudcontrols
  • curprev 11:25, 8 January 2021imported>Stashbot 78,396 bytes +1,559 arturo: rebooting both cloudnet2002-dev/cloudnet2003-dev to make sure interfaces are set up correctl (T271517)
  • curprev 15:19, 7 January 2021imported>Stashbot 76,837 bytes +447 dcaro: Finished speed tests on cloudcephosd2001-dev, reprovisioning the osd.0 sdc (T271417)
  • curprev 10:40, 5 January 2021imported>Stashbot 76,390 bytes +134 dcaro: removing dumps-[1..*] backups from cloudvirt1024 as they are not needed (T271094)
  • curprev 07:06, 3 January 2021imported>Stashbot 76,256 bytes +117 dcaro: Got a network hiccup on cloudnet1004, keeping track here T271058
  • curprev 12:32, 28 December 2020imported>Stashbot 76,139 bytes +567 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 (T260692)
  • curprev 15:38, 23 December 2020imported>Stashbot 75,572 bytes +333 andrewbogott: restarting rabbitmq on cloudcontrol1004; suspected leaks
  • curprev 15:30, 22 December 2020imported>Stashbot 75,239 bytes +231 dcaro: cleaning up 6778 dangling snapshots for glance images in eqiad (T270478)
  • curprev 16:18, 19 December 2020imported>Stashbot 75,008 bytes +84 dcaro: gzipped a bunch of logs on cloudvirt1004 due to / being out of space
  • curprev 00:14, 19 December 2020imported>Stashbot 74,924 bytes +2,096 bstorm: truncated /var/log/debug.1 on cloudcontrol1003 which appears to be the exact same content as the user.log files anyway
  • curprev 22:17, 17 December 2020imported>Stashbot 72,828 bytes +570 andrewbogott: correction to above, set the pg and pgp to 1024 for eqiad1-glance-images
  • curprev 09:31, 16 December 2020imported>Stashbot 72,258 bytes +121 dcaro: removing invalid backups from cloudvirt1024 (196 in total) (T269419)
  • curprev 17:42, 14 December 2020imported>Stashbot 72,137 bytes +359 dcaro: The removal freed ~12GB (still 100% usage :S) (T269419)
  • curprev 09:11, 13 December 2020imported>Stashbot 71,778 bytes +108 _dcaro: running backup purge script on cloudvirt1024 (T269419)
  • curprev 23:36, 10 December 2020imported>Stashbot 71,670 bytes +334 bstorm: cleaned up the logs for haproxy on cloudcontrol1003 by deleting all the gzipped ones and truncating the .1 file
  • curprev 18:01, 8 December 2020imported>Stashbot 71,336 bytes +387 dcaro: Host cloudvirt1030 up and running (T216195)
  • curprev 18:33, 7 December 2020imported>Stashbot 70,949 bytes +249 andrewbogott: putting cloudvirt1023 back into service T269467
  • curprev 00:35, 5 December 2020imported>Stashbot 70,700 bytes +1,279 andrewbogott: moving cloudvirt1023 back into maintenance because T269467 continues to puzzle
  • curprev 23:21, 3 December 2020imported>Stashbot 69,421 bytes +765 andrewbogott: removing all osds on cloudcephosd1004 for rebuild, T268746
  • curprev 20:04, 2 December 2020imported>Stashbot 68,656 bytes +909 andrewbogott: removing all osds on cloudcephosd1010 for rebuild, T268746
  • curprev 20:06, 1 December 2020imported>Stashbot 67,747 bytes +322 andrewbogott: removing all osds on cloudcephosd1014 for rebuild, T268746
  • curprev 18:12, 30 November 2020imported>Stashbot 67,425 bytes +131 andrewbogott: removing all osds from cloudcephosd1015 in order to investigate T268746
  • curprev 17:18, 29 November 2020imported>Stashbot 67,294 bytes +106 andrewbogott: cleaning up some logfiles in tools-sgecron-01 — drive is full
  • curprev 22:58, 26 November 2020imported>Stashbot 67,188 bytes +257 andrewbogott: deleting /var/log/haproxy logs older than 7 days in cloudcontrol100x. We need log rotation here it seems.
  • curprev 19:35, 25 November 2020imported>Stashbot 66,931 bytes +1,218 bstorm: repairing ceph pg `instructing pg 6.91 on osd.117 to repair`
  • curprev 17:40, 22 November 2020imported>Stashbot 65,713 bytes +161 andrewbogott: apt-get upgrade on cloudservices1003/1004
  • curprev 12:44, 20 November 2020imported>Stashbot 65,552 bytes +237 arturo: [codfw1dev] install conntrackd in cloudnet2003-dev/cloudnet2002-dev to research l3 agent HA reliability
  • curprev 19:21, 17 November 2020imported>Stashbot 65,315 bytes +103 andrewbogott: draining cloudvirt1012 to experiment with libvirt/cpu things
  • curprev 11:21, 15 November 2020imported>Stashbot 65,212 bytes +103 arturo: icinga downtime cloudbackup2002 for 48h (T267865)
  • curprev 16:38, 10 November 2020imported>Stashbot 65,109 bytes +243 arturo: icinga downtime toolschecker for 2h becasue toolsdb maintenance (T266587)
  • curprev 12:42, 9 November 2020imported>Stashbot 64,866 bytes +944 arturo: restarted neutron l3 agent in cloudnet1003 bc it still had the old default route (T265288)
  • curprev 13:36, 2 November 2020imported>Stashbot 63,922 bytes +127 arturo: (typo: dcaro)
  • curprev 16:57, 29 October 2020imported>Stashbot 63,795 bytes +167 bstorm: silenced deployment-prep project alerts for 60 days since the downtime expired
  • curprev 16:20, 25 October 2020imported>Stashbot 63,628 bytes +230 andrewbogott: adding cloudvirt1038 to the 'ceph' aggregate and removing from the 'spare' aggregate. We need this space while waiting on network upgrades for empty cloudvirts (T216195)
  • curprev 11:30, 23 October 2020imported>Stashbot 63,398 bytes +467 arturo: [codfw1dev] openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record nat.cloudgw.codfw1dev.wikimediacloud.org. --description "created by hand" 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa. (T261724)
  • curprev 10:46, 22 October 2020imported>Stashbot 62,931 bytes +285 arturo: [codfw1dev] rebooting cloudinfra-internal-puppetmaster-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud to try fixing some DNS weirdness
  • curprev 14:36, 21 October 2020imported>Stashbot 62,646 bytes +343 andrewbogott: running apt-get update && apt-get install -y facter on all cloud-vps instances
  • curprev 15:47, 20 October 2020imported>Stashbot 62,303 bytes +315 arturo: changing DNS recursor ACLs (https://gerrit.wikimedia.org/r/c/operations/puppet/+/635314) this can be reverted any time if it causes problems (T261724)
  • curprev 01:41, 19 October 2020imported>Stashbot 61,988 bytes +280 andrewbogott: deleting all Precise base images
  • curprev 09:29, 16 October 2020imported>Stashbot 61,708 bytes +678 arturo: [codfw1dev] still some DNS weirdness, investigating
  • curprev 15:17, 15 October 2020imported>Stashbot 61,030 bytes +258 arturo: [codfw1dev] try cleaning up anything related to address scopes in the neutron database (T261724)
  • curprev 17:54, 13 October 2020imported>Stashbot 60,772 bytes +373 andrewbogott: rebuilding cloudvirt1021 for backy support
  • curprev 10:15, 9 October 2020imported>Stashbot 60,399 bytes +1,133 arturo: [codfwd1ev] root@cloudcontrol2001-dev:~# openstack router set --disable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw (T261724)
  • curprev 16:17, 8 October 2020imported>Stashbot 59,266 bytes +546 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.8 --no-dhcp --subnet-range 185.15.57.8/31 cloud-gw-transport-codfw` (with a hack -- see task) (T263622)
  • curprev 21:30, 6 October 2020imported>Stashbot 58,720 bytes +326 andrewbogott: moved cloudvirt1013 out of the 'ceph' aggregate and into the 'maintenance' aggregate for T243414
  • curprev 17:40, 5 October 2020imported>Stashbot 58,394 bytes +129 bd808: `service uwsgi-labspuppetbackend restart` on cloud-puppetmaster-03 (T264649)
  • curprev 11:05, 2 October 2020imported>Stashbot 58,265 bytes +234 arturo: [codfw1dev] restarting rabbitmq-server in all 3 control nodes, the l3 agent was misbehaving
  • curprev 16:06, 1 October 2020imported>Stashbot 58,031 bytes +166 arturo: rebooting cloudvirt1024 to validate changes to /etc/network/interfaces file
  • curprev 16:47, 30 September 2020imported>Stashbot 57,865 bytes +1,958 andrewbogott: rebooting cloudvir1032, 1033, 1034 for T262979
  • curprev 14:55, 28 September 2020imported>Stashbot 55,907 bytes +256 arturo: [jbond42] upgraded facter to v3 across the VM fleet
  • curprev 15:47, 24 September 2020imported>Stashbot 55,651 bytes +294 arturo: stopping/restarting rabbitmq-server in all cloudcontrol servers
  • curprev 10:16, 18 September 2020imported>Stashbot 55,357 bytes +593 arturo: cloudvirt1039 libvirtd service issues were fixed with a reboot
  • curprev 20:32, 15 September 2020imported>Stashbot 54,764 bytes +181 andrewbogott: rebooting cloudvirt1038 to see if it resolves T262979
  • curprev 14:21, 14 September 2020imported>Stashbot 54,583 bytes +421 andrewbogott: draining cloudvirt1001, migrating all VMs with wmcs-ceph-migrate
  • curprev 18:13, 9 September 2020imported>Stashbot 54,162 bytes +433 andrewbogott: restarting ceph-mon@cloudcephmon1003 in hopes that the slow ops reported are phantoms
  • curprev 00:05, 9 September 2020imported>Stashbot 53,729 bytes +517 bd808: Running wmcs-novastats-dnsleaks (T262359)
  • curprev 09:32, 3 September 2020imported>Stashbot 53,212 bytes +106 arturo: icinga downtime cloud* servers for 30 mins (T261866)
  • curprev 08:46, 2 September 2020imported>Stashbot 53,106 bytes +131 arturo: [codfw1dev] reimaging spare server labtestvirt2003 as debian buster (T261724)
  • curprev 18:18, 1 September 2020imported>Stashbot 52,975 bytes +640 andrewbogott: adding drives on cloudcephosd100[3-5] to ceph osd pool
  • curprev 23:26, 31 August 2020imported>Stashbot 52,335 bytes +303 bd808: Removed stale lockfile at cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs:/var/lib/puppet/volatile/GeoIP/.geoipupdate.lock
  • curprev 20:12, 28 August 2020imported>Stashbot 52,032 bytes +100 bd808: Running `wmcs-novastats-dnsleaks --delete` from cloudcontrol1003
  • curprev 17:12, 26 August 2020imported>Stashbot 51,932 bytes +198 bstorm: Running 'ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%k KB %p\n" > tools_large_files_20200826.txt' on labstore1004 T261336
  • curprev 21:34, 21 August 2020imported>Stashbot 51,734 bytes +99 andrewbogott: restarting nova-compute on cloudvirt1033; it seems stuck
  • curprev 14:21, 19 August 2020imported>Stashbot 51,635 bytes +130 andrewbogott: rebooting cloudweb2001-dev, labweb1001, labweb1002 to address mediawiki-induced memleak
  • curprev 21:02, 6 August 2020imported>Stashbot 51,505 bytes +241 andrewbogott: removing cloudvirt1004/1006 from nova's list of hypervisors; rebuilding them to use as backup test hosts
  • curprev 18:54, 4 August 2020imported>Stashbot 51,264 bytes +105 bstorm: restarting mariadb on cloudcontrol1004 to setup parallel replication
  • curprev 17:02, 3 August 2020imported>Stashbot 51,159 bytes +137 bstorm: increased db connection limit to 800 across galera cluster because we were clearly hovering at limit
  • curprev 19:28, 31 July 2020imported>Stashbot 51,022 bytes +126 bd808: wmcs-novastats-dnsleaks --delete (lots of leaked fullstack-monitoring records to clean up)
  • curprev 22:17, 27 July 2020imported>Stashbot 50,896 bytes +150 andrewbogott: ceph osd pool set compute pg_num 2048
  • curprev 19:15, 24 July 2020imported>Stashbot 50,746 bytes +148 andrewbogott: ceph mgr module enable pg_autoscaler
  • curprev 08:55, 22 July 2020imported>Stashbot 50,598 bytes +291 jbond42: [codfw1dev] upgrading hiera to version5
  • curprev 10:48, 16 July 2020imported>Stashbot 50,307 bytes +158 arturo: merging change to neutron dmz_cidr https://gerrit.wikimedia.org/r/c/operations/puppet/+/613123 (T257534)
  • curprev 23:15, 15 July 2020imported>Stashbot 50,149 bytes +545 bd808: Removed Merlijn van Deen from toollabs-trusted Gerrit group (T255697)
  • curprev 15:19, 14 July 2020imported>Stashbot 49,604 bytes +504 arturo: briefly set root@cloudnet1003:~ # sysctl net.ipv4.conf.all.accept_local=1 (in neutron qrouter netns) (T257534)
  • curprev 16:17, 13 July 2020imported>Stashbot 49,100 bytes +127 arturo: icinga downtime cloudcontrol[1003-1005].wikimedia.org for 1h for galera database movements
  • curprev 17:39, 12 July 2020imported>Stashbot 48,973 bytes +98 andrewbogott: switched eqiad1 keystone from m5 to cloudcontrol galera
  • curprev 20:26, 10 July 2020imported>Stashbot 48,875 bytes +88 andrewbogott: disabling nova api to move database to galera
  • curprev 11:23, 9 July 2020imported>Stashbot 48,787 bytes +378 arturo: [codfw1dev] rebooting cloudnet2003-dev again for testing sysct/puppet behavior (T257552)
  • curprev 15:16, 6 July 2020imported>Stashbot 48,409 bytes +76 arturo: installing 'aptitude' in all cloudvirts
  • curprev 12:51, 3 July 2020imported>Stashbot 48,333 bytes +455 arturo: [codfw1dev] galera cluster should be up and running, openstack happy (T256283)
  • curprev 15:41, 2 July 2020imported>Stashbot 47,878 bytes +273 arturo: `sudo wmcs-openstack --os-compute-api-version 2.55 flavor create --private --vcpus 8 --disk 300 --ram 16384 --property aggregate_instance_extra_specs:ceph=true --description "for packaging envoy" bigdisk-ceph` (T256983)
  • curprev 14:24, 29 June 2020imported>Stashbot 47,605 bytes +162 arturo: starting rabbitmq-server in all 3 cloudcontrol servers
  • curprev 20:38, 18 June 2020imported>Stashbot 47,443 bytes +130 andrewbogott: rebooting cloudservices2003-dev due to a mysterious 'host down' alert on a secondary ip
  • curprev 15:38, 16 June 2020imported>Stashbot 47,313 bytes +159 arturo: created by hand neutron port 9c0a9a13-e409-49de-9ba3-bc8ec4801dbf `paws-haproxy-vip` (T295217)
  • curprev 13:23, 12 June 2020imported>Stashbot 47,154 bytes +202 arturo: DNS zone `paws.wmcloud.org` transferred to the PAWS project (T195217)
  • curprev 19:19, 11 June 2020imported>Stashbot 46,952 bytes +428 bstorm_: proceeding with failback to labstore1004 now that DRBD devices are consistent T224582
  • curprev 16:09, 10 June 2020imported>Stashbot 46,524 bytes +169 andrewbogott: deleting all old cloud-ns0.wikimedia.org and cloud-ns1.wikimedia.org ns records in designate database T254496
  • curprev 15:25, 9 June 2020imported>Stashbot 46,355 bytes +337 arturo: icinga downtime everything cloud* lab* for 2h more (T253780)
  • curprev 15:08, 5 June 2020imported>Stashbot 46,018 bytes +148 andrewbogott: trying to re-enable puppet without losing cumin contact, as per https://phabricator.wikimedia.org/T254589
  • curprev 14:24, 4 June 2020imported>Stashbot 45,870 bytes +180 andrewbogott: disabling puppet on all instances for /labs/private recovery
  • curprev 23:02, 28 May 2020imported>Stashbot 45,690 bytes +169 bd808: `/usr/local/sbin/maintain-dbusers --debug harvest-replicas` (T253930)
  • curprev 00:33, 28 May 2020imported>Stashbot 45,521 bytes +610 andrewbogott: shutting down cloudservices2002-dev to see if we can live without it. This is in anticipation or rebuilding it entirely for T253780
  • curprev 16:36, 25 May 2020imported>Stashbot 44,911 bytes +119 arturo: [codfw1dev] created zone `0-29.57.15.185.in-addr.arpa.` (T247972)
  • curprev 19:23, 21 May 2020imported>Stashbot 44,792 bytes +336 andrewbogott: disabling puppet on cloudbackup2001 to prevent the backup job from starting during maintenance
  • curprev 22:59, 19 May 2020imported>Stashbot 44,456 bytes +181 bd808: `apt-get install mariadb-client` on cloudcontrol1003
  • curprev 21:37, 18 May 2020imported>Stashbot 44,275 bytes +82 andrewbogott: rebuilding cloudnet2003-dev with Buster
  • curprev 22:10, 15 May 2020imported>Stashbot 44,193 bytes +375 bd808: Added reedy as projectadmin in cloudinfra project (T249774)
  • curprev 23:28, 14 May 2020imported>Stashbot 43,818 bytes +724 bstorm_: downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time T252831
  • curprev 20:33, 12 May 2020imported>Stashbot 43,094 bytes +747 andrewbogott: moving cloudvirt1023 to the 'standard' pool and out of the 'spare' pool
  • curprev 16:53, 9 May 2020imported>Stashbot 42,347 bytes +128 andrewbogott: rebuilding cloudcontrol2001-dev and 2003-dev with buster for T252121
  • curprev 19:02, 8 May 2020imported>Stashbot 42,219 bytes +118 bstorm_: moving tools-k8s-haproxy-2 from cloudvirt1021 to cloudvirt1017 to improve spread
  • curprev 13:58, 5 May 2020imported>Stashbot 42,101 bytes +101 andrewbogott: rebuilding cloudcontrol2004-dev to test new puppet changes
  • curprev 09:04, 4 May 2020imported>Stashbot 42,000 bytes +194 arturo: [codfw1dev] manually modify iptables ruleset to only allow SSH from WMF bastions on cloudservices2003-dev and cloudcontrol2004-dev (T251604)
  • curprev 22:12, 21 April 2020imported>Stashbot 41,806 bytes +205 andrewbogott: moving cloudvirt1004 out of the 'standard' aggregate and into the 'maintenance' aggregate
  • curprev 18:44, 15 April 2020imported>Stashbot 41,601 bytes +99 jeh: create indexes and views for grwikimedia T245912
  • curprev 15:07, 13 April 2020imported>Stashbot 41,502 bytes +110 jeh: restart memcached on labwebs to increase cache size T145703
  • curprev 19:57, 9 April 2020imported>Stashbot 41,392 bytes +156 andrewbogott: upgrading eqiad1 designate to rocky
  • curprev 19:20, 8 April 2020imported>Stashbot 41,236 bytes +239 andrewbogott: rotated password and api token for pdns servers on cloudservices1003 and cloudservices1004
  • curprev 20:57, 7 April 2020imported>Stashbot 40,997 bytes +128 andrewbogott: service sssd stop; rm -rf /var/lib/sss/db*; service sssd start on tools-sgebastion-08
  • curprev 22:39, 6 April 2020imported>Stashbot 40,869 bytes +634 andrewbogott: deleting bogus groups cn=b'project-bastion',ou=groups,dc=wikimedia,dc=org and cn=b'project-tools',ou=groups,dc=wikimedia,dc=org from ldap
  • curprev 20:59, 2 April 2020imported>Stashbot 40,235 bytes +112 jeh: codfw1dev clear VM error states and start bastions, puppet master and database
  • curprev 16:27, 1 April 2020imported>Stashbot 40,123 bytes +126 arturo: [codfw1dev] enable puppet across the fleet clean vxlan changes (T248881)
  • curprev 12:35, 31 March 2020imported>Stashbot 39,997 bytes +801 arturo: [codfw1dev] restarting VMs: designaterockytest14, bastion-codfw1dev-0[1,2] (T248881)
(newest | oldest) View (newer 250 | ) (20 | 50 | 100 | 250 | 500)