You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Nova Resource:Admin/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 50 | ) (20 | 50 | 100 | 250 | 500)

17 August 2022

  • curprev 10:5010:50, 17 August 2022imported>Stashbot 292,624 bytes +1,560 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] (T314870) - cookbook ran by fran@foz

16 August 2022

  • curprev 22:3922:39, 16 August 2022imported>Stashbot 291,064 bytes +2,291 andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate

14 August 2022

  • curprev 18:3618:36, 14 August 2022imported>Stashbot 288,773 bytes +105 taavi: deleted the http keystone endpoints from the keystone service catalog

11 August 2022

  • curprev 13:5713:57, 11 August 2022imported>Stashbot 288,668 bytes +1,152 andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.

10 August 2022

  • curprev 13:1013:10, 10 August 2022imported>Stashbot 287,516 bytes +578 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station

4 August 2022

  • curprev 17:1617:16, 4 August 2022imported>Stashbot 286,938 bytes +328 taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling

3 August 2022

  • curprev 20:5520:55, 3 August 2022imported>Stashbot 286,610 bytes +266 andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service

2 August 2022

  • curprev 14:0714:07, 2 August 2022imported>Stashbot 286,344 bytes +375 andrewbogott: shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html

27 July 2022

  • curprev 19:3219:32, 27 July 2022imported>Stashbot 285,969 bytes +286 andrewbogott: switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54

25 July 2022

  • curprev 13:4313:43, 25 July 2022imported>Stashbot 285,683 bytes +142 andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]

22 July 2022

  • curprev 16:4116:41, 22 July 2022imported>Stashbot 285,541 bytes +174 taavi: depool cloudweb1003/1004 since horizon seems to be having issues

21 July 2022

  • curprev 18:2618:26, 21 July 2022imported>Stashbot 285,367 bytes +216 andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes

20 July 2022

  • curprev 18:0218:02, 20 July 2022imported>Stashbot 285,151 bytes +785 dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 (T313400)

19 July 2022

  • curprev 16:3016:30, 19 July 2022imported>Stashbot 284,366 bytes +34,154 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • curprev 00:4600:46, 19 July 2022imported>Stashbot 250,212 bytes +15,548 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-bb4aa0a75e45, use this to unset). - cookbook ran by andrew@buster

13 July 2022

12 July 2022

  • curprev 12:0412:04, 12 July 2022imported>Stashbot 234,589 bytes +1,130 wm-bot2: Ceph cluster at {self.deployment} set out of maintenance. - cookbook ran by dcaro@vulcanus

8 July 2022

  • curprev 15:5715:57, 8 July 2022imported>Stashbot 233,459 bytes +311 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

7 July 2022

  • curprev 07:1707:17, 7 July 2022imported>Stashbot 233,148 bytes +271 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet (T312509) - cookbook ran by dcaro@vulcanus

6 July 2022

  • curprev 17:5017:50, 6 July 2022imported>Stashbot 232,877 bytes +258 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus

4 July 2022

  • curprev 13:2713:27, 4 July 2022imported>Stashbot 232,619 bytes +6,078 wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus

3 July 2022

  • curprev 21:2721:27, 3 July 2022imported>Stashbot 226,541 bytes +160 andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted

2 July 2022

  • curprev 11:0511:05, 2 July 2022imported>Stashbot 226,381 bytes +137 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

1 July 2022

  • curprev 15:3515:35, 1 July 2022imported>Stashbot 226,244 bytes +3,845 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus

30 June 2022

  • curprev 18:1718:17, 30 June 2022imported>Stashbot 222,399 bytes +533 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

29 June 2022

  • curprev 08:4508:45, 29 June 2022imported>Stashbot 221,866 bytes +217 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

28 June 2022

  • curprev 13:0313:03, 28 June 2022imported>Stashbot 221,649 bytes +137 taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor T301949

22 June 2022

  • curprev 16:4816:48, 22 June 2022imported>Stashbot 221,512 bytes +278 taavi: restart designate-*.service on both cloudservices nodes

21 June 2022

  • curprev 04:4804:48, 21 June 2022imported>Stashbot 221,234 bytes +230 andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow

17 June 2022

  • curprev 17:5317:53, 17 June 2022imported>Stashbot 221,004 bytes +251 andrewbogott: switching to a new python-based health check for galera and haproxy. This may make things more stable, or it may not. T310664

15 June 2022

  • curprev 11:3411:34, 15 June 2022imported>Stashbot 220,753 bytes +86 taavi: restart neutron-linuxbridge-agent on cloudvirt1022

14 June 2022

  • curprev 16:2616:26, 14 June 2022imported>Stashbot 220,667 bytes +1,288 wm-bot2: OSDs (['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloudcephosd1021', 'cl

13 June 2022

  • curprev 11:1411:14, 13 June 2022imported>Stashbot 219,379 bytes +876 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

6 June 2022

  • curprev 13:2113:21, 6 June 2022imported>Stashbot 218,503 bytes +133 andrewbogott: restarting mysql/galera on cloudcontrol100x in an attempt to stabilize some flapping there

2 June 2022

  • curprev 07:5107:51, 2 June 2022imported>Stashbot 218,370 bytes +99 taavi: restart neutron-linuxbridge-agent.service on cloudvirt1034 T309732
  • curprev 00:3700:37, 2 June 2022imported>Stashbot 218,271 bytes +388 andrewbogott: updated nameservers for codfw1dev instances via 'openstack subnet set --dns-nameserver etc.'

30 May 2022

  • curprev 17:4317:43, 30 May 2022imported>Stashbot 217,883 bytes +110 andrewbogott: restarting neutron-rpc and neutron-api services on cloudcontrol1xxx

29 May 2022

  • curprev 14:5514:55, 29 May 2022imported>Stashbot 217,773 bytes +223 andrewbogott: restarting nova services on all eqiad1 cloudcontrol nodes to recover from rabbit breakage

25 May 2022

  • curprev 20:0120:01, 25 May 2022imported>Stashbot 217,550 bytes +175 balloons: clean up cinder backup a bit, restart service due to network outage

19 May 2022

  • curprev 15:2115:21, 19 May 2022imported>Stashbot 217,375 bytes +176 andrewbogott: resetting password for the 'troveguest' rabbitmq user. I think I may have broken this during a recent rebuild of the rabbitmq cluster

18 May 2022

  • curprev 15:4215:42, 18 May 2022imported>Stashbot 217,199 bytes +109 andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build

14 May 2022

  • curprev 11:3311:33, 14 May 2022imported>Stashbot 217,090 bytes +103 taavi: deleted projects 'ores' and 'ores-staging' T308102

13 May 2022

  • curprev 06:2006:20, 13 May 2022imported>Stashbot 216,987 bytes +11,620 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • curprev 01:0001:00, 13 May 2022imported>Stashbot 205,367 bytes +14,074 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster

11 May 2022

  • curprev 18:4818:48, 11 May 2022imported>Stashbot 191,293 bytes +3,204 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster

10 May 2022

  • curprev 21:4321:43, 10 May 2022imported>Stashbot 188,089 bytes +3,450 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster

7 May 2022

  • curprev 01:3301:33, 7 May 2022imported>Stashbot 184,639 bytes +483 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster

3 May 2022

  • curprev 20:3820:38, 3 May 2022imported>Stashbot 184,156 bytes +173 andrewbogott: upgrading clouddb2001-dev in place

2 May 2022

29 April 2022

  • curprev 14:2214:22, 29 April 2022imported>Stashbot 183,921 bytes +270 andrewbogott: changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions T277653 https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline
(newest | oldest) View (newer 50 | ) (20 | 50 | 100 | 250 | 500)