You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Admin/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 50 | ) (20 | 50 | 100 | 250 | 500)

30 September 2022

  • curprev 14:5214:52, 30 September 2022imported>Stashbot 360,308 bytes +2,841 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169

27 September 2022

  • curprev 10:4810:48, 27 September 2022imported>Stashbot 357,467 bytes +2,068 wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169

26 September 2022

  • curprev 18:3718:37, 26 September 2022imported>Stashbot 355,399 bytes +13,416 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster

25 September 2022

  • curprev 15:0615:06, 25 September 2022imported>Stashbot 341,983 bytes +2,869 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster

24 September 2022

  • curprev 17:3717:37, 24 September 2022imported>Stashbot 339,114 bytes +186 andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown

22 September 2022

  • curprev 15:1415:14, 22 September 2022imported>Stashbot 338,928 bytes +14,397 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster

20 September 2022

  • curprev 21:0221:02, 20 September 2022imported>Stashbot 324,531 bytes +20,280 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster

19 September 2022

  • curprev 20:0720:07, 19 September 2022imported>Stashbot 304,251 bytes +1,214 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster

14 September 2022

  • curprev 16:5716:57, 14 September 2022imported>Stashbot 303,037 bytes +1,804 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro

13 September 2022

  • curprev 12:1612:16, 13 September 2022imported>Stashbot 301,233 bytes +1,263 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus

10 September 2022

  • curprev 15:3715:37, 10 September 2022imported>Stashbot 299,970 bytes +151 andrewbogott: restarting nova-conductor service (and possibly others, in response to lots of unanswered rabbitmq messages)

8 September 2022

  • curprev 19:1219:12, 8 September 2022imported>Stashbot 299,819 bytes +108 andrewbogott: restarting nginx on proxy-03.project-proxy.eqiad1.wikimedia.cloud

7 September 2022

  • curprev 10:1810:18, 7 September 2022imported>Stashbot 299,711 bytes +859 wm-bot2: Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

30 August 2022

  • curprev 14:5914:59, 30 August 2022imported>Stashbot 298,852 bytes +1,876 andrewbogott: manually marking most eqiad1 cloud* servers down in icinga for T296561

25 August 2022

  • curprev 15:1415:14, 25 August 2022imported>Stashbot 296,976 bytes +886 wm-bot2: Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

24 August 2022

  • curprev 22:0722:07, 24 August 2022imported>Stashbot 296,090 bytes +983 andrewbogott: replaced cloudservices1003 with cloudservices1005 T304888

23 August 2022

  • curprev 13:4613:46, 23 August 2022imported>Stashbot 295,107 bytes +886 wm-bot2: Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

21 August 2022

  • curprev 21:1221:12, 21 August 2022imported>Stashbot 294,221 bytes +167 andrewbogott: restarted neutron-dhcp-agent on cloudnet1003. it was claiming to be unable to contact Rabbit but seems happy after a restart

20 August 2022

  • curprev 07:3907:39, 20 August 2022imported>Stashbot 294,054 bytes +305 dcaro_away: cloudvirt1023 is back up, VMs are starting to recover (T315718)

19 August 2022

  • curprev 17:0617:06, 19 August 2022imported>Stashbot 293,749 bytes +145 taavi: [codfw1dev] restart mariadb on clouddb2002-dev to pick up certificate config changes T310795

18 August 2022

  • curprev 13:2513:25, 18 August 2022imported>Stashbot 293,604 bytes +980 wm-bot2: Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

17 August 2022

  • curprev 10:5010:50, 17 August 2022imported>Stashbot 292,624 bytes +1,560 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] (T314870) - cookbook ran by fran@foz

16 August 2022

  • curprev 22:3922:39, 16 August 2022imported>Stashbot 291,064 bytes +2,291 andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate

14 August 2022

  • curprev 18:3618:36, 14 August 2022imported>Stashbot 288,773 bytes +105 taavi: deleted the http keystone endpoints from the keystone service catalog

11 August 2022

  • curprev 13:5713:57, 11 August 2022imported>Stashbot 288,668 bytes +1,152 andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.

10 August 2022

  • curprev 13:1013:10, 10 August 2022imported>Stashbot 287,516 bytes +578 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station

4 August 2022

  • curprev 17:1617:16, 4 August 2022imported>Stashbot 286,938 bytes +328 taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling

3 August 2022

  • curprev 20:5520:55, 3 August 2022imported>Stashbot 286,610 bytes +266 andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service

2 August 2022

  • curprev 14:0714:07, 2 August 2022imported>Stashbot 286,344 bytes +375 andrewbogott: shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html

27 July 2022

  • curprev 19:3219:32, 27 July 2022imported>Stashbot 285,969 bytes +286 andrewbogott: switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54

25 July 2022

  • curprev 13:4313:43, 25 July 2022imported>Stashbot 285,683 bytes +142 andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]

22 July 2022

  • curprev 16:4116:41, 22 July 2022imported>Stashbot 285,541 bytes +174 taavi: depool cloudweb1003/1004 since horizon seems to be having issues

21 July 2022

  • curprev 18:2618:26, 21 July 2022imported>Stashbot 285,367 bytes +216 andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes

20 July 2022

  • curprev 18:0218:02, 20 July 2022imported>Stashbot 285,151 bytes +785 dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 (T313400)

19 July 2022

  • curprev 16:3016:30, 19 July 2022imported>Stashbot 284,366 bytes +34,154 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • curprev 00:4600:46, 19 July 2022imported>Stashbot 250,212 bytes +15,548 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-bb4aa0a75e45, use this to unset). - cookbook ran by andrew@buster

13 July 2022

12 July 2022

  • curprev 12:0412:04, 12 July 2022imported>Stashbot 234,589 bytes +1,130 wm-bot2: Ceph cluster at {self.deployment} set out of maintenance. - cookbook ran by dcaro@vulcanus

8 July 2022

  • curprev 15:5715:57, 8 July 2022imported>Stashbot 233,459 bytes +311 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

7 July 2022

  • curprev 07:1707:17, 7 July 2022imported>Stashbot 233,148 bytes +271 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet (T312509) - cookbook ran by dcaro@vulcanus

6 July 2022

  • curprev 17:5017:50, 6 July 2022imported>Stashbot 232,877 bytes +258 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus

4 July 2022

  • curprev 13:2713:27, 4 July 2022imported>Stashbot 232,619 bytes +6,078 wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus

3 July 2022

  • curprev 21:2721:27, 3 July 2022imported>Stashbot 226,541 bytes +160 andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted

2 July 2022

  • curprev 11:0511:05, 2 July 2022imported>Stashbot 226,381 bytes +137 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

1 July 2022

  • curprev 15:3515:35, 1 July 2022imported>Stashbot 226,244 bytes +3,845 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus

30 June 2022

  • curprev 18:1718:17, 30 June 2022imported>Stashbot 222,399 bytes +533 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

29 June 2022

  • curprev 08:4508:45, 29 June 2022imported>Stashbot 221,866 bytes +217 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

28 June 2022

  • curprev 13:0313:03, 28 June 2022imported>Stashbot 221,649 bytes +137 taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor T301949

22 June 2022

  • curprev 16:4816:48, 22 June 2022imported>Stashbot 221,512 bytes +278 taavi: restart designate-*.service on both cloudservices nodes

21 June 2022

  • curprev 04:4804:48, 21 June 2022imported>Stashbot 221,234 bytes +230 andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow
(newest | oldest) View (newer 50 | ) (20 | 50 | 100 | 250 | 500)