You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Admin/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 500 | ) (20 | 50 | 100 | 250 | 500)

7 December 2022

  • curprev 22:0722:07, 7 December 2022imported>Stashbot 403,988 bytes +151 andrewbogott: systemctl restart libvirt-guests.service on cloudvirt1019 to get ceph/rbd working on VMS on this hypervisor

3 December 2022

30 November 2022

  • curprev 20:0320:03, 30 November 2022imported>Stashbot 403,751 bytes +237 andrewbogott: changing all rabbitmq queues to quorum queues. Will be noisy! T318816

28 November 2022

  • curprev 13:0013:00, 28 November 2022imported>Stashbot 403,514 bytes +571 wm-bot2: unset cloudvirt1043.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo

25 November 2022

  • curprev 10:5410:54, 25 November 2022imported>Stashbot 402,943 bytes +520 wm-bot2: deleted VM canary2001-dev-2 from cloudvirt2001-dev - cookbook ran by arturo@nostromo

24 November 2022

  • curprev 16:5316:53, 24 November 2022imported>Stashbot 402,423 bytes +1,316 wm-bot2: deleted VM canary2001-dev-1 from cloudvirt2001-dev - cookbook ran by arturo@nostromo

23 November 2022

  • curprev 15:0015:00, 23 November 2022imported>Stashbot 401,107 bytes +1,734 wm-bot2: unset cloudvirt1045.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo

22 November 2022

  • curprev 13:2613:26, 22 November 2022imported>Stashbot 399,373 bytes +790 wm-bot2: unset cloudvirt1048.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo

21 November 2022

  • curprev 16:1916:19, 21 November 2022imported>Stashbot 398,583 bytes +1,730 wm-bot2: Drained cloudvirt1050.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo

18 November 2022

  • curprev 13:3713:37, 18 November 2022imported>Stashbot 396,853 bytes +376 arturo: [codfw1dev] reimaged cloudvirt2001-dev and cloudvirt2002-dev

16 November 2022

  • curprev 20:0720:07, 16 November 2022imported>Stashbot 396,477 bytes +682 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye

14 November 2022

  • curprev 20:2220:22, 14 November 2022imported>Stashbot 395,795 bytes +978 wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye

11 November 2022

  • curprev 13:2413:24, 11 November 2022imported>Stashbot 394,817 bytes +686 wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: edad3915-b7c6-4b23-bb9c-ab13b04a41c5, use this to unset) (T319184) - cookbook ran by arturo@nostromo

10 November 2022

  • curprev 16:1916:19, 10 November 2022imported>Stashbot 394,131 bytes +578 wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: 346013ec-ce4e-497e-ad65-2d215b14998c, use this to unset). (T319184) - cookbook ran by arturo@nostromo

8 November 2022

  • curprev 11:1711:17, 8 November 2022imported>Stashbot 393,553 bytes +128 taavi: backfilling security groups for metricsinfra access on all projects T288108

7 November 2022

  • curprev 21:0121:01, 7 November 2022imported>Stashbot 393,425 bytes +361 wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org (T305828) - cookbook ran by andrew@bullseye

4 November 2022

  • curprev 17:5717:57, 4 November 2022imported>Stashbot 393,064 bytes +172 andrewbogott: removing cinderv2 API endpoints from keystone catalog; this is deprecated and removed in Yoga. prep for T305828

3 November 2022

  • curprev 19:2419:24, 3 November 2022imported>Stashbot 392,892 bytes +274 wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet (T305828) - cookbook ran by andrew@bullseye
  • curprev 00:0800:08, 3 November 2022imported>Stashbot 392,618 bytes +1,091 wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org (T305828) - cookbook ran by andrew@bullseye

31 October 2022

  • curprev 13:0913:09, 31 October 2022imported>Stashbot 391,527 bytes +171 arturo: restart keepalived on all 4 cloudgw servers to run them with `-D` in /etc/default/keepalived to further debug T320975

26 October 2022

  • curprev 16:1816:18, 26 October 2022imported>Stashbot 391,356 bytes +313 wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:bf48880d-0c1b-4c2a-8e8b-778d28b16561) (T319446) - cookbook ran by dcaro@vulcanus

25 October 2022

  • curprev 16:0316:03, 25 October 2022imported>Stashbot 391,043 bytes +382 arturo: [codfw1dev] T321220 root@cloudcontrol2001-dev:~# openstack subnet create magnum --no-dhcp --network 57017d7c-3817-429a-8aa3-b028de82cdcc --ip-version 4 --gateway auto --subnet-range 192.168.0.0/24

24 October 2022

  • curprev 18:5018:50, 24 October 2022imported>Stashbot 390,661 bytes +1,101 wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by andrew@bullseye

20 October 2022

  • curprev 23:2323:23, 20 October 2022imported>Stashbot 389,560 bytes +24,091 wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye

15 October 2022

  • curprev 17:3817:38, 15 October 2022imported>Stashbot 365,469 bytes +179 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki Slevinski # T320867

13 October 2022

  • curprev 12:1912:19, 13 October 2022imported>Stashbot 365,290 bytes +719 wm-bot2: OSDs (['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034']) upgraded successfully B-) (T309786) - cookbook ran by dcaro@vulcanus

10 October 2022

9 October 2022

  • curprev 12:0412:04, 9 October 2022imported>Stashbot 364,412 bytes +176 taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki DatGuy # T320301

7 October 2022

  • curprev 13:4013:40, 7 October 2022imported>Stashbot 364,236 bytes +531 andrewbogott: dhinus is resetting rabbitmq cluster in an attempt to resolve a suspected (by Andrew) split-brain

6 October 2022

  • curprev 15:5515:55, 6 October 2022imported>Stashbot 363,705 bytes +663 arturo: cloudnet1005 & cloudnet1006 now in service. Secom cloudnet1003 & cloudnet1004. Drop neutron agents, etc. (T316284)

5 October 2022

  • curprev 14:4014:40, 5 October 2022imported>Stashbot 363,042 bytes +1,152 wm-bot2: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) (T319418) - cookbook ran by fran@wmf3169

4 October 2022

  • curprev 16:4016:40, 4 October 2022imported>Stashbot 361,890 bytes +1,582 wm-bot2: Added 1 new OSDs ['cloudcephosd1033.eqiad.wmnet'] (T314870) - cookbook ran by fran@wmf3169

30 September 2022

  • curprev 14:5214:52, 30 September 2022imported>Stashbot 360,308 bytes +2,841 wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169

27 September 2022

  • curprev 10:4810:48, 27 September 2022imported>Stashbot 357,467 bytes +2,068 wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169

26 September 2022

  • curprev 18:3718:37, 26 September 2022imported>Stashbot 355,399 bytes +13,416 wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster

25 September 2022

  • curprev 15:0615:06, 25 September 2022imported>Stashbot 341,983 bytes +2,869 wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster

24 September 2022

  • curprev 17:3717:37, 24 September 2022imported>Stashbot 339,114 bytes +186 andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown

22 September 2022

  • curprev 15:1415:14, 22 September 2022imported>Stashbot 338,928 bytes +14,397 wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster

20 September 2022

  • curprev 21:0221:02, 20 September 2022imported>Stashbot 324,531 bytes +20,280 wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster

19 September 2022

  • curprev 20:0720:07, 19 September 2022imported>Stashbot 304,251 bytes +1,214 wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster

14 September 2022

  • curprev 16:5716:57, 14 September 2022imported>Stashbot 303,037 bytes +1,804 wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro

13 September 2022

  • curprev 12:1612:16, 13 September 2022imported>Stashbot 301,233 bytes +1,263 wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus

10 September 2022

  • curprev 15:3715:37, 10 September 2022imported>Stashbot 299,970 bytes +151 andrewbogott: restarting nova-conductor service (and possibly others, in response to lots of unanswered rabbitmq messages)

8 September 2022

  • curprev 19:1219:12, 8 September 2022imported>Stashbot 299,819 bytes +108 andrewbogott: restarting nginx on proxy-03.project-proxy.eqiad1.wikimedia.cloud

7 September 2022

  • curprev 10:1810:18, 7 September 2022imported>Stashbot 299,711 bytes +859 wm-bot2: Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

30 August 2022

  • curprev 14:5914:59, 30 August 2022imported>Stashbot 298,852 bytes +1,876 andrewbogott: manually marking most eqiad1 cloud* servers down in icinga for T296561

25 August 2022

  • curprev 15:1415:14, 25 August 2022imported>Stashbot 296,976 bytes +886 wm-bot2: Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

24 August 2022

  • curprev 22:0722:07, 24 August 2022imported>Stashbot 296,090 bytes +983 andrewbogott: replaced cloudservices1003 with cloudservices1005 T304888

23 August 2022

  • curprev 13:4613:46, 23 August 2022imported>Stashbot 295,107 bytes +886 wm-bot2: Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

21 August 2022

  • curprev 21:1221:12, 21 August 2022imported>Stashbot 294,221 bytes +167 andrewbogott: restarted neutron-dhcp-agent on cloudnet1003. it was claiming to be unable to contact Rabbit but seems happy after a restart

20 August 2022

  • curprev 07:3907:39, 20 August 2022imported>Stashbot 294,054 bytes +305 dcaro_away: cloudvirt1023 is back up, VMs are starting to recover (T315718)

19 August 2022

  • curprev 17:0617:06, 19 August 2022imported>Stashbot 293,749 bytes +145 taavi: [codfw1dev] restart mariadb on clouddb2002-dev to pick up certificate config changes T310795

18 August 2022

  • curprev 13:2513:25, 18 August 2022imported>Stashbot 293,604 bytes +980 wm-bot2: Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

17 August 2022

  • curprev 10:5010:50, 17 August 2022imported>Stashbot 292,624 bytes +1,560 wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] (T314870) - cookbook ran by fran@foz

16 August 2022

  • curprev 22:3922:39, 16 August 2022imported>Stashbot 291,064 bytes +2,291 andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate

14 August 2022

  • curprev 18:3618:36, 14 August 2022imported>Stashbot 288,773 bytes +105 taavi: deleted the http keystone endpoints from the keystone service catalog

11 August 2022

  • curprev 13:5713:57, 11 August 2022imported>Stashbot 288,668 bytes +1,152 andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.

10 August 2022

  • curprev 13:1013:10, 10 August 2022imported>Stashbot 287,516 bytes +578 wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station

4 August 2022

  • curprev 17:1617:16, 4 August 2022imported>Stashbot 286,938 bytes +328 taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling

3 August 2022

  • curprev 20:5520:55, 3 August 2022imported>Stashbot 286,610 bytes +266 andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service

2 August 2022

  • curprev 14:0714:07, 2 August 2022imported>Stashbot 286,344 bytes +375 andrewbogott: shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html

27 July 2022

  • curprev 19:3219:32, 27 July 2022imported>Stashbot 285,969 bytes +286 andrewbogott: switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54

25 July 2022

  • curprev 13:4313:43, 25 July 2022imported>Stashbot 285,683 bytes +142 andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]

22 July 2022

  • curprev 16:4116:41, 22 July 2022imported>Stashbot 285,541 bytes +174 taavi: depool cloudweb1003/1004 since horizon seems to be having issues

21 July 2022

  • curprev 18:2618:26, 21 July 2022imported>Stashbot 285,367 bytes +216 andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes

20 July 2022

  • curprev 18:0218:02, 20 July 2022imported>Stashbot 285,151 bytes +785 dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 (T313400)

19 July 2022

  • curprev 16:3016:30, 19 July 2022imported>Stashbot 284,366 bytes +34,154 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • curprev 00:4600:46, 19 July 2022imported>Stashbot 250,212 bytes +15,548 wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-bb4aa0a75e45, use this to unset). - cookbook ran by andrew@buster

13 July 2022

12 July 2022

  • curprev 12:0412:04, 12 July 2022imported>Stashbot 234,589 bytes +1,130 wm-bot2: Ceph cluster at {self.deployment} set out of maintenance. - cookbook ran by dcaro@vulcanus

8 July 2022

  • curprev 15:5715:57, 8 July 2022imported>Stashbot 233,459 bytes +311 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

7 July 2022

  • curprev 07:1707:17, 7 July 2022imported>Stashbot 233,148 bytes +271 wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet (T312509) - cookbook ran by dcaro@vulcanus

6 July 2022

  • curprev 17:5017:50, 6 July 2022imported>Stashbot 232,877 bytes +258 wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus

4 July 2022

  • curprev 13:2713:27, 4 July 2022imported>Stashbot 232,619 bytes +6,078 wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus

3 July 2022

  • curprev 21:2721:27, 3 July 2022imported>Stashbot 226,541 bytes +160 andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted

2 July 2022

  • curprev 11:0511:05, 2 July 2022imported>Stashbot 226,381 bytes +137 wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

1 July 2022

  • curprev 15:3515:35, 1 July 2022imported>Stashbot 226,244 bytes +3,845 wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus

30 June 2022

  • curprev 18:1718:17, 30 June 2022imported>Stashbot 222,399 bytes +533 wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

29 June 2022

  • curprev 08:4508:45, 29 June 2022imported>Stashbot 221,866 bytes +217 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

28 June 2022

  • curprev 13:0313:03, 28 June 2022imported>Stashbot 221,649 bytes +137 taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor T301949

22 June 2022

  • curprev 16:4816:48, 22 June 2022imported>Stashbot 221,512 bytes +278 taavi: restart designate-*.service on both cloudservices nodes

21 June 2022

  • curprev 04:4804:48, 21 June 2022imported>Stashbot 221,234 bytes +230 andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow

17 June 2022

  • curprev 17:5317:53, 17 June 2022imported>Stashbot 221,004 bytes +251 andrewbogott: switching to a new python-based health check for galera and haproxy. This may make things more stable, or it may not. T310664

15 June 2022

  • curprev 11:3411:34, 15 June 2022imported>Stashbot 220,753 bytes +86 taavi: restart neutron-linuxbridge-agent on cloudvirt1022

14 June 2022

  • curprev 16:2616:26, 14 June 2022imported>Stashbot 220,667 bytes +1,288 wm-bot2: OSDs (['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloudcephosd1021', 'cl

13 June 2022

  • curprev 11:1411:14, 13 June 2022imported>Stashbot 219,379 bytes +876 wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

6 June 2022

  • curprev 13:2113:21, 6 June 2022imported>Stashbot 218,503 bytes +133 andrewbogott: restarting mysql/galera on cloudcontrol100x in an attempt to stabilize some flapping there

2 June 2022

  • curprev 07:5107:51, 2 June 2022imported>Stashbot 218,370 bytes +99 taavi: restart neutron-linuxbridge-agent.service on cloudvirt1034 T309732
  • curprev 00:3700:37, 2 June 2022imported>Stashbot 218,271 bytes +388 andrewbogott: updated nameservers for codfw1dev instances via 'openstack subnet set --dns-nameserver etc.'

30 May 2022

  • curprev 17:4317:43, 30 May 2022imported>Stashbot 217,883 bytes +110 andrewbogott: restarting neutron-rpc and neutron-api services on cloudcontrol1xxx

29 May 2022

  • curprev 14:5514:55, 29 May 2022imported>Stashbot 217,773 bytes +223 andrewbogott: restarting nova services on all eqiad1 cloudcontrol nodes to recover from rabbit breakage

25 May 2022

  • curprev 20:0120:01, 25 May 2022imported>Stashbot 217,550 bytes +175 balloons: clean up cinder backup a bit, restart service due to network outage

19 May 2022

  • curprev 15:2115:21, 19 May 2022imported>Stashbot 217,375 bytes +176 andrewbogott: resetting password for the 'troveguest' rabbitmq user. I think I may have broken this during a recent rebuild of the rabbitmq cluster

18 May 2022

  • curprev 15:4215:42, 18 May 2022imported>Stashbot 217,199 bytes +109 andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build

14 May 2022

  • curprev 11:3311:33, 14 May 2022imported>Stashbot 217,090 bytes +103 taavi: deleted projects 'ores' and 'ores-staging' T308102

13 May 2022

  • curprev 06:2006:20, 13 May 2022imported>Stashbot 216,987 bytes +11,620 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • curprev 01:0001:00, 13 May 2022imported>Stashbot 205,367 bytes +14,074 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster

11 May 2022

  • curprev 18:4818:48, 11 May 2022imported>Stashbot 191,293 bytes +3,204 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster

10 May 2022

  • curprev 21:4321:43, 10 May 2022imported>Stashbot 188,089 bytes +3,450 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster

7 May 2022

  • curprev 01:3301:33, 7 May 2022imported>Stashbot 184,639 bytes +483 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster

3 May 2022

  • curprev 20:3820:38, 3 May 2022imported>Stashbot 184,156 bytes +173 andrewbogott: upgrading clouddb2001-dev in place

2 May 2022

29 April 2022

  • curprev 14:2214:22, 29 April 2022imported>Stashbot 183,921 bytes +270 andrewbogott: changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions T277653 https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline

27 April 2022

  • curprev 14:5114:51, 27 April 2022imported>Stashbot 183,651 bytes +8,234 wm-bot: Finished rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloud

26 April 2022

  • curprev 10:3610:36, 26 April 2022imported>Stashbot 175,417 bytes +227 taavi: [codfw1dev] updated designate pool to 2004/2005-dev according to the instructions on https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#Initial_designate/pdns_node_setup

22 April 2022

  • curprev 10:3310:33, 22 April 2022imported>Stashbot 175,190 bytes +130 taavi: [codfw1dev] restart designate-sink on both new cloudservices host to fix rabbitmq connectivity

21 April 2022

  • curprev 05:3805:38, 21 April 2022imported>Stashbot 175,060 bytes +100 andrewbogott: replaced cloudservices200[2,3] with cloudservices200[4,5]

19 April 2022

  • curprev 15:2915:29, 19 April 2022imported>Stashbot 174,960 bytes +92 andrewbogott: stopping all VMs on cloudvirt1019, reimaging host

18 April 2022

  • curprev 15:2315:23, 18 April 2022imported>Stashbot 174,868 bytes +205 andrewbogott: reimaging cloudvirt1020, leaving VMs in place

14 April 2022

  • curprev 20:1420:14, 14 April 2022imported>Stashbot 174,663 bytes +147 andrewbogott: restarting nova-api and nova-conductor services in a superstitious attempt to reduce open DB connections

13 April 2022

  • curprev 22:0122:01, 13 April 2022imported>Stashbot 174,516 bytes +116 andrewbogott: restarting galera on cloudcontrols (one by one) to clear open connections

11 April 2022

9 April 2022

  • curprev 19:5519:55, 9 April 2022imported>Stashbot 174,329 bytes +236 andrewbogott: reimaging cloudbackup1001-dev to bullseye

7 April 2022

  • curprev 12:5112:51, 7 April 2022imported>Stashbot 174,093 bytes +152 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T305631) - cookbook ran by arturo@nostromo

6 April 2022

  • curprev 09:1209:12, 6 April 2022imported>Stashbot 173,941 bytes +713 arturo: [codf1dev] installing python3-eventlet 0.30.2-5~bpo11+1 on all required servers (cloudvirt, cloudnet, cloudcontrol) (T305157)

30 March 2022

  • curprev 11:2011:20, 30 March 2022imported>Stashbot 173,228 bytes +114 arturo: apply urpf strict filter to eqiad cloud-hosts vlan - T285461

29 March 2022

23 March 2022

  • curprev 22:5322:53, 23 March 2022imported>Stashbot 173,032 bytes +6,374 wm-bot: Drained 'cloudvirt1045.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

22 March 2022

  • curprev 22:5922:59, 22 March 2022imported>Stashbot 166,658 bytes +263 wm-bot: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

17 March 2022

  • curprev 01:0901:09, 17 March 2022imported>Stashbot 166,395 bytes +509 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

15 March 2022

  • curprev 20:5820:58, 15 March 2022imported>Stashbot 165,886 bytes +941 wm-bot: Drained 'cloudvirt1026.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

14 March 2022

  • curprev 21:2421:24, 14 March 2022imported>Stashbot 164,945 bytes +3,074 wm-bot: Drained 'cloudvirt1025.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

8 March 2022

  • curprev 18:2918:29, 8 March 2022imported>Stashbot 161,871 bytes +3,117 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

3 March 2022

  • curprev 08:4908:49, 3 March 2022imported>Stashbot 158,754 bytes +105 taavi: deploying cloudmetrics grafana to grafana 8, T282863

2 March 2022

  • curprev 09:0609:06, 2 March 2022imported>Stashbot 158,649 bytes +138 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/701347

28 February 2022

  • curprev 15:3015:30, 28 February 2022imported>Stashbot 158,511 bytes +132 dcaro: cleaning up leftover snapshots from failed backups of the maps volume (T302720)

24 February 2022

  • curprev 17:0417:04, 24 February 2022imported>Stashbot 158,379 bytes +464 andrewbogott: upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server'

23 February 2022

  • curprev 20:3920:39, 23 February 2022imported>Stashbot 157,915 bytes +1,620 taavi: added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account T295246

22 February 2022

18 February 2022

  • curprev 21:5721:57, 18 February 2022imported>Stashbot 156,024 bytes +484 andrewbogott: leaving cloudcontrol1003 downtimed with disabled puppet for the weekend. Everything there should be stable and fine save rabbit which needs an upgrade.

17 February 2022

  • curprev 23:0223:02, 17 February 2022imported>Stashbot 155,540 bytes +116 andrewbogott: in-place upgrade to Bullseye on cloudcontrol1005 T281276

15 February 2022

  • curprev 14:1514:15, 15 February 2022imported>Stashbot 155,424 bytes +175 taavi: [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account T295246

4 February 2022

  • curprev 10:1210:12, 4 February 2022imported>Stashbot 155,249 bytes +107 arturo: restart backup_vms service in cloudvirt1024 (T300956)

3 February 2022

  • curprev 08:2108:21, 3 February 2022imported>Stashbot 155,142 bytes +226 taavi: cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing "performing a change every puppet run" alert)

2 February 2022

31 January 2022

  • curprev 10:1510:15, 31 January 2022imported>Stashbot 154,835 bytes +146 arturo: cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert)

29 January 2022

27 January 2022

  • curprev 13:2413:24, 27 January 2022imported>Stashbot 154,587 bytes +146 arturo: cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service (T300138)

26 January 2022

  • curprev 19:0919:09, 26 January 2022imported>Stashbot 154,441 bytes +157 andrewbogott: bootstrapping a fresh galera node on cloudcontrol1004

25 January 2022

  • curprev 10:4910:49, 25 January 2022imported>Stashbot 154,284 bytes +170 arturo: made cloudmetrics1001/1002 primary/backup respectively (T299744, T297814, T300011)

19 January 2022

  • curprev 16:3816:38, 19 January 2022imported>Stashbot 154,114 bytes +121 andrewbogott: moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud

5 January 2022

  • curprev 03:1103:11, 5 January 2022imported>Stashbot 153,993 bytes +227 andrewbogott: 'cp /etc/apt/sources.list /etc/apt/sources.list.prepuppet' on all VMs. Backing up state before puppetizing sources.list with https://gerrit.wikimedia.org/r/c/operations/puppet/+/751498

4 January 2022

26 December 2021

  • curprev 16:5516:55, 26 December 2021imported>Stashbot 153,683 bytes +110 majavah: run attachLdapUser.php on wikitech for developer account "Karthiksripal"

24 December 2021

23 December 2021

  • curprev 21:4221:42, 23 December 2021imported>Stashbot 153,474 bytes +116 majavah: deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies

21 December 2021

15 December 2021

  • curprev 12:4412:44, 15 December 2021imported>Stashbot 153,248 bytes +143 dcaro: Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed (T297454)

14 December 2021

  • curprev 10:2610:26, 14 December 2021imported>Stashbot 153,105 bytes +258 dcaro: Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/<canary_image_id>) to the root disk on cloudvirt-wdqs1001 to temporary free some space (T297454)

13 December 2021

  • curprev 18:0818:08, 13 December 2021imported>Stashbot 152,847 bytes +1,373 wm-bot: Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse

3 December 2021

  • curprev 18:5618:56, 3 December 2021imported>Stashbot 151,474 bytes +176 andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020

2 December 2021

  • curprev 01:1701:17, 2 December 2021imported>Stashbot 151,298 bytes +1,832 wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster

28 November 2021

  • curprev 17:4817:48, 28 November 2021imported>Stashbot 149,466 bytes +209 andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for T296592. It will need to be moved back after the raid is rebuilt.
(newest | oldest) View (newer 500 | ) (20 | 50 | 100 | 250 | 500)