Nova Resource:Admin/SAL: Revision history

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

curprev 13:5613:56, 28 March 2023‎ imported>Stashbot‎ 452,928 bytes +361‎ andrewbogott: depooling cloudweb1003 before switch upgrade

curprev 14:3014:30, 27 March 2023‎ imported>Stashbot‎ 452,567 bytes +1,800‎ wm-bot2: Drained cloudvirt1024.eqiad.wmnet - cookbook ran by andrew@bullseye

curprev 12:4112:41, 22 March 2023‎ imported>Stashbot‎ 450,767 bytes +96‎ taavi: delete wmde-templates-alpha project T332773

curprev 21:4521:45, 8 March 2023‎ imported>Stashbot‎ 450,671 bytes +172‎ bd808: maintain-kubeusers container in CrashLoopBackoff, investigating

curprev 16:0616:06, 7 March 2023‎ imported>Stashbot‎ 450,499 bytes +691‎ andrewbogott: updated application credential roles, replacing 'user' with 'reader' and 'projectadmin' with 'member': update application_credential_role set role_id='f75a3c410bca4e96a1cf6ac103b0ccaf' where role_id='f473273fac7146b3bdbf22e5d4504f95' and update application_credential_role set role_id='38676f30eaeb44518bf7e144a73c8da6' where role_id='4d8cad783d6342efa8414d7d36fbc034'

curprev 12:1312:13, 3 March 2023‎ imported>Stashbot‎ 449,808 bytes +90‎ arturo: installing haproxy updates (T331119)

curprev 16:1716:17, 2 March 2023‎ imported>Stashbot‎ 449,718 bytes +1,892‎ wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1010.eqiad.wmnet'] (T329504) - cookbook ran by dcaro@vulcanus

curprev 22:4722:47, 28 February 2023‎ imported>Stashbot‎ 447,826 bytes +1,434‎ andrewbogott: adding new 'member' role assignment to every user/project pair that currently has the 'user' assignment. T330759

curprev 21:0121:01, 27 February 2023‎ imported>Stashbot‎ 446,392 bytes +2,790‎ wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1004.eqiad.wmnet'] (T329502) - cookbook ran by dcaro@vulcanus

curprev 08:2508:25, 22 February 2023‎ imported>Stashbot‎ 443,602 bytes +1,244‎ wm-bot2: Depooled and destroyed OSD daemons [31, 30, 29, 28, 27, 26, 25, 24] and removed the OSD host cloudcephosd1004 from the CRUSH map. (T329502) - cookbook ran by dcaro@vulcanus

curprev 20:2720:27, 21 February 2023‎ imported>Stashbot‎ 442,358 bytes +879‎ andrewbogott: deleted 200 more orphaned VM images with wmcs-novastats-cephleaks

curprev 19:1419:14, 16 February 2023‎ imported>Stashbot‎ 441,479 bytes +3,906‎ wm-bot2: The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1002.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus

curprev 14:5314:53, 15 February 2023‎ imported>Stashbot‎ 437,573 bytes +906‎ andrewbogott: deleting another 100 leaked VM images with wmcs-novastats-cephleaks

curprev 13:1713:17, 14 February 2023‎ imported>Stashbot‎ 436,667 bytes +135‎ andrewbogott: restarting all eqiad1 openstack services because that seems to sometimes help things *shrug*

curprev 14:0614:06, 13 February 2023‎ imported>Stashbot‎ 436,532 bytes +285‎ wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 8fbf6bfd-eec1-4d81-8e0d-ea431d8411ee (T329498) - cookbook ran by dcaro@vulcanus

curprev 21:1721:17, 9 February 2023‎ imported>Stashbot‎ 436,247 bytes +152‎ andrewbogott: deleted 10% of leaked VM ceph images using wmcs-novastats-cephleaks (only 10% out of an abundance of caution)

curprev 17:0817:08, 8 February 2023‎ imported>Stashbot‎ 436,095 bytes +112‎ arturo: changing to cloudgw network setup, make VIPs /32 (T295774)

curprev 11:2611:26, 7 February 2023‎ imported>Stashbot‎ 435,983 bytes +138‎ arturo: [codfw1dev] testing network changes in cloudgw, expect unrealiable network (T295774)

curprev 13:4413:44, 4 February 2023‎ imported>Stashbot‎ 435,845 bytes +118‎ taavi: drop old columns from oathauth_users table on labtestwiki T328131

curprev 15:0015:00, 3 February 2023‎ imported>Stashbot‎ 435,727 bytes +205‎ andrewbogott: restarted nova services in eqiad1 in an attempt to eke out another day or two of stability

curprev 13:1413:14, 2 February 2023‎ imported>Stashbot‎ 435,522 bytes +302‎ dcaro_away: draining osd.48 from node cloudcephosd1001 (T316544)

curprev 22:3422:34, 30 January 2023‎ imported>Stashbot‎ 435,220 bytes +276‎ wm-bot2: Upgraded and rebooted host cloudrabbit1002.wikimedia.org - cookbook ran by andrew@bullseye

curprev 20:0820:08, 27 January 2023‎ imported>Stashbot‎ 434,944 bytes +461‎ wm-bot2: Upgraded and rebooted host cloudcontrol2005-dev.wikimedia.org - cookbook ran by andrew@bullseye

curprev 20:3420:34, 26 January 2023‎ imported>Stashbot‎ 434,483 bytes +136‎ andrewbogott: shutting down mariadb on cloudbackup2001-dev, testing the waters for T328079

curprev 03:4203:42, 22 January 2023‎ imported>Stashbot‎ 434,347 bytes +111‎ andrewbogott: reset eqiad1 rabbitmq in an attempt to resolve some mild instability

curprev 15:2615:26, 20 January 2023‎ imported>Stashbot‎ 434,236 bytes +801‎ wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye

curprev 18:0618:06, 19 January 2023‎ imported>Stashbot‎ 433,435 bytes +680‎ wm-bot2: Removed cloudweb hosts (cloudweb2002-dev.wikimedia.org) from maintenance mode. - cookbook ran by andrew@bullseye

curprev 22:3222:32, 18 January 2023‎ imported>Stashbot‎ 432,755 bytes +7,287‎ wm-bot2: Set cloudweb cloudweb2002-dev.wikimedia.org maintenance (downtime id: 347cb75e-215e-4b85-ae14-4ce1934c70c7, use this to unset) - cookbook ran by andrew@bullseye

curprev 19:3219:32, 17 January 2023‎ imported>Stashbot‎ 425,468 bytes +556‎ wm-bot2: Upgraded and rebooted host cloudbackup2002.codfw.wmnet - cookbook ran by andrew@bullseye

curprev 17:3617:36, 13 January 2023‎ imported>Stashbot‎ 424,912 bytes +2,718‎ wm-bot2: Restarting openstack services on cloudcontrol2005-dev: ['nova-conductor', 'nova-scheduler', 'nova-api', 'nova-api-metadata'] - cookbook ran by andrew@bullseye

curprev 22:3422:34, 12 January 2023‎ imported>Stashbot‎ 422,194 bytes +253‎ andrewbogott: updated the Bullseye base image with the upstream 20221219 build

curprev 18:4218:42, 6 January 2023‎ imported>Stashbot‎ 421,941 bytes +886‎ wm-bot2: Safe reboot of cloudvirt2003-dev.codfw.wmnet finished successfully - cookbook ran by andrew@bullseye
curprev 00:1400:14, 6 January 2023‎ imported>Stashbot‎ 421,055 bytes +995‎ wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet - cookbook ran by andrew@bullseye

curprev 21:1821:18, 4 January 2023‎ imported>Stashbot‎ 420,060 bytes +1,344‎ wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org - cookbook ran by andrew@bullseye

curprev 22:1122:11, 3 January 2023‎ imported>Stashbot‎ 418,716 bytes +325‎ wm-bot2: Upgraded and rebooted host cloudservices2005-dev.wikimedia.org - cookbook ran by andrew@bullseye

curprev 14:2114:21, 25 December 2022‎ imported>Stashbot‎ 418,391 bytes +193‎ taavi: register developer account 'instance-puppet-user-dev' to update the codfw1dev instance-puppet repo without access to the eqiad1 repo T318504

curprev 15:1615:16, 22 December 2022‎ imported>Stashbot‎ 418,198 bytes +98‎ dcaro: added submit rights for JenkinsBot on all cloud/* gerrit repos

curprev 04:5904:59, 21 December 2022‎ imported>Stashbot‎ 418,100 bytes +2,530‎ wm-bot2: Rebooting node cloudcephosd1030.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye
curprev 00:5800:58, 21 December 2022‎ imported>Stashbot‎ 415,570 bytes +11,131‎ wm-bot2: Rebooting node cloudcephosd1020.eqiad.wmnet (T325132) - cookbook ran by andrew@bullseye

curprev 07:5007:50, 17 December 2022‎ imported>Stashbot‎ 404,439 bytes +151‎ taavi: deleted project packagist-mirror per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge#packagist-mirror

curprev 19:3619:36, 16 December 2022‎ imported>Stashbot‎ 404,288 bytes +300‎ volans: restarted sshd twice on bastion-restricted-eqiad1-02 to debug SSH connections for T319401

curprev 22:0722:07, 7 December 2022‎ imported>Stashbot‎ 403,988 bytes +151‎ andrewbogott: systemctl restart libvirt-guests.service on cloudvirt1019 to get ceph/rbd working on VMS on this hypervisor

curprev 19:2419:24, 3 December 2022‎ imported>Stashbot‎ 403,837 bytes +86‎ taavi: restart designate-sink on both cloudservices hosts

curprev 20:0320:03, 30 November 2022‎ imported>Stashbot‎ 403,751 bytes +237‎ andrewbogott: changing all rabbitmq queues to quorum queues. Will be noisy! T318816

curprev 13:0013:00, 28 November 2022‎ imported>Stashbot‎ 403,514 bytes +571‎ wm-bot2: unset cloudvirt1043.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo

curprev 10:5410:54, 25 November 2022‎ imported>Stashbot‎ 402,943 bytes +520‎ wm-bot2: deleted VM canary2001-dev-2 from cloudvirt2001-dev - cookbook ran by arturo@nostromo

curprev 16:5316:53, 24 November 2022‎ imported>Stashbot‎ 402,423 bytes +1,316‎ wm-bot2: deleted VM canary2001-dev-1 from cloudvirt2001-dev - cookbook ran by arturo@nostromo

curprev 15:0015:00, 23 November 2022‎ imported>Stashbot‎ 401,107 bytes +1,734‎ wm-bot2: unset cloudvirt1045.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo

curprev 13:2613:26, 22 November 2022‎ imported>Stashbot‎ 399,373 bytes +790‎ wm-bot2: unset cloudvirt1048.eqiad.wmnet maintenance (aggregates: ceph) - cookbook ran by arturo@nostromo

curprev 16:1916:19, 21 November 2022‎ imported>Stashbot‎ 398,583 bytes +1,730‎ wm-bot2: Drained cloudvirt1050.eqiad.wmnet (T319184) - cookbook ran by arturo@nostromo

curprev 13:3713:37, 18 November 2022‎ imported>Stashbot‎ 396,853 bytes +376‎ arturo: [codfw1dev] reimaged cloudvirt2001-dev and cloudvirt2002-dev

curprev 20:0720:07, 16 November 2022‎ imported>Stashbot‎ 396,477 bytes +682‎ wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org - cookbook ran by andrew@bullseye

curprev 20:2220:22, 14 November 2022‎ imported>Stashbot‎ 395,795 bytes +978‎ wm-bot2: Upgraded and rebooted host cloudnet1005.eqiad.wmnet - cookbook ran by andrew@bullseye

curprev 13:2413:24, 11 November 2022‎ imported>Stashbot‎ 394,817 bytes +686‎ wm-bot2: Set cloudvirt cloudvirt2003-dev.codfw.wmnet maintenance (downtime id: edad3915-b7c6-4b23-bb9c-ab13b04a41c5, use this to unset) (T319184) - cookbook ran by arturo@nostromo

curprev 16:1916:19, 10 November 2022‎ imported>Stashbot‎ 394,131 bytes +578‎ wm-bot2: Set cloudvirt 'cloudvirt2002-dev.codfw.wmnet' maintenance (downtime id: 346013ec-ce4e-497e-ad65-2d215b14998c, use this to unset). (T319184) - cookbook ran by arturo@nostromo

curprev 11:1711:17, 8 November 2022‎ imported>Stashbot‎ 393,553 bytes +128‎ taavi: backfilling security groups for metricsinfra access on all projects T288108

curprev 21:0121:01, 7 November 2022‎ imported>Stashbot‎ 393,425 bytes +361‎ wm-bot2: Upgraded and rebooted host cloudservices1004.wikimedia.org (T305828) - cookbook ran by andrew@bullseye

curprev 17:5717:57, 4 November 2022‎ imported>Stashbot‎ 393,064 bytes +172‎ andrewbogott: removing cinderv2 API endpoints from keystone catalog; this is deprecated and removed in Yoga. prep for T305828

curprev 19:2419:24, 3 November 2022‎ imported>Stashbot‎ 392,892 bytes +274‎ wm-bot2: Upgraded and rebooted host cloudbackup1002-dev.eqiad.wmnet (T305828) - cookbook ran by andrew@bullseye
curprev 00:0800:08, 3 November 2022‎ imported>Stashbot‎ 392,618 bytes +1,091‎ wm-bot2: Upgraded and rebooted host cloudcontrol2004-dev.wikimedia.org (T305828) - cookbook ran by andrew@bullseye

curprev 13:0913:09, 31 October 2022‎ imported>Stashbot‎ 391,527 bytes +171‎ arturo: restart keepalived on all 4 cloudgw servers to run them with `-D` in /etc/default/keepalived to further debug T320975

curprev 16:1816:18, 26 October 2022‎ imported>Stashbot‎ 391,356 bytes +313‎ wm-bot2: Created new flavor: g3.cores1.ram1.disk20 (id:bf48880d-0c1b-4c2a-8e8b-778d28b16561) (T319446) - cookbook ran by dcaro@vulcanus

curprev 16:0316:03, 25 October 2022‎ imported>Stashbot‎ 391,043 bytes +382‎ arturo: [codfw1dev] T321220 root@cloudcontrol2001-dev:~# openstack subnet create magnum --no-dhcp --network 57017d7c-3817-429a-8aa3-b028de82cdcc --ip-version 4 --gateway auto --subnet-range 192.168.0.0/24

curprev 18:5018:50, 24 October 2022‎ imported>Stashbot‎ 390,661 bytes +1,101‎ wm-bot2: Rebooting node cloudcephmon1002.eqiad.wmnet - cookbook ran by andrew@bullseye

curprev 23:2323:23, 20 October 2022‎ imported>Stashbot‎ 389,560 bytes +24,091‎ wm-bot2: Safe reboot of 'cloudvirt1021.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye

curprev 17:3817:38, 15 October 2022‎ imported>Stashbot‎ 365,469 bytes +179‎ taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki Slevinski # T320867

curprev 12:1912:19, 13 October 2022‎ imported>Stashbot‎ 365,290 bytes +719‎ wm-bot2: OSDs (['cloudcephosd1027', 'cloudcephosd1028', 'cloudcephosd1029', 'cloudcephosd1030', 'cloudcephosd1031', 'cloudcephosd1032', 'cloudcephosd1033', 'cloudcephosd1034']) upgraded successfully B-) (T309786) - cookbook ran by dcaro@vulcanus

curprev 14:0114:01, 10 October 2022‎ imported>Stashbot‎ 364,571 bytes +159‎ dcaro: test2

curprev 12:0412:04, 9 October 2022‎ imported>Stashbot‎ 364,412 bytes +176‎ taavi: taavi@cloudweb1003 ~ $ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki DatGuy # T320301

curprev 13:4013:40, 7 October 2022‎ imported>Stashbot‎ 364,236 bytes +531‎ andrewbogott: dhinus is resetting rabbitmq cluster in an attempt to resolve a suspected (by Andrew) split-brain

curprev 15:5515:55, 6 October 2022‎ imported>Stashbot‎ 363,705 bytes +663‎ arturo: cloudnet1005 & cloudnet1006 now in service. Secom cloudnet1003 & cloudnet1004. Drop neutron agents, etc. (T316284)

curprev 14:4014:40, 5 October 2022‎ imported>Stashbot‎ 363,042 bytes +1,152‎ wm-bot2: Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) (T319418) - cookbook ran by fran@wmf3169

curprev 16:4016:40, 4 October 2022‎ imported>Stashbot‎ 361,890 bytes +1,582‎ wm-bot2: Added 1 new OSDs ['cloudcephosd1033.eqiad.wmnet'] (T314870) - cookbook ran by fran@wmf3169

curprev 14:5214:52, 30 September 2022‎ imported>Stashbot‎ 360,308 bytes +2,841‎ wm-bot2: Added 1 new OSDs ['cloudcephosd1031.eqiad.wmnet'] - cookbook ran by fran@wmf3169

curprev 10:4810:48, 27 September 2022‎ imported>Stashbot‎ 357,467 bytes +2,068‎ wm-bot2: Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169

curprev 18:3718:37, 26 September 2022‎ imported>Stashbot‎ 355,399 bytes +13,416‎ wm-bot2: Safe reboot of 'cloudvirt1024.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster

curprev 15:0615:06, 25 September 2022‎ imported>Stashbot‎ 341,983 bytes +2,869‎ wm-bot2: Drained 'cloudvirt1052.eqiad.wmnet'. (T317391) - cookbook ran by andrew@buster

curprev 17:3717:37, 24 September 2022‎ imported>Stashbot‎ 339,114 bytes +186‎ andrewbogott: restarting neutron api on cloudcontrol1006; cause of outage unknown

curprev 15:1415:14, 22 September 2022‎ imported>Stashbot‎ 338,928 bytes +14,397‎ wm-bot2: Safe reboot of 'cloudvirt1025.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster

curprev 21:0221:02, 20 September 2022‎ imported>Stashbot‎ 324,531 bytes +20,280‎ wm-bot2: Safe reboot of 'cloudvirt1037.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster

curprev 20:0720:07, 19 September 2022‎ imported>Stashbot‎ 304,251 bytes +1,214‎ wm-bot2: Safe reboot of 'cloudvirt1026.eqiad.wmnet' finished successfully. (T317391) - cookbook ran by andrew@buster

curprev 16:5716:57, 14 September 2022‎ imported>Stashbot‎ 303,037 bytes +1,804‎ wm-bot2: Finished rebooting node cloudcephosd1030.eqiad.wmnet - cookbook ran by fran@Francesco’s-MacBook-Pro

curprev 12:1612:16, 13 September 2022‎ imported>Stashbot‎ 301,233 bytes +1,263‎ wm-bot2: Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) - cookbook ran by dcaro@vulcanus

curprev 15:3715:37, 10 September 2022‎ imported>Stashbot‎ 299,970 bytes +151‎ andrewbogott: restarting nova-conductor service (and possibly others, in response to lots of unanswered rabbitmq messages)

curprev 19:1219:12, 8 September 2022‎ imported>Stashbot‎ 299,819 bytes +108‎ andrewbogott: restarting nginx on proxy-03.project-proxy.eqiad1.wikimedia.cloud

curprev 10:1810:18, 7 September 2022‎ imported>Stashbot‎ 299,711 bytes +859‎ wm-bot2: Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

curprev 14:5914:59, 30 August 2022‎ imported>Stashbot‎ 298,852 bytes +1,876‎ andrewbogott: manually marking most eqiad1 cloud* servers down in icinga for T296561

curprev 15:1415:14, 25 August 2022‎ imported>Stashbot‎ 296,976 bytes +886‎ wm-bot2: Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

curprev 22:0722:07, 24 August 2022‎ imported>Stashbot‎ 296,090 bytes +983‎ andrewbogott: replaced cloudservices1003 with cloudservices1005 T304888

curprev 13:4613:46, 23 August 2022‎ imported>Stashbot‎ 295,107 bytes +886‎ wm-bot2: Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

curprev 21:1221:12, 21 August 2022‎ imported>Stashbot‎ 294,221 bytes +167‎ andrewbogott: restarted neutron-dhcp-agent on cloudnet1003. it was claiming to be unable to contact Rabbit but seems happy after a restart

curprev 07:3907:39, 20 August 2022‎ imported>Stashbot‎ 294,054 bytes +305‎ dcaro_away: cloudvirt1023 is back up, VMs are starting to recover (T315718)

curprev 17:0617:06, 19 August 2022‎ imported>Stashbot‎ 293,749 bytes +145‎ taavi: [codfw1dev] restart mariadb on clouddb2002-dev to pick up certificate config changes T310795

curprev 13:2513:25, 18 August 2022‎ imported>Stashbot‎ 293,604 bytes +980‎ wm-bot2: Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

curprev 10:5010:50, 17 August 2022‎ imported>Stashbot‎ 292,624 bytes +1,560‎ wm-bot2: Added 1 new OSDs ['cloudcephosd1025.eqiad.wmnet'] (T314870) - cookbook ran by fran@foz

curprev 22:3922:39, 16 August 2022‎ imported>Stashbot‎ 291,064 bytes +2,291‎ andrewbogott: replacing the now-rebuilt cloudvirt1025 in 'ceph' aggregate and removing it from the 'maintenance' aggregate

curprev 18:3618:36, 14 August 2022‎ imported>Stashbot‎ 288,773 bytes +105‎ taavi: deleted the http keystone endpoints from the keystone service catalog

curprev 13:5713:57, 11 August 2022‎ imported>Stashbot‎ 288,668 bytes +1,152‎ andrewbogott: decommissioning cloudcontrol1003 + cloudcontrl1004. I backed up $home in case anyone needs their files.

curprev 13:1013:10, 10 August 2022‎ imported>Stashbot‎ 287,516 bytes +578‎ wm-bot2: Finished rebooting node cloudcephosd1025.eqiad.wmnet (T314870) - cookbook ran by fran@MacBook-Pro.station

curprev 17:1617:16, 4 August 2022‎ imported>Stashbot‎ 286,938 bytes +328‎ taavi: deleted all scheduler_fanout_ rabbit queues in an attempt to fix scheduling

curprev 20:5520:55, 3 August 2022‎ imported>Stashbot‎ 286,610 bytes +266‎ andrewbogott: root@tools-checker-04:~# systemctl restart uwsgi-toolschecker_cron.service

curprev 14:0714:07, 2 August 2022‎ imported>Stashbot‎ 286,344 bytes +375‎ andrewbogott: shutting down codfw1dev ceph cluster according to https://docs.mirantis.com/mcp/q4-18/mcp-operations-guide/scheduled-maintenance-power-outage/power-off-ceph-cluster.html

curprev 19:3219:32, 27 July 2022‎ imported>Stashbot‎ 285,969 bytes +286‎ andrewbogott: switching the openstack.eqiad1.wikimedia.cloud endpoint from cloudcontrol1004 to 1006, https://gerrit.wikimedia.org/r/c/operations/dns/+/817878/2/templates/wikimediacloud.org#54

curprev 13:4313:43, 25 July 2022‎ imported>Stashbot‎ 285,683 bytes +142‎ andrewbogott: pooling cloudweb100[34] and depooling labweb100[12] for testing in prep for decomming labweb100[12]

curprev 16:4116:41, 22 July 2022‎ imported>Stashbot‎ 285,541 bytes +174‎ taavi: depool cloudweb1003/1004 since horizon seems to be having issues

curprev 18:2618:26, 21 July 2022‎ imported>Stashbot‎ 285,367 bytes +216‎ andrewbogott: depooling cloudweb1003 and 1004 for wikitech, horizon, striker -- pending db grant changes

curprev 18:0218:02, 20 July 2022‎ imported>Stashbot‎ 285,151 bytes +785‎ dcaro: things seem stable, trying to bring up a the last rabbit node, cloudcontrol1007 (T313400)

curprev 16:3016:30, 19 July 2022‎ imported>Stashbot‎ 284,366 bytes +34,154‎ wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
curprev 00:4600:46, 19 July 2022‎ imported>Stashbot‎ 250,212 bytes +15,548‎ wm-bot2: Set cloudvirt 'cloudvirt1024.eqiad.wmnet' maintenance (downtime id: 301309a2-a8b5-4698-98d6-bb4aa0a75e45, use this to unset). - cookbook ran by andrew@buster

curprev 14:4814:48, 13 July 2022‎ imported>Stashbot‎ 234,664 bytes +75‎ bd808: Added Tavvi as member of #acl*wmcs-team

curprev 12:0412:04, 12 July 2022‎ imported>Stashbot‎ 234,589 bytes +1,130‎ wm-bot2: Ceph cluster at {self.deployment} set out of maintenance. - cookbook ran by dcaro@vulcanus

curprev 15:5715:57, 8 July 2022‎ imported>Stashbot‎ 233,459 bytes +311‎ wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

curprev 07:1707:17, 7 July 2022‎ imported>Stashbot‎ 233,148 bytes +271‎ wm-bot2: Finished rebooting node cloudcephosd1015.eqiad.wmnet (T312509) - cookbook ran by dcaro@vulcanus

curprev 17:5017:50, 6 July 2022‎ imported>Stashbot‎ 232,877 bytes +258‎ wm-bot2: Set the ceph cluster for eqiad1 in maintenance, alert silence ids: ['8a5b9eee-48c0-474d-8277-faeb05a2ea61', '65aad0fc-d887-47a3-b20c-d1ed461a2411', '86b078ae-3a27-4063-8c7c-198a2fe0c172'] - cookbook ran by dcaro@vulcanus

curprev 13:2713:27, 4 July 2022‎ imported>Stashbot‎ 232,619 bytes +6,078‎ wm-bot2: Rebooting cloudgw host cloudgw1002.eqiad.wmnet - cookbook ran by dcaro@vulcanus

curprev 21:2721:27, 3 July 2022‎ imported>Stashbot‎ 226,541 bytes +160‎ andrewbogott: rebuilding rabbit cluster in codfw1dev to get rid of some queues so unresponsive that they can't otherwise be deleted

curprev 11:0511:05, 2 July 2022‎ imported>Stashbot‎ 226,381 bytes +137‎ wm-bot2: Rebooting all the cloudnet nodes cloudnet2006-dev,cloudnet2005-dev - cookbook ran by dcaro@vulcanus

curprev 15:3515:35, 1 July 2022‎ imported>Stashbot‎ 226,244 bytes +3,845‎ wm-bot2: Finished rebooting the cloudnet nodes ['cloudnet2006-dev', 'cloudnet2005-dev'] - cookbook ran by dcaro@vulcanus

curprev 18:1718:17, 30 June 2022‎ imported>Stashbot‎ 222,399 bytes +533‎ wm-bot2: Rebooted cloudnet host cloudnet2005-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus

curprev 08:4508:45, 29 June 2022‎ imported>Stashbot‎ 221,866 bytes +217‎ wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet - cookbook ran by dcaro@vulcanus

curprev 13:0313:03, 28 June 2022‎ imported>Stashbot‎ 221,649 bytes +137‎ taavi: grant the tools project access to the g3.cores16.ram64.disk20.10xiops flavor T301949

curprev 16:4816:48, 22 June 2022‎ imported>Stashbot‎ 221,512 bytes +278‎ taavi: restart designate-*.service on both cloudservices nodes

curprev 04:4804:48, 21 June 2022‎ imported>Stashbot‎ 221,234 bytes +230‎ andrewbogott: stopping nova-fullstack agent on cloudcontrol1003; it's going to page us otherwise and we're all AFK tomorrow

curprev 17:5317:53, 17 June 2022‎ imported>Stashbot‎ 221,004 bytes +251‎ andrewbogott: switching to a new python-based health check for galera and haproxy. This may make things more stable, or it may not. T310664

curprev 11:3411:34, 15 June 2022‎ imported>Stashbot‎ 220,753 bytes +86‎ taavi: restart neutron-linuxbridge-agent on cloudvirt1022

curprev 16:2616:26, 14 June 2022‎ imported>Stashbot‎ 220,667 bytes +1,288‎ wm-bot2: OSDs (['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloudcephosd1021', 'cl

curprev 11:1411:14, 13 June 2022‎ imported>Stashbot‎ 219,379 bytes +876‎ wm-bot2: Finished rebooting node cloudcephosd1021.eqiad.wmnet (T309789) - cookbook ran by dcaro@vulcanus

curprev 13:2113:21, 6 June 2022‎ imported>Stashbot‎ 218,503 bytes +133‎ andrewbogott: restarting mysql/galera on cloudcontrol100x in an attempt to stabilize some flapping there

curprev 07:5107:51, 2 June 2022‎ imported>Stashbot‎ 218,370 bytes +99‎ taavi: restart neutron-linuxbridge-agent.service on cloudvirt1034 T309732
curprev 00:3700:37, 2 June 2022‎ imported>Stashbot‎ 218,271 bytes +388‎ andrewbogott: updated nameservers for codfw1dev instances via 'openstack subnet set --dns-nameserver etc.'

curprev 17:4317:43, 30 May 2022‎ imported>Stashbot‎ 217,883 bytes +110‎ andrewbogott: restarting neutron-rpc and neutron-api services on cloudcontrol1xxx

curprev 14:5514:55, 29 May 2022‎ imported>Stashbot‎ 217,773 bytes +223‎ andrewbogott: restarting nova services on all eqiad1 cloudcontrol nodes to recover from rabbit breakage

curprev 20:0120:01, 25 May 2022‎ imported>Stashbot‎ 217,550 bytes +175‎ balloons: clean up cinder backup a bit, restart service due to network outage

curprev 15:2115:21, 19 May 2022‎ imported>Stashbot‎ 217,375 bytes +176‎ andrewbogott: resetting password for the 'troveguest' rabbitmq user. I think I may have broken this during a recent rebuild of the rabbitmq cluster

curprev 15:4215:42, 18 May 2022‎ imported>Stashbot‎ 217,199 bytes +109‎ andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build

curprev 11:3311:33, 14 May 2022‎ imported>Stashbot‎ 217,090 bytes +103‎ taavi: deleted projects 'ores' and 'ores-staging' T308102

curprev 06:2006:20, 13 May 2022‎ imported>Stashbot‎ 216,987 bytes +11,620‎ wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
curprev 01:0001:00, 13 May 2022‎ imported>Stashbot‎ 205,367 bytes +14,074‎ wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster

curprev 18:4818:48, 11 May 2022‎ imported>Stashbot‎ 191,293 bytes +3,204‎ wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster

curprev 21:4321:43, 10 May 2022‎ imported>Stashbot‎ 188,089 bytes +3,450‎ wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster

curprev 01:3301:33, 7 May 2022‎ imported>Stashbot‎ 184,639 bytes +483‎ wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster

curprev 20:3820:38, 3 May 2022‎ imported>Stashbot‎ 184,156 bytes +173‎ andrewbogott: upgrading clouddb2001-dev in place

curprev 16:5616:56, 2 May 2022‎ imported>Stashbot‎ 183,983 bytes +62‎ dcaro: rebooting cloudmetrics1001

curprev 14:2214:22, 29 April 2022‎ imported>Stashbot‎ 183,921 bytes +270‎ andrewbogott: changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions T277653 https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline

curprev 14:5114:51, 27 April 2022‎ imported>Stashbot‎ 183,651 bytes +8,234‎ wm-bot: Finished rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloud

curprev 10:3610:36, 26 April 2022‎ imported>Stashbot‎ 175,417 bytes +227‎ taavi: [codfw1dev] updated designate pool to 2004/2005-dev according to the instructions on https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#Initial_designate/pdns_node_setup

curprev 10:3310:33, 22 April 2022‎ imported>Stashbot‎ 175,190 bytes +130‎ taavi: [codfw1dev] restart designate-sink on both new cloudservices host to fix rabbitmq connectivity

curprev 05:3805:38, 21 April 2022‎ imported>Stashbot‎ 175,060 bytes +100‎ andrewbogott: replaced cloudservices200[2,3] with cloudservices200[4,5]

curprev 15:2915:29, 19 April 2022‎ imported>Stashbot‎ 174,960 bytes +92‎ andrewbogott: stopping all VMs on cloudvirt1019, reimaging host

curprev 15:2315:23, 18 April 2022‎ imported>Stashbot‎ 174,868 bytes +205‎ andrewbogott: reimaging cloudvirt1020, leaving VMs in place

curprev 20:1420:14, 14 April 2022‎ imported>Stashbot‎ 174,663 bytes +147‎ andrewbogott: restarting nova-api and nova-conductor services in a superstitious attempt to reduce open DB connections

curprev 22:0122:01, 13 April 2022‎ imported>Stashbot‎ 174,516 bytes +116‎ andrewbogott: restarting galera on cloudcontrols (one by one) to clear open connections

curprev 15:5915:59, 11 April 2022‎ imported>Stashbot‎ 174,400 bytes +71‎ taavi: created cloudinfra.wmcloud.org zone

curprev 19:5519:55, 9 April 2022‎ imported>Stashbot‎ 174,329 bytes +236‎ andrewbogott: reimaging cloudbackup1001-dev to bullseye

curprev 12:5112:51, 7 April 2022‎ imported>Stashbot‎ 174,093 bytes +152‎ wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T305631) - cookbook ran by arturo@nostromo

curprev 09:1209:12, 6 April 2022‎ imported>Stashbot‎ 173,941 bytes +713‎ arturo: [codf1dev] installing python3-eventlet 0.30.2-5~bpo11+1 on all required servers (cloudvirt, cloudnet, cloudcontrol) (T305157)

curprev 11:2011:20, 30 March 2022‎ imported>Stashbot‎ 173,228 bytes +114‎ arturo: apply urpf strict filter to eqiad cloud-hosts vlan - T285461

curprev 10:0210:02, 29 March 2022‎ imported>Stashbot‎ 173,114 bytes +82‎ dcaro: restarting keystone (T304918)

curprev 22:5322:53, 23 March 2022‎ imported>Stashbot‎ 173,032 bytes +6,374‎ wm-bot: Drained 'cloudvirt1045.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

curprev 22:5922:59, 22 March 2022‎ imported>Stashbot‎ 166,658 bytes +263‎ wm-bot: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

curprev 01:0901:09, 17 March 2022‎ imported>Stashbot‎ 166,395 bytes +509‎ wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

curprev 20:5820:58, 15 March 2022‎ imported>Stashbot‎ 165,886 bytes +941‎ wm-bot: Drained 'cloudvirt1026.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

curprev 21:2421:24, 14 March 2022‎ imported>Stashbot‎ 164,945 bytes +3,074‎ wm-bot: Drained 'cloudvirt1025.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

curprev 18:2918:29, 8 March 2022‎ imported>Stashbot‎ 161,871 bytes +3,117‎ wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

curprev 08:4908:49, 3 March 2022‎ imported>Stashbot‎ 158,754 bytes +105‎ taavi: deploying cloudmetrics grafana to grafana 8, T282863

curprev 09:0609:06, 2 March 2022‎ imported>Stashbot‎ 158,649 bytes +138‎ arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/701347

curprev 15:3015:30, 28 February 2022‎ imported>Stashbot‎ 158,511 bytes +132‎ dcaro: cleaning up leftover snapshots from failed backups of the maps volume (T302720)

curprev 17:0417:04, 24 February 2022‎ imported>Stashbot‎ 158,379 bytes +464‎ andrewbogott: upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server'

curprev 20:3920:39, 23 February 2022‎ imported>Stashbot‎ 157,915 bytes +1,620‎ taavi: added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account T295246

curprev 22:1022:10, 22 February 2022‎ imported>Stashbot‎ 156,295 bytes +271‎ andrewbogott: raising project 'maps' quota by two tb -- T300160

curprev 21:5721:57, 18 February 2022‎ imported>Stashbot‎ 156,024 bytes +484‎ andrewbogott: leaving cloudcontrol1003 downtimed with disabled puppet for the weekend. Everything there should be stable and fine save rabbit which needs an upgrade.

curprev 23:0223:02, 17 February 2022‎ imported>Stashbot‎ 155,540 bytes +116‎ andrewbogott: in-place upgrade to Bullseye on cloudcontrol1005 T281276

curprev 14:1514:15, 15 February 2022‎ imported>Stashbot‎ 155,424 bytes +175‎ taavi: [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account T295246

curprev 10:1210:12, 4 February 2022‎ imported>Stashbot‎ 155,249 bytes +107‎ arturo: restart backup_vms service in cloudvirt1024 (T300956)

curprev 08:2108:21, 3 February 2022‎ imported>Stashbot‎ 155,142 bytes +226‎ taavi: cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing "performing a change every puppet run" alert)

curprev 02:3602:36, 2 February 2022‎ imported>Stashbot‎ 154,916 bytes +81‎ andrewbogott: restarting mariadb on cloudcontrol1004

curprev 10:1510:15, 31 January 2022‎ imported>Stashbot‎ 154,835 bytes +146‎ arturo: cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert)

curprev 18:2418:24, 29 January 2022‎ imported>Stashbot‎ 154,689 bytes +102‎ taavi: delete 2 puppet prefixes in a weird state T299750

curprev 13:2413:24, 27 January 2022‎ imported>Stashbot‎ 154,587 bytes +146‎ arturo: cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service (T300138)

curprev 19:0919:09, 26 January 2022‎ imported>Stashbot‎ 154,441 bytes +157‎ andrewbogott: bootstrapping a fresh galera node on cloudcontrol1004

curprev 10:4910:49, 25 January 2022‎ imported>Stashbot‎ 154,284 bytes +170‎ arturo: made cloudmetrics1001/1002 primary/backup respectively (T299744, T297814, T300011)

curprev 16:3816:38, 19 January 2022‎ imported>Stashbot‎ 154,114 bytes +121‎ andrewbogott: moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud

curprev 03:1103:11, 5 January 2022‎ imported>Stashbot‎ 153,993 bytes +227‎ andrewbogott: 'cp /etc/apt/sources.list /etc/apt/sources.list.prepuppet' on all VMs. Backing up state before puppetizing sources.list with https://gerrit.wikimedia.org/r/c/operations/puppet/+/751498

curprev 12:4412:44, 4 January 2022‎ imported>Stashbot‎ 153,766 bytes +83‎ dcaro: increasing the size_limit for labs ldap servers

curprev 16:5516:55, 26 December 2021‎ imported>Stashbot‎ 153,683 bytes +110‎ majavah: run attachLdapUser.php on wikitech for developer account "Karthiksripal"

curprev 22:5122:51, 24 December 2021‎ imported>Stashbot‎ 153,573 bytes +99‎ majavah: ran the wikireplica dns script on s5 T298303

curprev 21:4221:42, 23 December 2021‎ imported>Stashbot‎ 153,474 bytes +116‎ majavah: deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies

curprev 10:3910:39, 21 December 2021‎ imported>Stashbot‎ 153,358 bytes +110‎ arturo: dropped egress NAT exceptions for WMF apt repos, T298042

curprev 12:4412:44, 15 December 2021‎ imported>Stashbot‎ 153,248 bytes +143‎ dcaro: Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed (T297454)

curprev 10:2610:26, 14 December 2021‎ imported>Stashbot‎ 153,105 bytes +258‎ dcaro: Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/<canary_image_id>) to the root disk on cloudvirt-wdqs1001 to temporary free some space (T297454)

curprev 18:0818:08, 13 December 2021‎ imported>Stashbot‎ 152,847 bytes +1,373‎ wm-bot: Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse

curprev 18:5618:56, 3 December 2021‎ imported>Stashbot‎ 151,474 bytes +176‎ andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020

curprev 01:1701:17, 2 December 2021‎ imported>Stashbot‎ 151,298 bytes +1,832‎ wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster

curprev 17:4817:48, 28 November 2021‎ imported>Stashbot‎ 149,466 bytes +209‎ andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for T296592. It will need to be moved back after the raid is rebuilt.

curprev 07:1907:19, 21 November 2021‎ imported>Stashbot‎ 149,257 bytes +120‎ dcaro_away: restarting designate-sink with some extra logs in it (T296144)

curprev 15:4815:48, 17 November 2021‎ imported>Stashbot‎ 149,137 bytes +280‎ andrewbogott: upgrading mariadb packages on eqiad1 cloudcontrols

curprev 13:3113:31, 12 November 2021‎ imported>Stashbot‎ 148,857 bytes +142‎ arturo: restarting glance-api services to make sure they work with new ceph auth creds (T293752)

curprev 21:5021:50, 8 November 2021‎ imported>Stashbot‎ 148,715 bytes +620‎ andrewbogott: returned clouddb pools back to normal after maintain_views run: https://gerrit.wikimedia.org/r/c/operations/puppet/+/737505 T216481

curprev 11:1811:18, 5 November 2021‎ imported>Stashbot‎ 148,095 bytes +742‎ wm-bot: Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance

curprev 16:3916:39, 4 November 2021‎ imported>Stashbot‎ 147,353 bytes +2,597‎ wm-bot: Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance

curprev 17:2217:22, 3 November 2021‎ imported>Stashbot‎ 144,756 bytes +279‎ arturo: [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev (T294956)

curprev 10:5410:54, 2 November 2021‎ imported>Stashbot‎ 144,477 bytes +179‎ arturo: rebooting cloudnet1004/1003 for T291813

curprev 00:4700:47, 24 October 2021‎ imported>Stashbot‎ 144,298 bytes +166‎ andrewbogott: deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738

curprev 10:1910:19, 21 October 2021‎ imported>Stashbot‎ 144,132 bytes +227‎ arturo: drop firewall exception on core routers for wiki replicas legacy setup (T293897)

curprev 21:0621:06, 20 October 2021‎ imported>Stashbot‎ 143,905 bytes +99‎ andrewbogott: creating cloudinfra-nfs project T293936

curprev 19:2119:21, 18 October 2021‎ imported>Stashbot‎ 143,806 bytes +252‎ andrewbogott: also ticked the 'admin' box on wikitech for majavah T292827

curprev 12:2812:28, 14 October 2021‎ imported>Stashbot‎ 143,554 bytes +149‎ arturo: [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB (T292546)

curprev 10:4610:46, 13 October 2021‎ imported>Stashbot‎ 143,405 bytes +105‎ arturo: updating python3-neutron across the fleet (T292936)

curprev 09:0609:06, 12 October 2021‎ imported>Stashbot‎ 143,300 bytes +200‎ dcaro: upgrading eqiad cloudnet hosts neutron packages (T292936)

curprev 09:3909:39, 5 October 2021‎ imported>Stashbot‎ 143,100 bytes +152‎ arturo: [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) T291257

curprev 14:5014:50, 30 September 2021‎ imported>Stashbot‎ 142,948 bytes +391‎ andrewbogott: sudo cumin "cloud*" "ps -ef | grep nslcd && service nslcd restart" and sudo cumin "lab*" "ps -ef | grep nslcd && service nslcd restart" T292202

curprev 09:4109:41, 29 September 2021‎ imported>Stashbot‎ 142,557 bytes +196‎ arturo: [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured (T291257)

curprev 16:2316:23, 28 September 2021‎ imported>Stashbot‎ 142,361 bytes +531‎ bstorm: downtime for clouddb1020 to reduce re-pages in case this goes badly T291963

curprev 10:0710:07, 27 September 2021‎ imported>Stashbot‎ 141,830 bytes +169‎ arturo: cloudcontrol1004 apparently healthy T291446

curprev 13:0213:02, 24 September 2021‎ imported>Stashbot‎ 141,661 bytes +211‎ arturo: [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev

curprev 12:1312:13, 21 September 2021‎ imported>Stashbot‎ 141,450 bytes +677‎ arturo: [codfw1dev] trying to create a manila service image (T291257)

curprev 23:0823:08, 20 September 2021‎ imported>Stashbot‎ 140,773 bytes +408‎ bstorm: ran `echo check > /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid

curprev 11:3511:35, 17 September 2021‎ imported>Stashbot‎ 140,365 bytes +114‎ arturo: [codfw1dev] install manila on cloudcontrol2001-dev (T291257)

curprev 15:5615:56, 16 September 2021‎ imported>Stashbot‎ 140,251 bytes +134‎ bstorm: removing downtime for labstore1005 so we'll know if it has another issue T290318

curprev 22:0322:03, 9 September 2021‎ imported>Stashbot‎ 140,117 bytes +315‎ bstorm: restarted the prometheus-mysqld-exporter@s1 service as it was not working T290630

curprev 15:3415:34, 3 September 2021‎ imported>Stashbot‎ 139,802 bytes +365‎ bstorm: rebooting labstore1005 to disconnect the drives from labstore1004 T290318

curprev 16:1616:16, 30 August 2021‎ imported>Stashbot‎ 139,437 bytes +825‎ wm-bot: Added 1 new OSDs ['cloudcephosd1018.eqiad.wmnet'] - cookbook ran by andrew@buster

curprev 18:5718:57, 27 August 2021‎ imported>Stashbot‎ 138,612 bytes +126‎ andrewbogott: raising toolsbeta ram/core/instances quotas so majavah can experiment with bullseye

curprev 14:4514:45, 25 August 2021‎ imported>Stashbot‎ 138,486 bytes +534‎ wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster

curprev 17:3917:39, 19 August 2021‎ imported>Stashbot‎ 137,952 bytes +93‎ bstorm: restarting glance image backup to try and clear the page

curprev 16:2116:21, 18 August 2021‎ imported>Stashbot‎ 137,859 bytes +899‎ wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster

curprev 15:1115:11, 17 August 2021‎ imported>Stashbot‎ 136,960 bytes +119‎ andrewbogott: rebooting cloudcephosd1008 to force raid rebuild -- T287838

curprev 13:5113:51, 11 August 2021‎ imported>Stashbot‎ 136,841 bytes +480‎ wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus

curprev 15:1515:15, 10 August 2021‎ imported>Stashbot‎ 136,361 bytes +214‎ andrewbogott: restarting all designate services in eqiad1

curprev 09:3709:37, 5 August 2021‎ imported>Stashbot‎ 136,147 bytes +106‎ dcaro: Taking one osd daemon down ot codfw cluster (T288203)

curprev 19:2019:20, 4 August 2021‎ imported>Stashbot‎ 136,041 bytes +126‎ bd808: Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki

curprev 17:4017:40, 3 August 2021‎ imported>Stashbot‎ 135,915 bytes +85‎ bstorm: rerunning the glance backup script after failure

curprev 00:1000:10, 31 July 2021‎ imported>Stashbot‎ 135,830 bytes +233‎ andrewbogott: "systemctl reset-failed cloud-init.service" on all VMs for T287309

curprev 21:3221:32, 27 July 2021‎ imported>Stashbot‎ 135,597 bytes +313‎ andrewbogott: putting cloudvirt1012 back into service T286748

curprev 15:2215:22, 23 July 2021‎ imported>Stashbot‎ 135,284 bytes +88‎ bstorm: update wikireplicas-dns for s7 fix for web replicas

curprev 17:0717:07, 20 July 2021‎ imported>Stashbot‎ 135,196 bytes +215‎ andrewbogott: reloading haproxy on dbproxy1018 for T286598
curprev 00:1000:10, 20 July 2021‎ imported>Stashbot‎ 134,981 bytes +465‎ bstorm: restarting nova-api on cloudcontrol1003 to try and recover whatever it's doing with designate_floating_ip_ptr_records_updater

curprev 09:5509:55, 16 July 2021‎ imported>Stashbot‎ 134,516 bytes +103‎ dcaro: checking HP raid issues on coludvirt1012 (T286766)

curprev 21:0821:08, 14 July 2021‎ imported>Stashbot‎ 134,413 bytes +316‎ andrewbogott: restarting lots of openstack services while trying to resolve T286675

curprev 10:1210:12, 2 July 2021‎ imported>Stashbot‎ 134,097 bytes +1,731‎ wm-bot: The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus

curprev 16:2716:27, 1 July 2021‎ imported>Stashbot‎ 132,366 bytes +2,402‎ bstorm: failed over cloudstore1009 to cloudstore1008 T224747

curprev 21:4821:48, 30 June 2021‎ imported>Stashbot‎ 129,964 bytes +115‎ bstorm: downtimed space alerts for scratch on cloudstore1008 until after the migration

curprev 15:2815:28, 25 June 2021‎ imported>Stashbot‎ 129,849 bytes +238‎ andrewbogott: restarting openstack services on cloudcontrol1005

curprev 13:5413:54, 21 June 2021‎ imported>Stashbot‎ 129,611 bytes +228‎ dcaro: puppet fix merged and deployed, servers are back to normal

curprev 22:2122:21, 20 June 2021‎ imported>Stashbot‎ 129,383 bytes +144‎ andrewbogott: clearing admin-monitoring VMs; puppet has been failing lately due to a full drive on the puppetmaster

curprev 01:1801:18, 15 June 2021‎ imported>Stashbot‎ 129,239 bytes +130‎ bstorm: running a modified version of the prometheus dir size cron in screen T284964

curprev 10:1310:13, 14 June 2021‎ imported>Stashbot‎ 129,109 bytes +110‎ dcaro: setting ssd to debug mode on tools-sgeexec-0917 (T284130)

curprev 10:5810:58, 10 June 2021‎ imported>Stashbot‎ 128,999 bytes +3,910‎ wm-bot: Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] (T281248) - cookbook ran by dcaro@vulcanus

curprev 17:3317:33, 9 June 2021‎ imported>Stashbot‎ 125,089 bytes +1,815‎ arturo: removed icinga downtime for cloudmetrics1002 -- to see if hardware is healthy (T281881)

curprev 23:1923:19, 8 June 2021‎ imported>Stashbot‎ 123,274 bytes +2,253‎ bd808: Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 (T281881)

Nova Resource:Admin/SAL: Revision history

28 March 2023

27 March 2023

22 March 2023

8 March 2023

7 March 2023

3 March 2023

2 March 2023

28 February 2023

27 February 2023

22 February 2023

21 February 2023

16 February 2023

15 February 2023

14 February 2023

13 February 2023

9 February 2023

8 February 2023

7 February 2023

4 February 2023

3 February 2023

2 February 2023

30 January 2023

27 January 2023

26 January 2023

22 January 2023

20 January 2023

19 January 2023

18 January 2023

17 January 2023

13 January 2023

12 January 2023

6 January 2023

4 January 2023

3 January 2023

25 December 2022

22 December 2022

21 December 2022

17 December 2022

16 December 2022

7 December 2022

3 December 2022

30 November 2022

28 November 2022

25 November 2022

24 November 2022

23 November 2022

22 November 2022

21 November 2022

18 November 2022

16 November 2022

14 November 2022

11 November 2022

10 November 2022

8 November 2022

7 November 2022

4 November 2022

3 November 2022

31 October 2022

26 October 2022

25 October 2022

24 October 2022

20 October 2022

15 October 2022

13 October 2022

10 October 2022

9 October 2022

7 October 2022

6 October 2022

5 October 2022

4 October 2022

30 September 2022

27 September 2022

26 September 2022

25 September 2022

24 September 2022

22 September 2022

20 September 2022

19 September 2022

14 September 2022