You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Admin/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)

18 May 2022

  • curprev 15:4215:42, 18 May 2022imported>Stashbot 217,199 bytes +109 andrewbogott: updated the 'debian-11.0-bullseye' glance image with a fresh build

14 May 2022

  • curprev 11:3311:33, 14 May 2022imported>Stashbot 217,090 bytes +103 taavi: deleted projects 'ores' and 'ores-staging' T308102

13 May 2022

  • curprev 06:2006:20, 13 May 2022imported>Stashbot 216,987 bytes +11,620 wm-bot2: Safe reboot of 'cloudvirt1045.eqiad.wmnet' finished successfully. - cookbook ran by andrew@buster
  • curprev 01:0001:00, 13 May 2022imported>Stashbot 205,367 bytes +14,074 wm-bot2: Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@buster

11 May 2022

  • curprev 18:4818:48, 11 May 2022imported>Stashbot 191,293 bytes +3,204 wm-bot2: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster

10 May 2022

  • curprev 21:4321:43, 10 May 2022imported>Stashbot 188,089 bytes +3,450 wm-bot: Set cloudvirt 'cloudvirt1021.eqiad.wmnet' maintenance. - cookbook ran by andrew@buster

7 May 2022

  • curprev 01:3301:33, 7 May 2022imported>Stashbot 184,639 bytes +483 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. - cookbook ran by andrew@buster

3 May 2022

  • curprev 20:3820:38, 3 May 2022imported>Stashbot 184,156 bytes +173 andrewbogott: upgrading clouddb2001-dev in place

2 May 2022

29 April 2022

  • curprev 14:2214:22, 29 April 2022imported>Stashbot 183,921 bytes +270 andrewbogott: changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions T277653 https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline

27 April 2022

  • curprev 14:5114:51, 27 April 2022imported>Stashbot 183,651 bytes +8,234 wm-bot: Finished rebooting the nodes ['cloudcephosd1001', 'cloudcephosd1002', 'cloudcephosd1003', 'cloudcephosd1004', 'cloudcephosd1005', 'cloudcephosd1006', 'cloudcephosd1007', 'cloudcephosd1008', 'cloudcephosd1009', 'cloudcephosd1010', 'cloudcephosd1011', 'cloudcephosd1012', 'cloudcephosd1013', 'cloudcephosd1014', 'cloudcephosd1015', 'cloudcephosd1016', 'cloudcephosd1017', 'cloudcephosd1018', 'cloudcephosd1019', 'cloudcephosd1020', 'cloud

26 April 2022

  • curprev 10:3610:36, 26 April 2022imported>Stashbot 175,417 bytes +227 taavi: [codfw1dev] updated designate pool to 2004/2005-dev according to the instructions on https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/DNS/Designate#Initial_designate/pdns_node_setup

22 April 2022

  • curprev 10:3310:33, 22 April 2022imported>Stashbot 175,190 bytes +130 taavi: [codfw1dev] restart designate-sink on both new cloudservices host to fix rabbitmq connectivity

21 April 2022

  • curprev 05:3805:38, 21 April 2022imported>Stashbot 175,060 bytes +100 andrewbogott: replaced cloudservices200[2,3] with cloudservices200[4,5]

19 April 2022

  • curprev 15:2915:29, 19 April 2022imported>Stashbot 174,960 bytes +92 andrewbogott: stopping all VMs on cloudvirt1019, reimaging host

18 April 2022

  • curprev 15:2315:23, 18 April 2022imported>Stashbot 174,868 bytes +205 andrewbogott: reimaging cloudvirt1020, leaving VMs in place

14 April 2022

  • curprev 20:1420:14, 14 April 2022imported>Stashbot 174,663 bytes +147 andrewbogott: restarting nova-api and nova-conductor services in a superstitious attempt to reduce open DB connections

13 April 2022

  • curprev 22:0122:01, 13 April 2022imported>Stashbot 174,516 bytes +116 andrewbogott: restarting galera on cloudcontrols (one by one) to clear open connections

11 April 2022

9 April 2022

  • curprev 19:5519:55, 9 April 2022imported>Stashbot 174,329 bytes +236 andrewbogott: reimaging cloudbackup1001-dev to bullseye

7 April 2022

  • curprev 12:5112:51, 7 April 2022imported>Stashbot 174,093 bytes +152 wm-bot: Set cloudvirt 'cloudvirt1016.eqiad.wmnet' maintenance. (T305631) - cookbook ran by arturo@nostromo

6 April 2022

  • curprev 09:1209:12, 6 April 2022imported>Stashbot 173,941 bytes +713 arturo: [codf1dev] installing python3-eventlet 0.30.2-5~bpo11+1 on all required servers (cloudvirt, cloudnet, cloudcontrol) (T305157)

30 March 2022

  • curprev 11:2011:20, 30 March 2022imported>Stashbot 173,228 bytes +114 arturo: apply urpf strict filter to eqiad cloud-hosts vlan - T285461

29 March 2022

23 March 2022

  • curprev 22:5322:53, 23 March 2022imported>Stashbot 173,032 bytes +6,374 wm-bot: Drained 'cloudvirt1045.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

22 March 2022

  • curprev 22:5922:59, 22 March 2022imported>Stashbot 166,658 bytes +263 wm-bot: Set cloudvirt 'cloudvirt1027.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

17 March 2022

  • curprev 01:0901:09, 17 March 2022imported>Stashbot 166,395 bytes +509 wm-bot: Drained 'cloudvirt1016.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

15 March 2022

  • curprev 20:5820:58, 15 March 2022imported>Stashbot 165,886 bytes +941 wm-bot: Drained 'cloudvirt1026.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

14 March 2022

  • curprev 21:2421:24, 14 March 2022imported>Stashbot 164,945 bytes +3,074 wm-bot: Drained 'cloudvirt1025.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

8 March 2022

  • curprev 18:2918:29, 8 March 2022imported>Stashbot 161,871 bytes +3,117 wm-bot: Set cloudvirt 'cloudvirt1022.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

3 March 2022

  • curprev 08:4908:49, 3 March 2022imported>Stashbot 158,754 bytes +105 taavi: deploying cloudmetrics grafana to grafana 8, T282863

2 March 2022

  • curprev 09:0609:06, 2 March 2022imported>Stashbot 158,649 bytes +138 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/701347

28 February 2022

  • curprev 15:3015:30, 28 February 2022imported>Stashbot 158,511 bytes +132 dcaro: cleaning up leftover snapshots from failed backups of the maps volume (T302720)

24 February 2022

  • curprev 17:0417:04, 24 February 2022imported>Stashbot 158,379 bytes +464 andrewbogott: upgrading eqiad1 and codfw1dev to mariadb 10.5.15+maria~bullseye via 'apt-get install libmariadb3:amd64 galera-4 mariadb-server'

23 February 2022

  • curprev 20:3920:39, 23 February 2022imported>Stashbot 157,915 bytes +1,620 taavi: added domain-wide 'designateadmin' and 'observer' roles to project-proxy-dns-manager service account T295246

22 February 2022

18 February 2022

  • curprev 21:5721:57, 18 February 2022imported>Stashbot 156,024 bytes +484 andrewbogott: leaving cloudcontrol1003 downtimed with disabled puppet for the weekend. Everything there should be stable and fine save rabbit which needs an upgrade.

17 February 2022

  • curprev 23:0223:02, 17 February 2022imported>Stashbot 155,540 bytes +116 andrewbogott: in-place upgrade to Bullseye on cloudcontrol1005 T281276

15 February 2022

  • curprev 14:1514:15, 15 February 2022imported>Stashbot 155,424 bytes +175 taavi: [codfw1dev] added domain-wide 'designateadmin' and 'observer' roles to codfw1dev-proxy-dns-manager service account T295246

4 February 2022

  • curprev 10:1210:12, 4 February 2022imported>Stashbot 155,249 bytes +107 arturo: restart backup_vms service in cloudvirt1024 (T300956)

3 February 2022

  • curprev 08:2108:21, 3 February 2022imported>Stashbot 155,142 bytes +226 taavi: cloudmetrics1004: manually added an empty line to /etc/prometheus/blackbox.yml to make /usr/local/bin/blackbox-exporter-assemble happy (clearing "performing a change every puppet run" alert)

2 February 2022

31 January 2022

  • curprev 10:1510:15, 31 January 2022imported>Stashbot 154,835 bytes +146 arturo: cloudcontrol1005:~$ sudo systemctl restart backup_glance_images.service (failed state, no logs, icinga alert)

29 January 2022

27 January 2022

  • curprev 13:2413:24, 27 January 2022imported>Stashbot 154,587 bytes +146 arturo: cloudmetrics1004:~ $ sudo systemctl restart wmcs_monitoring_graphite_rsync.service (T300138)

26 January 2022

  • curprev 19:0919:09, 26 January 2022imported>Stashbot 154,441 bytes +157 andrewbogott: bootstrapping a fresh galera node on cloudcontrol1004

25 January 2022

  • curprev 10:4910:49, 25 January 2022imported>Stashbot 154,284 bytes +170 arturo: made cloudmetrics1001/1002 primary/backup respectively (T299744, T297814, T300011)

19 January 2022

  • curprev 16:3816:38, 19 January 2022imported>Stashbot 154,114 bytes +121 andrewbogott: moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud

5 January 2022

  • curprev 03:1103:11, 5 January 2022imported>Stashbot 153,993 bytes +227 andrewbogott: 'cp /etc/apt/sources.list /etc/apt/sources.list.prepuppet' on all VMs. Backing up state before puppetizing sources.list with https://gerrit.wikimedia.org/r/c/operations/puppet/+/751498

4 January 2022

26 December 2021

  • curprev 16:5516:55, 26 December 2021imported>Stashbot 153,683 bytes +110 majavah: run attachLdapUser.php on wikitech for developer account "Karthiksripal"

24 December 2021

23 December 2021

  • curprev 21:4221:42, 23 December 2021imported>Stashbot 153,474 bytes +116 majavah: deployed horizon wmf-proxy-dashboard update to fix editing of existing proxies

21 December 2021

15 December 2021

  • curprev 12:4412:44, 15 December 2021imported>Stashbot 153,248 bytes +143 dcaro: Downtiming cloudvirt-wdqs1001 as it has no VMs running until disk space is fixed (T297454)

14 December 2021

  • curprev 10:2610:26, 14 December 2021imported>Stashbot 153,105 bytes +258 dcaro: Moved the nova cache (/var/lib/nova/instances/_base) and the canary image local data (/var/lib/nova/instance/<canary_image_id>) to the root disk on cloudvirt-wdqs1001 to temporary free some space (T297454)

13 December 2021

  • curprev 18:0818:08, 13 December 2021imported>Stashbot 152,847 bytes +1,373 wm-bot: Drained 'cloudvirt1014.eqiad.wmnet'. - cookbook ran by michael@mouse

3 December 2021

  • curprev 18:5618:56, 3 December 2021imported>Stashbot 151,474 bytes +176 andrewbogott: maintain-views and maintain-meta-p on clouddb1013-1020

2 December 2021

  • curprev 01:1701:17, 2 December 2021imported>Stashbot 151,298 bytes +1,832 wm-bot: Drained 'cloudvirt1028.eqiad.wmnet'. (T296790) - cookbook ran by andrew@buster

28 November 2021

  • curprev 17:4817:48, 28 November 2021imported>Stashbot 149,466 bytes +209 andrewbogott: moved cloudvirt1018 out of the 'localstorage' aggregate and into 'maintenance' for T296592. It will need to be moved back after the raid is rebuilt.

21 November 2021

  • curprev 07:1907:19, 21 November 2021imported>Stashbot 149,257 bytes +120 dcaro_away: restarting designate-sink with some extra logs in it (T296144)

17 November 2021

12 November 2021

  • curprev 13:3113:31, 12 November 2021imported>Stashbot 148,857 bytes +142 arturo: restarting glance-api services to make sure they work with new ceph auth creds (T293752)

8 November 2021

  • curprev 21:5021:50, 8 November 2021imported>Stashbot 148,715 bytes +620 andrewbogott: returned clouddb pools back to normal after maintain_views run: https://gerrit.wikimedia.org/r/c/operations/puppet/+/737505 T216481

5 November 2021

  • curprev 11:1811:18, 5 November 2021imported>Stashbot 148,095 bytes +742 wm-bot: Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance

4 November 2021

  • curprev 16:3916:39, 4 November 2021imported>Stashbot 147,353 bytes +2,597 wm-bot: Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance

3 November 2021

  • curprev 17:2217:22, 3 November 2021imported>Stashbot 144,756 bytes +279 arturo: [codfw1dev] installing keepalived 2.1.5 from buster-backports on cloudgw2001-dev/2002-dev (T294956)

2 November 2021

24 October 2021

  • curprev 00:4700:47, 24 October 2021imported>Stashbot 144,298 bytes +166 andrewbogott: deploying a change so that openstack clients use tls endpoints: https://gerrit.wikimedia.org/r/c/operations/puppet/+/732738

21 October 2021

  • curprev 10:1910:19, 21 October 2021imported>Stashbot 144,132 bytes +227 arturo: drop firewall exception on core routers for wiki replicas legacy setup (T293897)

20 October 2021

18 October 2021

  • curprev 19:2119:21, 18 October 2021imported>Stashbot 143,806 bytes +252 andrewbogott: also ticked the 'admin' box on wikitech for majavah T292827

14 October 2021

  • curprev 12:2812:28, 14 October 2021imported>Stashbot 143,554 bytes +149 arturo: [codfw1dev] add DB grants for cloudbackup2002.codfw.wmnet IP address to the cinder DB (T292546)

13 October 2021

12 October 2021

  • curprev 09:0609:06, 12 October 2021imported>Stashbot 143,300 bytes +200 dcaro: upgrading eqiad cloudnet hosts neutron packages (T292936)

5 October 2021

  • curprev 09:3909:39, 5 October 2021imported>Stashbot 143,100 bytes +152 arturo: [codfw1dev] cleaning up manila stuff from openstack (db, endpoints, tenant, VMs, and such) T291257

30 September 2021

  • curprev 14:5014:50, 30 September 2021imported>Stashbot 142,948 bytes +391 andrewbogott: sudo cumin "cloud*" "ps -ef | grep nslcd && service nslcd restart" and sudo cumin "lab*" "ps -ef | grep nslcd && service nslcd restart" T292202

29 September 2021

  • curprev 09:4109:41, 29 September 2021imported>Stashbot 142,557 bytes +196 arturo: [codfw1dev] cleanup manila shares definitions for a clean start now that the manila-sharecontroller VM is apparently well configured (T291257)

28 September 2021

  • curprev 16:2316:23, 28 September 2021imported>Stashbot 142,361 bytes +531 bstorm: downtime for clouddb1020 to reduce re-pages in case this goes badly T291963

27 September 2021

24 September 2021

  • curprev 13:0213:02, 24 September 2021imported>Stashbot 141,661 bytes +211 arturo: [codfw1dev] create VM manila-share-controller-01 on cloudinfra-codfw1dev

21 September 2021

  • curprev 12:1312:13, 21 September 2021imported>Stashbot 141,450 bytes +677 arturo: [codfw1dev] trying to create a manila service image (T291257)

20 September 2021

  • curprev 23:0823:08, 20 September 2021imported>Stashbot 140,773 bytes +408 bstorm: ran `echo check > /sys/block/md0/md/sync_action` on cloudcontrol1004 to check raid

17 September 2021

16 September 2021

  • curprev 15:5615:56, 16 September 2021imported>Stashbot 140,251 bytes +134 bstorm: removing downtime for labstore1005 so we'll know if it has another issue T290318

9 September 2021

  • curprev 22:0322:03, 9 September 2021imported>Stashbot 140,117 bytes +315 bstorm: restarted the prometheus-mysqld-exporter@s1 service as it was not working T290630

3 September 2021

  • curprev 15:3415:34, 3 September 2021imported>Stashbot 139,802 bytes +365 bstorm: rebooting labstore1005 to disconnect the drives from labstore1004 T290318

30 August 2021

  • curprev 16:1616:16, 30 August 2021imported>Stashbot 139,437 bytes +825 wm-bot: Added 1 new OSDs ['cloudcephosd1018.eqiad.wmnet'] - cookbook ran by andrew@buster

27 August 2021

  • curprev 18:5718:57, 27 August 2021imported>Stashbot 138,612 bytes +126 andrewbogott: raising toolsbeta ram/core/instances quotas so majavah can experiment with bullseye

25 August 2021

  • curprev 14:4514:45, 25 August 2021imported>Stashbot 138,486 bytes +534 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster

19 August 2021

  • curprev 17:3917:39, 19 August 2021imported>Stashbot 137,952 bytes +93 bstorm: restarting glance image backup to try and clear the page

18 August 2021

  • curprev 16:2116:21, 18 August 2021imported>Stashbot 137,859 bytes +899 wm-bot: Rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by andrew@buster

17 August 2021

  • curprev 15:1115:11, 17 August 2021imported>Stashbot 136,960 bytes +119 andrewbogott: rebooting cloudcephosd1008 to force raid rebuild -- T287838

11 August 2021

  • curprev 13:5113:51, 11 August 2021imported>Stashbot 136,841 bytes +480 wm-bot: Finished rebooting node cloudcephosd1018.eqiad.wmnet - cookbook ran by dcaro@vulcanus

10 August 2021

  • curprev 15:1515:15, 10 August 2021imported>Stashbot 136,361 bytes +214 andrewbogott: restarting all designate services in eqiad1

5 August 2021

  • curprev 09:3709:37, 5 August 2021imported>Stashbot 136,147 bytes +106 dcaro: Taking one osd daemon down ot codfw cluster (T288203)

4 August 2021

  • curprev 19:2019:20, 4 August 2021imported>Stashbot 136,041 bytes +126 bd808: Running deleteBatch.php on cloudweb2001-dev to remove legacy Heira: pages from labtestwiki

3 August 2021

  • curprev 17:4017:40, 3 August 2021imported>Stashbot 135,915 bytes +85 bstorm: rerunning the glance backup script after failure

31 July 2021

  • curprev 00:1000:10, 31 July 2021imported>Stashbot 135,830 bytes +233 andrewbogott: "systemctl reset-failed cloud-init.service" on all VMs for T287309

27 July 2021

  • curprev 21:3221:32, 27 July 2021imported>Stashbot 135,597 bytes +313 andrewbogott: putting cloudvirt1012 back into service T286748
(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)