You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Admin/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

17 March 2021

  • curprev 17:2817:28, 17 March 2021imported>Stashbot 100,320 bytes +432 bstorm: restarted the backup-glance-images job to clear errors in systemd T271782

10 March 2021

9 March 2021

5 March 2021

  • curprev 21:4021:40, 5 March 2021imported>Stashbot 97,855 bytes +748 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1 T276018

4 March 2021

  • curprev 18:3618:36, 4 March 2021imported>Stashbot 97,107 bytes +1,044 andrewbogott: rebooting cloudmetrics1002; the console is hanging

3 March 2021

  • curprev 17:1617:16, 3 March 2021imported>Stashbot 96,063 bytes +1,804 andrewbogott: restarting rabbitmq-server on cloudcontrol1003,1004,1005; trying to explain amqp errors in scheduler logs

2 March 2021

  • curprev 17:1617:16, 2 March 2021imported>Stashbot 94,259 bytes +717 andrewbogott: rebooting cloudvirt1039 to see if I can trigger T276208

1 March 2021

  • curprev 20:1220:12, 1 March 2021imported>Stashbot 93,542 bytes +347 andrewbogott: removing novaadmin from all projects save 'admin' for T274385

28 February 2021

  • curprev 04:5404:54, 28 February 2021imported>Stashbot 93,195 bytes +162 andrewbogott: restarted redis-server on tools-redis-1003 and tools-redis-1004 in an attempt to reduce replag, no real change detected

27 February 2021

  • curprev 00:3300:33, 27 February 2021imported>Stashbot 93,033 bytes +4,713 andrewbogott: sudo cumin --timeout 500 "A:all and not O{project:clouddb-services}" 'lsb_release -c | grep -i buster && uname -r | grep -v 4.19.0-14-amd64 && reboot'

25 February 2021

  • curprev 14:5614:56, 25 February 2021imported>Stashbot 88,320 bytes +121 arturo: deployed wmcs-netns-events daemon to all cloudnet servers (T275483)

24 February 2021

  • curprev 11:0711:07, 24 February 2021imported>Stashbot 88,199 bytes +112 arturo: force-reboot cloudmetrics1002, add icinga downtime for 2 hours. Investigating some server issue
  • curprev 00:1700:17, 24 February 2021imported>Stashbot 88,087 bytes +717 bstorm: set --property hw_scsi_model=virtio-scsi and --property hw_disk_bus=scsi on the main stretch image in glance on eqiad1 T275430

22 February 2021

  • curprev 17:1517:15, 22 February 2021imported>Stashbot 87,370 bytes +368 bstorm: restarting nova-compute on cloudvirt1016 and cloudvirt1036 in case it helps T275411

18 February 2021

17 February 2021

  • curprev 15:5815:58, 17 February 2021imported>Stashbot 86,576 bytes +153 arturo: deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/664845 to cloudnet servers (T268335)

15 February 2021

  • curprev 16:2516:25, 15 February 2021imported>Stashbot 86,423 bytes +395 arturo: [codfw1dev] rebooting all cloudgw200x-dev / cloudnet200x-dev servers (T272963)

11 February 2021

  • curprev 12:0112:01, 11 February 2021imported>Stashbot 86,028 bytes +692 arturo: [codfw1dev] drop instance `tools-codfw1dev-bastion-1` in `tools-codfw1dev` (was buster, cannot use it yet)

9 February 2021

  • curprev 15:2315:23, 9 February 2021imported>Stashbot 85,336 bytes +224 arturo: icinga-downtime for 2h everything *labs *cloud for openstack upgrades

8 February 2021

5 February 2021

  • curprev 10:5910:59, 5 February 2021imported>Stashbot 84,859 bytes +334 arturo: icinga-downtime labstore1004 tools share space check for 1 week (T272247)

4 February 2021

  • curprev 10:1210:12, 4 February 2021imported>Stashbot 84,525 bytes +147 dcaro: Increasing the memory limit of osds in eqiad from 8589934592(8G) to 12884901888(12G) (T273851)

3 February 2021

  • curprev 09:5909:59, 3 February 2021imported>Stashbot 84,378 bytes +203 dcaro: Doing a full vm backup on cloudvirt1024 with the new script (T260692)

2 February 2021

29 January 2021

  • curprev 15:3615:36, 29 January 2021imported>Stashbot 83,829 bytes +155 andrewbogott: disabling puppet and some services on eqiad1 cloudcontrol nodes; replacing nova-placement-api with placement-api

28 January 2021

  • curprev 19:4419:44, 28 January 2021imported>Stashbot 83,674 bytes +158 andrewbogott: shutting down cloudcontrol2001-dev because it's in a partially upgraded state; will revive when it's time for Train

27 January 2021

22 January 2021

  • curprev 16:4416:44, 22 January 2021imported>Stashbot 83,415 bytes +191 andrewbogott: upgrading designate on cloudvirt1003/1004 to OpenStack 'train'

21 January 2021

  • curprev 11:3511:35, 21 January 2021imported>Stashbot 83,224 bytes +338 arturo: merging core router firewall changes https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657439 (T209082)

20 January 2021

  • curprev 10:4910:49, 20 January 2021imported>Stashbot 82,886 bytes +1,118 arturo: merging core router firewall change https://gerrit.wikimedia.org/r/c/operations/homer/public/+/657302 (T209082)

19 January 2021

18 January 2021

  • curprev 16:0016:00, 18 January 2021imported>Stashbot 81,665 bytes +865 dcaro: Codfw1 ceph cluster uprgaded, will wait until tomorrow to see if there's any instability, but everything looks fine (T272303)

17 January 2021

  • curprev 16:5316:53, 17 January 2021imported>Stashbot 80,800 bytes +126 arturo: icinga downtime labstore1004 /srv/tools space check for 3 days (T272247)

15 January 2021

  • curprev 13:4113:41, 15 January 2021imported>Stashbot 80,674 bytes +405 arturo: icinga downtime labstore1004 maintain-dbuser alert until 2021-01-19 (T272125)

13 January 2021

  • curprev 17:0317:03, 13 January 2021imported>Stashbot 80,269 bytes +927 arturo: remove cloudvirt1013 cloudvirt1032 cloudvirt1037 to the 'toobusy' host aggregate to prevent further CPU oversubscribing

12 January 2021

11 January 2021

  • curprev 10:2210:22, 11 January 2021imported>Stashbot 79,167 bytes +573 arturo: doubling size of conntrack table in cloudnet servers https://gerrit.wikimedia.org/r/c/operations/puppet/+/655407 (T271058)

10 January 2021

  • curprev 16:0216:02, 10 January 2021imported>Stashbot 78,594 bytes +198 andrewbogott: restarting rabbitmq-server on all eqiad1 cloudcontrols

8 January 2021

  • curprev 11:2511:25, 8 January 2021imported>Stashbot 78,396 bytes +1,559 arturo: rebooting both cloudnet2002-dev/cloudnet2003-dev to make sure interfaces are set up correctl (T271517)

7 January 2021

  • curprev 15:1915:19, 7 January 2021imported>Stashbot 76,837 bytes +447 dcaro: Finished speed tests on cloudcephosd2001-dev, reprovisioning the osd.0 sdc (T271417)

5 January 2021

  • curprev 10:4010:40, 5 January 2021imported>Stashbot 76,390 bytes +134 dcaro: removing dumps-[1..*] backups from cloudvirt1024 as they are not needed (T271094)

3 January 2021

  • curprev 07:0607:06, 3 January 2021imported>Stashbot 76,256 bytes +117 dcaro: Got a network hiccup on cloudnet1004, keeping track here T271058

28 December 2020

  • curprev 12:3212:32, 28 December 2020imported>Stashbot 76,139 bytes +567 arturo: stop doing backups for the dumps project https://gerrit.wikimedia.org/r/c/operations/puppet/+/652182 (T260692)

23 December 2020

  • curprev 15:3815:38, 23 December 2020imported>Stashbot 75,572 bytes +333 andrewbogott: restarting rabbitmq on cloudcontrol1004; suspected leaks

22 December 2020

  • curprev 15:3015:30, 22 December 2020imported>Stashbot 75,239 bytes +231 dcaro: cleaning up 6778 dangling snapshots for glance images in eqiad (T270478)

19 December 2020

  • curprev 16:1816:18, 19 December 2020imported>Stashbot 75,008 bytes +84 dcaro: gzipped a bunch of logs on cloudvirt1004 due to / being out of space
  • curprev 00:1400:14, 19 December 2020imported>Stashbot 74,924 bytes +2,096 bstorm: truncated /var/log/debug.1 on cloudcontrol1003 which appears to be the exact same content as the user.log files anyway

17 December 2020

  • curprev 22:1722:17, 17 December 2020imported>Stashbot 72,828 bytes +570 andrewbogott: correction to above, set the pg and pgp to 1024 for eqiad1-glance-images

16 December 2020

  • curprev 09:3109:31, 16 December 2020imported>Stashbot 72,258 bytes +121 dcaro: removing invalid backups from cloudvirt1024 (196 in total) (T269419)

14 December 2020

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)