Revision history of "Nova Resource:Admin/SAL"

Jump to navigation Jump to search

Diff selection: Mark the radio boxes of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)
  • curprev 14:58, 27 May 2021imported>Stashbot 120,590 bytes +77 wm-bot: Testing - cookbook ran by dcaro@vulcanus
  • curprev 19:10, 26 May 2021imported>Stashbot 120,513 bytes +688 andrewbogott: reimaging cloudvirt1018 to support local VM storage
  • curprev 16:14, 25 May 2021imported>Stashbot 119,825 bytes +412 bd808: Closed #wikimedia-cloud-admin on f***node
  • curprev 22:32, 24 May 2021imported>Stashbot 119,413 bytes +302 andrewbogott: changing the default ttl for eqiad1.wikimedia.cloud. from 3600 to 60; this should help us avoid madness when re-using hostnames.
  • curprev 02:14, 22 May 2021imported>Stashbot 119,111 bytes +159 bstorm: downtiming SMART alerts on dumps server labstore1007 for the weekend because it has been flapping T281045
  • curprev 21:25, 13 May 2021imported>Stashbot 118,952 bytes +245 bstorm: converted the maps and scratch volumes on cloudstore1008 (standby) to drbd T224747
  • curprev 14:23, 12 May 2021imported>Stashbot 118,707 bytes +189 arturo: [codfw1dev] cleanup old unused agents (bgp, ovs)
  • curprev 18:00, 11 May 2021imported>Stashbot 118,518 bytes +198 andrewbogott: adding 'trove' service project in advance of deploying trove in eqiad1
  • curprev 10:53, 9 May 2021imported>Stashbot 118,320 bytes +109 arturo: icinga-downtime cloudmetrics1002 for 3 months (T275605)
  • curprev 13:51, 7 May 2021imported>Stashbot 118,211 bytes +252 andrewbogott: add inherited 'admin' right to novaadmin user throughout eqiad1. I was trying to narrow down the rights here but lack of admin breaks some workflows, e.g. T281894 and T282235
  • curprev 15:31, 6 May 2021imported>Stashbot 117,959 bytes +249 arturo: about to migrating CloudVPS network to the cloudgw architecture T270704
  • curprev 16:07, 5 May 2021imported>Stashbot 117,710 bytes +4,552 dcaro: disallowing insecure global ids on the eqiad ceph cluster (T280641)
  • curprev 16:05, 4 May 2021imported>Stashbot 113,158 bytes +1,656 wm-bot: Safe reboot of 'cloudvirt1028.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus
  • curprev 23:53, 3 May 2021imported>Stashbot 111,502 bytes +1,153 bstorm: running `maintain-dbusers harvest-replicas` on labstore1004 T281287
  • curprev 11:16, 30 April 2021imported>Stashbot 110,349 bytes +267 dcaro: draining and rebooting coludvirt1017, last one today (T280641)
  • curprev 15:11, 29 April 2021imported>Stashbot 110,082 bytes +404 dcaro: hard rebooting cloudmetrics1002, got hung again (T275605)
  • curprev 21:11, 28 April 2021imported>Stashbot 109,678 bytes +2,619 andrewbogott: cleaning up more references to deleted hypervisors with delete from services where topic='compute' and version != 53;
  • curprev 14:10, 27 April 2021imported>Stashbot 107,059 bytes +1,057 dcaro: codfw.openstack upgraded ceph libraries to 15.2.11 (T280641)
  • curprev 20:56, 26 April 2021imported>Stashbot 106,002 bytes +265 andrewbogott: deleting spurious 'codfw1dev' and 'codw1dev-4' regions in the dallas deployment; regions without endpoints break a bunch of things
  • curprev 13:49, 23 April 2021imported>Stashbot 105,737 bytes +569 dcaro: testing the drain_cloudvirt cookbook on codfw1 openstack cluster, draining cloudvirt2001 (T280641)
  • curprev 17:59, 21 April 2021imported>Stashbot 105,168 bytes +439 dcaro: all monitors upgraded on codfw1 with one cookbook `cookbook --verbose -c ~/.config/spicerack/cookbook.yaml wmcs.ceph.upgrade_mons --monitor-node-fqdn cloudcephmon2002-dev.codfw.wmnet` (T280641)
  • curprev 20:21, 20 April 2021imported>Stashbot 104,729 bytes +114 andrewbogott: reboot cloudservices1003
  • curprev 08:40, 19 April 2021imported>Stashbot 104,615 bytes +218 dcaro: enabling puppet on labstore1004 after mysql restart (T279657)
  • curprev 10:48, 14 April 2021imported>Stashbot 104,397 bytes +588 dcaro: Upgrade of codfw ceph to octopus 15.2.20 done, will run some performance tests now (T274566)
  • curprev 16:42, 13 April 2021imported>Stashbot 103,809 bytes +989 dcaro: Ceph balancer got the cluster to eval 0.014916, that is 88-77% usage for compute pool, and 28-19% usage for the cinder one \o/ (T274573)
  • curprev 21:33, 7 April 2021imported>Stashbot 102,820 bytes +84 andrewbogott: upgrading codfw1dev designate to Victoria
  • curprev 17:36, 4 April 2021imported>Stashbot 102,736 bytes +79 andrewbogott: upgrading eqiad1 designate to Ussuri
  • curprev 14:12, 2 April 2021imported>Stashbot 102,657 bytes +90 andrewbogott: upgrading codfw1dev to OpenStack version Ussuri
  • curprev 12:15, 1 April 2021imported>Stashbot 102,567 bytes +431 dcaro: Restoring the 4.9 kernel on cloudcephosd2003-dev and upgrading (T274565)
  • curprev 08:47, 31 March 2021imported>Stashbot 102,136 bytes +109 dcaro: upgrading cinder on codfw cloudcontrol2* nodes (T278845)
  • curprev 09:53, 30 March 2021imported>Stashbot 102,027 bytes +119 arturo: rebooting cloudnet1003 to cleanup conntrack table, it wouldn't cleanup by hand ...
  • curprev 15:42, 28 March 2021imported>Stashbot 101,908 bytes +80 andrewbogott: updated debian-10.0-buster base image
  • curprev 09:54, 27 March 2021imported>Stashbot 101,828 bytes +102 arturo: cleanup conntrack table in qrouter nents in cloudnet1003 (backup)
  • curprev 19:03, 25 March 2021imported>Stashbot 101,726 bytes +576 andrewbogott: deleting all unused (per wmcs-imageusage) Jessie base images from Glance
  • curprev 09:19, 24 March 2021imported>Stashbot 101,150 bytes +158 dcaro: restarted wmcs-backup on cloudvirt1024 as it failed due to an image being removed while running (T276892)
  • curprev 11:33, 23 March 2021imported>Stashbot 100,992 bytes +94 arturo: root@cloudcontrol1005:~# wmcs-novastats-dnsleaks --delete
  • curprev 10:10, 22 March 2021imported>Stashbot 100,898 bytes +191 arturo: cleanup conntrack table in standby node: aborrero@cloudnet1003:~ $ sudo ip netns exec qrouter-d93771ba-2711-4f88-804a-8df6fd03978a conntrack -F
  • curprev 17:18, 19 March 2021imported>Stashbot 100,707 bytes +293 bstorm: running `ALTER TABLE account MODIFY COLUMN type ENUM('user','tool','paws');` against the labsdbaccounts database on m5 T276284
  • curprev 00:30, 19 March 2021imported>Stashbot 100,414 bytes +94 bstorm: downtimed labstore1004 to check some things in debug mode
  • curprev 17:28, 17 March 2021imported>Stashbot 100,320 bytes +432 bstorm: restarted the backup-glance-images job to clear errors in systemd T271782
  • curprev 16:51, 10 March 2021imported>Stashbot 99,888 bytes +1,162 arturo: rebooting cloudvirt1030 for T275753
  • curprev 16:27, 9 March 2021imported>Stashbot 98,726 bytes +871 arturo: rebooting cloudvirt1027 (T275753)
  • curprev 21:40, 5 March 2021imported>Stashbot 97,855 bytes +748 andrewbogott: replacing 'observer' role with 'reader' role in eqiad1 T276018
  • curprev 18:36, 4 March 2021imported>Stashbot 97,107 bytes +1,044 andrewbogott: rebooting cloudmetrics1002; the console is hanging
  • curprev 17:16, 3 March 2021imported>Stashbot 96,063 bytes +1,804 andrewbogott: restarting rabbitmq-server on cloudcontrol1003,1004,1005; trying to explain amqp errors in scheduler logs
  • curprev 17:16, 2 March 2021imported>Stashbot 94,259 bytes +717 andrewbogott: rebooting cloudvirt1039 to see if I can trigger T276208
  • curprev 20:12, 1 March 2021imported>Stashbot 93,542 bytes +347 andrewbogott: removing novaadmin from all projects save 'admin' for T274385
  • curprev 04:54, 28 February 2021imported>Stashbot 93,195 bytes +162 andrewbogott: restarted redis-server on tools-redis-1003 and tools-redis-1004 in an attempt to reduce replag, no real change detected
  • curprev 00:33, 27 February 2021imported>Stashbot 93,033 bytes +4,713 andrewbogott: sudo cumin --timeout 500 "A:all and not O{project:clouddb-services}" 'lsb_release -c | grep -i buster && uname -r | grep -v 4.19.0-14-amd64 && reboot'
  • curprev 14:56, 25 February 2021imported>Stashbot 88,320 bytes +121 arturo: deployed wmcs-netns-events daemon to all cloudnet servers (T275483)
(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)