Nova Resource:Admin/SAL: Revision history

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

curprev 18:3318:33, 7 December 2020‎ imported>Stashbot‎ 70,949 bytes +249‎ andrewbogott: putting cloudvirt1023 back into service T269467

curprev 00:3500:35, 5 December 2020‎ imported>Stashbot‎ 70,700 bytes +1,279‎ andrewbogott: moving cloudvirt1023 back into maintenance because T269467 continues to puzzle

curprev 23:2123:21, 3 December 2020‎ imported>Stashbot‎ 69,421 bytes +765‎ andrewbogott: removing all osds on cloudcephosd1004 for rebuild, T268746

curprev 20:0420:04, 2 December 2020‎ imported>Stashbot‎ 68,656 bytes +909‎ andrewbogott: removing all osds on cloudcephosd1010 for rebuild, T268746

curprev 20:0620:06, 1 December 2020‎ imported>Stashbot‎ 67,747 bytes +322‎ andrewbogott: removing all osds on cloudcephosd1014 for rebuild, T268746

curprev 18:1218:12, 30 November 2020‎ imported>Stashbot‎ 67,425 bytes +131‎ andrewbogott: removing all osds from cloudcephosd1015 in order to investigate T268746

curprev 17:1817:18, 29 November 2020‎ imported>Stashbot‎ 67,294 bytes +106‎ andrewbogott: cleaning up some logfiles in tools-sgecron-01 — drive is full

curprev 22:5822:58, 26 November 2020‎ imported>Stashbot‎ 67,188 bytes +257‎ andrewbogott: deleting /var/log/haproxy logs older than 7 days in cloudcontrol100x. We need log rotation here it seems.

curprev 19:3519:35, 25 November 2020‎ imported>Stashbot‎ 66,931 bytes +1,218‎ bstorm: repairing ceph pg `instructing pg 6.91 on osd.117 to repair`

curprev 17:4017:40, 22 November 2020‎ imported>Stashbot‎ 65,713 bytes +161‎ andrewbogott: apt-get upgrade on cloudservices1003/1004

curprev 12:4412:44, 20 November 2020‎ imported>Stashbot‎ 65,552 bytes +237‎ arturo: [codfw1dev] install conntrackd in cloudnet2003-dev/cloudnet2002-dev to research l3 agent HA reliability

curprev 19:2119:21, 17 November 2020‎ imported>Stashbot‎ 65,315 bytes +103‎ andrewbogott: draining cloudvirt1012 to experiment with libvirt/cpu things

curprev 11:2111:21, 15 November 2020‎ imported>Stashbot‎ 65,212 bytes +103‎ arturo: icinga downtime cloudbackup2002 for 48h (T267865)

curprev 16:3816:38, 10 November 2020‎ imported>Stashbot‎ 65,109 bytes +243‎ arturo: icinga downtime toolschecker for 2h becasue toolsdb maintenance (T266587)

curprev 12:4212:42, 9 November 2020‎ imported>Stashbot‎ 64,866 bytes +944‎ arturo: restarted neutron l3 agent in cloudnet1003 bc it still had the old default route (T265288)

curprev 13:3613:36, 2 November 2020‎ imported>Stashbot‎ 63,922 bytes +127‎ arturo: (typo: dcaro)

curprev 16:5716:57, 29 October 2020‎ imported>Stashbot‎ 63,795 bytes +167‎ bstorm: silenced deployment-prep project alerts for 60 days since the downtime expired

curprev 16:2016:20, 25 October 2020‎ imported>Stashbot‎ 63,628 bytes +230‎ andrewbogott: adding cloudvirt1038 to the 'ceph' aggregate and removing from the 'spare' aggregate. We need this space while waiting on network upgrades for empty cloudvirts (T216195)

curprev 11:3011:30, 23 October 2020‎ imported>Stashbot‎ 63,398 bytes +467‎ arturo: [codfw1dev] openstack --os-project-id cloudinfra-codfw1dev recordset create --type PTR --record nat.cloudgw.codfw1dev.wikimediacloud.org. --description "created by hand" 0-29.57.15.185.in-addr.arpa. 1.0-29.57.15.185.in-addr.arpa. (T261724)

curprev 10:4610:46, 22 October 2020‎ imported>Stashbot‎ 62,931 bytes +285‎ arturo: [codfw1dev] rebooting cloudinfra-internal-puppetmaster-01.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud to try fixing some DNS weirdness

curprev 14:3614:36, 21 October 2020‎ imported>Stashbot‎ 62,646 bytes +343‎ andrewbogott: running apt-get update && apt-get install -y facter on all cloud-vps instances

curprev 15:4715:47, 20 October 2020‎ imported>Stashbot‎ 62,303 bytes +315‎ arturo: changing DNS recursor ACLs (https://gerrit.wikimedia.org/r/c/operations/puppet/+/635314) this can be reverted any time if it causes problems (T261724)

curprev 01:4101:41, 19 October 2020‎ imported>Stashbot‎ 61,988 bytes +280‎ andrewbogott: deleting all Precise base images

curprev 09:2909:29, 16 October 2020‎ imported>Stashbot‎ 61,708 bytes +678‎ arturo: [codfw1dev] still some DNS weirdness, investigating

curprev 15:1715:17, 15 October 2020‎ imported>Stashbot‎ 61,030 bytes +258‎ arturo: [codfw1dev] try cleaning up anything related to address scopes in the neutron database (T261724)

curprev 17:5417:54, 13 October 2020‎ imported>Stashbot‎ 60,772 bytes +373‎ andrewbogott: rebuilding cloudvirt1021 for backy support

curprev 10:1510:15, 9 October 2020‎ imported>Stashbot‎ 60,399 bytes +1,133‎ arturo: [codfwd1ev] root@cloudcontrol2001-dev:~# openstack router set --disable-snat cloudinstances2b-gw --external-gateway wan-transport-codfw (T261724)

curprev 16:1716:17, 8 October 2020‎ imported>Stashbot‎ 59,266 bytes +546‎ arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# openstack subnet create --network wan-transport-codfw --gateway 185.15.57.8 --no-dhcp --subnet-range 185.15.57.8/31 cloud-gw-transport-codfw` (with a hack -- see task) (T263622)

curprev 21:3021:30, 6 October 2020‎ imported>Stashbot‎ 58,720 bytes +326‎ andrewbogott: moved cloudvirt1013 out of the 'ceph' aggregate and into the 'maintenance' aggregate for T243414

curprev 17:4017:40, 5 October 2020‎ imported>Stashbot‎ 58,394 bytes +129‎ bd808: `service uwsgi-labspuppetbackend restart` on cloud-puppetmaster-03 (T264649)

curprev 11:0511:05, 2 October 2020‎ imported>Stashbot‎ 58,265 bytes +234‎ arturo: [codfw1dev] restarting rabbitmq-server in all 3 control nodes, the l3 agent was misbehaving

curprev 16:0616:06, 1 October 2020‎ imported>Stashbot‎ 58,031 bytes +166‎ arturo: rebooting cloudvirt1024 to validate changes to /etc/network/interfaces file

curprev 16:4716:47, 30 September 2020‎ imported>Stashbot‎ 57,865 bytes +1,958‎ andrewbogott: rebooting cloudvir1032, 1033, 1034 for T262979

curprev 14:5514:55, 28 September 2020‎ imported>Stashbot‎ 55,907 bytes +256‎ arturo: [jbond42] upgraded facter to v3 across the VM fleet

curprev 15:4715:47, 24 September 2020‎ imported>Stashbot‎ 55,651 bytes +294‎ arturo: stopping/restarting rabbitmq-server in all cloudcontrol servers

curprev 10:1610:16, 18 September 2020‎ imported>Stashbot‎ 55,357 bytes +593‎ arturo: cloudvirt1039 libvirtd service issues were fixed with a reboot

curprev 20:3220:32, 15 September 2020‎ imported>Stashbot‎ 54,764 bytes +181‎ andrewbogott: rebooting cloudvirt1038 to see if it resolves T262979

curprev 14:2114:21, 14 September 2020‎ imported>Stashbot‎ 54,583 bytes +421‎ andrewbogott: draining cloudvirt1001, migrating all VMs with wmcs-ceph-migrate

curprev 18:1318:13, 9 September 2020‎ imported>Stashbot‎ 54,162 bytes +433‎ andrewbogott: restarting ceph-mon@cloudcephmon1003 in hopes that the slow ops reported are phantoms
curprev 00:0500:05, 9 September 2020‎ imported>Stashbot‎ 53,729 bytes +517‎ bd808: Running wmcs-novastats-dnsleaks (T262359)

curprev 09:3209:32, 3 September 2020‎ imported>Stashbot‎ 53,212 bytes +106‎ arturo: icinga downtime cloud* servers for 30 mins (T261866)

curprev 08:4608:46, 2 September 2020‎ imported>Stashbot‎ 53,106 bytes +131‎ arturo: [codfw1dev] reimaging spare server labtestvirt2003 as debian buster (T261724)

curprev 18:1818:18, 1 September 2020‎ imported>Stashbot‎ 52,975 bytes +640‎ andrewbogott: adding drives on cloudcephosd100[3-5] to ceph osd pool

curprev 23:2623:26, 31 August 2020‎ imported>Stashbot‎ 52,335 bytes +303‎ bd808: Removed stale lockfile at cloud-puppetmaster-03.cloudinfra.eqiad.wmflabs:/var/lib/puppet/volatile/GeoIP/.geoipupdate.lock

curprev 20:1220:12, 28 August 2020‎ imported>Stashbot‎ 52,032 bytes +100‎ bd808: Running `wmcs-novastats-dnsleaks --delete` from cloudcontrol1003

curprev 17:1217:12, 26 August 2020‎ imported>Stashbot‎ 51,932 bytes +198‎ bstorm: Running 'ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%k KB %p\n" > tools_large_files_20200826.txt' on labstore1004 T261336

curprev 21:3421:34, 21 August 2020‎ imported>Stashbot‎ 51,734 bytes +99‎ andrewbogott: restarting nova-compute on cloudvirt1033; it seems stuck

curprev 14:2114:21, 19 August 2020‎ imported>Stashbot‎ 51,635 bytes +130‎ andrewbogott: rebooting cloudweb2001-dev, labweb1001, labweb1002 to address mediawiki-induced memleak

curprev 21:0221:02, 6 August 2020‎ imported>Stashbot‎ 51,505 bytes +241‎ andrewbogott: removing cloudvirt1004/1006 from nova's list of hypervisors; rebuilding them to use as backup test hosts

curprev 18:5418:54, 4 August 2020‎ imported>Stashbot‎ 51,264 bytes +105‎ bstorm: restarting mariadb on cloudcontrol1004 to setup parallel replication

curprev 17:0217:02, 3 August 2020‎ imported>Stashbot‎ 51,159 bytes +137‎ bstorm: increased db connection limit to 800 across galera cluster because we were clearly hovering at limit

curprev 19:2819:28, 31 July 2020‎ imported>Stashbot‎ 51,022 bytes +126‎ bd808: wmcs-novastats-dnsleaks --delete (lots of leaked fullstack-monitoring records to clean up)

curprev 22:1722:17, 27 July 2020‎ imported>Stashbot‎ 50,896 bytes +150‎ andrewbogott: ceph osd pool set compute pg_num 2048

curprev 19:1519:15, 24 July 2020‎ imported>Stashbot‎ 50,746 bytes +148‎ andrewbogott: ceph mgr module enable pg_autoscaler

curprev 08:5508:55, 22 July 2020‎ imported>Stashbot‎ 50,598 bytes +291‎ jbond42: [codfw1dev] upgrading hiera to version5

curprev 10:4810:48, 16 July 2020‎ imported>Stashbot‎ 50,307 bytes +158‎ arturo: merging change to neutron dmz_cidr https://gerrit.wikimedia.org/r/c/operations/puppet/+/613123 (T257534)

curprev 23:1523:15, 15 July 2020‎ imported>Stashbot‎ 50,149 bytes +545‎ bd808: Removed Merlijn van Deen from toollabs-trusted Gerrit group (T255697)

curprev 15:1915:19, 14 July 2020‎ imported>Stashbot‎ 49,604 bytes +504‎ arturo: briefly set root@cloudnet1003:~ # sysctl net.ipv4.conf.all.accept_local=1 (in neutron qrouter netns) (T257534)

curprev 16:1716:17, 13 July 2020‎ imported>Stashbot‎ 49,100 bytes +127‎ arturo: icinga downtime cloudcontrol[1003-1005].wikimedia.org for 1h for galera database movements

curprev 17:3917:39, 12 July 2020‎ imported>Stashbot‎ 48,973 bytes +98‎ andrewbogott: switched eqiad1 keystone from m5 to cloudcontrol galera

curprev 20:2620:26, 10 July 2020‎ imported>Stashbot‎ 48,875 bytes +88‎ andrewbogott: disabling nova api to move database to galera

curprev 11:2311:23, 9 July 2020‎ imported>Stashbot‎ 48,787 bytes +378‎ arturo: [codfw1dev] rebooting cloudnet2003-dev again for testing sysct/puppet behavior (T257552)

curprev 15:1615:16, 6 July 2020‎ imported>Stashbot‎ 48,409 bytes +76‎ arturo: installing 'aptitude' in all cloudvirts

curprev 12:5112:51, 3 July 2020‎ imported>Stashbot‎ 48,333 bytes +455‎ arturo: [codfw1dev] galera cluster should be up and running, openstack happy (T256283)

curprev 15:4115:41, 2 July 2020‎ imported>Stashbot‎ 47,878 bytes +273‎ arturo: `sudo wmcs-openstack --os-compute-api-version 2.55 flavor create --private --vcpus 8 --disk 300 --ram 16384 --property aggregate_instance_extra_specs:ceph=true --description "for packaging envoy" bigdisk-ceph` (T256983)

curprev 14:2414:24, 29 June 2020‎ imported>Stashbot‎ 47,605 bytes +162‎ arturo: starting rabbitmq-server in all 3 cloudcontrol servers

curprev 20:3820:38, 18 June 2020‎ imported>Stashbot‎ 47,443 bytes +130‎ andrewbogott: rebooting cloudservices2003-dev due to a mysterious 'host down' alert on a secondary ip

curprev 15:3815:38, 16 June 2020‎ imported>Stashbot‎ 47,313 bytes +159‎ arturo: created by hand neutron port 9c0a9a13-e409-49de-9ba3-bc8ec4801dbf `paws-haproxy-vip` (T295217)

curprev 13:2313:23, 12 June 2020‎ imported>Stashbot‎ 47,154 bytes +202‎ arturo: DNS zone `paws.wmcloud.org` transferred to the PAWS project (T195217)

curprev 19:1919:19, 11 June 2020‎ imported>Stashbot‎ 46,952 bytes +428‎ bstorm_: proceeding with failback to labstore1004 now that DRBD devices are consistent T224582

curprev 16:0916:09, 10 June 2020‎ imported>Stashbot‎ 46,524 bytes +169‎ andrewbogott: deleting all old cloud-ns0.wikimedia.org and cloud-ns1.wikimedia.org ns records in designate database T254496

curprev 15:2515:25, 9 June 2020‎ imported>Stashbot‎ 46,355 bytes +337‎ arturo: icinga downtime everything cloud* lab* for 2h more (T253780)

curprev 15:0815:08, 5 June 2020‎ imported>Stashbot‎ 46,018 bytes +148‎ andrewbogott: trying to re-enable puppet without losing cumin contact, as per https://phabricator.wikimedia.org/T254589

curprev 14:2414:24, 4 June 2020‎ imported>Stashbot‎ 45,870 bytes +180‎ andrewbogott: disabling puppet on all instances for /labs/private recovery

curprev 23:0223:02, 28 May 2020‎ imported>Stashbot‎ 45,690 bytes +169‎ bd808: `/usr/local/sbin/maintain-dbusers --debug harvest-replicas` (T253930)
curprev 00:3300:33, 28 May 2020‎ imported>Stashbot‎ 45,521 bytes +610‎ andrewbogott: shutting down cloudservices2002-dev to see if we can live without it. This is in anticipation or rebuilding it entirely for T253780

curprev 16:3616:36, 25 May 2020‎ imported>Stashbot‎ 44,911 bytes +119‎ arturo: [codfw1dev] created zone `0-29.57.15.185.in-addr.arpa.` (T247972)

curprev 19:2319:23, 21 May 2020‎ imported>Stashbot‎ 44,792 bytes +336‎ andrewbogott: disabling puppet on cloudbackup2001 to prevent the backup job from starting during maintenance

curprev 22:5922:59, 19 May 2020‎ imported>Stashbot‎ 44,456 bytes +181‎ bd808: `apt-get install mariadb-client` on cloudcontrol1003

curprev 21:3721:37, 18 May 2020‎ imported>Stashbot‎ 44,275 bytes +82‎ andrewbogott: rebuilding cloudnet2003-dev with Buster

curprev 22:1022:10, 15 May 2020‎ imported>Stashbot‎ 44,193 bytes +375‎ bd808: Added reedy as projectadmin in cloudinfra project (T249774)

curprev 23:2823:28, 14 May 2020‎ imported>Stashbot‎ 43,818 bytes +724‎ bstorm_: downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time T252831

curprev 20:3320:33, 12 May 2020‎ imported>Stashbot‎ 43,094 bytes +747‎ andrewbogott: moving cloudvirt1023 to the 'standard' pool and out of the 'spare' pool

curprev 16:5316:53, 9 May 2020‎ imported>Stashbot‎ 42,347 bytes +128‎ andrewbogott: rebuilding cloudcontrol2001-dev and 2003-dev with buster for T252121

curprev 19:0219:02, 8 May 2020‎ imported>Stashbot‎ 42,219 bytes +118‎ bstorm_: moving tools-k8s-haproxy-2 from cloudvirt1021 to cloudvirt1017 to improve spread

curprev 13:5813:58, 5 May 2020‎ imported>Stashbot‎ 42,101 bytes +101‎ andrewbogott: rebuilding cloudcontrol2004-dev to test new puppet changes

curprev 09:0409:04, 4 May 2020‎ imported>Stashbot‎ 42,000 bytes +194‎ arturo: [codfw1dev] manually modify iptables ruleset to only allow SSH from WMF bastions on cloudservices2003-dev and cloudcontrol2004-dev (T251604)

curprev 22:1222:12, 21 April 2020‎ imported>Stashbot‎ 41,806 bytes +205‎ andrewbogott: moving cloudvirt1004 out of the 'standard' aggregate and into the 'maintenance' aggregate

curprev 18:4418:44, 15 April 2020‎ imported>Stashbot‎ 41,601 bytes +99‎ jeh: create indexes and views for grwikimedia T245912

curprev 15:0715:07, 13 April 2020‎ imported>Stashbot‎ 41,502 bytes +110‎ jeh: restart memcached on labwebs to increase cache size T145703

curprev 19:5719:57, 9 April 2020‎ imported>Stashbot‎ 41,392 bytes +156‎ andrewbogott: upgrading eqiad1 designate to rocky

curprev 19:2019:20, 8 April 2020‎ imported>Stashbot‎ 41,236 bytes +239‎ andrewbogott: rotated password and api token for pdns servers on cloudservices1003 and cloudservices1004

curprev 20:5720:57, 7 April 2020‎ imported>Stashbot‎ 40,997 bytes +128‎ andrewbogott: service sssd stop; rm -rf /var/lib/sss/db*; service sssd start on tools-sgebastion-08

curprev 22:3922:39, 6 April 2020‎ imported>Stashbot‎ 40,869 bytes +634‎ andrewbogott: deleting bogus groups cn=b'project-bastion',ou=groups,dc=wikimedia,dc=org and cn=b'project-tools',ou=groups,dc=wikimedia,dc=org from ldap

curprev 20:5920:59, 2 April 2020‎ imported>Stashbot‎ 40,235 bytes +112‎ jeh: codfw1dev clear VM error states and start bastions, puppet master and database

curprev 16:2716:27, 1 April 2020‎ imported>Stashbot‎ 40,123 bytes +126‎ arturo: [codfw1dev] enable puppet across the fleet clean vxlan changes (T248881)

curprev 12:3512:35, 31 March 2020‎ imported>Stashbot‎ 39,997 bytes +801‎ arturo: [codfw1dev] restarting VMs: designaterockytest14, bastion-codfw1dev-0[1,2] (T248881)

curprev 23:4223:42, 30 March 2020‎ imported>Stashbot‎ 39,196 bytes +399‎ bstorm_: deleted "Kubernetes Cluster" and "Kubernetes Performance" dashboards T246689

curprev 21:2821:28, 27 March 2020‎ imported>Stashbot‎ 38,797 bytes +181‎ bd808: Created huggle.wmcloud.org Designate zone and allocated it to the huggle project

curprev 15:0115:01, 26 March 2020‎ imported>Stashbot‎ 38,616 bytes +325‎ arturo: icinga downtime cloudvirt* cloudcontrol* cloudnet* lab* cloudstore*

curprev 19:2919:29, 25 March 2020‎ imported>Stashbot‎ 38,291 bytes +285‎ andrewbogott: dumping a bunch of VMs on cloudvirt1015 to see if it still crashes

curprev 19:4119:41, 24 March 2020‎ imported>Stashbot‎ 38,006 bytes +225‎ jeh: switch cloudvirt1016 from maintenance to standard host aggregate T243327

curprev 21:4121:41, 23 March 2020‎ imported>Stashbot‎ 37,781 bytes +305‎ jeh: restart neutron-l3-agent on cloudnet100[3,4] to pickup policy.yaml changes

curprev 14:2314:23, 21 March 2020‎ imported>Stashbot‎ 37,476 bytes +84‎ andrewbogott: restarting apache2 on labweb1001 and 1002

curprev 19:1719:17, 18 March 2020‎ imported>Stashbot‎ 37,392 bytes +334‎ andrewbogott: deleted a bunch of records from the pdns database on cloudservices1003/1004 which had a record name but the content (where an IP address should be) was NULL, e.g. m.wikidata.beta.wmflabs.org.

curprev 17:4017:40, 14 March 2020‎ imported>Stashbot‎ 37,058 bytes +99‎ jeh: restart maintain-dbusers on labstore1004 T247654

curprev 12:3912:39, 13 March 2020‎ imported>Stashbot‎ 36,959 bytes +302‎ arturo: [codfw1dev] reintroduce address scopes for another round of testing T244851

curprev 22:2922:29, 12 March 2020‎ imported>Stashbot‎ 36,657 bytes +130‎ bstorm_: running puppet across all dumps mounts to make sure active links are shifted to labstore1006

curprev 18:3818:38, 11 March 2020‎ imported>Stashbot‎ 36,527 bytes +579‎ jeh: set icingia downtime until 2020-03-23 on CODFW cloud[control,net,virt] hosts during openstack upgrades

curprev 17:0217:02, 10 March 2020‎ imported>Stashbot‎ 35,948 bytes +272‎ arturo: [codfw1dev] deleting address scopes, bad interaction with our custom NAT setup T247135

curprev 18:0918:09, 9 March 2020‎ imported>Stashbot‎ 35,676 bytes +343‎ arturo: enabling puppet in cloudvirt1006, all services have been restored

curprev 14:5414:54, 6 March 2020‎ imported>Stashbot‎ 35,333 bytes +115‎ andrewbogott: draining all instances off of cloudvirt1006 for T246908

curprev 14:2414:24, 5 March 2020‎ imported>Stashbot‎ 35,218 bytes +475‎ arturo: [codfw1dev] we just enabled BGP session between cloudnet2xxx-dev and cr1-codfw (T245606)

curprev 22:2222:22, 4 March 2020‎ imported>Stashbot‎ 34,743 bytes +776‎ andrewbogott: upgrading designate on cloudservices1003/1004 to Queens

curprev 16:5416:54, 2 March 2020‎ imported>Stashbot‎ 33,967 bytes +159‎ arturo: [codfw1dev] deleted python3-os-ken debian package in cloudnet2003-dev which was installed by hand and had depedency issues

curprev 16:3216:32, 29 February 2020‎ imported>Stashbot‎ 33,808 bytes +160‎ bstorm_: downtimed the smart alert on cloudvirt1009 until Monday since apparently predictive failures flap T244986

curprev 22:0322:03, 26 February 2020‎ imported>Stashbot‎ 33,648 bytes +86‎ jeh: powering down cloudvirt1014 for hardware maintenance

curprev 16:0816:08, 25 February 2020‎ imported>Stashbot‎ 33,562 bytes +458‎ andrewbogott: changing neutron's rabbitmq password because oslo is having trouble parsing some of the characters in the password

curprev 12:1612:16, 24 February 2020‎ imported>Stashbot‎ 33,104 bytes +1,060‎ arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr2-codfw` (T245606)

curprev 12:4812:48, 21 February 2020‎ imported>Stashbot‎ 32,044 bytes +771‎ arturo: [codfw1dev] running `root@cloudcontrol2001-dev:~# neutron bgp-speaker-network-add bgpspeaker wan-transport-codfw` (T245606)

curprev 19:2219:22, 20 February 2020‎ imported>Stashbot‎ 31,273 bytes +478‎ andrewbogott: updating designate pool config for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/572213/

curprev 22:1922:19, 18 February 2020‎ imported>Stashbot‎ 30,795 bytes +429‎ andrewbogott: transferred the tools.wmcloud.org. to the tools project

curprev 10:3510:35, 14 February 2020‎ imported>Stashbot‎ 30,366 bytes +477‎ arturo: running `root@cloudcontrol2001-dev:~# designate server-create --name ns1.openstack.codfw1dev.wikimediacloud.org.` (T243766)

curprev 13:3813:38, 12 February 2020‎ imported>Stashbot‎ 29,889 bytes +270‎ arturo: [codfw1dev] add reference to subnetpool to the instance subnet `MariaDB [neutron]> update subnets set subnetpool_id='d129650d-d4be-4fe1-b13e-6edb5565cb4a' where id = '7adfcebe-b3d0-4315-92fe-e8365cc80668';` (T244851)

curprev 13:4613:46, 11 February 2020‎ imported>Stashbot‎ 29,619 bytes +570‎ arturo: [codfw1dev] creating some neutron objects to investigate T244851 (subnets, subnet pools, address scopes, ...)

curprev 18:1118:11, 7 February 2020‎ imported>Stashbot‎ 29,049 bytes +106‎ jeh: shutdown cloudvirt1016 for hardware maintenance T241882

curprev 14:4414:44, 6 February 2020‎ imported>Stashbot‎ 28,943 bytes +169‎ jeh: update apt packages on cloudvirt1015 T220853

curprev 17:2417:24, 28 January 2020‎ imported>Stashbot‎ 28,774 bytes +1,040‎ arturo: [codfw1dev] root@cloudcontrol2001-dev:~# designate server-create --name ns0.openstack.codfw1dev.wikimediacloud.org. (T243766)

curprev 12:4512:45, 27 January 2020‎ imported>Stashbot‎ 27,734 bytes +495‎ arturo: [codfw1dev] manually move the new domain to the `cloudinfra-codfw1dev` project clouddb2001-dev: `[designate]> update zones set tenant_id='cloudinfra-codfw1dev' where id = '4c75410017904858a5839de93c9e8b3d';` T243556

curprev 15:1015:10, 24 January 2020‎ imported>Stashbot‎ 27,239 bytes +185‎ jeh: remove icinga downtime for cloudvirt1013 T241313

curprev 17:4317:43, 21 January 2020‎ imported>Stashbot‎ 27,054 bytes +253‎ bstorm_: remounting /mnt/nfs/dumps-labstore1007.wikimedia.org/ on all dumps-mounting projects

curprev 16:5916:59, 15 January 2020‎ imported>Stashbot‎ 26,801 bytes +144‎ bd808: Changed the config for cloud-announce mailing list so that lsit admins do not get bounce unsubscribe notices

curprev 14:0314:03, 14 January 2020‎ imported>Stashbot‎ 26,657 bytes +395‎ arturo: icinga downtime all cloudvirts for another 2h for fixing some icinga checks

curprev 13:3413:34, 13 January 2020‎ imported>Stashbot‎ 26,262 bytes +269‎ arturo: [¢odfw1dev] prevent neutron from allocating floating IPs from the wrong subnet by doing `neutron subnet-update --allocation-pool start=208.80.153.190,end=208.80.153.190 cloud-instances-transport1-b-codfw` (T242594)

curprev 13:2713:27, 10 January 2020‎ imported>Stashbot‎ 25,993 bytes +167‎ arturo: cloudvirt1009: virsh undefine i-000069b6. This is tools-elastic-01 which is running on cloudvirt1008 (so, leaked on cloudvirt1009)

curprev 11:1211:12, 9 January 2020‎ imported>Stashbot‎ 25,826 bytes +397‎ arturo: running `MariaDB [nova_eqiad1]> update quota_usages set in_use='0' where project_id='etytree';` (T242332)

curprev 10:5310:53, 8 January 2020‎ imported>Stashbot‎ 25,429 bytes +111‎ arturo: icinga downtime all cloudvirts for 30 minutes to re-create all canary VMs"

curprev 11:1211:12, 7 January 2020‎ imported>Stashbot‎ 25,318 bytes +228‎ arturo: icinga-downtime everything cloud* for 30 minutes to merge nova scheduler changes

curprev 13:4513:45, 6 January 2020‎ imported>Stashbot‎ 25,090 bytes +110‎ andrewbogott: restarting nova-api and nova-conductor on cloudcontrol1003 and 1004

curprev 16:3416:34, 4 January 2020‎ imported>Stashbot‎ 24,980 bytes +130‎ arturo: icinga downtime cloudvirt1024 for 2 months because hardware errors (T241884)

curprev 11:4611:46, 31 December 2019‎ imported>Stashbot‎ 24,850 bytes +158‎ andrewbogott: I couldn't!

curprev 10:1310:13, 25 December 2019‎ imported>Stashbot‎ 24,692 bytes +205‎ arturo: icinga downtime for 30 minutes the whole cloud* lab* fleet to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/560575 (will restart some openstack components)

curprev 15:1315:13, 24 December 2019‎ imported>Stashbot‎ 24,487 bytes +188‎ arturo: icinga downtime all the lab* fleet for nova password change for 1h

curprev 11:1311:13, 23 December 2019‎ imported>Stashbot‎ 24,299 bytes +174‎ arturo: enable puppet in cloudcontrol1003/1004

curprev 23:4823:48, 22 December 2019‎ imported>Stashbot‎ 24,125 bytes +290‎ andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and 1004

curprev 12:4312:43, 20 December 2019‎ imported>Stashbot‎ 23,835 bytes +83‎ arturo: icinga downtime cloudmetrics1001 for 128 hours

curprev 12:5512:55, 18 December 2019‎ imported>Stashbot‎ 23,752 bytes +191‎ arturo: [codfw1dev] created a new subnet neutron object to hold the new CIDR for floating IPs (cloud-codfw1dev-floating - 185.15.57.0/29) T239347

curprev 07:2107:21, 17 December 2019‎ imported>Stashbot‎ 23,561 bytes +85‎ andrewbogott: deploying horizon/train to labweb1001/1002

curprev 06:1106:11, 12 December 2019‎ imported>Stashbot‎ 23,476 bytes +173‎ arturo: schedule 4h downtime for labstores

curprev 06:2806:28, 2 December 2019‎ imported>Stashbot‎ 23,303 bytes +611‎ andrewbogott: running nova-manage db sync on eqiad1

curprev 19:2719:27, 18 November 2019‎ imported>Stashbot‎ 22,692 bytes +897‎ andrewbogott: repooling labsdb1011

curprev 20:0420:04, 15 November 2019‎ imported>Stashbot‎ 21,795 bytes +681‎ andrewbogott: repool labdb1011 (T237509)

curprev 13:1013:10, 11 November 2019‎ imported>Stashbot‎ 21,114 bytes +217‎ arturo: cloudweb2001-dev: disable puppet and redirect stderr in the loadExitNodes.php cron script to prevent cronspam while we investigate the cause of the issue (T237971)

curprev 11:5911:59, 5 November 2019‎ imported>Stashbot‎ 20,897 bytes +171‎ arturo: icinga downtime for 1h cloudcontrol1004, cloudnet1003, cloudvirt1017/1020/1022 for PDU operations in the rack T227542

curprev 21:5521:55, 4 November 2019‎ imported>Stashbot‎ 20,726 bytes +145‎ andrewbogott: deleting a ton of wikitech hiera pages that were either no-ops or refer to nonexistent VMs or prefixes

curprev 11:0111:01, 31 October 2019‎ imported>Stashbot‎ 20,581 bytes +151‎ arturo: icinga-downtimed cloudvirt1030 and cloudservices1003 for 1h due to PDU upgrade operations T227543

curprev 22:4322:43, 30 October 2019‎ imported>Stashbot‎ 20,430 bytes +232‎ jeh: reboot cloud-bootstrapvz-stretch to resolve bad bootstrapvz build

curprev 10:4510:45, 25 October 2019‎ imported>Stashbot‎ 20,198 bytes +176‎ arturo: icinga downtime toolschecker for 1 to upgrade clouddb1002 mariadb (toolsdb secondary) (T236384 , T236420)

curprev 12:3012:30, 24 October 2019‎ imported>Stashbot‎ 20,022 bytes +517‎ arturo: starting cloudvirt1019, PDU operations ended (T227540)

curprev 09:2309:23, 23 October 2019‎ imported>Stashbot‎ 19,505 bytes +293‎ arturo: cloudvirt1026 reboot ended OK

curprev 16:0116:01, 18 October 2019‎ imported>Stashbot‎ 19,212 bytes +420‎ arturo: created the `eqiad1.wikimedia.cloud` DNS zone (T235846)

curprev 21:5921:59, 16 October 2019‎ imported>Stashbot‎ 18,792 bytes +512‎ jeh: resync wiki replica tool and user accounts T235697

curprev 13:3013:30, 15 October 2019‎ imported>Stashbot‎ 18,280 bytes +97‎ jeh: creating indexes and views for banwiki T234770

curprev 18:5518:55, 10 October 2019‎ imported>Stashbot‎ 18,183 bytes +250‎ bd808: Created indexes and views for nqowiki (T230543)

curprev 10:4410:44, 9 October 2019‎ imported>Stashbot‎ 17,933 bytes +531‎ arturo: cloudvirt1013 rebooted well

curprev 14:0714:07, 7 October 2019‎ imported>Stashbot‎ 17,402 bytes +196‎ arturo: horizon is disabled for maintenance (T212302)

curprev 15:2315:23, 2 October 2019‎ imported>Stashbot‎ 17,206 bytes +779‎ arturo: codfw1dev renaming net/subnet objects to a more modern naming scheme T233665

curprev 10:2110:21, 30 September 2019‎ imported>Stashbot‎ 16,427 bytes +290‎ arturo: we installed ferm in every VM by mistake. Deleting it and forcing a puppet agent run to try to go back to a clean state.

curprev 10:3910:39, 18 August 2019‎ imported>Stashbot‎ 16,137 bytes +149‎ arturo: rebooting cloudvirt1023 for new interface names configuration

curprev 17:1717:17, 5 August 2019‎ imported>Stashbot‎ 15,988 bytes +136‎ bd808: Set downtime on gridengine and kubernetes webservice checks in icinga until 2019-09-02 (flaky tests)

curprev 20:1420:14, 29 July 2019‎ imported>Stashbot‎ 15,852 bytes +114‎ bd808: Restarted maintain-kubeusers on tools-k8s-master-01 (T194859)

curprev 12:3212:32, 25 July 2019‎ imported>Stashbot‎ 15,738 bytes +688‎ arturo: eqiad1/glance: debian-9.9-stretch image deprecates debian-9.8-stretch (T228983)

curprev 19:4319:43, 23 July 2019‎ imported>Stashbot‎ 15,050 bytes +98‎ andrewbogott: restarting rabbitmq-server on cloudcontrol1003 and 1004

curprev 23:4423:44, 22 July 2019‎ imported>Stashbot‎ 14,952 bytes +114‎ bd808: Restarted maintain-kubeusers on tools-k8s-master-01 (T228529)

curprev 22:0722:07, 11 July 2019‎ imported>Stashbot‎ 14,838 bytes +343‎ bd808: Ran `sudo systemctl stop designate_floating_ip_ptr_records_updater.service` on cloudcontrol1003

curprev 16:0516:05, 25 June 2019‎ imported>Stashbot‎ 14,495 bytes +213‎ bstorm_: updated python3.4 to update4 wherever it was installed on Jessie VMs to prevent issues with broken update3.

curprev 15:5615:56, 3 June 2019‎ imported>Stashbot‎ 14,282 bytes +7,003‎ arturo: T221769 rebooting cloudservices1003 after bootstrapping is apparently completed

curprev 20:0020:00, 13 January 2019‎ imported>Stashbot‎ 7,279 bytes +441‎ andrewbogott: VPS proxies are now running in eqiad1 on proxy-01. Old VMs will wait a bit for deletion. T213540

curprev 22:2122:21, 9 January 2019‎ imported>Stashbot‎ 6,838 bytes +85‎ bd808: neutron quota-update --tenant-id tools --port 256

curprev 18:5918:59, 8 January 2019‎ imported>Stashbot‎ 6,753 bytes +257‎ bd808: Definately did NOT delete uid=novaadmin,ou=people,dc=wikimedia,dc=org

curprev 22:0322:03, 6 January 2019‎ imported>Stashbot‎ 6,496 bytes +126‎ bd808: Set floatingip quota of 60 for tools project in eqiad1-r region (T212360)

curprev 17:1017:10, 20 December 2018‎ imported>Stashbot‎ 6,370 bytes +100‎ arturo: T207663 renumbered transport network in eqiad1

curprev 17:5917:59, 5 December 2018‎ imported>Stashbot‎ 6,270 bytes +130‎ arturo: T207663 changed labtestn transport network addressing from private to public

curprev 13:2513:25, 3 December 2018‎ imported>Stashbot‎ 6,140 bytes +107‎ arturo: T202886 create again PTR records after dnsleak.py fix

curprev 14:0814:08, 30 November 2018‎ imported>Stashbot‎ 6,033 bytes +126‎ arturo: running dns leaks cleanup `root@cloudcontrol1003:~# /root/novastats/dnsleaks.py --delete`

curprev 17:3317:33, 28 November 2018‎ imported>Stashbot‎ 5,907 bytes +94‎ gtirloni: deleted contintcloud project (T209644)

curprev 13:3213:32, 27 November 2018‎ imported>Stashbot‎ 5,813 bytes +113‎ gtirloni: enabled DRBD stats collection on labstore100[4-5] T208446

curprev 07:1207:12, 22 November 2018‎ imported>Stashbot‎ 5,700 bytes +76‎ gtirloni: deployed new debian-9.6-stretch image

curprev 10:4810:48, 21 November 2018‎ imported>Stashbot‎ 5,624 bytes +134‎ arturo: re-created compat-net as not shared in labtestn to test stuff related to T209954

curprev 12:4312:43, 16 November 2018‎ imported>Stashbot‎ 5,490 bytes +244‎ gtirloni: armed keyholder on labpuppetmaster1001/1002 after reboots

curprev 17:1917:19, 14 November 2018‎ imported>Stashbot‎ 5,246 bytes +869‎ gtirloni: added cloudvirt1016 to scheduler pool (T209426)

curprev 18:1718:17, 9 November 2018‎ imported>Stashbot‎ 4,377 bytes +96‎ gtirloni: restarted neutron-linuxbridge-agent on cloudvirt1018/1023

curprev 11:0011:00, 8 November 2018‎ imported>Stashbot‎ 4,281 bytes +123‎ gtirloni: Added novaproxy-02 to $CACHES

curprev 13:4913:49, 7 November 2018‎ imported>Stashbot‎ 4,158 bytes +144‎ arturo: T208733 moving labvirt1017 from main deployment to eqiad1 and renaming it to cloudvirt1017

curprev 16:2416:24, 22 October 2018‎ imported>Stashbot‎ 4,014 bytes +248‎ arturo: T206261 another update to dmz_cidr in eqiad1

curprev 12:0212:02, 19 October 2018‎ imported>Stashbot‎ 3,766 bytes +392‎ arturo: revert change in dmz_cidr in eqiad1 for now (T206261)

curprev 10:4010:40, 26 September 2018‎ imported>Stashbot‎ 3,374 bytes +753‎ arturo: T205524 all sorts of restarts in all neutron daemons

curprev 22:0822:08, 17 September 2018‎ imported>Stashbot‎ 2,621 bytes +2,384‎ bd808: Granted gtirloni project roles of admin, projectadmin, and user

curprev 00:5500:55, 9 February 2018‎ imported>Stashbot‎ 237 bytes +211‎ bd808: Added Arturo Borrero Gonzalez and Bstorm as project members

curprev 21:0221:02, 8 October 2015‎ imported>Andrew Bogott‎ 26 bytes +26‎ No edit summary

Nova Resource:Admin/SAL: Revision history

7 December 2020

5 December 2020

3 December 2020

2 December 2020

1 December 2020

30 November 2020

29 November 2020

26 November 2020

25 November 2020

22 November 2020

20 November 2020

17 November 2020

15 November 2020

10 November 2020

9 November 2020

2 November 2020

29 October 2020

25 October 2020

23 October 2020

22 October 2020

21 October 2020

20 October 2020

19 October 2020

16 October 2020

15 October 2020

13 October 2020

9 October 2020

8 October 2020

6 October 2020

5 October 2020

2 October 2020

1 October 2020

30 September 2020

28 September 2020

24 September 2020

18 September 2020

15 September 2020

14 September 2020

9 September 2020

3 September 2020

2 September 2020

1 September 2020

31 August 2020

28 August 2020

26 August 2020

21 August 2020

19 August 2020

6 August 2020

4 August 2020

3 August 2020

31 July 2020

27 July 2020

24 July 2020

22 July 2020

16 July 2020

15 July 2020

14 July 2020

13 July 2020

12 July 2020

10 July 2020

9 July 2020

6 July 2020

3 July 2020

2 July 2020

29 June 2020

18 June 2020

16 June 2020

12 June 2020

11 June 2020

10 June 2020

9 June 2020

5 June 2020

4 June 2020

28 May 2020

25 May 2020

21 May 2020

19 May 2020

18 May 2020

15 May 2020