You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Admin/SAL: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

3 August 2020

  • curprev 17:0217:02, 3 August 2020imported>Stashbot 51,159 bytes +137 bstorm: increased db connection limit to 800 across galera cluster because we were clearly hovering at limit

31 July 2020

  • curprev 19:2819:28, 31 July 2020imported>Stashbot 51,022 bytes +126 bd808: wmcs-novastats-dnsleaks --delete (lots of leaked fullstack-monitoring records to clean up)

27 July 2020

  • curprev 22:1722:17, 27 July 2020imported>Stashbot 50,896 bytes +150 andrewbogott: ceph osd pool set compute pg_num 2048

24 July 2020

  • curprev 19:1519:15, 24 July 2020imported>Stashbot 50,746 bytes +148 andrewbogott: ceph mgr module enable pg_autoscaler

22 July 2020

16 July 2020

  • curprev 10:4810:48, 16 July 2020imported>Stashbot 50,307 bytes +158 arturo: merging change to neutron dmz_cidr https://gerrit.wikimedia.org/r/c/operations/puppet/+/613123 (T257534)

15 July 2020

  • curprev 23:1523:15, 15 July 2020imported>Stashbot 50,149 bytes +545 bd808: Removed Merlijn van Deen from toollabs-trusted Gerrit group (T255697)

14 July 2020

  • curprev 15:1915:19, 14 July 2020imported>Stashbot 49,604 bytes +504 arturo: briefly set root@cloudnet1003:~ # sysctl net.ipv4.conf.all.accept_local=1 (in neutron qrouter netns) (T257534)

13 July 2020

  • curprev 16:1716:17, 13 July 2020imported>Stashbot 49,100 bytes +127 arturo: icinga downtime cloudcontrol[1003-1005].wikimedia.org for 1h for galera database movements

12 July 2020

  • curprev 17:3917:39, 12 July 2020imported>Stashbot 48,973 bytes +98 andrewbogott: switched eqiad1 keystone from m5 to cloudcontrol galera

10 July 2020

  • curprev 20:2620:26, 10 July 2020imported>Stashbot 48,875 bytes +88 andrewbogott: disabling nova api to move database to galera

9 July 2020

  • curprev 11:2311:23, 9 July 2020imported>Stashbot 48,787 bytes +378 arturo: [codfw1dev] rebooting cloudnet2003-dev again for testing sysct/puppet behavior (T257552)

6 July 2020

  • curprev 15:1615:16, 6 July 2020imported>Stashbot 48,409 bytes +76 arturo: installing 'aptitude' in all cloudvirts

3 July 2020

  • curprev 12:5112:51, 3 July 2020imported>Stashbot 48,333 bytes +455 arturo: [codfw1dev] galera cluster should be up and running, openstack happy (T256283)

2 July 2020

  • curprev 15:4115:41, 2 July 2020imported>Stashbot 47,878 bytes +273 arturo: `sudo wmcs-openstack --os-compute-api-version 2.55 flavor create --private --vcpus 8 --disk 300 --ram 16384 --property aggregate_instance_extra_specs:ceph=true --description "for packaging envoy" bigdisk-ceph` (T256983)

29 June 2020

  • curprev 14:2414:24, 29 June 2020imported>Stashbot 47,605 bytes +162 arturo: starting rabbitmq-server in all 3 cloudcontrol servers

18 June 2020

  • curprev 20:3820:38, 18 June 2020imported>Stashbot 47,443 bytes +130 andrewbogott: rebooting cloudservices2003-dev due to a mysterious 'host down' alert on a secondary ip

16 June 2020

  • curprev 15:3815:38, 16 June 2020imported>Stashbot 47,313 bytes +159 arturo: created by hand neutron port 9c0a9a13-e409-49de-9ba3-bc8ec4801dbf `paws-haproxy-vip` (T295217)

12 June 2020

  • curprev 13:2313:23, 12 June 2020imported>Stashbot 47,154 bytes +202 arturo: DNS zone `paws.wmcloud.org` transferred to the PAWS project (T195217)

11 June 2020

  • curprev 19:1919:19, 11 June 2020imported>Stashbot 46,952 bytes +428 bstorm_: proceeding with failback to labstore1004 now that DRBD devices are consistent T224582

10 June 2020

  • curprev 16:0916:09, 10 June 2020imported>Stashbot 46,524 bytes +169 andrewbogott: deleting all old cloud-ns0.wikimedia.org and cloud-ns1.wikimedia.org ns records in designate database T254496

9 June 2020

  • curprev 15:2515:25, 9 June 2020imported>Stashbot 46,355 bytes +337 arturo: icinga downtime everything cloud* lab* for 2h more (T253780)

5 June 2020

  • curprev 15:0815:08, 5 June 2020imported>Stashbot 46,018 bytes +148 andrewbogott: trying to re-enable puppet without losing cumin contact, as per https://phabricator.wikimedia.org/T254589

4 June 2020

  • curprev 14:2414:24, 4 June 2020imported>Stashbot 45,870 bytes +180 andrewbogott: disabling puppet on all instances for /labs/private recovery

28 May 2020

  • curprev 23:0223:02, 28 May 2020imported>Stashbot 45,690 bytes +169 bd808: `/usr/local/sbin/maintain-dbusers --debug harvest-replicas` (T253930)
  • curprev 00:3300:33, 28 May 2020imported>Stashbot 45,521 bytes +610 andrewbogott: shutting down cloudservices2002-dev to see if we can live without it. This is in anticipation or rebuilding it entirely for T253780

25 May 2020

  • curprev 16:3616:36, 25 May 2020imported>Stashbot 44,911 bytes +119 arturo: [codfw1dev] created zone `0-29.57.15.185.in-addr.arpa.` (T247972)

21 May 2020

  • curprev 19:2319:23, 21 May 2020imported>Stashbot 44,792 bytes +336 andrewbogott: disabling puppet on cloudbackup2001 to prevent the backup job from starting during maintenance

19 May 2020

  • curprev 22:5922:59, 19 May 2020imported>Stashbot 44,456 bytes +181 bd808: `apt-get install mariadb-client` on cloudcontrol1003

18 May 2020

  • curprev 21:3721:37, 18 May 2020imported>Stashbot 44,275 bytes +82 andrewbogott: rebuilding cloudnet2003-dev with Buster

15 May 2020

  • curprev 22:1022:10, 15 May 2020imported>Stashbot 44,193 bytes +375 bd808: Added reedy as projectadmin in cloudinfra project (T249774)

14 May 2020

  • curprev 23:2823:28, 14 May 2020imported>Stashbot 43,818 bytes +724 bstorm_: downtimed cloudvirt1004/6 and cloudvirt-wdqs1003 until tomorrow around this time T252831

12 May 2020

  • curprev 20:3320:33, 12 May 2020imported>Stashbot 43,094 bytes +747 andrewbogott: moving cloudvirt1023 to the 'standard' pool and out of the 'spare' pool

9 May 2020

  • curprev 16:5316:53, 9 May 2020imported>Stashbot 42,347 bytes +128 andrewbogott: rebuilding cloudcontrol2001-dev and 2003-dev with buster for T252121

8 May 2020

  • curprev 19:0219:02, 8 May 2020imported>Stashbot 42,219 bytes +118 bstorm_: moving tools-k8s-haproxy-2 from cloudvirt1021 to cloudvirt1017 to improve spread

5 May 2020

  • curprev 13:5813:58, 5 May 2020imported>Stashbot 42,101 bytes +101 andrewbogott: rebuilding cloudcontrol2004-dev to test new puppet changes

4 May 2020

  • curprev 09:0409:04, 4 May 2020imported>Stashbot 42,000 bytes +194 arturo: [codfw1dev] manually modify iptables ruleset to only allow SSH from WMF bastions on cloudservices2003-dev and cloudcontrol2004-dev (T251604)

21 April 2020

  • curprev 22:1222:12, 21 April 2020imported>Stashbot 41,806 bytes +205 andrewbogott: moving cloudvirt1004 out of the 'standard' aggregate and into the 'maintenance' aggregate

15 April 2020

13 April 2020

  • curprev 15:0715:07, 13 April 2020imported>Stashbot 41,502 bytes +110 jeh: restart memcached on labwebs to increase cache size T145703

9 April 2020

8 April 2020

  • curprev 19:2019:20, 8 April 2020imported>Stashbot 41,236 bytes +239 andrewbogott: rotated password and api token for pdns servers on cloudservices1003 and cloudservices1004

7 April 2020

  • curprev 20:5720:57, 7 April 2020imported>Stashbot 40,997 bytes +128 andrewbogott: service sssd stop; rm -rf /var/lib/sss/db*; service sssd start on tools-sgebastion-08

6 April 2020

  • curprev 22:3922:39, 6 April 2020imported>Stashbot 40,869 bytes +634 andrewbogott: deleting bogus groups cn=b'project-bastion',ou=groups,dc=wikimedia,dc=org and cn=b'project-tools',ou=groups,dc=wikimedia,dc=org from ldap

2 April 2020

  • curprev 20:5920:59, 2 April 2020imported>Stashbot 40,235 bytes +112 jeh: codfw1dev clear VM error states and start bastions, puppet master and database

1 April 2020

  • curprev 16:2716:27, 1 April 2020imported>Stashbot 40,123 bytes +126 arturo: [codfw1dev] enable puppet across the fleet clean vxlan changes (T248881)

31 March 2020

  • curprev 12:3512:35, 31 March 2020imported>Stashbot 39,997 bytes +801 arturo: [codfw1dev] restarting VMs: designaterockytest14, bastion-codfw1dev-0[1,2] (T248881)

30 March 2020

  • curprev 23:4223:42, 30 March 2020imported>Stashbot 39,196 bytes +399 bstorm_: deleted "Kubernetes Cluster" and "Kubernetes Performance" dashboards T246689

27 March 2020

  • curprev 21:2821:28, 27 March 2020imported>Stashbot 38,797 bytes +181 bd808: Created huggle.wmcloud.org Designate zone and allocated it to the huggle project

26 March 2020

  • curprev 15:0115:01, 26 March 2020imported>Stashbot 38,616 bytes +325 arturo: icinga downtime cloudvirt* cloudcontrol* cloudnet* lab* cloudstore*

25 March 2020

  • curprev 19:2919:29, 25 March 2020imported>Stashbot 38,291 bytes +285 andrewbogott: dumping a bunch of VMs on cloudvirt1015 to see if it still crashes

24 March 2020

  • curprev 19:4119:41, 24 March 2020imported>Stashbot 38,006 bytes +225 jeh: switch cloudvirt1016 from maintenance to standard host aggregate T243327

23 March 2020

  • curprev 21:4121:41, 23 March 2020imported>Stashbot 37,781 bytes +305 jeh: restart neutron-l3-agent on cloudnet100[3,4] to pickup policy.yaml changes

21 March 2020

  • curprev 14:2314:23, 21 March 2020imported>Stashbot 37,476 bytes +84 andrewbogott: restarting apache2 on labweb1001 and 1002

18 March 2020

  • curprev 19:1719:17, 18 March 2020imported>Stashbot 37,392 bytes +334 andrewbogott: deleted a bunch of records from the pdns database on cloudservices1003/1004 which had a record name but the content (where an IP address should be) was NULL, e.g. m.wikidata.beta.wmflabs.org.

14 March 2020

13 March 2020

  • curprev 12:3912:39, 13 March 2020imported>Stashbot 36,959 bytes +302 arturo: [codfw1dev] reintroduce address scopes for another round of testing T244851

12 March 2020

  • curprev 22:2922:29, 12 March 2020imported>Stashbot 36,657 bytes +130 bstorm_: running puppet across all dumps mounts to make sure active links are shifted to labstore1006

11 March 2020

  • curprev 18:3818:38, 11 March 2020imported>Stashbot 36,527 bytes +579 jeh: set icingia downtime until 2020-03-23 on CODFW cloud[control,net,virt] hosts during openstack upgrades

10 March 2020

  • curprev 17:0217:02, 10 March 2020imported>Stashbot 35,948 bytes +272 arturo: [codfw1dev] deleting address scopes, bad interaction with our custom NAT setup T247135

9 March 2020

  • curprev 18:0918:09, 9 March 2020imported>Stashbot 35,676 bytes +343 arturo: enabling puppet in cloudvirt1006, all services have been restored

6 March 2020

  • curprev 14:5414:54, 6 March 2020imported>Stashbot 35,333 bytes +115 andrewbogott: draining all instances off of cloudvirt1006 for T246908

5 March 2020

  • curprev 14:2414:24, 5 March 2020imported>Stashbot 35,218 bytes +475 arturo: [codfw1dev] we just enabled BGP session between cloudnet2xxx-dev and cr1-codfw (T245606)

4 March 2020

  • curprev 22:2222:22, 4 March 2020imported>Stashbot 34,743 bytes +776 andrewbogott: upgrading designate on cloudservices1003/1004 to Queens

2 March 2020

  • curprev 16:5416:54, 2 March 2020imported>Stashbot 33,967 bytes +159 arturo: [codfw1dev] deleted python3-os-ken debian package in cloudnet2003-dev which was installed by hand and had depedency issues

29 February 2020

  • curprev 16:3216:32, 29 February 2020imported>Stashbot 33,808 bytes +160 bstorm_: downtimed the smart alert on cloudvirt1009 until Monday since apparently predictive failures flap T244986

26 February 2020

25 February 2020

  • curprev 16:0816:08, 25 February 2020imported>Stashbot 33,562 bytes +458 andrewbogott: changing neutron's rabbitmq password because oslo is having trouble parsing some of the characters in the password

24 February 2020

  • curprev 12:1612:16, 24 February 2020imported>Stashbot 33,104 bytes +1,060 arturo: [codfw1dev] `root@cloudcontrol2001-dev:~# neutron bgp-speaker-peer-add bgpspeaker cr2-codfw` (T245606)

21 February 2020

  • curprev 12:4812:48, 21 February 2020imported>Stashbot 32,044 bytes +771 arturo: [codfw1dev] running `root@cloudcontrol2001-dev:~# neutron bgp-speaker-network-add bgpspeaker wan-transport-codfw` (T245606)

20 February 2020

  • curprev 19:2219:22, 20 February 2020imported>Stashbot 31,273 bytes +478 andrewbogott: updating designate pool config for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/572213/

18 February 2020

  • curprev 22:1922:19, 18 February 2020imported>Stashbot 30,795 bytes +429 andrewbogott: transferred the tools.wmcloud.org. to the tools project

14 February 2020

  • curprev 10:3510:35, 14 February 2020imported>Stashbot 30,366 bytes +477 arturo: running `root@cloudcontrol2001-dev:~# designate server-create --name ns1.openstack.codfw1dev.wikimediacloud.org.` (T243766)

12 February 2020

  • curprev 13:3813:38, 12 February 2020imported>Stashbot 29,889 bytes +270 arturo: [codfw1dev] add reference to subnetpool to the instance subnet `MariaDB [neutron]> update subnets set subnetpool_id='d129650d-d4be-4fe1-b13e-6edb5565cb4a' where id = '7adfcebe-b3d0-4315-92fe-e8365cc80668';` (T244851)

11 February 2020

  • curprev 13:4613:46, 11 February 2020imported>Stashbot 29,619 bytes +570 arturo: [codfw1dev] creating some neutron objects to investigate T244851 (subnets, subnet pools, address scopes, ...)

7 February 2020

6 February 2020

28 January 2020

  • curprev 17:2417:24, 28 January 2020imported>Stashbot 28,774 bytes +1,040 arturo: [codfw1dev] root@cloudcontrol2001-dev:~# designate server-create --name ns0.openstack.codfw1dev.wikimediacloud.org. (T243766)

27 January 2020

  • curprev 12:4512:45, 27 January 2020imported>Stashbot 27,734 bytes +495 arturo: [codfw1dev] manually move the new domain to the `cloudinfra-codfw1dev` project clouddb2001-dev: `[designate]> update zones set tenant_id='cloudinfra-codfw1dev' where id = '4c75410017904858a5839de93c9e8b3d';` T243556

24 January 2020

21 January 2020

  • curprev 17:4317:43, 21 January 2020imported>Stashbot 27,054 bytes +253 bstorm_: remounting /mnt/nfs/dumps-labstore1007.wikimedia.org/ on all dumps-mounting projects

15 January 2020

  • curprev 16:5916:59, 15 January 2020imported>Stashbot 26,801 bytes +144 bd808: Changed the config for cloud-announce mailing list so that lsit admins do not get bounce unsubscribe notices

14 January 2020

  • curprev 14:0314:03, 14 January 2020imported>Stashbot 26,657 bytes +395 arturo: icinga downtime all cloudvirts for another 2h for fixing some icinga checks

13 January 2020

  • curprev 13:3413:34, 13 January 2020imported>Stashbot 26,262 bytes +269 arturo: [¢odfw1dev] prevent neutron from allocating floating IPs from the wrong subnet by doing `neutron subnet-update --allocation-pool start=208.80.153.190,end=208.80.153.190 cloud-instances-transport1-b-codfw` (T242594)

10 January 2020

  • curprev 13:2713:27, 10 January 2020imported>Stashbot 25,993 bytes +167 arturo: cloudvirt1009: virsh undefine i-000069b6. This is tools-elastic-01 which is running on cloudvirt1008 (so, leaked on cloudvirt1009)

9 January 2020

  • curprev 11:1211:12, 9 January 2020imported>Stashbot 25,826 bytes +397 arturo: running `MariaDB [nova_eqiad1]> update quota_usages set in_use='0' where project_id='etytree';` (T242332)

8 January 2020

  • curprev 10:5310:53, 8 January 2020imported>Stashbot 25,429 bytes +111 arturo: icinga downtime all cloudvirts for 30 minutes to re-create all canary VMs"

7 January 2020

  • curprev 11:1211:12, 7 January 2020imported>Stashbot 25,318 bytes +228 arturo: icinga-downtime everything cloud* for 30 minutes to merge nova scheduler changes

6 January 2020

  • curprev 13:4513:45, 6 January 2020imported>Stashbot 25,090 bytes +110 andrewbogott: restarting nova-api and nova-conductor on cloudcontrol1003 and 1004

4 January 2020

  • curprev 16:3416:34, 4 January 2020imported>Stashbot 24,980 bytes +130 arturo: icinga downtime cloudvirt1024 for 2 months because hardware errors (T241884)

31 December 2019

25 December 2019

  • curprev 10:1310:13, 25 December 2019imported>Stashbot 24,692 bytes +205 arturo: icinga downtime for 30 minutes the whole cloud* lab* fleet to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/560575 (will restart some openstack components)

24 December 2019

  • curprev 15:1315:13, 24 December 2019imported>Stashbot 24,487 bytes +188 arturo: icinga downtime all the lab* fleet for nova password change for 1h

23 December 2019

22 December 2019

  • curprev 23:4823:48, 22 December 2019imported>Stashbot 24,125 bytes +290 andrewbogott: restarting nova-conductor and nova-api on cloudcontrol1003 and 1004

20 December 2019

18 December 2019

  • curprev 12:5512:55, 18 December 2019imported>Stashbot 23,752 bytes +191 arturo: [codfw1dev] created a new subnet neutron object to hold the new CIDR for floating IPs (cloud-codfw1dev-floating - 185.15.57.0/29) T239347

17 December 2019

12 December 2019

2 December 2019

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)