You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Nova Resource:Cloudinfra/SAL: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(Krenair: Drop internal-puppetmaster old cherry-pick of https://gerrit.wikimedia.org/r/c/operations/puppet/+/545567 which was preventing rebase and blocking cloud-puppetmaster machines from getting puppet commits themselves - new version of that change will have appeared there in the rebase)
imported>Stashbot
(taavi: delete old cloudinfra-db instances (cloudinfra-db01 cloudinfra-db02))
 
(33 intermediate revisions by the same user not shown)
Line 1: Line 1:
=== 2022-02-16 ===
* 14:44 taavi: delete old cloudinfra-db instances (cloudinfra-db01 cloudinfra-db02)
* 12:10 taavi: enable gtid (master_use_gtid=current_pos) on cloudinfra-db04
=== 2022-02-13 ===
* 17:14 taavi: encapi db switchover: cloudinfra-db01 -> cloudinfra-db03
* 11:38 taavi: generate new profile::mariadb::grants::cloudinfra::repl_pass, previous one seems to have been lost in the 2020-06-04 labs/private incident
* 10:48 taavi: shutdown and delete ntp-01 and ntp-02 running stretch, replaced by -03 and -04
=== 2022-02-11 ===
* 15:41 taavi: switch floating ip 185.15.56.3 from ntp-01 to ntp-03
* 15:18 taavi: switch floating ip 185.15.56.27 from ntp-02 to ntp-04
=== 2022-02-09 ===
* 18:33 taavi: deleted cloud-cumin-01 [[phab:T255980|T255980]]
* 18:31 taavi: deleted cloud-cumin-02 [[phab:T255980|T255980]]
=== 2022-01-06 ===
* 13:18 taavi: shut down cloud-cumin-02 [[phab:T255980|T255980]]
=== 2021-11-11 ===
* 10:50 arturo: add user `srv-networktests` as project user ([[phab:T294955|T294955]])
=== 2021-11-04 ===
* 16:42 majavah: provisioning cloud-cumin-03 as bullseye
=== 2021-10-06 ===
* 09:47 dcaro: upgraded cloudmetrics to grafana 7.5 ([[phab:T292614|T292614]])
=== 2021-07-19 ===
* 22:49 bstorm: fixed rebase conflicts in labs-private
* 22:43 bstorm: disabling puppet on entire project before meddling with labs-private rebase
=== 2021-05-05 ===
* 14:33 arturo: delete also old VMs mx-out01/02
* 14:33 arturo: delete zones 'mx-out01.wmflabs.org' and 'mx-out02.wmflabs.org'
=== 2021-05-04 ===
* 15:58 arturo: shutoff mx-out01 and mx-out02 (migrated to mx-out03/mx-out04)
* 15:56 arturo: relocate floating IPs 185.15.56.18 and .19 from mx-out01/mx-out02 to mx-out03/mx-out04
* 15:33 arturo: created VMs mx-out03/mx-out04 as debian buster
* 15:31 arturo: bump instance quota from 12 to 14
* 15:28 arturo: created anti-affinity server group 'mx'
=== 2021-03-01 ===
* 10:37 dcaro: rebooting cloudinfra-acme-chief-01 to ensure hostname stability ([[phab:T276041|T276041]])
=== 2021-02-26 ===
* 20:46 andrewbogott: rebooting all hosts
=== 2021-01-08 ===
* 09:04 dcaro: manually testing patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/655019 to the puppetmaster to test ([[phab:T271509|T271509]])
=== 2021-01-07 ===
* 09:49 dcaro: Added recordset for mx-out02.wmflabs.org ([[phab:T271322|T271322]])
* 09:46 dcaro: Added recordset for mx-out01.wmflabs.org ([[phab:T271322|T271322]])
=== 2021-01-05 ===
* 14:11 dcaro: finished ssl tests for enc, cleaned up cloud-puppetmaster-03 ([[phab:T268877|T268877]])
* 13:07 dcaro: adding custom nginx config for labspuppetbackend on cloud-puppetmaster-03 to test ssl ([[phab:T268877|T268877]])
* 12:41 arturo: live-hacking cloudinfra-internal-puppetmaster02 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/654415 ([[phab:T260834|T260834]])
* 12:31 arturo: refresh acme-chief config for mx certs https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/949f1b4e81f3a1c6d4f4825292343f1ee17c48a1%5E%21/ ([[phab:T260834|T260834]])
* 12:21 arturo: resolve git merge conflicts and rebase cloudinfra-internal-puppetmaster-02 /var/lib/git/labs/private
* 12:12 arturo: created puppet prefix `mx-out` and added hiera to use internal puppetmaster ([[phab:T260834|T260834]])
=== 2020-11-09 ===
* 10:23 arturo: added jmm (moritz) as user & projectadmin
=== 2020-11-05 ===
* 16:28 dcaro: add myself as user and projectadmin
=== 2020-10-21 ===
* 21:17 andrewbogott: deleting broken cloud-puppetmaster-04
* 21:14 andrewbogott: switching secondary puppetmaster from cloud-puppetmaster-04 to cloud-puppetmaster-05
=== 2020-10-06 ===
* 11:31 arturo: cleanup local changes in ops/puppet git repo in cloud-puppetmaster-03
=== 2020-10-05 ===
* 21:58 bstorm: setting "mtail::from_component: true" on both mx-out servers to make puppet work again
=== 2020-09-02 ===
* 15:01 arturo: linvehacking ended
* 14:20 arturo: live-hacking cloud-puppetmaster-03
=== 2020-08-28 ===
* 19:12 bstorm: moving aside weird old mitaka-jessie sources.list file on cloudinfra-db02
=== 2020-08-04 ===
* 18:24 bd808: Made DNS entry lists.wmcloud.org A 185.15.56.49 (Tried CNAME to mailman.wmcloud.org first but Horizon didn't like that) ([[phab:T259444|T259444]])
=== 2020-07-31 ===
* 20:08 bd808: Added nskaggs as projectadmin
=== 2020-07-20 ===
* 21:33 bd808: Created CNAME record for www.wmcloud.org pointing to  wmcloud.org ([[phab:T258415|T258415]])
* 21:25 bd808: Created A record for wmcloud.org pointing to proxy-proxy well known IP ([[phab:T258415|T258415]])
=== 2020-04-15 ===
* 20:10 jeh: update default security group to allow prometheus01.metricsinfra.eqiad.wmflabs TCP 9100 [[phab:T250206|T250206]]
=== 2020-03-30 ===
* 16:55 arturo: dropping `_psl.wmcloud.org` record ([[phab:T168677|T168677]])
=== 2020-03-04 ===
* 22:33 Krenair: Shutoff cloudinfra-internal-puppetmaster01, replaced with -02 per [[phab:T241719|T241719]]
=== 2020-02-21 ===
* 15:46 jeh: cloud-puppetmaster-03 cleanup puppet agent certs that were missed by wmfsink
* 00:18 andrewbogott: temporarily shutting down cloud-puppetmaster-01 and -02 as part of debugging the new puppetmasters
=== 2020-02-20 ===
* 22:45 Krenair: Swapped 185.15.56.64 floating IP (backing puppetmaster.cloudinfra.wmflabs.org) over to cloud-puppetmaster-03 from cloud-puppetmaster-01
=== 2020-01-28 ===
* 10:03 arturo: delegated `codfw1dev.wmcloud.org` to designate @ codfw1dev ns0.openstack.codfw1dev.wikimediacloud.org ([[phab:T242976|T242976]] and [[phab:T243766|T243766]])
* 09:53 arturo: the DNS zone wmcloud.org now belongs to this project ([[phab:T242976|T242976]])
=== 2020-01-02 ===
* 23:34 mutante: cloud-puppetmaster-01 puppet cert clean puppetmaster-1001.devtools.eqiad.wmflabs
=== 2019-11-29 ===
* 10:29 arturo: re-arm keyholder in cloud-cumin-02 (password in pwstore)
* 10:13 arturo: re-arm keyholder in cloud-cumin-01
=== 2019-11-09 ===
=== 2019-11-09 ===
* 18:30 Krenair: Drop internal-puppetmaster old cherry-pick of https://gerrit.wikimedia.org/r/c/operations/puppet/+/545567 which was preventing rebase and blocking cloud-puppetmaster machines from getting puppet commits themselves - new version of that change will have appeared there in the rebase
* 18:30 Krenair: Drop internal-puppetmaster old cherry-pick of https://gerrit.wikimedia.org/r/c/operations/puppet/+/545567 which was preventing rebase and blocking cloud-puppetmaster machines from getting puppet commits themselves - new version of that change will have appeared there in the rebase

Latest revision as of 14:44, 16 February 2022

2022-02-16

  • 14:44 taavi: delete old cloudinfra-db instances (cloudinfra-db01 cloudinfra-db02)
  • 12:10 taavi: enable gtid (master_use_gtid=current_pos) on cloudinfra-db04

2022-02-13

  • 17:14 taavi: encapi db switchover: cloudinfra-db01 -> cloudinfra-db03
  • 11:38 taavi: generate new profile::mariadb::grants::cloudinfra::repl_pass, previous one seems to have been lost in the 2020-06-04 labs/private incident
  • 10:48 taavi: shutdown and delete ntp-01 and ntp-02 running stretch, replaced by -03 and -04

2022-02-11

  • 15:41 taavi: switch floating ip 185.15.56.3 from ntp-01 to ntp-03
  • 15:18 taavi: switch floating ip 185.15.56.27 from ntp-02 to ntp-04

2022-02-09

  • 18:33 taavi: deleted cloud-cumin-01 T255980
  • 18:31 taavi: deleted cloud-cumin-02 T255980

2022-01-06

  • 13:18 taavi: shut down cloud-cumin-02 T255980

2021-11-11

  • 10:50 arturo: add user `srv-networktests` as project user (T294955)

2021-11-04

  • 16:42 majavah: provisioning cloud-cumin-03 as bullseye

2021-10-06

  • 09:47 dcaro: upgraded cloudmetrics to grafana 7.5 (T292614)

2021-07-19

  • 22:49 bstorm: fixed rebase conflicts in labs-private
  • 22:43 bstorm: disabling puppet on entire project before meddling with labs-private rebase

2021-05-05

  • 14:33 arturo: delete also old VMs mx-out01/02
  • 14:33 arturo: delete zones 'mx-out01.wmflabs.org' and 'mx-out02.wmflabs.org'

2021-05-04

  • 15:58 arturo: shutoff mx-out01 and mx-out02 (migrated to mx-out03/mx-out04)
  • 15:56 arturo: relocate floating IPs 185.15.56.18 and .19 from mx-out01/mx-out02 to mx-out03/mx-out04
  • 15:33 arturo: created VMs mx-out03/mx-out04 as debian buster
  • 15:31 arturo: bump instance quota from 12 to 14
  • 15:28 arturo: created anti-affinity server group 'mx'

2021-03-01

  • 10:37 dcaro: rebooting cloudinfra-acme-chief-01 to ensure hostname stability (T276041)

2021-02-26

  • 20:46 andrewbogott: rebooting all hosts

2021-01-08

2021-01-07

  • 09:49 dcaro: Added recordset for mx-out02.wmflabs.org (T271322)
  • 09:46 dcaro: Added recordset for mx-out01.wmflabs.org (T271322)

2021-01-05

2020-11-09

  • 10:23 arturo: added jmm (moritz) as user & projectadmin

2020-11-05

  • 16:28 dcaro: add myself as user and projectadmin

2020-10-21

  • 21:17 andrewbogott: deleting broken cloud-puppetmaster-04
  • 21:14 andrewbogott: switching secondary puppetmaster from cloud-puppetmaster-04 to cloud-puppetmaster-05

2020-10-06

  • 11:31 arturo: cleanup local changes in ops/puppet git repo in cloud-puppetmaster-03

2020-10-05

  • 21:58 bstorm: setting "mtail::from_component: true" on both mx-out servers to make puppet work again

2020-09-02

  • 15:01 arturo: linvehacking ended
  • 14:20 arturo: live-hacking cloud-puppetmaster-03

2020-08-28

  • 19:12 bstorm: moving aside weird old mitaka-jessie sources.list file on cloudinfra-db02

2020-08-04

  • 18:24 bd808: Made DNS entry lists.wmcloud.org A 185.15.56.49 (Tried CNAME to mailman.wmcloud.org first but Horizon didn't like that) (T259444)

2020-07-31

  • 20:08 bd808: Added nskaggs as projectadmin

2020-07-20

  • 21:33 bd808: Created CNAME record for www.wmcloud.org pointing to wmcloud.org (T258415)
  • 21:25 bd808: Created A record for wmcloud.org pointing to proxy-proxy well known IP (T258415)

2020-04-15

  • 20:10 jeh: update default security group to allow prometheus01.metricsinfra.eqiad.wmflabs TCP 9100 T250206

2020-03-30

  • 16:55 arturo: dropping `_psl.wmcloud.org` record (T168677)

2020-03-04

  • 22:33 Krenair: Shutoff cloudinfra-internal-puppetmaster01, replaced with -02 per T241719

2020-02-21

  • 15:46 jeh: cloud-puppetmaster-03 cleanup puppet agent certs that were missed by wmfsink
  • 00:18 andrewbogott: temporarily shutting down cloud-puppetmaster-01 and -02 as part of debugging the new puppetmasters

2020-02-20

  • 22:45 Krenair: Swapped 185.15.56.64 floating IP (backing puppetmaster.cloudinfra.wmflabs.org) over to cloud-puppetmaster-03 from cloud-puppetmaster-01

2020-01-28

  • 10:03 arturo: delegated `codfw1dev.wmcloud.org` to designate @ codfw1dev ns0.openstack.codfw1dev.wikimediacloud.org (T242976 and T243766)
  • 09:53 arturo: the DNS zone wmcloud.org now belongs to this project (T242976)

2020-01-02

  • 23:34 mutante: cloud-puppetmaster-01 puppet cert clean puppetmaster-1001.devtools.eqiad.wmflabs

2019-11-29

  • 10:29 arturo: re-arm keyholder in cloud-cumin-02 (password in pwstore)
  • 10:13 arturo: re-arm keyholder in cloud-cumin-01

2019-11-09

  • 18:30 Krenair: Drop internal-puppetmaster old cherry-pick of https://gerrit.wikimedia.org/r/c/operations/puppet/+/545567 which was preventing rebase and blocking cloud-puppetmaster machines from getting puppet commits themselves - new version of that change will have appeared there in the rebase

2019-10-23

  • 09:55 arturo: manually restart mariadb in cloudinfra-db02 to fix replication
  • 09:41 arturo: the cloudinfra-db01 VM was rebooted bc the hypervisor rebooted
  • 09:41 arturo: manually start mariadb in cloudinfra-db01

2019-10-17

  • 16:29 jeh: cleanup old fullstackd puppet certs
  • 12:52 arturo: add phamhi as user and projectadmin

2019-09-09

  • 18:23 Krenair: Started mariadb service on cloudinfra-db02 - was not running for some reason

2019-04-08

  • 14:17 andrewbogott: moved cloudinfra-internal-puppetmaster01 to cloudvirt1026

2019-04-02

  • 20:07 andrewbogott: moving cloudinfra-db02 to cloudvirt1017

2019-03-26

  • 17:43 andrewbogott: removing "profile::ldap::client::labs::restricted_to: ops" from project puppet to allow volunteer admin access
  • 17:21 andrewbogott: adding Krenair as projectadmin as per T218448

2019-01-07

  • 21:08 gtirloni: upgraded kernel and rebooted mx-out{01,02}