Nova Resource:Devtools/SAL

2021-07-28

• 16:39 andrewbogott: rebooting gerrit-prod-1001; seemingly unreachable

2021-03-10

• 10:58 arturo: briefly stopped VM 'doc' to disable VMX cpu flag and live-migrate it

2021-02-22

• 20:58 mutante: fixed puppet run on deploy-1002 by adding empty array of wikimedia-sites to hiera
• 20:01 mutante: deploy-1002 is broken because mediawiki::sites is not in Hiera (yet)

2020-10-28

• 17:01 andrewbogott: fixed puppet runs on phabricator-stage-1001 (previously puppetmaster name mismatch)

2020-09-01

• 00:16 mutante: - unbreaking puppet run on the local deployment after it was broken since July due to changes in prod deployment_server role

2020-06-30

• 20:22 mutante: managed to let certbot get LE certs for gerrit.devtools.wmflabs.org and the floating IP

2020-06-17

• 19:55 paladox: ran `iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT` on phabricator-prod-1001

2020-05-08

• 07:01 mutante: phabricator-prod-1001 - removing cron for public task dump (though puppet should have removed it)

2020-05-07

• 09:24 mutante: - cloud puppetmasters still affected by https://phabricator.wikimedia.org/T83447#5807825
• 09:07 mutante: - puppetmaster-1001 - Permission denied @ rb_sysopen - /var/lib/puppet/volatile/GeoIP/.geoipupdate.lock
• 09:06 mutante: - avoiding the need for a second role for deployment_servers in cloud with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594903
• 09:05 mutante: - puppet fixed on deploy-1002 with https://gerrit.wikimedia.org/r/c/operations/puppet/+/594900
• 08:04 mutante: - broken puppet again from prod changes. this time: deploy-1002 - []' is not applicable to an Undef Value. mediawiki/mcrouter_wancache.pp, line: 19

2020-04-13

• 10:00 mutante: - phabricator-stage-1001: replace deployment-tin.deployment-rep with deploy-1002.devtools in deployment-cache/.config
• 09:40 mutante: set missing (and new) profile::tlsproxy::envoy::capitalize_headers: true to fix puppet errors
• 09:35 mutante: set phabricator::vcs::address::v6 to fe80 local address to fix puppet error on phabricator-stage-1001

2020-01-16

• 00:53 mutante: deploy-1002 - become 'trebuchet' user and ssh to phabricator scap targets. to fix ssh host key verification issue on first deploy
• 00:30 mutante: deploy-1002 live hack /srv/deployment/phabricator/deployment/scap/phabricator-targets and replace prod server with cloud instances; scap deploy in phabricator repo

2020-01-15

• 23:51 paladox: deploy-1002 rm -rf /srv/deployment
• 23:44 mutante: deploy-1002 sudo git init in /srv/deployment ; scap deploy --init (now fails with 'fatal: Not a valid object name HEAD')
• 23:42 mutante: deploy-1002 mkdir /srv/deployments/.git ; chown trebuchet:wikidev .git ; manually run "scap deploy --init" as trebuchet user in an attempt to fix initial puppet run on deployment_server

2020-01-14

• 00:59 mutante: deleting instance codesearch-buster
• 00:54 mutante: - deleting instance codesearch-stretch, creating codesearch-buster

2020-01-11

• 00:35 mutante: deleting instance codesearch-buster, creating codesearch-stretch
• 00:05 mutante: s/cloudsearch/codesearch/g
• 00:04 mutante: creating throwaway instance "cloudsearch"
• 00:04 mutante: deleting instance deploy1001 (buster), creating deploy-1002 (stretch) instead

2020-01-04

• 16:01 bstorm_: moving vm puppetmaster-1001 from cloudvirt1024 to cloudvirt1009 due to hardware error T241884

2020-01-03

• 22:37 mutante: - sudo vi /srv/deployment/phabricator/deployment-cache/.config on both phabricator instances to fix deployment server (remove deployment-tin (!))
• 21:50 mutante: assigned 172.16.0.198/32 on eth0 on phabricator-prod-1001
• 21:50 jeh: add secondary interface to phabricator-prod-1001
• 21:32 mutante: configure 172.16.0.189 as "vcs" address v4 for phabricator-stage-1001
• 21:24 jeh: add secondary interface to phabricator-stage-1001
• 00:47 paladox: puppet cert generate puppetmaster-1001.devtools.eqiad.wmflabs
• 00:30 paladox: set puppetmaster: puppetmaster-1001.devtools.eqiad.wmflabs in hiera

2020-01-02

• 23:42 paladox: puppetmaster-1001: ln -s /var/lib/puppet/ssl/private_keys/puppetmaster-1001.devtools.eqiad.wmflabs.pem /var/lib/puppet/server/ssl/private_keys/puppetmaster-1001.devtools.eqiad.wmflabs.pem
• 23:40 paladox: puppetmaster-1001: ln -s /var/lib/puppet/ssl/certs/puppetmaster-1001.devtools.eqiad.wmflabs.pem /var/lib/puppet/server/ssl/certs/puppetmaster-1001.devtools.eqiad.wmflabs.pem
• 23:35 mutante: attempting to create local puppetmaster (formerly puppetmaster::self)

2019-12-23

• 04:14 mutante: - turns out instance creation doesn't work using Brave browser but does work using Firefox (T241345) - created phabricator-prod-1001
• 03:45 mutante: - creating new instance seems to fail - nothing shows up at all
• 03:42 mutante: - launching medium sized buster instance gerrit-prod-01 (T236309)