Difference between revisions of "Nova Resource:Codesearch/SAL"

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(majavah: hard reboot codesearch8 after OOM crash)
imported>Stashbot
(legoktm: one more restart of everything, switched hound container over to bullseye for newer git)
Line 1: Line 1:
=== 2021-10-24 ===
* 22:39 legoktm: one more restart of everything, switched hound container over to bullseye for newer git
* 22:20 legoktm: restarted everything to add new prometheus metrics endpoint: https://codesearch.wmcloud.org/_metrics
=== 2021-10-23 ===
=== 2021-10-23 ===
* 16:07 majavah: hard reboot codesearch8 after OOM crash
* 16:07 majavah: hard reboot codesearch8 after OOM crash

Revision as of 22:39, 24 October 2021

2021-10-24

  • 22:39 legoktm: one more restart of everything, switched hound container over to bullseye for newer git
  • 22:20 legoktm: restarted everything to add new prometheus metrics endpoint: https://codesearch.wmcloud.org/_metrics

2021-10-23

  • 16:07 majavah: hard reboot codesearch8 after OOM crash

2021-05-21

  • 16:08 legoktm: restarting everything for T283317

2021-05-20

  • 23:45 legoktm: restarting to switch docker images

2021-04-14

  • 01:38 legoktm: deleting codesearch6 instance (T279763)

2021-04-13

  • 20:53 legoktm: switched codesearch.wmcloud.org and codesearch-beta.wmcloud.org to point to codesearch8 (T279763)

2021-04-09

  • 08:57 legoktm: deleting codesearch7, creating codesearch8 as the correct size (T279763)
  • 08:52 legoktm: creating codesearch7 instance (T279763)
  • 08:48 legoktm: moved role::labs::lvm::srv from project puppet to codesearch06's puppet with the goal of moving to cinder

2021-03-05

  • 23:45 legoktm: restarting for new shouthow profile (T253597)

2021-01-26

  • 02:01 legoktm: restarting for new puppet profile (T272947)

2021-01-09

  • 00:52 legoktm: bunch of restarts, repos with non-master default branches should be properly supported

2020-11-25

  • 19:03 legoktm: fixed permissions on /srv/codesearch/.git/objects, forcing puppet run which is triggering restarts

2020-11-18

  • 22:45 Amir1: restart codesearch to pick up new config (T268187)

2020-11-09

  • 11:05 Amir1: restart codesearch

2020-10-30

  • 22:43 GarbageMonster: restarting codesearch to pick up new config (T266909)

2020-10-05

  • 04:57 legoktm: restarting to deploy T264557

2020-09-22

  • 20:33 legoktm: (re)starting all hound instances now that gerrit-replica is back up

2020-08-28

  • 19:17 Amir1: restarting codesearch to pick up the new config (T261517)

2020-08-12

  • 08:57 legoktm: restarting proxy to implement etag changes

2020-08-11

  • 23:48 legoktm: restarting hound-deployed for the same reason
  • 21:30 Amir1: restarting hound-search to pick up the new config for services (T260124)

2020-07-26

2020-07-24

  • 13:39 Amir1: restarting codesearch to pick up the new config (T231572)

2020-07-22

  • 14:03 Amir1: restart codesearch to pick up new config (adding mediawiki/vagrant)

2020-07-13

  • 21:04 Amir1: create codesearch.wmcloud.org dns proxy (T256206)

2020-06-28

  • 09:31 legoktm: root@codesearch6:~# chown codesearch:codesearch -R /srv/codesearch

2020-06-11

  • 11:04 legoktm: restarting everything after gerrit-replica 502s fixed T255094 T255125

2020-03-06

  • 18:03 bd808: Forced Puppet run on codesearch6 to apply Puppet fix for T246017

2020-02-25

  • 21:52 Amir1: restart hound proxy
  • 21:50 Amir1: sudo service hound-libraries restart (T246120 T246121)

2020-02-24

  • 17:51 bd808: `systemctl restart docker` on codesearch6 (T246017)
  • 17:37 bd808: `systemctl restart hound-search.service` on codesearch6 (T246017)

2020-02-22

  • 16:54 Amir1: hard reboot of codesearch6 (T245920)

2020-02-19

  • 14:48 Amir1: restarting hound_proxy to pick up the new changes

2020-02-13

  • 19:02 Reedy: is down "Hound is still starting up, please wait a few minutes for the initial indexing to complete."

2020-01-25

  • 04:14 legoktm: restarted for new libraries search profile

2020-01-23

  • 02:00 legoktm: shutting down and deleting temporary codesearch5 instance
  • 01:59 legoktm: shutting down old codesearch4 instance (but not deleting yet)
  • 01:57 legoktm: deleting temporary codesearch6.wmflabs.org domain
  • 01:56 legoktm: retargetting codesearch.wmflabs.org to codesearch6 (T242319)

2020-01-19

  • 06:08 legoktm: temporarily taking everything down to free up disk space (T243121)

2020-01-17

  • 07:37 legoktm: rolling restart of hound instances to test new bootstrapping overload protection

2020-01-16

  • 20:17 legoktm: legoktm@codesearch5:~$ sudo crontab -u codesearch -r

2020-01-11

  • 03:57 legoktm: added mutante as project admin to help with puppetization efforts

2019-12-22

  • 12:29 Amir1: ladsgroup@codesearch4:~$ sudo service hound-armchairgm restart (T241319)

2019-12-15

  • 01:30 legoktm: rolling restart of all hound instances because of gerrit-replica outage (T240776, T240763)
  • 00:40 Amir1: sudo service hound-search restart (T240776)

2019-10-16

  • 10:17 Amir1: ladsgroup@codesearch4:~$ sudo service hound-services restart (T235596)

2019-09-30

  • 16:14 Amir1: soft-rebooting codesearch4
  • 15:49 Amir1: ladsgroup@codesearch4:~$ sudo MODE=restart /srv/codesearch/manage.sh (T234211)

2019-09-29

  • 21:49 Amir1: git pull for puppet (T219248)

2019-09-18

  • 00:06 Amir1: another restart

2019-09-17

  • 23:48 Amir1: restart to pick up the shiny new "services" section

2019-09-16

  • 15:56 Amir1: ladsgroup@codesearch4:~$ sudo MODE=restart /srv/codesearch/manage.sh (It seems it's down, forcing a restart)

2019-09-03

  • 22:39 Amir1: "sudo MODE=restart /srv/codesearch/manage.sh"

2019-08-07

  • 20:10 legoktm: deleting now unneeded ggmirror-02 instance
  • 20:04 legoktm: switched over to using gerrit-replica

2019-07-17

  • 00:53 legoktm: manual rolling restart of everything to start using new gerrit mirror

2019-07-16

  • 23:04 legoktm: deleted medium sized ggmirror-01 instance, it ran out of disk space. recreated as ggmirror-02 with size large

2019-07-11

  • 15:01 legoktm: legoktm@codesearch4:/srv/puppet$ git pull

2019-06-06

  • 18:15 legoktm: force updated config and doing a rolling restart (T223771)

2019-05-24

  • 21:54 legoktm: maintenance to reduce disk space

2019-03-28

  • 08:13 legoktm: legoktm@codesearch4:/srv/puppet$ git pull # T219248

2019-01-03

  • 00:35 legoktm: restarting hound instances for T211999

2018-12-03

  • 06:17 legoktm: deleted codesearch3 instance (T210372)

2018-11-26

  • 18:33 duh: codesearch4: deleted all of /srv/hound/hound-core/data/ and restarted the hound-core service for a fresh clone (T210409)
  • 06:36 legoktm: retarget codesearch.wmflabs.org to codesearch4 instance
  • 06:14 legoktm: created and setting up codesearch4 image based on Debian Stretch
  • 06:03 legoktm: deleted codesearch-sourcegraph1 instance

2018-11-16

  • 13:51 andrewbogott: migrating project to eqiad1

2018-11-05

  • 07:46 legoktm: Implemented automatic restarts of hound to regularly pick up new config changes

2018-09-27

2018-09-21

  • 07:36 legoktm: restarted to pick up new config / repository lists

2018-09-05

  • 05:25 legoktm: restarted all hound backends after deleting all of their data
  • 05:16 legoktm: stopping hound backends to clean up disk space
  • 04:52 legoktm: restarted hound for gerrit:458105

2018-08-17

  • 18:17 legoktm: rolling restart to pick up new config

2018-07-09

  • 13:07 legoktm: restarting everything for config update

2018-04-13

  • 17:41 legoktm: booted up codesearch3, it was down for some reason

2018-03-02

  • 02:12 legoktm: deleted codesearch2 instance
  • 02:09 legoktm: pointed codesearch.wmflabs.org at codesearch3:3002
  • 02:05 legoktm: ran `mkfs.ext4 -T news /dev/mapper/vd-second--local--disk` (h/t bd808) to quintuple the number of inodes
  • 01:52 legoktm: applied role::labs::lvm::srv role to codesearch3
  • 01:52 legoktm: created codesearch3 instance (jessie, medium flavor)

2018-02-28

  • 04:44 legoktm: restarted hound-search service (T188477), also I restarted hound-extensions earlier (T188450)

2018-02-24

  • 08:47 legoktm: added MaxSem as a project admin

2018-02-23

  • 22:23 legoktm: added d3r1ck's correct account as project admin

2018-02-14

  • 20:19 legoktm: added Addshore, D3r1ck01, Ladsgroup, and Prtksxna to Gerrit labs-codesearch project
  • 20:12 legoktm: added Addshore, D3r1ck01, Ladsgroup, and Prtksxna as project admins of codesearch

2018-01-18

  • 13:32 legoktm: rebooting codesearch2 to see how well it recovers
  • 11:36 legoktm: legoktm@codesearch2:~$ sudo systemctl enable hound_proxy

2018-01-16

  • 22:49 chasemp: restarted everything I could find in the stack :)

2017-12-21

  • 23:28 legoktm: restarting hound instances to pick up config change