You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Nova Resource:Quarry/SAL

From Wikitech-static
Jump to navigation Jump to search

2020-07-08

  • 19:23 Framawiki: `framawiki@quarry-web-01:/tmp$ find /tmp/* -mtime +360 -user www-data -exec sudo rm -v {} \;` 778 files deleted for 10G.

2020-03-06

  • 19:32 zhuyifei1999_: changed to analytics replica for database queries and restarted celery workers T246970

2020-02-26

  • 20:12 jeh: restart quarry-web-01 and quarry-worker-01

2020-01-14

2020-01-01

2019-11-14

  • 21:59 zhuyifei1999_: `zhuyifei1999@quarry-web-01:/tmp$ find /tmp/* -mtime +360 -user www-data -exec sudo rm -v {} \;` T238375

2019-10-19

  • 15:39 wm-bot: framawiki: Deployed 90a1bef on -web-01 (`query-status.html: fix compiled.js`)
  • 15:18 wm-bot: framawiki: Deployed 1f297c9 on -web-01 T205214 (`query-status.html: hide Explain button until bug is solved`)

2019-10-02

  • 17:20 mutante: - mariadb::packages should now work on buster too, fyi

2019-10-01

  • 11:51 zhuyifei1999_: restart celery-quarry-worker.service

2019-06-28

  • 18:45 zhuyifei1999_: Deployed 2f7ee60 to quarry-web-01
  • 14:37 bstorm_: changed to web replica for database queries and restarted celery workers

2019-06-21

  • 21:34 wm-bot: framawiki: Deployed 5d6844e on -web-01

2019-06-14

  • 20:23 wm-bot: framawiki: Deployed b303ce8 on -web-01

2019-05-25

  • 12:58 framawiki: block spammer `INSERT INTO user_group (user_id, group_name) VALUES (3765, "blocked");`
  • 12:22 wm-bot: framawiki: Deployed cc0c0a7 on -web-01 T224300

2019-05-24

  • 21:00 zhuyifei1999_: masked uwsgi service on quarry-web-01 to prevent future mess-ups
  • 20:59 zhuyifei1999_: reenabled puppet on quarry-web-01, should use uwsgi-quarry-web service not uwsgi service
  • 20:51 zhuyifei1999_: disabled puppet on quarry-web-01 because it wants uwsgi dead

2019-05-12

2019-05-11

  • 22:10 framawiki: re-enable puppet T223018
  • 22:00 framawiki: disabling puppet temporary on -web-01 to test nginx conf T223018
  • 19:25 framawiki: block spammer `INSERT INTO user_group (user_id, group_name) VALUES (3927, "blocked"), (3958, "blocked"), (3984, "blocked"), (3985, "blocked"), (3986, "blocked");`

2019-05-10

  • 14:00 andrewbogott: restarting uwsgi-quarry-web and nginx on quarry-web-01
  • 07:09 zhuyifei1999_: restarted uwsgi. nginx reports 502

2019-04-05

  • 18:48 zhuyifei1999_: checked out FETCH_HEAD on quarry-web-01 T209226
  • 18:43 zhuyifei1999_: applied 0001-SECURITY-escape-CSV-injections.patch on quarry-web-01 and restarted uwsgi T209226

2019-03-16

  • 08:31 framawiki: restarted uwsgi to deal with 502 nginx errors `sudo systemctl restart uwsgi-quarry-web`

2019-03-02

2019-02-21

  • 09:29 gtirloni: applied CSP change T214637
  • 09:22 gtirloni: updated and rebooted all servers (debian 9.8)

2019-02-20

  • 20:59 wm-bot: framawiki: Deployed 8f72587 on -web-01 T216581
  • 20:38 framawiki: re-activating puppet on -web-01, csp conf looks good T214637
  • 20:15 framawiki: disabling puppet temporary on -web-01 to test csp conf T214637

2019-02-18

2018-12-24

2018-12-02

2018-11-27

  • 18:54 zhuyifei1999_: triggered OOM killer on quarry-worker-02 9 times T188564

2018-11-16

2018-11-12

2018-11-05

  • 18:49 zhuyifei1999_: `UPDATE query join query_revision on query.latest_rev_id = query_revision.id join query_run on latest_run_id = query_run.id SET status=1 where (status = 2 or status = 0) and query_run.timestamp <= DATE_ADD(NOW(), INTERVAL -30 MINUTE);` 286 rows affected.
  • 18:43 framawiki: migration is over T207677
  • 18:33 zhuyifei1999_: flushed redis with flushall
  • 18:22 zhuyifei1999_: unset db read-only `SET GLOBAL read_only = 0; UNLOCK TABLES;` T207677
  • 18:01 andrewbogott: moving instances from eqiad to eqiad1-r
  • 17:40 zhuyifei1999_: set db read-only `FLUSH TABLES WITH READ LOCK; SET GLOBAL read_only = 1;` T207677
  • 17:24 zhuyifei1999_: shutting down all workers `sudo kill -TERM $(systemctl show -p MainPID celery-quarry-worker.service | cut -d= -f2)` T207677

2018-11-04

  • 22:24 zhuyifei1999_: checked out FETCH_HEAD 8c065d0, previous head was 71643b6 on quarry-web-01
  • 17:37 framawiki: deployed c10fc32 and 71643b6 on quarry-web-01

2018-10-21

2018-10-15

  • 18:52 framawiki: deployed c1dfde7 on quarry-web-01, quarry-worker-0{1,2} T126888

2018-10-06

  • 16:30 framawiki: deployed 8550956 on quarry-web-01

2018-09-26

  • 13:24 zhuyifei1999_: restarted mariadb on -db-01 after max_allowed_packet fix

2018-09-25

  • 12:09 arturo: make myself projectadmin

2018-09-24

  • 22:33 framawiki: manually clear queries and resultsets where userid=3214 T205286
  • 22:31 framawiki: `update query_run set status=3 where id=290865;` on quarry-db-01 to mark a ghost query as killed on the ui
  • 21:50 framawiki: `select * from query_run where id=290865;` on quarry-db-01 to mark a ghost query as killed on the ui
  • 17:54 framawiki: quarry-db-01: `INSERT INTO user_group (user_id, group_name) VALUES (3214, "blocked");` T205286
  • 17:53 framawiki: deployed 028a292 on quarry-web-01 T205286 T104322

2018-09-23

2018-09-21

2018-09-19

2018-09-17

2018-09-16

2018-09-13

  • 22:10 zhuyifei1999_: purging stuffs created by labs_debrepo T153615
  • 21:40 zhuyifei1999_: deployed 8b4bde0 to quarry-web-01 T204277
  • 19:19 framawiki: deleted legacy instances quarry-main-01 and quarry-runner-0{1,2}, migration is over T202588
  • 19:10 framawiki: copy /var/log/nginx from legacy main-01 to /data/project/nginx-logs-legacy-20180913-framawiki for further analysis T202588 T197256

2018-09-12

  • 21:21 zhuyifei1999_: unset read-only again on new database
  • 21:15 zhuyifei1999_: `sudo chown quarry:quarry /data/project/quarry/ -Rv` T202588
  • 21:13 zhuyifei1999_: set read-only again on new database because new quarry's UID is 498 T202588
  • 21:10 zhuyifei1999_: unset read-only again on new database
  • 21:09 zhuyifei1999_: rm'ed /var/lib/mysql on -db-01, we are using /srv/sqldata/ now, and if something goes really better have a loud failure
  • 21:03 zhuyifei1999_: deployed 8b4bde0 to -web-01 T204161
  • 21:01 zhuyifei1999_: deployed 461e56c
  • 20:45 zhuyifei1999_: set read-only again on new database due to a bug in worker code
  • 20:42 zhuyifei1999_: unset read-only on new database T202588
  • 20:41 framawiki: switched quarry.wmflabs.org proxy to new quarry-web-01.quarry.eqiad.wmflabs T202588
  • 20:27 zhuyifei1999_: backed up old db to /data/project/dump-2018-09-12.sql and restoring to new server T202588
  • 20:03 zhuyifei1999_: set quarry-main-01 mariadb read-only T202588
  • 20:02 zhuyifei1999_: stopped celery-quarry-worker on quarry-runner-0{1,2} T202588
  • 19:45 zhuyifei1999_: created new quarry database and user in quarry-db-01.quarry.eqiad.wmflabs T202588

2018-09-11

  • 17:22 zhuyifei1999_: doing another backup of main db: `sudo mysqldump quarry | sudo tee /data/project/dump-$(date '+%Y-%m-%d').sql > /dev/null` T202588
  • 17:14 zhuyifei1999_: disabling puppet on quarry-main-01, quarry-runner-0{1,2} T202588

2018-09-07

  • 21:06 zhuyifei1999_: reverted hotpatch, deployed till 3375dc3
  • 20:47 zhuyifei1999_: hotpatch /etc/uwsgi/apps-enabled/quarry-web.ini processes 8 -> 1 for some gdb-ing
  • 19:56 framawiki: deployed 501695f to quarry-main-01 (T202588)
  • 18:11 framawiki: deployed 769cace to quarry-main-01 (T202588)

2018-08-24

2018-05-31

  • 15:52 zhuyifei1999_: live-patch `/srv/quarry/quarry/web/connections.py` on `quarry-main-01` and restart uwsgi

2018-05-07

  • 22:40 framawiki: deployed 24038e3 to quarry-main-01

2018-05-04

  • 02:05 zhuyifei1999_: Deployed 6069904

2018-05-02

  • 18:23 framawiki: deployed af2f7e6 to quarry-main-01
  • 18:04 zhuyifei1999_: Deployed f4e86f1 and restarted everything

2018-04-24

  • 21:36 framawiki: removing old /srv/venv on quarry-main-01 T192731

2018-04-23

  • 22:50 zhuyifei1999_: Does quarry only have an effective concurrency limit of 3, despite having a few dozen celery worker processes?!
  • 22:46 zhuyifei1999_: behaving abnormally. https://quarry.wmflabs.org/query/26629 has been queued for 16 mins... (hopefully) have some time to investigate

2018-04-18

  • 23:21 framawiki: deployed 02049d9 to quarry-main-01
  • 22:21 zhuyifei1999_: +Framawiki project admin & Gerrit +2
  • 19:35 zhuyifei1999_: deployed c6cd55e to quarry-main-01
  • 17:36 zhuyifei1999_: deployed 8eeeff8 to quarry-main-01

2018-04-17

  • 23:00 zhuyifei1999_: forgot to restart uwsgi on last deployment. restarted it now
  • 00:34 zhuyifei1999_: Deploy b5fd6b0 on quarry-main-01

2018-03-24

2018-03-16

2018-03-15

  • 19:27 zhuyifei1999_: switch back to d9cc1c8 on both hosts
  • 16:29 zhuyifei1999_: quarry-runner-02 is on d9cc1c8
  • 16:21 zhuyifei1999_: installed python-dbg on quarry-runner-02 because it's so good
  • 16:18 zhuyifei1999_: depool quarry-runner-01
  • 15:56 zhuyifei1999_: deploying d653400 to quarry-runner-0{1,2} T188564

2018-03-01

  • 18:41 zhuyifei1999_: deploying d5e2845 to quarry-runner-01 & 02
  • 00:37 zhuyifei1999_: `UPDATE query join query_revision on query.latest_rev_id = query_revision.id join query_run on latest_run_id = query_run.id SET status=1 where (status = 2 or status = 1) and query_run.timestamp <= DATE_ADD(NOW(), INTERVAL -1 HOUR);` 251 rows affected (1.81 sec) T139162 T172086 T188564

2018-02-28

  • 22:57 zhuyifei1999_: killed two IO-intensive query saves

2018-02-09

  • 01:06 bd808: Removed TestingAccount2 at user request (T186289)
  • 01:06 bd808: Removed Yuvipanda at user request (T186289)

2018-01-02

  • 10:52 zhuyifei1999_: deploying d9cc1c8 to quarry-runner-01 & 02 T172143

2017-12-13

  • 19:19 zhuyifei1999_: Deployed 62676f2 to quarry-main-01 and restarted uwsgi

2017-12-10

  • 05:52 zhuyifei1999_: deployed e835a46 to quarry-main-01 and restarted uwsgi T165169
  • 05:49 zhuyifei1999_: quarry-main-01: `ALTER IGNORE TABLE star ADD UNIQUE INDEX star_user_query_index (user_id, query_id);` Records: 728 Duplicates: 17 Warnings: 0 T165169

2017-12-05

  • 18:56 zhuyifei1999_: quarry-main-01: `MariaDB [quarry]> UPDATE user SET username = '-revi' WHERE username = 'Hym411';` T182064

2017-10-02

  • 05:15 zhuyifei1999_: Deployed 644b293 to quarry-main-01 and restarted uwsgi

2017-09-26

  • 04:08 zhuyifei1999_: Restarting service 'uwsgi-quarry-web' on quarry-main-01, 'celery-quarry-worker' on quarry-runner-01 & quarry-runner-02 T176694
  • 03:59 zhuyifei1999_: Switching REPLICA_HOST from 'enwiki.labsdb' to 'enwiki.analytics.db.svc.eqiad.wmflabs' T176694 (Executing `sudo -- sudo -u quarry sed -i 's/enwiki.labsdb/enwiki.analytics.db.svc.eqiad.wmflabs/' /srv/quarry/quarry/config.yaml` on all hosts)

2017-09-10

2017-09-08

2017-09-06

  • 00:05 zhuyifei1999_: backup quarry main database to /data/project/dump-2017-09-05.sql because I fear it die :(. Executing `sudo mysqldump quarry | sudo tee /data/project/dump-2017-09-05.sql > /dev/null`

2017-08-11

  • 11:12 zhuyifei1999_: deployed 2834160 on quarry-main-01

2017-08-01

2017-07-31

  • 16:31 zhuyifei1999_: Repeated for quarry-main-01, but restarted uwsgi T146483
  • 16:30 zhuyifei1999_: Repeated for quarry-runner-02
  • 16:28 zhuyifei1999_: Restarted celery-quarry-worker on quarry-runner-01 T146483
  • 16:22 zhuyifei1999_: `zhuyifei1999@quarry-runner-01:/srv/quarry$ sudo git fetch; sudo git checkout 6447943` T146483
  • 01:01 zhuyifei1999_: `zhuyifei1999@quarry-main-01:/srv/quarry$ sudo git fetch; sudo git checkout 7dd8c60; sudo service uwsgi restart` T101424

2017-07-30

  • 21:17 zhuyifei1999_: `sudo service uwsgi restart` T76126
  • 21:14 zhuyifei1999_: `sudo git fetch; sudo git checkout 172eb7e` on /srv/quarry T76126
  • 20:10 zhuyifei1999_: Yuvi gave me access after I asked about T76126

2017-06-30

  • 22:34 bd808: Added BryanDavis (self) as project admin
  • 18:09 bd808: Ran service uwsgi-quarry-web restart on quarry-main-01. People seeing intermittent 502s

2017-06-26

  • 18:33 milimetric: Restarted celery workers on quarry-runner-01 and quarry-runner-02 (systemctl restart celery-quarry-worker.service)
  • 17:20 madhuvishy: Add milimetric as project admin

2017-01-19

  • 12:07 yuvipanda: run chown -R 998:998 quarry/ on labstore1004

2016-10-05

  • 19:48 mutante: quarry-runner-01 has a problem starting exim4
  • 19:47 mutante: merged gerrit 308313 - should definitely be no-op, but noticed that puppet is disabled on quarry-main-01

2016-10-04

  • 19:33 valhallasw`cloud: removed myself as admin

2016-09-23

  • 20:41 yuvipanda: add halfak as projectadmin

2016-05-08

  • 21:35 Krenair: restarted quarry-runner-01 to attempt to get the queue working again - new queries are going through but some old ones (from the last few hours) seem stuck

2016-04-02

  • 14:06 valhallasw`cloud: systemctl restart celery-quarry-worker.service hangs. Will now reboot quarry-runner-02.
  • 14:02 valhallasw`cloud: killed 100% CPU using process on quarry-runner-02 (ptrace suggested some sort of idle loop). Let's see if that has any effect.

July 4

  • 15:45 YuviPanda: deploying to latest master and hoping!

April 30

  • 21:57 andrewbogott: moving quarry-main-01 to labvirt1003
  • 20:53 andrewbogott: moving quarry-runner-01 to labvirt1004
  • 19:41 andrewbogott: cold-migrating quarry-runner-test to labvirt1003

December 8

  • 19:36 YuviPanda: deploying to get in valhallasw`cloud’s patches

September 30

  • 15:44 andrewbogott: enabled puppet on quarry-runner-test, updated, installed a bunch of maria stuff, rebooted

September 16

  • 21:53 YuviPanda: appllying db, web and redis roles to quarry-runner-test, will act as db and web host until labs issues clear up

August 21

  • 19:47 YuviPanda: upgraded all text and varchar columns to utf8

August 17

  • 02:03 YuviPanda: increased mysql connection limit manually to 1024, re-running all old query-runs to produce output in new sqlite format

August 14

  • 20:21 YuviPanda: upgrade to MariaDB 10.1, because fuck offline ALTER TABLEs