You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 500 | ) (20 | 50 | 100 | 250 | 500)

18 August 2022

  • curprev 00:4900:49, 18 August 2022imported>Stashbot 425,191 bytes +51,429 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['kubernetes2023.codfw.wmnet']

17 August 2022

  • curprev 01:2301:23, 17 August 2022imported>Stashbot 373,762 bytes +37,026 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['kafka-logging2005']

16 August 2022

  • curprev 00:1800:18, 16 August 2022imported>Stashbot 336,736 bytes +19,280 tstarling@deploy1002: Synchronized wmf-config/InitialiseSettings.php: replaceableSettings g 820247 (duration: 03m 18s)

14 August 2022

  • curprev 08:5408:54, 14 August 2022imported>Stashbot 317,456 bytes +542 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T312863)', diff saved to https://phabricator.wikimedia.org/P32380 and previous config saved to /var/cache/conftool/dbconfig/20220814-085443-ladsgroup.json

13 August 2022

  • curprev 13:3713:37, 13 August 2022imported>Stashbot 316,914 bytes +1,309 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1145.eqiad.wmnet with reason: Maintenance

12 August 2022

  • curprev 23:4123:41, 12 August 2022imported>Stashbot 315,605 bytes +10,763 mutante: wikistats-bullseye:~$ /usr/lib/wikistats/update.php wp prefix blk ; /usr/lib/wikistats/update.php wp prefix kcg T315121
  • curprev 01:0301:03, 12 August 2022imported>Stashbot 304,842 bytes +25,321 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T312863)', diff saved to https://phabricator.wikimedia.org/P32369 and previous config saved to /var/cache/conftool/dbconfig/20220812-010312-ladsgroup.json

11 August 2022

  • curprev 00:5800:58, 11 August 2022imported>Stashbot 279,521 bytes +35,722 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp2042.codfw.wmnet,service=varnish-fe

9 August 2022

  • curprev 23:1723:17, 9 August 2022imported>Stashbot 243,799 bytes +16,655 bking@cumin1001: conftool action : set/weight=10:pooled=yes; selector: name=wdqs1011.eqiad.wmnet

8 August 2022

  • curprev 23:5223:52, 8 August 2022imported>Stashbot 227,144 bytes +14,190 tstarling@deploy1002: Synchronized wmf-config/InitialiseSettings.php: clean up testwiki experiments T314750 (duration: 03m 19s)

7 August 2022

  • curprev 19:5819:58, 7 August 2022imported>Stashbot 212,954 bytes +3,230 taavi: taavi@mwmaint1002 ~ $ echo "https://upload.wikimedia.org/wikipedia/commons/1/15/Keep_tidy_ask.svg" | mwscript purgeList.php --wiki enwiki # T314712

6 August 2022

  • curprev 17:5917:59, 6 August 2022imported>Stashbot 209,724 bytes +2,395 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T312863)', diff saved to https://phabricator.wikimedia.org/P32295 and previous config saved to /var/cache/conftool/dbconfig/20220806-175916-ladsgroup.json

5 August 2022

  • curprev 22:2022:20, 5 August 2022imported>Stashbot 207,329 bytes +9,421 dcausse@deploy1002: Finished deploy [wikimedia/discovery/analytics@71fe016]: Fix schedule_interval for image_recommendation_weekly (duration: 02m 01s)
  • curprev 00:5300:53, 5 August 2022imported>Stashbot 197,908 bytes +49,951 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8 days, 0:00:00 on gerrit2001.wikimedia.org with reason: decom, replaced by gerrit2002

4 August 2022

  • curprev 01:2301:23, 4 August 2022imported>Stashbot 147,957 bytes +66,383 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T312972)', diff saved to https://phabricator.wikimedia.org/P32278 and previous config saved to /var/cache/conftool/dbconfig/20220804-012341-marostegui.json

2 August 2022

  • curprev 22:3922:39, 2 August 2022imported>Stashbot 81,574 bytes +58,081 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 00:4100:41, 2 August 2022imported>Stashbot 23,493 bytes −741,414 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply

1 August 2022

  • curprev 01:0001:00, 1 August 2022imported>Stashbot 764,907 bytes +3,017 krinkle@deploy1002: Synchronized multiversion/: Ic0dbcba9f60f20a (duration: 03m 31s)

30 July 2022

  • curprev 01:4401:44, 30 July 2022imported>Stashbot 761,890 bytes +392 bking@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reimage (bullseye upgrade) - bking@cumin1001 - T289135
  • curprev 00:5500:55, 30 July 2022imported>Stashbot 761,498 bytes +10,804 bking@cumin1001: START - Cookbook sre.hosts.reimage for host elastic2028.codfw.wmnet with OS bullseye

29 July 2022

  • curprev 00:4800:48, 29 July 2022imported>Stashbot 750,694 bytes +35,853 TimStarling: slowly restarting (with batch 1 sleep 5) trafficserver on text caches to fully deploy g 817086 T313578

28 July 2022

  • curprev 01:2601:26, 28 July 2022imported>Stashbot 714,841 bytes +36,833 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply

26 July 2022

  • curprev 23:5923:59, 26 July 2022imported>Stashbot 678,008 bytes +28,393 tzatziki: removing one file for legal compliance
  • curprev 00:1100:11, 26 July 2022imported>Stashbot 649,615 bytes +31,213 TimStarling: restarted php7.2-fpm on the 9 canary hosts in eqiad T313770

24 July 2022

  • curprev 20:5420:54, 24 July 2022imported>Stashbot 618,402 bytes +4,271 btullis@cumin1001: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM archiva1002.wikimedia.org
  • curprev 00:3700:37, 24 July 2022imported>Stashbot 614,131 bytes +18,168 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T312863)', diff saved to https://phabricator.wikimedia.org/P31802 and previous config saved to /var/cache/conftool/dbconfig/20220724-003718-ladsgroup.json

23 July 2022

  • curprev 01:3701:37, 23 July 2022imported>Stashbot 595,963 bytes +27,357 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P31750 and previous config saved to /var/cache/conftool/dbconfig/20220723-013755-ladsgroup.json

22 July 2022

  • curprev 00:4400:44, 22 July 2022imported>Stashbot 568,606 bytes +68,956 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply

21 July 2022

  • curprev 00:4400:44, 21 July 2022imported>Stashbot 499,650 bytes +51,704 bking@cumin1001: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster search_codfw: codfw cluster reimage (bullseye upgrade) - bking@cumin1001 - T289135

20 July 2022

  • curprev 01:2701:27, 20 July 2022imported>Stashbot 447,946 bytes +56,169 bking@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2052.codfw.wmnet with reason: host reimage

18 July 2022

  • curprev 23:5823:58, 18 July 2022imported>Stashbot 391,777 bytes +61,137 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudvirt1050.eqiad.wmnet

17 July 2022

  • curprev 18:0518:05, 17 July 2022imported>Stashbot 330,640 bytes +10,758 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T312984)', diff saved to https://phabricator.wikimedia.org/P31256 and previous config saved to /var/cache/conftool/dbconfig/20220717-180539-ladsgroup.json
  • curprev 00:4800:48, 17 July 2022imported>Stashbot 319,882 bytes +13,275 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P31225 and previous config saved to /var/cache/conftool/dbconfig/20220717-004804-ladsgroup.json

16 July 2022

  • curprev 00:4700:47, 16 July 2022imported>Stashbot 306,607 bytes +27,543 ryankemper@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2064.codfw.wmnet with OS bullseye

15 July 2022

  • curprev 00:3000:30, 15 July 2022imported>Stashbot 279,064 bytes +31,156 TimStarling: on ms-fe1010 restarting swift-proxy

14 July 2022

  • curprev 00:4400:44, 14 July 2022imported>Stashbot 247,908 bytes +16,545 krinkle@deploy1002: Synchronized php-1.39.0-wmf.19/includes/ResourceLoader/: Ie11bdfdcf5e6724 (duration: 02m 55s)

12 July 2022

  • curprev 22:3222:32, 12 July 2022imported>Stashbot 231,363 bytes +15,830 bking@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2039.codfw.wmnet with OS bullseye
  • curprev 00:1000:10, 12 July 2022imported>Stashbot 215,533 bytes +11,642 ejegg: updated payments-wiki from 53a7b7bd to 2f95d8b4

11 July 2022

  • curprev 00:2300:23, 11 July 2022imported>Stashbot 203,891 bytes +379 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply

9 July 2022

  • curprev 13:3413:34, 9 July 2022imported>Stashbot 203,512 bytes +504 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 01:4401:44, 9 July 2022imported>Stashbot 203,008 bytes +12,736 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply

8 July 2022

  • curprev 00:0200:02, 8 July 2022imported>Stashbot 190,272 bytes +47,463 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2181.codfw.wmnet with OS bullseye

7 July 2022

  • curprev 00:5800:58, 7 July 2022imported>Stashbot 142,809 bytes +50,193 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply

5 July 2022

  • curprev 23:3023:30, 5 July 2022imported>Stashbot 92,616 bytes +32,932 ebernhardson: start restore of commonswiki_file from thanos-swift to cloudelastic

4 July 2022

  • curprev 20:0920:09, 4 July 2022imported>Stashbot 59,684 bytes +22,620 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cloudcontrol1004.wikimedia.org

3 July 2022

  • curprev 11:3611:36, 3 July 2022imported>Stashbot 37,064 bytes +255 _joe_: temporarily raised replicas for shellbox to 24

2 July 2022

  • curprev 05:3605:36, 2 July 2022imported>Stashbot 36,809 bytes +2,607 bmansurov@deploy1002: Finished deploy [airflow-dags/research@b3fe77c]: (no justification provided) (duration: 00m 09s)
  • curprev 00:4500:45, 2 July 2022imported>Stashbot 34,202 bytes +30,284 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance

1 July 2022

  • curprev 01:3901:39, 1 July 2022imported>Stashbot 3,918 bytes −776,405 krinkle@deploy1002: Synchronized tests/: I60edfb0f60 (1/3) (duration: 03m 32s)

30 June 2022

  • curprev 01:3601:36, 30 June 2022imported>Stashbot 780,323 bytes +52,007 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2158.codfw.wmnet with OS bullseye

29 June 2022

  • curprev 00:1800:18, 29 June 2022imported>Stashbot 728,316 bytes +67,167 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw2003-dev.codfw.wmnet with OS bullseye

27 June 2022

  • curprev 23:5123:51, 27 June 2022imported>Stashbot 661,149 bytes +85,388 cmjohnson@cumin1001: START - Cookbook sre.hosts.provision for host mw1474.mgmt.eqiad.wmnet with reboot policy FORCED
  • curprev 01:2501:25, 27 June 2022imported>Stashbot 575,761 bytes +7,333 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker1008.mgmt.eqiad.wmnet with reboot policy FORCED

25 June 2022

  • curprev 18:1718:17, 25 June 2022imported>Stashbot 568,428 bytes +2,028 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1002.eqiad.wmnet

24 June 2022

  • curprev 19:3519:35, 24 June 2022imported>Stashbot 566,400 bytes +54,579 dancy@deploy1002: backport aborted: (duration: 00m 12s)

23 June 2022

  • curprev 21:2321:23, 23 June 2022imported>Stashbot 511,821 bytes +39,618 mutante: restbase-dev1006 has manually installed packages (wrk, maybe others)
  • curprev 00:3500:35, 23 June 2022imported>Stashbot 472,203 bytes +29,515 brennen: end of phabricator maintenance window

22 June 2022

  • curprev 01:1801:18, 22 June 2022imported>Stashbot 442,688 bytes +15,474 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply

20 June 2022

  • curprev 07:1407:14, 20 June 2022imported>Stashbot 427,214 bytes +308 SandraEbele: Started Airflow 3 Wikidata metrics jobs (Articleplaceholder, Reliability and SpecialEntityData metrics).

19 June 2022

  • curprev 10:2810:28, 19 June 2022imported>Stashbot 426,906 bytes +493 ayounsi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1132.eqiad.wmnet with reason: depooled

17 June 2022

  • curprev 22:0522:05, 17 June 2022imported>Stashbot 426,413 bytes +16,273 AndyRussG: update payments-wiki revision 10304f69 -> ef53c82e
  • curprev 01:4301:43, 17 June 2022imported>Stashbot 410,140 bytes +29,970 pt1979@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aqs1017.eqiad.wmnet with reason: host reimage

15 June 2022

  • curprev 22:4822:48, 15 June 2022imported>Stashbot 380,170 bytes +61,049 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1184 (T310011)', diff saved to https://phabricator.wikimedia.org/P29867 and previous config saved to /var/cache/conftool/dbconfig/20220615-224845-marostegui.json

14 June 2022

  • curprev 23:5223:52, 14 June 2022imported>Stashbot 319,121 bytes +44,141 mutante: gitlab-runner1001/1002 - clean revert not possible, icinga alerting about failed buildkitd service, manually deleting systemd unit and trying to clean up T308271
  • curprev 00:3600:36, 14 June 2022imported>Stashbot 274,980 bytes +45,898 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T310011)', diff saved to https://phabricator.wikimedia.org/P29701 and previous config saved to /var/cache/conftool/dbconfig/20220614-003608-marostegui.json

12 June 2022

  • curprev 18:3118:31, 12 June 2022imported>Stashbot 229,082 bytes +4,306 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddumps1002.wikimedia.org with OS bullseye
  • curprev 01:4601:46, 12 June 2022imported>Stashbot 224,776 bytes +4,304 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddumps1001.wikimedia.org with reason: host reimage

11 June 2022

  • curprev 01:1701:17, 11 June 2022imported>Stashbot 220,472 bytes +8,628 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply

10 June 2022

  • curprev 00:3300:33, 10 June 2022imported>Stashbot 211,844 bytes +35,139 ejegg: rolled back payments-wiki from 05139a0c to 8c6208c2

9 June 2022

  • curprev 00:4900:49, 9 June 2022imported>Stashbot 176,705 bytes +52,552 krinkle@deploy1002: Synchronized php-1.39.0-wmf.15/includes/libs/rdbms/: I99b817b3d50ffcdf56, T310214 (duration: 03m 23s)

8 June 2022

  • curprev 01:4301:43, 8 June 2022imported>Stashbot 124,153 bytes +33,565 cstone: civicrm revision changed from de12571a to b0b400ae

6 June 2022

  • curprev 23:1723:17, 6 June 2022imported>Stashbot 90,588 bytes +16,595 tzatziki: removing one file for legal compliance

5 June 2022

  • curprev 22:2122:21, 5 June 2022imported>Stashbot 73,993 bytes +6,438 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T298560)', diff saved to https://phabricator.wikimedia.org/P29417 and previous config saved to /var/cache/conftool/dbconfig/20220605-222110-ladsgroup.json
  • curprev 01:3701:37, 5 June 2022imported>Stashbot 67,555 bytes +6,227 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host clouddumps1001.wikimedia.org with OS bullseye

3 June 2022

  • curprev 22:1922:19, 3 June 2022imported>Stashbot 61,328 bytes +9,538 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • curprev 01:2001:20, 3 June 2022imported>Stashbot 51,790 bytes +28,593 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T298560)', diff saved to https://phabricator.wikimedia.org/P29365 and previous config saved to /var/cache/conftool/dbconfig/20220603-012045-ladsgroup.json

2 June 2022

  • curprev 01:4701:47, 2 June 2022imported>Stashbot 23,197 bytes −1,118,222 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply

1 June 2022

  • curprev 01:4101:41, 1 June 2022imported>Stashbot 1,141,419 bytes +62,344 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance

31 May 2022

  • curprev 00:4000:40, 31 May 2022imported>Stashbot 1,079,075 bytes +101,499 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance

30 May 2022

  • curprev 01:4501:45, 30 May 2022imported>Stashbot 977,576 bytes +7,857 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3316', diff saved to https://phabricator.wikimedia.org/P28904 and previous config saved to /var/cache/conftool/dbconfig/20220530-014458-ladsgroup.json

28 May 2022

  • curprev 23:3623:36, 28 May 2022imported>Stashbot 969,719 bytes +50,883 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T298560)', diff saved to https://phabricator.wikimedia.org/P28882 and previous config saved to /var/cache/conftool/dbconfig/20220528-233650-ladsgroup.json
  • curprev 01:3201:32, 28 May 2022imported>Stashbot 918,836 bytes +45,130 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1122 (T309311)', diff saved to https://phabricator.wikimedia.org/P28737 and previous config saved to /var/cache/conftool/dbconfig/20220528-013212-ladsgroup.json

27 May 2022

  • curprev 00:4500:45, 27 May 2022imported>Stashbot 873,706 bytes +31,398 mutante: rsyncing /srv/gitlab-backup from gitlab1004 to gitlab2002 | systemctl status full-backup ..in progress on gitlab1001 - T274463

26 May 2022

  • curprev 00:5800:58, 26 May 2022imported>Stashbot 842,308 bytes +49,509 mutante: gitlab1001 - T308089 T274463 - gitlab1001 - systemctl start full-backup

25 May 2022

  • curprev 00:1500:15, 25 May 2022imported>Stashbot 792,799 bytes +52,401 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3315 (T298560)', diff saved to https://phabricator.wikimedia.org/P28462 and previous config saved to /var/cache/conftool/dbconfig/20220525-001552-ladsgroup.json

24 May 2022

  • curprev 00:5200:52, 24 May 2022imported>Stashbot 740,398 bytes +67,605 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1113:3315', diff saved to https://phabricator.wikimedia.org/P28379 and previous config saved to /var/cache/conftool/dbconfig/20220524-005257-ladsgroup.json

22 May 2022

  • curprev 20:4620:46, 22 May 2022imported>Stashbot 672,793 bytes +13,528 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 00:2100:21, 22 May 2022imported>Stashbot 659,265 bytes +20,709 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T298560)', diff saved to https://phabricator.wikimedia.org/P28249 and previous config saved to /var/cache/conftool/dbconfig/20220522-002120-ladsgroup.json

21 May 2022

  • curprev 01:0601:06, 21 May 2022imported>Stashbot 638,556 bytes +27,942 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T298555)', diff saved to https://phabricator.wikimedia.org/P28208 and previous config saved to /var/cache/conftool/dbconfig/20220521-010640-ladsgroup.json

20 May 2022

  • curprev 01:3101:31, 20 May 2022imported>Stashbot 610,614 bytes +72,169 cmooney@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

19 May 2022

  • curprev 00:5800:58, 19 May 2022imported>Stashbot 538,445 bytes +60,753 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance

18 May 2022

  • curprev 01:0501:05, 18 May 2022imported>Stashbot 477,692 bytes +34,747 ejegg: updated fundraising CiviCRM from d45afdfc to b8b8c177

16 May 2022

  • curprev 22:1422:14, 16 May 2022imported>Stashbot 442,945 bytes +16,328 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx2001.wikimedia.org with reason: exim debugging

15 May 2022

  • curprev 21:4721:47, 15 May 2022imported>Stashbot 426,617 bytes +1,183 aqu@deploy1002: Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 07s)

14 May 2022

  • curprev 08:3408:34, 14 May 2022imported>Stashbot 425,434 bytes +205 jynus@cumin1001: dbctl commit (dc=all): 'Depool db1172', diff saved to https://phabricator.wikimedia.org/P27830 and previous config saved to /var/cache/conftool/dbconfig/20220514-083421-jynus.json
  • curprev 00:5300:53, 14 May 2022imported>Stashbot 425,229 bytes +4,537 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on an-tool1005.eqiad.wmnet with reason: Server need to be downgraded to stretch, on monday

12 May 2022

  • curprev 21:5621:56, 12 May 2022imported>Stashbot 420,692 bytes +26,145 razzi@deploy1002: Finished deploy [analytics/turnilo/deploy@a2bdc3e]: (no justification provided) (duration: 02m 08s)

11 May 2022

  • curprev 22:2822:28, 11 May 2022imported>Stashbot 394,547 bytes +16,527 robh: cp305[67] returned to service and all green in icinga, cp305[89] depooling for firmware update T243167
  • curprev 01:4101:41, 11 May 2022imported>Stashbot 378,020 bytes +25,757 mutante: gitlab2001 - starting backup-restore service that had failed on previous automatic run

9 May 2022

  • curprev 21:5821:58, 9 May 2022imported>Stashbot 352,263 bytes +25,329 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx2001.wikimedia.org with reason: new kernel round deux

8 May 2022

  • curprev 07:1607:16, 8 May 2022imported>Stashbot 326,934 bytes +81 godog: silence probedown for thumbor:8800 until monday

7 May 2022

  • curprev 21:2921:29, 7 May 2022imported>Stashbot 326,853 bytes +2,312 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: seeking consistency between codfw1dev and eqiad1 (duration: 04m 04s)

6 May 2022

  • curprev 19:1619:16, 6 May 2022imported>Stashbot 324,541 bytes +16,729 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-airflow1002.eqiad.wmnet
  • curprev 00:4600:46, 6 May 2022imported>Stashbot 307,812 bytes +83,868 rook@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudvirt1016.eqiad.wmnet

5 May 2022

  • curprev 01:4201:42, 5 May 2022imported>Stashbot 223,944 bytes +94,859 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P27586 and previous config saved to /var/cache/conftool/dbconfig/20220505-014205-ladsgroup.json

4 May 2022

  • curprev 00:5000:50, 4 May 2022imported>Stashbot 129,085 bytes +36,153 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance

2 May 2022

  • curprev 23:1523:15, 2 May 2022imported>Stashbot 92,932 bytes +77,432 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb2002.codfw.wmnet with OS bullseye
  • curprev 00:5900:59, 2 May 2022imported>Stashbot 15,500 bytes +15,385 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P27203 and previous config saved to /var/cache/conftool/dbconfig/20220502-005940-ladsgroup.json

1 May 2022

29 April 2022

  • curprev 23:1123:11, 29 April 2022imported>Stashbot 1,095,940 bytes +87,748 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P27163 and previous config saved to /var/cache/conftool/dbconfig/20220429-231136-ladsgroup.json
  • curprev 00:5700:57, 29 April 2022imported>Stashbot 1,008,192 bytes +87,341 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26967 and previous config saved to /var/cache/conftool/dbconfig/20220429-005702-ladsgroup.json

28 April 2022

  • curprev 01:4701:47, 28 April 2022imported>Stashbot 920,851 bytes +89,525 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26857 and previous config saved to /var/cache/conftool/dbconfig/20220428-014723-ladsgroup.json

27 April 2022

  • curprev 01:4301:43, 27 April 2022imported>Stashbot 831,326 bytes +86,215 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26663 and previous config saved to /var/cache/conftool/dbconfig/20220427-014355-ladsgroup.json

25 April 2022

  • curprev 23:0523:05, 25 April 2022imported>Stashbot 745,111 bytes +49,942 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 00:5400:54, 25 April 2022imported>Stashbot 695,169 bytes +25,636 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26408 and previous config saved to /var/cache/conftool/dbconfig/20220425-005432-ladsgroup.json

24 April 2022

  • curprev 01:2801:28, 24 April 2022imported>Stashbot 669,533 bytes +28,655 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance

23 April 2022

  • curprev 01:3401:34, 23 April 2022imported>Stashbot 640,878 bytes +45,596 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26246 and previous config saved to /var/cache/conftool/dbconfig/20220423-013450-ladsgroup.json

22 April 2022

  • curprev 01:4701:47, 22 April 2022imported>Stashbot 595,282 bytes −356,134 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance

21 April 2022

  • curprev 00:5200:52, 21 April 2022imported>Stashbot 951,416 bytes +154,237 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25837 and previous config saved to /var/cache/conftool/dbconfig/20220421-005225-ladsgroup.json

20 April 2022

  • curprev 01:3101:31, 20 April 2022imported>Stashbot 797,179 bytes +136,676 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

19 April 2022

  • curprev 00:5300:53, 19 April 2022imported>Stashbot 660,503 bytes +82,833 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25214 and previous config saved to /var/cache/conftool/dbconfig/20220419-005334-ladsgroup.json

18 April 2022

  • curprev 01:4001:40, 18 April 2022imported>Stashbot 577,670 bytes +74,955 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24982 and previous config saved to /var/cache/conftool/dbconfig/20220418-014003-ladsgroup.json

17 April 2022

  • curprev 00:5100:51, 17 April 2022imported>Stashbot 502,715 bytes +28,006 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24761 and previous config saved to /var/cache/conftool/dbconfig/20220417-005150-ladsgroup.json

16 April 2022

  • curprev 00:3500:35, 16 April 2022imported>Stashbot 474,709 bytes +12,679 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P24681 and previous config saved to /var/cache/conftool/dbconfig/20220416-003538-ladsgroup.json

14 April 2022

  • curprev 22:2822:28, 14 April 2022imported>Stashbot 462,030 bytes +16,537 mutante: gitlab - deleting runner-1018, runner-1019, creating runner-1029, runner-1030 T297659
  • curprev 00:3700:37, 14 April 2022imported>Stashbot 445,493 bytes +50,258 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance

13 April 2022

  • curprev 01:4201:42, 13 April 2022imported>Stashbot 395,235 bytes +39,686 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P24545 and previous config saved to /var/cache/conftool/dbconfig/20220413-014214-ladsgroup.json

12 April 2022

  • curprev 00:4900:49, 12 April 2022imported>Stashbot 355,549 bytes +52,260 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P24477 and previous config saved to /var/cache/conftool/dbconfig/20220412-004933-ladsgroup.json

11 April 2022

  • curprev 01:4301:43, 11 April 2022imported>Stashbot 303,289 bytes +5,989 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T298565)', diff saved to https://phabricator.wikimedia.org/P24355 and previous config saved to /var/cache/conftool/dbconfig/20220411-014316-ladsgroup.json

9 April 2022

  • curprev 12:3912:39, 9 April 2022imported>Stashbot 297,300 bytes +1,710 godog: bounce prometheus@ops on prometheus5001
  • curprev 00:5300:53, 9 April 2022imported>Stashbot 295,590 bytes +31,885 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T298565)', diff saved to https://phabricator.wikimedia.org/P24333 and previous config saved to /var/cache/conftool/dbconfig/20220409-005351-ladsgroup.json

7 April 2022

  • curprev 22:1822:18, 7 April 2022imported>Stashbot 263,705 bytes +77,590 ejegg: restarted fundraising scheduled jobs
  • curprev 00:5800:58, 7 April 2022imported>Stashbot 186,115 bytes +64,418 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T297189)', diff saved to https://phabricator.wikimedia.org/P24195 and previous config saved to /var/cache/conftool/dbconfig/20220407-005817-marostegui.json

6 April 2022

  • curprev 01:3401:34, 6 April 2022imported>Stashbot 121,697 bytes +61,769 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P24142 and previous config saved to /var/cache/conftool/dbconfig/20220406-013420-ladsgroup.json

5 April 2022

  • curprev 00:5800:58, 5 April 2022imported>Stashbot 59,928 bytes +52,310 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4034.ulsfo.wmnet

2 April 2022

  • curprev 11:2611:26, 2 April 2022imported>Stashbot 7,618 bytes +272 akosiaris: disable zotero paging until T291707 is resolved.

1 April 2022

  • curprev 23:2523:25, 1 April 2022imported>Stashbot 7,346 bytes −1,236,074 mutante: DNS - new project language 'kcg'. 'Tyap is a regionally important dialect cluster of Plateau languages in Nigeria's Middle Belt, named after its prestige dialect. It is also known by its Hausa exonym as Katab or Kataf.' T305279

31 March 2022

  • curprev 23:4523:45, 31 March 2022imported>Stashbot 1,243,420 bytes +66,130 mutante: gitlab2001 - fdisk /dev/vdb (g, w) (create partition table), (n, w) (create partition) ; mkfs.ext4 /dev/vdb1 (create filesystem); systemctl reset-failed (fix Icinga alert); mkdir /mnt/gitlab-backup; mount /dev/vdb1 /mnt/gitlab-backup ; blkid (get UUID); edit /etc/fstab and insert "UUID=c5235682-ac21-46a9-85ee-9603f694a6a4 /mnt/gitlab-backup ext4 errors=remount-ro 0 2" T274463
  • curprev 01:4401:44, 31 March 2022imported>Stashbot 1,177,290 bytes +118,543 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P23948 and previous config saved to /var/cache/conftool/dbconfig/20220331-014403-ladsgroup.json

30 March 2022

  • curprev 01:4601:46, 30 March 2022imported>Stashbot 1,058,747 bytes +98,930 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23664 and previous config saved to /var/cache/conftool/dbconfig/20220330-014621-ladsgroup.json

28 March 2022

  • curprev 23:1523:15, 28 March 2022imported>Stashbot 959,817 bytes +54,742 eileen: civicrm revision 15d22bd1 -> 1c5d10e1
  • curprev 00:5500:55, 28 March 2022imported>Stashbot 905,075 bytes +30,248 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23315 and previous config saved to /var/cache/conftool/dbconfig/20220328-005533-ladsgroup.json

27 March 2022

  • curprev 00:5000:50, 27 March 2022imported>Stashbot 874,827 bytes +29,440 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P23228 and previous config saved to /var/cache/conftool/dbconfig/20220327-005010-ladsgroup.json

26 March 2022

  • curprev 01:1201:12, 26 March 2022imported>Stashbot 845,387 bytes +31,577 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P23147 and previous config saved to /var/cache/conftool/dbconfig/20220326-011216-ladsgroup.json

25 March 2022

  • curprev 00:3900:39, 25 March 2022imported>Stashbot 813,810 bytes +36,424 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase2027.codfw.wmnet with OS buster

24 March 2022

  • curprev 00:3300:33, 24 March 2022imported>Stashbot 777,386 bytes +38,343 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1046.eqiad.wmnet with OS bullseye

23 March 2022

  • curprev 01:2001:20, 23 March 2022imported>Stashbot 739,043 bytes +42,681 ejegg: updated payments-wiki from 3048f0aa to 28e24856

22 March 2022

  • curprev 01:3501:35, 22 March 2022imported>Stashbot 696,362 bytes +39,479 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1047.eqiad.wmnet with OS bullseye

20 March 2022

  • curprev 23:4423:44, 20 March 2022imported>Stashbot 656,883 bytes +3,079 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T300775)', diff saved to https://phabricator.wikimedia.org/P22857 and previous config saved to /var/cache/conftool/dbconfig/20220320-234358-marostegui.json

19 March 2022

  • curprev 17:1817:18, 19 March 2022imported>Stashbot 653,804 bytes +4,978 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T300775)', diff saved to https://phabricator.wikimedia.org/P22845 and previous config saved to /var/cache/conftool/dbconfig/20220319-171757-marostegui.json
  • curprev 01:4601:46, 19 March 2022imported>Stashbot 648,826 bytes +15,864 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1016.eqiad.wmnet with reason: host reimage

17 March 2022

  • curprev 22:5522:55, 17 March 2022imported>Stashbot 632,962 bytes +44,817 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 01:1101:11, 17 March 2022imported>Stashbot 588,145 bytes +54,021 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1016.eqiad.wmnet with OS bullseye

16 March 2022

  • curprev 00:3600:36, 16 March 2022imported>Stashbot 534,124 bytes +72,992 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6011.drmrs.wmnet with reason: host reimage

15 March 2022

  • curprev 01:3001:30, 15 March 2022imported>Stashbot 461,132 bytes +46,179 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T300775)', diff saved to https://phabricator.wikimedia.org/P22465 and previous config saved to /var/cache/conftool/dbconfig/20220315-013013-marostegui.json

11 March 2022

  • curprev 15:5615:56, 11 March 2022imported>Stashbot 414,953 bytes +11,582 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2014.codfw.wmnet with OS bullseye
  • curprev 00:3300:33, 11 March 2022imported>Stashbot 403,371 bytes +68,747 TimStarling: on mwmaint1002 running populateGlobalEditCount.php

10 March 2022

  • curprev 00:2600:26, 10 March 2022imported>Stashbot 334,624 bytes +40,477 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@7975c27]: (no justification provided) (duration: 00m 08s)

9 March 2022

  • curprev 01:3201:32, 9 March 2022imported>Stashbot 294,147 bytes +75,530 marostegui@cumin2002: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P22170 and previous config saved to /var/cache/conftool/dbconfig/20220309-013256-marostegui.json

8 March 2022

  • curprev 00:3400:34, 8 March 2022imported>Stashbot 218,617 bytes +83,815 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@c8a753b]: (no justification provided) (duration: 00m 07s)

4 March 2022

  • curprev 17:5917:59, 4 March 2022imported>Stashbot 134,802 bytes +23,275 btullis@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:3501:35, 4 March 2022imported>Stashbot 111,527 bytes +40,357 rzl@deploy1002: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply

3 March 2022

  • curprev 01:4201:42, 3 March 2022imported>Stashbot 71,170 bytes +51,552 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on datahubsearch[1001-1003].eqiad.wmnet with reason: Still having errors setting up opensearch

2 March 2022

  • curprev 00:1500:15, 2 March 2022imported>Stashbot 19,618 bytes +18,689 topranks: Re-enabling Lumen AS3356 BGP session over IPv4 on cr3-ulsfo to assess affect on currently broken routing to ulsfo.

1 March 2022

  • curprev 01:1401:14, 1 March 2022imported>Stashbot 929 bytes −955,327 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1104 (T302185)', diff saved to https://phabricator.wikimedia.org/P21614 and previous config saved to /var/cache/conftool/dbconfig/20220301-011404-ladsgroup.json

27 February 2022

25 February 2022

  • curprev 23:3223:32, 25 February 2022imported>Stashbot 956,175 bytes +19,462 dzahn@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply

24 February 2022

  • curprev 23:3523:35, 24 February 2022imported>Stashbot 936,713 bytes +51,509 ryankemper: T302526 Deployed https://gerrit.wikimedia.org/r/765652 and ran puppet across wcqs*
  • curprev 00:5900:59, 24 February 2022imported>Stashbot 885,204 bytes +60,442 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2074.codfw.wmnet with OS bullseye

23 February 2022

  • curprev 01:4101:41, 23 February 2022imported>Stashbot 824,762 bytes +60,474 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2068.codfw.wmnet with reason: host reimage

21 February 2022

  • curprev 22:3022:30, 21 February 2022imported>Stashbot 764,288 bytes +74,792 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T300381)', diff saved to https://phabricator.wikimedia.org/P21231 and previous config saved to /var/cache/conftool/dbconfig/20220221-223015-marostegui.json
  • curprev 01:3901:39, 21 February 2022imported>Stashbot 689,496 bytes +3,448 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db2152.codfw.wmnet with OS bullseye

19 February 2022

  • curprev 16:5016:50, 19 February 2022imported>Stashbot 686,048 bytes +5,104 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 00:5900:59, 19 February 2022imported>Stashbot 680,944 bytes +28,538 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2020.codfw.wmnet with OS bullseye

17 February 2022

  • curprev 22:2822:28, 17 February 2022imported>Stashbot 652,406 bytes +28,397 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:3601:36, 17 February 2022imported>Stashbot 624,009 bytes +52,457 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T300381)', diff saved to https://phabricator.wikimedia.org/P20954 and previous config saved to /var/cache/conftool/dbconfig/20220217-013607-marostegui.json

15 February 2022

  • curprev 23:4723:47, 15 February 2022imported>Stashbot 571,552 bytes +59,268 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host restbase-dev2003.mgmt.codfw.wmnet with reboot policy FORCED

14 February 2022

  • curprev 22:0422:04, 14 February 2022imported>Stashbot 512,284 bytes +52,126 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

13 February 2022

  • curprev 23:1723:17, 13 February 2022imported>Stashbot 460,158 bytes +3,305 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T300775)', diff saved to https://phabricator.wikimedia.org/P20627 and previous config saved to /var/cache/conftool/dbconfig/20220213-231742-marostegui.json

12 February 2022

  • curprev 22:5822:58, 12 February 2022imported>Stashbot 456,853 bytes +2,897 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T300775)', diff saved to https://phabricator.wikimedia.org/P20617 and previous config saved to /var/cache/conftool/dbconfig/20220212-225806-marostegui.json

11 February 2022

10 February 2022

  • curprev 00:4200:42, 10 February 2022imported>Stashbot 362,781 bytes +25,944 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

8 February 2022

  • curprev 23:5223:52, 8 February 2022imported>Stashbot 336,837 bytes +73,681 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2055.codfw.wmnet with OS buster
  • curprev 00:1200:12, 8 February 2022imported>Stashbot 263,156 bytes +33,177 ryankemper: T294805 Re-enabling puppet across eqiad elastic fleet: `ryankemper@cumin1001:~$ sudo cumin -b 8 'elastic1*' 'sudo enable-puppet "Add new eqiad replacement hosts elastic10[68-83] - T294805 - root" && sudo run-puppet-agent'` tmux session `elastic`

5 February 2022

  • curprev 22:1022:10, 5 February 2022imported>Stashbot 229,979 bytes +1,284 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2003-dev.codfw.wmnet with OS bullseye

4 February 2022

  • curprev 23:4323:43, 4 February 2022imported>Stashbot 228,695 bytes +5,568 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mirror1001.wikimedia.org with reason: new kernel
  • curprev 01:0801:08, 4 February 2022imported>Stashbot 223,127 bytes +72,959 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

3 February 2022

2 February 2022

  • curprev 00:5300:53, 2 February 2022imported>Stashbot 90,142 bytes −738,181 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

1 February 2022

  • curprev 00:3100:31, 1 February 2022imported>Stashbot 828,323 bytes +72,250 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

29 January 2022

  • curprev 21:0821:08, 29 January 2022imported>Stashbot 756,073 bytes +1,014 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices2003-dev.wikimedia.org with OS bullseye
  • curprev 00:1400:14, 29 January 2022imported>Stashbot 755,059 bytes +14,112 ebernhardson: restart elasticsearch_6@production-search-psi-eqiad on elastic1049 to address CirrusSearchJVMGCOldPoolFlatlined alert

28 January 2022

  • curprev 01:4701:47, 28 January 2022imported>Stashbot 740,947 bytes +73,300 andrew@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcontrol2001-dev.wikimedia.org with OS bullseye

27 January 2022

26 January 2022

25 January 2022

  • curprev 00:3100:31, 25 January 2022imported>Stashbot 525,956 bytes +53,597 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

23 January 2022

  • curprev 22:0222:02, 23 January 2022imported>Stashbot 472,359 bytes +500 ebysans@deploy1002: Finished deploy [airflow-dags/analytics-test@37937f6]: (no justification provided) (duration: 00m 08s)

22 January 2022

  • curprev 22:3822:38, 22 January 2022imported>Stashbot 471,859 bytes +812 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx1001.wikimedia.org with reason: kernel testing
  • curprev 01:3001:30, 22 January 2022imported>Stashbot 471,047 bytes +18,324 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on mx1001.wikimedia.org with reason: kernel testing

20 January 2022

  • curprev 22:4022:40, 20 January 2022imported>Stashbot 452,723 bytes +39,372 inflatador: running puppet-merge for https://gerrit.wikimedia.org/r/755810

19 January 2022

17 January 2022

  • curprev 23:2723:27, 17 January 2022imported>Stashbot 327,176 bytes +12,624 jynus: forced session revocation on phab for a user T299315

16 January 2022

  • curprev 08:2108:21, 16 January 2022imported>Stashbot 314,552 bytes +684 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: sync on production

15 January 2022

  • curprev 08:5508:55, 15 January 2022imported>Stashbot 313,868 bytes +1,296 legoktm: finished running recountCategories on s4 wikis (T299244)
  • curprev 01:2201:22, 15 January 2022imported>Stashbot 312,572 bytes +10,517 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn

14 January 2022

  • curprev 00:3600:36, 14 January 2022imported>Stashbot 302,055 bytes +32,093 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

13 January 2022

  • curprev 00:3500:35, 13 January 2022imported>Stashbot 269,962 bytes +64,421 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

12 January 2022

  • curprev 00:5500:55, 12 January 2022imported>Stashbot 205,541 bytes +59,425 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

11 January 2022

8 January 2022

  • curprev 10:5110:51, 8 January 2022imported>Stashbot 107,107 bytes +180 elukey: restart hive daemons on an-coord1002 (after my last upgrade/rollback of packages the prometheus agent settings were not picked up, so no metrics)

7 January 2022

6 January 2022

5 January 2022

  • curprev 00:5900:59, 5 January 2022imported>Stashbot 64,897 bytes +32,691 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

4 January 2022

  • curprev 00:5400:54, 4 January 2022imported>Stashbot 32,206 bytes +32,091 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P18329 and previous config saved to /var/cache/conftool/dbconfig/20220104-005456-marostegui.json

1 January 2022

29 December 2021

  • curprev 10:3010:30, 29 December 2021imported>Stashbot 664,919 bytes +126 elukey: kill tcpdump process on kubestagemaster1001 (kept a big pcap file opened that kept growing)

28 December 2021

24 December 2021

  • curprev 20:0820:08, 24 December 2021imported>Stashbot 663,863 bytes +325 mforns@deploy1002: Finished deploy [airflow-dags/analytics@e282d2d]: (no justification provided) (duration: 00m 06s)
  • curprev 00:5700:57, 24 December 2021imported>Stashbot 663,538 bytes +3,353 ejegg: updated fundraising CiviCRM from 47dd67f2 to aaceb4ab

23 December 2021

  • curprev 00:0400:04, 23 December 2021imported>Stashbot 660,185 bytes +4,302 bking@cumin1001: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T297986

21 December 2021

19 December 2021

18 December 2021

  • curprev 13:5713:57, 18 December 2021imported>Stashbot 633,583 bytes +93 dcausse: restarting blazegraph on wdqs1013 (jvm stuck for 10hours)

17 December 2021

16 December 2021

  • curprev 00:3700:37, 16 December 2021imported>Stashbot 600,611 bytes +14,349 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

15 December 2021

14 December 2021

  • curprev 01:4201:42, 14 December 2021imported>Stashbot 545,410 bytes +41,246 ryankemper: T297468 `sudo cookbook sre.elasticsearch.rolling-operation search_eqiad "eqiad rolling restart" --nodes-per-run 3 --start-datetime 2021-12-14T01:27:58 --task-id T297468` on `ryankemper@cumin1001` tmux `elastic_restarts`

12 December 2021

  • curprev 14:3514:35, 12 December 2021imported>Stashbot 504,164 bytes +844 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host graphite1004.eqiad.wmnet

11 December 2021

  • curprev 19:0419:04, 11 December 2021imported>Stashbot 503,320 bytes +131 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster
  • curprev 00:0400:04, 11 December 2021imported>Stashbot 503,189 bytes +18,770 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

10 December 2021

9 December 2021

  • curprev 00:2600:26, 9 December 2021imported>Stashbot 469,358 bytes +7,967 rzl: graphite1004.mgmt: /admin1-> racadm serveraction powercycle (T297265)

8 December 2021

  • curprev 00:5100:51, 8 December 2021imported>Stashbot 461,391 bytes +29,464 ebernhardson@deploy1002: Synchronized php-1.38.0-wmf.12/extensions/GrowthExperiments/includes/NewcomerTasks/AddImage/AddImageSubmissionHandler.php: backport window for 744896 (duration: 01m 05s)

7 December 2021

4 December 2021

  • curprev 01:1401:14, 4 December 2021imported>Stashbot 424,523 bytes +12,137 mutante: mx2001 - did not come back from reboot, did not get IP on interface, could not start ferm, logged in via console with root password, in /etc/network/interfaces replaced all "ens5" with "ens13", rebooted again, selected previous kernel version

3 December 2021

2 December 2021

  • curprev 01:2101:21, 2 December 2021imported>Stashbot 394,618 bytes +24,122 ryankemper: T280001 Rolling restart of low-traffic pybal hosts complete. All of `wcqs` is pooled and the pybal / ipvs related alerts have cleared

1 December 2021

  • curprev 00:3500:35, 1 December 2021imported>Stashbot 370,496 bytes +9,705 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

30 November 2021

  • curprev 00:2200:22, 30 November 2021imported>Stashbot 360,791 bytes +17,004 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

28 November 2021

  • curprev 17:1417:14, 28 November 2021imported>Stashbot 343,787 bytes +293 elukey@deploy1002: Finished deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 (duration: 02m 11s)

27 November 2021

  • curprev 19:5519:55, 27 November 2021imported>Stashbot 343,494 bytes +1,043 andrew@deploy1002: Finished deploy [horizon/deploy@6115b3b]: network UI updates for T296548 (duration: 04m 14s)

26 November 2021

  • curprev 16:1116:11, 26 November 2021imported>Stashbot 342,451 bytes +4,657 arnoldokoth: drain kubestage1002 node in prep for decommissioning

25 November 2021

  • curprev 20:4420:44, 25 November 2021imported>Stashbot 337,794 bytes +25,467 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17872 and previous config saved to /var/cache/conftool/dbconfig/20211125-204357-ladsgroup.json
  • curprev 00:3700:37, 25 November 2021imported>Stashbot 312,327 bytes +56,025 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2066.codfw.wmnet with OS buster

24 November 2021

  • curprev 00:2500:25, 24 November 2021imported>Stashbot 256,302 bytes +24,082 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS buster

23 November 2021

  • curprev 00:5100:51, 23 November 2021imported>Stashbot 232,220 bytes +27,289 urbanecm@deploy1002: Started scap: 69aa4a7: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 4/4)

21 November 2021

20 November 2021

  • curprev 01:0201:02, 20 November 2021imported>Stashbot 204,605 bytes +91 mutante: lists1001 - restarted apache, icinga alerts for the web UI, but recovered
  • curprev 00:2700:27, 20 November 2021imported>Stashbot 204,514 bytes +9,398 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)

19 November 2021

  • curprev 01:4501:45, 19 November 2021imported>Stashbot 195,116 bytes +21,872 mutante: I think git-ssh6_22 is down (see alerts lvs2008/2009) due to the v6 issue from ongoing lvs maintenance. depooled in conftool

18 November 2021

  • curprev 01:4701:47, 18 November 2021imported>Stashbot 173,244 bytes +19,375 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2005.codfw.wmnet

17 November 2021

  • curprev 00:1900:19, 17 November 2021imported>Stashbot 153,869 bytes +12,582 ryankemper: T276198 `ryankemper@cumin1001:~$ sudo cumin -b 3 '*elastic*' 'sudo run-puppet-agent --force'` Change looks good (no complaints from systemd), rolling out to rest of fleet / reenabling puppet

16 November 2021

14 November 2021

  • curprev 11:4811:48, 14 November 2021imported>Stashbot 125,456 bytes +111 paravoid: disable cr1-eqiad:xe-3/0/6 (IXP port) to mitigate T295650

13 November 2021

  • curprev 18:4318:43, 13 November 2021imported>Stashbot 125,345 bytes +836 AndyRussG: Enabled debug logging for PayPal IPN listener (updated SmashPig config a9e30591 -> 9567cc4a on frpig1001)

12 November 2021

  • curprev 21:0021:00, 12 November 2021imported>Stashbot 124,509 bytes +5,102 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:1900:19, 12 November 2021imported>Stashbot 119,407 bytes +11,807 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

11 November 2021

  • curprev 00:2200:22, 11 November 2021imported>Stashbot 107,600 bytes +12,063 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

10 November 2021

  • curprev 01:0701:07, 10 November 2021imported>Stashbot 95,537 bytes +10,338 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

8 November 2021

  • curprev 23:3923:39, 8 November 2021imported>Stashbot 85,199 bytes +11,520 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

6 November 2021

5 November 2021

  • curprev 00:1600:16, 5 November 2021imported>Stashbot 66,667 bytes +20,851 mutante: phab1001 - sudo systemctl start phabricator_clean_tmp_files.service because Icinga alerted it had failed... worked fine

4 November 2021

2 November 2021

  • curprev 23:4723:47, 2 November 2021imported>Stashbot 22,959 bytes +11,789 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:2101:21, 2 November 2021imported>Stashbot 11,170 bytes −844,596 ejegg: updated SmashPig standalone deploy from dd3a81c7c2 to be68299b92

31 October 2021

  • curprev 21:4921:49, 31 October 2021imported>Stashbot 855,766 bytes +371 urbanecm: urbanecm@mwmaint1002:~$ mwscript userOptions.php --wiki=dewiki --nowarn --touserid 3802752 --old 'linkrecommendation' --new 'control' 'growthexperiments-homepage-variant' # T294712

30 October 2021

29 October 2021

  • curprev 22:5722:57, 29 October 2021imported>Stashbot 855,227 bytes +3,905 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=foundationwiki --userlist users.txt # T205347, users.txt is at P17641

28 October 2021

  • curprev 23:5023:50, 28 October 2021imported>Stashbot 851,322 bytes +21,486 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4033.ulsfo.wmnet with OS buster

27 October 2021

  • curprev 23:5523:55, 27 October 2021imported>Stashbot 829,836 bytes +10,201 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

26 October 2021

  • curprev 22:5922:59, 26 October 2021imported>Stashbot 819,635 bytes +12,871 legoktm: uploaded python-logstash to buster-wikimedia for T294393

25 October 2021

  • curprev 23:1223:12, 25 October 2021imported>Stashbot 806,764 bytes +11,982 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Create alias for Appendix and Appendix_talk namespaces on mywiktionary (T291146) (duration: 00m 55s)

23 October 2021

  • curprev 16:4016:40, 23 October 2021imported>Stashbot 794,782 bytes +253 dcausse: restarting blazegraph on wdqs1004 and wdqs1006 (free allocators alert)

22 October 2021

  • curprev 23:1723:17, 22 October 2021imported>Stashbot 794,529 bytes +3,761 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

21 October 2021

  • curprev 23:4023:40, 21 October 2021imported>Stashbot 790,768 bytes +17,384 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:0600:06, 21 October 2021imported>Stashbot 773,384 bytes +14,742 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

20 October 2021

  • curprev 01:3101:31, 20 October 2021imported>Stashbot 758,642 bytes +27,590 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

19 October 2021

  • curprev 00:3800:38, 19 October 2021imported>Stashbot 731,052 bytes +11,211 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer

16 October 2021

  • curprev 03:5603:56, 16 October 2021imported>Stashbot 719,841 bytes +266 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

15 October 2021

  • curprev 23:4823:48, 15 October 2021imported>Stashbot 719,575 bytes +5,052 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
  • curprev 00:0900:09, 15 October 2021imported>Stashbot 714,523 bytes +19,109 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

14 October 2021

  • curprev 01:4101:41, 14 October 2021imported>Stashbot 695,414 bytes +17,092 ejegg: updated payments-wiki from b329d2dea2 to 19d18c1852

13 October 2021

  • curprev 00:3800:38, 13 October 2021imported>Stashbot 678,322 bytes +15,911 ejegg: updated payments-wiki from 030b11da1a to b329d2dea2

12 October 2021

  • curprev 00:1100:11, 12 October 2021imported>Stashbot 662,411 bytes +3,728 eileen: civicrm revision changed from 598b59b0ee to 96090e4bd2, config revision is 85277466ed

9 October 2021

  • curprev 05:0105:01, 9 October 2021imported>Stashbot 658,683 bytes +224 jiji@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:3201:32, 9 October 2021imported>Stashbot 658,459 bytes +8,808 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

8 October 2021

7 October 2021

  • curprev 00:1100:11, 7 October 2021imported>Stashbot 637,764 bytes +11,864 mutante: [grafana2001:~] $ sudo systemctl start rsync-var-lib-grafana because of "PROBLEM - Check systemd state on grafana2001 is CRITICAL: CRITICAL - degraded" because of some race condition where a file vanished during sync

6 October 2021

  • curprev 01:3901:39, 6 October 2021imported>Stashbot 625,900 bytes +10,381 legoktm: legoktm@mwmaint1002:~$ echo "https://en.wikiversity.org/static/images/mobile/copyright/wikiversity.svg" |mwscript purgeList.php

4 October 2021

  • curprev 23:3023:30, 4 October 2021imported>Stashbot 615,519 bytes +9,952 foks: resetting some emails used for abuse by a globally-banned user

3 October 2021

2 October 2021

  • curprev 17:2817:28, 2 October 2021imported>Stashbot 605,040 bytes +230 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

1 October 2021

  • curprev 23:1923:19, 1 October 2021imported>Stashbot 604,810 bytes +38,661 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

29 September 2021

  • curprev 23:2023:20, 29 September 2021imported>Stashbot 566,149 bytes +14,880 bd808@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 00:5200:52, 29 September 2021imported>Stashbot 551,269 bytes +27,094 eileen: civicrm revision changed from a1929b3dfd to a480bf03c9, config revision is 77cb7ec866

27 September 2021

  • curprev 23:4223:42, 27 September 2021imported>Stashbot 524,175 bytes +27,476 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

26 September 2021

  • curprev 14:5114:51, 26 September 2021imported>Stashbot 496,699 bytes +432 volker-e@deploy1002: Finished deploy [design/style-guide@aac0ae9]: Deploy design/style-guide: aac0ae9 “Apps”: Fix image path (#490) (duration: 00m 06s)

25 September 2021

  • curprev 02:0002:00, 25 September 2021imported>Stashbot 496,267 bytes +374 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .

24 September 2021

  • curprev 20:0020:00, 24 September 2021imported>Stashbot 495,893 bytes +10,338 volker-e@deploy1002: Finished deploy [design/style-guide@362c6b1]: Deploy design/style-guide: 362c6b1 “Components”: Fix index link (#489) (duration: 00m 06s)
  • curprev 00:3900:39, 24 September 2021imported>Stashbot 485,555 bytes +19,157 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

23 September 2021

  • curprev 00:3500:35, 23 September 2021imported>Stashbot 466,398 bytes +12,744 catrope@deploy1002: Synchronized php-1.38.0-wmf.1/extensions/MediaSearch/: Use text() instead of parse() for MediaSearch UI messages (T291590) (duration: 01m 08s)

21 September 2021

  • curprev 23:2023:20, 21 September 2021imported>Stashbot 453,654 bytes +9,016 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:1600:16, 21 September 2021imported>Stashbot 444,638 bytes +11,548 tgr: Evening deploys done

18 September 2021

  • curprev 01:4701:47, 18 September 2021imported>Stashbot 433,090 bytes +322 ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.23/includes/libs/rdbms/database/Database.php: (no justification provided) (duration: 00m 57s)

17 September 2021

  • curprev 21:2821:28, 17 September 2021imported>Stashbot 432,768 bytes +3,000 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:0400:04, 17 September 2021imported>Stashbot 429,768 bytes +13,476 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .

15 September 2021

14 September 2021

  • curprev 23:0123:01, 14 September 2021imported>Stashbot 406,930 bytes +15,959 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Re-enable VipsScaler (2 of 2) (duration: 01m 04s)
  • curprev 00:0400:04, 14 September 2021imported>Stashbot 390,971 bytes +8,994 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

12 September 2021

11 September 2021

  • curprev 19:0219:02, 11 September 2021imported>Stashbot 381,555 bytes +650 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 27814b8eaacb5ba2fee1b6167a36ea14356a1ecf: testwiki: Fully remove securepoll-related groups (T290808) (duration: 00m 57s)

10 September 2021

  • curprev 21:2921:29, 10 September 2021imported>Stashbot 380,905 bytes +5,151 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • curprev 00:3500:35, 10 September 2021imported>Stashbot 375,754 bytes +15,347 tgr: Deployed patch for T290692

9 September 2021

  • curprev 00:5000:50, 9 September 2021imported>Stashbot 360,407 bytes +8,465 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Don't set default to Score (try #2) (duration: 00m 58s)

8 September 2021

  • curprev 00:0000:00, 8 September 2021imported>Stashbot 351,942 bytes +12,900 legoktm: legoktm@lists1001:~$ sudo rm -rf /etc/mailman # cleanup as part of 4869d91b0be / T282303

6 September 2021

  • curprev 23:5223:52, 6 September 2021imported>Stashbot 339,042 bytes +8,019 tstarling@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/SecurePoll/includes/Talliers/STVTallier.php: T290000 (duration: 00m 58s)

5 September 2021

  • curprev 18:5418:54, 5 September 2021imported>Stashbot 331,023 bytes +201 urbanecm: wikiadmin@10.192.0.119(ptwiki)> update protected_titles set pt_create_perm='editautoreviewprotected' where pt_create_perm='autoreviewer'; # T290396

4 September 2021

  • curprev 13:3513:35, 4 September 2021imported>Stashbot 330,822 bytes +1,989 marostegui@cumin1001: dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: Slowly repool T290374', diff saved to https://phabricator.wikimedia.org/P17217 and previous config saved to /var/cache/conftool/dbconfig/20210904-133532-root.json

3 September 2021

1 September 2021

  • curprev 23:5023:50, 1 September 2021imported>Stashbot 297,294 bytes +11,478 Amir1: mwscript createAndPromote.php --wiki=test2wiki --sysop --force Ladsgroup
  • curprev 00:5300:53, 1 September 2021imported>Stashbot 285,816 bytes +13,363 eileen: civicrm revision changed from e567b4c289 to 7da3eba4f9, config revision is 5f004d94d7

30 August 2021

  • curprev 23:1423:14, 30 August 2021imported>Stashbot 272,453 bytes +10,249 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

29 August 2021

  • curprev 00:1500:15, 29 August 2021imported>Stashbot 262,204 bytes +714 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

27 August 2021

  • curprev 16:4616:46, 27 August 2021imported>Stashbot 261,490 bytes +3,953 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps1005.eqiad.wmnet with reason: Resyncing from master
  • curprev 00:4600:46, 27 August 2021imported>Stashbot 257,537 bytes +9,223 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

25 August 2021

  • curprev 23:2323:23, 25 August 2021imported>Stashbot 248,314 bytes +23,982 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:5300:53, 25 August 2021imported>Stashbot 224,332 bytes +6,127 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .

24 August 2021

  • curprev 00:1700:17, 24 August 2021imported>Stashbot 218,205 bytes +10,888 ryankemper: [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'`

22 August 2021

21 August 2021

  • curprev 15:1215:12, 21 August 2021imported>Stashbot 1,011,788 bytes +262 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

20 August 2021

  • curprev 23:1823:18, 20 August 2021imported>Stashbot 1,011,526 bytes +4,817 legoktm: deployed patch for T289385
  • curprev 00:1800:18, 20 August 2021imported>Stashbot 1,006,709 bytes +10,542 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

19 August 2021

  • curprev 01:1301:13, 19 August 2021imported>Stashbot 996,167 bytes +6,256 ejegg: updated fundraising CiviCRM from 73f6ec9190 to 8ed303f2d1

18 August 2021

  • curprev 00:3900:39, 18 August 2021imported>Stashbot 989,911 bytes +13,880 dpifke@deploy1002: Finished deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243) (duration: 00m 05s)

17 August 2021

  • curprev 00:4500:45, 17 August 2021imported>Stashbot 976,031 bytes +12,823 eileen: civicrm revision changed from ba0c7705bb to 175a3101f7, config revision is 7bdc78073d

15 August 2021

14 August 2021

  • curprev 03:5403:54, 14 August 2021imported>Stashbot 962,909 bytes +121 legoktm[m]: restarting mailman3 on lists1001, bounce runner crashed (T288880)

13 August 2021

  • curprev 18:4318:43, 13 August 2021imported>Stashbot 962,788 bytes +9,163 bblack: reprepro: uploaded gdnsd-3.8.0-1~wmf1 to buster-wikimedia - T252132

12 August 2021

11 August 2021

  • curprev 23:3023:30, 11 August 2021imported>Stashbot 931,568 bytes +15,885 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:4701:47, 11 August 2021imported>Stashbot 915,683 bytes +19,823 dpifke@deploy1002: Finished deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 06s)

9 August 2021

  • curprev 16:1216:12, 9 August 2021imported>Stashbot 895,860 bytes +14,288 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .

6 August 2021

  • curprev 19:1719:17, 6 August 2021imported>Stashbot 881,572 bytes +8,445 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:0101:01, 6 August 2021imported>Stashbot 873,127 bytes +23,598 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be2065.codfw.wmnet with reason: REIMAGE

5 August 2021

  • curprev 01:2601:26, 5 August 2021imported>Stashbot 849,529 bytes +19,135 Krinkle: krinkle@mwmaint1002 Temporarily grant myself `translationadmin` on wikimania2016wiki in order to approve an edit given FlaggedRevs-like nature of Translate

3 August 2021

  • curprev 23:3423:34, 3 August 2021imported>Stashbot 830,394 bytes +13,656 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:4400:44, 3 August 2021imported>Stashbot 816,738 bytes +12,154 reedy@deploy1002: Finished deploy [integration/docroot@f9d225d]: with less gref (duration: 00m 05s)

31 July 2021

  • curprev 12:4012:40, 31 July 2021imported>Stashbot 804,584 bytes +151 reedy@deploy1002: Synchronized php-1.37.0-wmf.16/extensions/SecurePoll/: T287780 T287782 (duration: 00m 58s)
  • curprev 00:0100:01, 31 July 2021imported>Stashbot 804,433 bytes +8,409 eileen: civicrm revision changed from 158ed65e00 to d6baf291f4, config revision is 6011d9c471

29 July 2021

28 July 2021

27 July 2021

26 July 2021

  • curprev 23:3723:37, 26 July 2021imported>Stashbot 765,495 bytes +4,200 legoktm@deploy1002: Synchronized php-1.37.0-wmf.15/extensions/Score/includes/Score.php: Increase lilypond version cache TTL to 1 hour (duration: 00m 57s)

24 July 2021

  • curprev 11:0411:04, 24 July 2021imported>Stashbot 761,295 bytes +338 urbanecm: [urbanecm@mwmaint2002 ~]$ mwscript extensions/Translate/scripts/moveTranslatablePage.php --wiki=commonswiki --reason='OTRS -> VRTS renaming process; see Phab:T280392 and Phab:T280397' --move-subpages 'Commons:OTRS' 'Commons:Volunteer Response Team' 'Martin Urbanec' # T287321

23 July 2021

  • curprev 19:1119:11, 23 July 2021imported>Stashbot 760,957 bytes +5,409 topranks: Successfully re-pooled eqiad - reversed change from yesterday after successful line card replacement in cr2-codfw - T287110

22 July 2021

20 July 2021

  • curprev 20:5320:53, 20 July 2021imported>Stashbot 730,828 bytes +12,637 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: caa5a076f39b051b01622aa3e4c9d716a8643eef: Set wgGEMentorDashboardBackendEnabled properly (T285811) (duration: 00m 57s)

19 July 2021

16 July 2021

  • curprev 19:5019:50, 16 July 2021imported>Stashbot 704,075 bytes +7,705 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on copernicium.wikimedia.org with reason: REIMAGE
  • curprev 00:0600:06, 16 July 2021imported>Stashbot 696,370 bytes +21,594 hoo: Updated the Wikidata property suggester with data from the 2021-07-12 JSON dump (with pre-applied T132839 workarounds)

15 July 2021

14 July 2021

  • curprev 00:5800:58, 14 July 2021imported>Stashbot 662,656 bytes +13,088 eileen: process control updated to c291b3c6890364281d

12 July 2021

  • curprev 23:5723:57, 12 July 2021imported>Stashbot 649,568 bytes +13,413 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 1896efc27f3de39659673091bc4c43ad874da0c5: Add sayahna.org to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T286163) (duration: 00m 56s)

9 July 2021

  • curprev 23:2823:28, 9 July 2021imported>Stashbot 636,155 bytes +1,014 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • curprev 00:4700:47, 9 July 2021imported>Stashbot 635,141 bytes +9,116 legoktm: zotero rolling restart didn't help, filed T286360 for DNS issues

7 July 2021

  • curprev 20:2220:22, 7 July 2021imported>Stashbot 626,025 bytes +5,635 legoktm: repooling eqiad - https://gerrit.wikimedia.org/r/703561

6 July 2021

  • curprev 18:3418:34, 6 July 2021imported>Stashbot 620,390 bytes +5,171 otto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
  • curprev 00:5600:56, 6 July 2021imported>Stashbot 615,219 bytes +1,686 eileen: process-control config revision is 8d46b52ed4

5 July 2021

  • curprev 00:5300:53, 5 July 2021imported>Stashbot 613,533 bytes +528 eileen: process-control config revision is a1717c7fde

3 July 2021

  • curprev 17:4617:46, 3 July 2021imported>Stashbot 613,005 bytes +322 elukey: depool eqsin due to loss of power redundancy (equinix maintenance) - T286113

2 July 2021

  • curprev 22:0622:06, 2 July 2021imported>Stashbot 612,683 bytes +8,804 foks: removing three files for legal compliance
  • curprev 01:4701:47, 2 July 2021imported>Stashbot 603,879 bytes +9,552 eileen: civicrm revision changed from e07c2be1a7 to bb62188ec6, config revision is 1739c53fcb

30 June 2021

  • curprev 23:2823:28, 30 June 2021imported>Stashbot 594,327 bytes +11,040 urbanecm: Evening B&C window finished
  • curprev 00:3600:36, 30 June 2021imported>Stashbot 583,287 bytes +15,990 tstarling@deploy1002: Synchronized wmf-config/db-labs.php: gerrit 701995 SQL query log (duration: 01m 05s)

29 June 2021

  • curprev 00:2600:26, 29 June 2021imported>Stashbot 567,297 bytes +16,261 Krinkle: krinkle@mwmaint1002: purgeParserCache.php --tag pc1, ref T282761

27 June 2021

  • curprev 09:1909:19, 27 June 2021imported>Elukey 551,036 bytes +741 Add missing restart entry (logged on a different chan)

26 June 2021

  • curprev 21:2821:28, 26 June 2021imported>Stashbot 550,295 bytes +1,342 volans: upgraded spicerack to v0.0.56 on the cumin hosts (includes only bug fixes for the switchdc)

25 June 2021

  • curprev 21:3721:37, 25 June 2021imported>Stashbot 548,953 bytes +6,188 ebernhardson@deploy1002: Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s)

24 June 2021

  • curprev 23:0223:02, 24 June 2021imported>Stashbot 542,765 bytes +11,139 legoktm: reverted cumin1001 spicerack live hacks
  • curprev 00:2500:25, 24 June 2021imported>Stashbot 531,626 bytes +19,731 eileen: civicrm revision changed from bd906975f0 to 6d3dd6e5a5, config revision is 821e5889f7

23 June 2021

  • curprev 00:5000:50, 23 June 2021imported>Stashbot 511,895 bytes +14,361 eileen: civicrm revision changed from c745d4f075 to 03bead707d, config revision is 4ab72c1033

22 June 2021

  • curprev 01:0201:02, 22 June 2021imported>Stashbot 497,534 bytes +26,215 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE

18 June 2021

  • curprev 20:5520:55, 18 June 2021imported>Stashbot 471,319 bytes +9,077 Krinkle: Remove doc1001:/srv/doc/mediawiki-core/wmf-1.36.0-wmf.31-testing

17 June 2021

  • curprev 21:4921:49, 17 June 2021imported>Stashbot 462,242 bytes +15,302 legoktm: regenerating pipermail redirects to skip those with duplicate message-ids (T280731)

16 June 2021

  • curprev 21:3521:35, 16 June 2021imported>Stashbot 446,940 bytes +4,876 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .

15 June 2021

  • curprev 17:5417:54, 15 June 2021imported>Stashbot 442,064 bytes +4,022 dancy: testing upcoming Scap release on beta
  • curprev 00:3700:37, 15 June 2021imported>Stashbot 438,042 bytes +26,468 eileen: civicrm revision changed from 31d07115a0 to 28ace1b86f, config revision is 2aed6ff89b

12 June 2021

  • curprev 13:4913:49, 12 June 2021imported>Stashbot 411,574 bytes +317 rzl@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: alert noise, no impact, x2 is unused

11 June 2021

  • curprev 23:3723:37, 11 June 2021imported>Stashbot 411,257 bytes +6,591 mutante: removing firewall hole for mgmt networks to install* because it turned out it cant be used for firmware upgrades
  • curprev 01:1001:10, 11 June 2021imported>Stashbot 404,666 bytes +17,032 eileen: process-control config revision is 2aed6ff89b

9 June 2021

7 June 2021

  • curprev 21:2621:26, 7 June 2021imported>Stashbot 350,845 bytes +11,147 otto@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0)

5 June 2021

  • curprev 16:1616:16, 5 June 2021imported>Stashbot 339,698 bytes +307 Amir1: deleting all private archives of mm2. All are inaccessible now (T282303)
  • curprev 00:2100:21, 5 June 2021imported>Stashbot 339,391 bytes +11,476 mutante: backup1001 - systemctl baclua-dir works again (restoring backup for non-existing host)

4 June 2021

  • curprev 00:1200:12, 4 June 2021imported>Stashbot 327,915 bytes +24,727 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

3 June 2021

1 June 2021

  • curprev 21:0921:09, 1 June 2021imported>Stashbot 287,834 bytes +5,348 andrewbogott: dropping a bunch of tables from the labswiki db as per T284108
  • curprev 00:4600:46, 1 June 2021imported>Stashbot 282,486 bytes +974 legoktm@deploy1002: Synchronized logos/config.yaml: Revert "Use eswiki 20th anniversary logos" (T280908) (duration: 01m 07s)

29 May 2021

  • curprev 14:4414:44, 29 May 2021imported>Stashbot 281,512 bytes +194 elukey: execute apt-get clean on an-airflow1001 to free space

28 May 2021

  • curprev 08:0608:06, 28 May 2021imported>Stashbot 281,318 bytes +368 oblivian@cumin1001: conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet,dc=eqiad

27 May 2021

  • curprev 23:5623:56, 27 May 2021imported>Stashbot 280,950 bytes +10,488 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab1004.eqiad.wmnet with reason: REIMAGE

26 May 2021

25 May 2021

  • curprev 00:4800:48, 25 May 2021imported>Stashbot 245,293 bytes +11,934 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

23 May 2021

  • curprev 14:2514:25, 23 May 2021imported>Stashbot 233,359 bytes +262 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: EMERGENCY: f752f8b80c57a5b7e41b91873b3eac388535ac46: enwiktionary: Raise AF emergency disable treshold+count (T283460) (duration: 00m 57s)

22 May 2021

  • curprev 22:1322:13, 22 May 2021imported>Stashbot 233,097 bytes +265 legoktm: reset 2FA for User:Yuvipanda on wikitech

21 May 2021

  • curprev 22:3222:32, 21 May 2021imported>Stashbot 232,832 bytes +12,510 bstorm: upload nfsd-ldap: 1.2+deb10u1 to buster-wikimedia T283385

20 May 2021

  • curprev 21:4521:45, 20 May 2021imported>Stashbot 220,322 bytes +13,158 herron@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:0101:01, 20 May 2021imported>Stashbot 207,164 bytes +12,243 mutante: signing puppet certs for doh2001 and doh2002.wikimedia.org (T283192)

18 May 2021

  • curprev 18:4018:40, 18 May 2021imported>Stashbot 194,921 bytes +14,463 razzi@deploy1002: Finished deploy [analytics/refinery@9392f1d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@9392f1db6e66975304c8e9b2b7031acd3ed87fa7] (duration: 05m 16s)
  • curprev 00:5600:56, 18 May 2021imported>Stashbot 180,458 bytes +19,706 eileen: civicrm revision changed from 38ac15233f to b3fb3c9cb0, config revision is 1f8d0a6bfa

17 May 2021

16 May 2021

  • curprev 00:5300:53, 16 May 2021imported>Stashbot 160,344 bytes +324 legoktm: restarted mailman3-web on lists1001, uwsgi looked like it got stuck, consuming all CPU/memory

14 May 2021

  • curprev 20:4220:42, 14 May 2021imported>Stashbot 160,020 bytes +3,428 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people1002.eqiad.wmnet
  • curprev 00:3900:39, 14 May 2021imported>Stashbot 156,592 bytes +8,035 ryankemper: T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2003.codfw.wmnet` on `ryankemper@cumin2001` tmux session `wdqs_reimage`

12 May 2021

  • curprev 23:4823:48, 12 May 2021imported>Stashbot 148,557 bytes +16,709 urbanecm@deploy1002: Synchronized php-1.37.0-wmf.4/extensions/WikiEditor/includes/WikiEditorHooks.php: 2f6af514c49d47bbec5ce51f9f7263015e039003? PHP VisualEditorFeatureUse logging: properly record session id (T281409) (duration: 01m 07s)
  • curprev 01:4201:42, 12 May 2021imported>Stashbot 131,848 bytes +17,502 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host

10 May 2021

  • curprev 23:3823:38, 10 May 2021imported>Stashbot 114,346 bytes +15,358 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: 779fb53bfd7a4d9b11f865df14f8a72adb97f33b: Update messages used for tech CoC (T280886) (duration: 00m 56s)

9 May 2021

  • curprev 21:4421:44, 9 May 2021imported>Stashbot 98,988 bytes −1,654,362 legoktm: restarted mailman3 again (T282348) pymysql.err.InternalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

8 May 2021

  • curprev 17:1817:18, 8 May 2021imported>Stashbot 1,753,350 bytes +105 Amir1: starting upgrade of batch G of mailing lists (T280322)

7 May 2021

  • curprev 21:4021:40, 7 May 2021imported>Stashbot 1,753,245 bytes +7,667 legoktm: deleted education@ from MM3, didn't import properly

6 May 2021

  • curprev 23:5023:50, 6 May 2021imported>Stashbot 1,745,578 bytes +25,003 brennen@deploy1002: rebuilt and synchronized wikiversions files: Rollback group1 and group2 to 1.37.0-wmf.3 (T282193)
  • curprev 00:3500:35, 6 May 2021imported>Stashbot 1,720,575 bytes +28,723 Amir1: sudo service mailman3-web restart

5 May 2021

  • curprev 01:4501:45, 5 May 2021imported>Stashbot 1,691,852 bytes +24,478 ryankemper: T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1011.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph` on `ryankemper@cumin1001` tmux session `reimage`

3 May 2021

  • curprev 23:1823:18, 3 May 2021imported>Stashbot 1,667,374 bytes +11,155 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 230ef5716b34ca83348667f289180313b76ce8a3: Prepare for new configuration option (T277951) (duration: 00m 57s)

2 May 2021

  • curprev 13:4013:40, 2 May 2021imported>Stashbot 1,656,219 bytes +311 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cloudmetrics1002.eqiad.wmnet with reason: Flaky host

1 May 2021

  • curprev 19:1219:12, 1 May 2021imported>Stashbot 1,655,908 bytes +557 Urbanecm: Invalidate password for MaraBot@SUL (T281586)

30 April 2021

  • curprev 21:5521:55, 30 April 2021imported>Stashbot 1,655,351 bytes +6,117 mutante: people1003 - rsycncing /home from peopel1002
  • curprev 01:0801:08, 30 April 2021imported>Stashbot 1,649,234 bytes +23,564 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw reboot - ryankemper@cumin1001 - T280563

29 April 2021

  • curprev 00:4100:41, 29 April 2021imported>Stashbot 1,625,670 bytes +24,040 tstarling@deploy1002: Synchronized php-1.37.0-wmf.3/includes/specials/pagers/ImageListPager.php: T281405 (duration: 01m 08s)

28 April 2021

  • curprev 01:0601:06, 28 April 2021imported>Stashbot 1,601,630 bytes +35,800 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

26 April 2021

  • curprev 23:2823:28, 26 April 2021imported>Stashbot 1,565,830 bytes +16,596 mutante: renewing TLS cert for peopleweb.discovery.wmnet, adding *3 hosts

25 April 2021

  • curprev 15:2315:23, 25 April 2021imported>Stashbot 1,549,234 bytes +138 Amir1: sudo -u list /var/lib/mailman/bin/change_pw -l wikica-l -p $(pwgen -c1 -s 12) (T281066)

24 April 2021

23 April 2021

22 April 2021

  • curprev 17:2617:26, 22 April 2021imported>Stashbot 1,545,052 bytes +1,074 marostegui: Stop mysql on tendril/dbtree database
  • curprev 01:3401:34, 22 April 2021imported>Stashbot 1,543,978 bytes +14,300 eileen: civicrm revision changed from 35a8dd33ba to 42ca3cf65a, config revision is cf07e7ba0b

21 April 2021

  • curprev 00:3800:38, 21 April 2021imported>Stashbot 1,529,678 bytes +27,920 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE

19 April 2021

  • curprev 22:5622:56, 19 April 2021imported>Stashbot 1,501,758 bytes +29,254 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE

18 April 2021

  • curprev 06:4006:40, 18 April 2021imported>Stashbot 1,472,504 bytes +106 Amir1: cleaning watchlist of User:Mr._Ibrahem in wikidatawiki (in main ns only)

17 April 2021

  • curprev 16:1616:16, 17 April 2021imported>Stashbot 1,472,398 bytes +61 Amir1: cleaning SuccuBot's watchlist in wikidatawiki
  • curprev 00:5300:53, 17 April 2021imported>Stashbot 1,472,337 bytes +38,446 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet

15 April 2021

  • curprev 01:1901:19, 15 April 2021imported>Stashbot 1,433,891 bytes +21,435 Amir1: mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P8671 --new-data-type external-id (T278427)

14 April 2021

  • curprev 00:1100:11, 14 April 2021imported>Stashbot 1,412,456 bytes +25,158 legoktm@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw2411.codfw.wmnet

12 April 2021

  • curprev 23:2523:25, 12 April 2021imported>Stashbot 1,387,298 bytes +4,421 krinkle@deploy1002: Synchronized wmf-config/mc.php: I390b4726d01037107 (duration: 00m 58s)

10 April 2021

  • curprev 14:2114:21, 10 April 2021imported>Stashbot 1,382,877 bytes +815 andrew@deploy1002: Finished deploy [horizon/deploy@ee1be56]: fix for T279699 (duration: 04m 12s)

9 April 2021

  • curprev 14:0714:07, 9 April 2021imported>Stashbot 1,382,062 bytes +59 jynus: retry es4 backup dump on eqiad (backup1002)
  • curprev 01:2501:25, 9 April 2021imported>Stashbot 1,382,003 bytes +23,659 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on moss-be2002.codfw.wmnet with reason: REIMAGE

7 April 2021

  • curprev 23:3823:38, 7 April 2021imported>Stashbot 1,358,344 bytes +22,437 ejegg: updated payments-wiki from b06009c099 to 70f5163816,
  • curprev 01:4601:46, 7 April 2021imported>Stashbot 1,335,907 bytes +16,883 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: REIMAGE

5 April 2021

  • curprev 23:1723:17, 5 April 2021imported>Stashbot 1,319,024 bytes +12,454 AaronSchulz: Running importMissingLocalNames.php on mwmaint1002 in a screen

4 April 2021

  • curprev 14:4714:47, 4 April 2021imported>Stashbot 1,306,570 bytes +265 andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 01m 36s)

3 April 2021

  • curprev 19:2019:20, 3 April 2021imported>Stashbot 1,306,305 bytes +1,408 andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 02m 11s)

2 April 2021

  • curprev 22:3122:31, 2 April 2021imported>Stashbot 1,304,897 bytes +10,597 bstorm@cumin1001: END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0)

1 April 2021

  • curprev 23:3223:32, 1 April 2021imported>Stashbot 1,294,300 bytes +18,843 thcipriani@deploy1002: Synchronized php-1.36.0-wmf.37/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: Backport: Revert "Turn on glent m1 AB test" T262612 (duration: 00m 58s)
  • curprev 00:5600:56, 1 April 2021imported>Stashbot 1,275,457 bytes +7,735 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2381.codfw.wmnet with reason: REIMAGE

30 March 2021

  • curprev 23:5923:59, 30 March 2021imported>Stashbot 1,267,722 bytes +13,942 Trey314159: reindexing English wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T274200)

29 March 2021

  • curprev 19:0619:06, 29 March 2021imported>Stashbot 1,253,780 bytes +3,056 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=aqs1004.eqiad.wmnet

27 March 2021

26 March 2021

25 March 2021

  • curprev 23:4723:47, 25 March 2021imported>Stashbot 1,243,504 bytes +12,592 thcipriani@deploy1002: Synchronized php-1.36.0-wmf.36/extensions/3D/package.json: No-op demo sync (duration: 01m 07s)
  • curprev 01:3301:33, 25 March 2021imported>Stashbot 1,230,912 bytes +22,060 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)

24 March 2021

  • curprev 00:2200:22, 24 March 2021imported>Stashbot 1,208,852 bytes +46,207 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2378.codfw.wmnet with reason: REIMAGE

23 March 2021

  • curprev 00:0800:08, 23 March 2021imported>Stashbot 1,162,645 bytes +17,284 tstarling@deploy1002: Synchronized wmf-config: use RequestTimeout library step 3: clean up (duration: 00m 58s)

21 March 2021

  • curprev 10:2510:25, 21 March 2021imported>Stashbot 1,145,361 bytes +368 _joe_: restarting gerrit on gerrit1001, using 45G of reserved memory

20 March 2021

  • curprev 00:2200:22, 20 March 2021imported>Stashbot 1,144,993 bytes +6,348 tzatziki: altering emails for STei (WMF) and SGrabarczuk (WMF)

19 March 2021

  • curprev 00:5100:51, 19 March 2021imported>Stashbot 1,138,645 bytes +9,921 dancy@deploy1002: Synchronized php-1.36.0-wmf.35/extensions/LiquidThreads/classes/Thread.php: T277772 (duration: 00m 58s)

18 March 2021

  • curprev 00:0500:05, 18 March 2021imported>Stashbot 1,128,724 bytes +22,211 eileen: tools revision changed from b7b4060c30 to ef54260b0d

16 March 2021

  • curprev 23:5623:56, 16 March 2021imported>Stashbot 1,106,513 bytes +44,018 krinkle@deploy1002: Synchronized php-1.36.0-wmf.35/includes/Revision/: I8619ab9e92b, T277362, T275531 (duration: 00m 58s)

15 March 2021

  • curprev 23:3123:31, 15 March 2021imported>Stashbot 1,062,495 bytes +21,243 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Remove back-compat from when IRC feed servers was a string (T224579) (duration: 00m 59s)

14 March 2021

  • curprev 17:5717:57, 14 March 2021imported>Stashbot 1,041,252 bytes +1,114 marostegui@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Repool db1146:3314', diff saved to https://phabricator.wikimedia.org/P14827 and previous config saved to /var/cache/conftool/dbconfig/20210314-175751-root.json

13 March 2021

  • curprev 19:0219:02, 13 March 2021imported>Stashbot 1,040,138 bytes +699 Amir1: change default charset of all core tables in labstestwiki to binary (T269348)
  • curprev 00:5500:55, 13 March 2021imported>Stashbot 1,039,439 bytes +10,754 mutante: [wdqs1009:/etc/envoy] $ sudo /usr/local/sbin/build-envoy-config -c /etc/envoy/

12 March 2021

  • curprev 01:3001:30, 12 March 2021imported>Stashbot 1,028,685 bytes +31,013 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2215.codfw.wmnet

11 March 2021

  • curprev 00:5000:50, 11 March 2021imported>Stashbot 997,672 bytes +37,248 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2219.codfw.wmnet

10 March 2021

  • curprev 01:0801:08, 10 March 2021imported>Stashbot 960,424 bytes +27,385 krinkle@deploy1002: Synchronized php-1.36.0-wmf.34/extensions/NavigationTiming/modules/ext.navigationTiming.js: T276826 Ibd9ddf14d64 (duration: 01m 14s)

9 March 2021

  • curprev 00:5800:58, 9 March 2021imported>Stashbot 933,039 bytes +16,107 Krinkle: krinkle@mwmaint1002 Ran invalidateUserSesssions.php for one user

7 March 2021

  • curprev 08:0108:01, 7 March 2021imported>Stashbot 916,932 bytes +170 elukey: "megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll" on analytics1066 - BBU looks fine, but the raid controller was using WriteThrough

5 March 2021

  • curprev 23:1623:16, 5 March 2021imported>Stashbot 916,762 bytes +8,775 legoktm: imported pygments 2.8.0+dfsg-1 to apt.wm.o buster-wikimedia component/pygments (T276298)
  • curprev 00:5900:59, 5 March 2021imported>Stashbot 907,987 bytes +16,215 legoktm: depooled registry1001/registry1002 (old stretch VMs) - T272550

4 March 2021

3 March 2021

  • curprev 00:4200:42, 3 March 2021imported>Stashbot 861,710 bytes +24,132 Urbanecm: Finished deployment in Evening B&C window; logmsgbot is currently down, and a simple restart did not bring it back up

2 March 2021

  • curprev 00:5900:59, 2 March 2021imported>Stashbot 837,578 bytes +21,612 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 0f08e8bbe1f74220a7a01a7606b67e0f75734a53: Update the Persian Wikipedia logos (T261033; 2/2) (duration: 00m 56s)

28 February 2021

27 February 2021

  • curprev 21:1921:19, 27 February 2021imported>Stashbot 815,895 bytes +260 dwisehaupt: ran the following on frdb2002 to allow replication to continue after conversion to utf8mb4 charset: set global slave_type_conversions = ALL_NON_LOSSY;
  • curprev 00:0800:08, 27 February 2021imported>Stashbot 815,635 bytes +26,358 mutante: deploy1002 - rsyncing home dirs from deploy1001

26 February 2021

  • curprev 00:1400:14, 26 February 2021imported>Stashbot 789,277 bytes +25,251 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.32/extensions/Graph/: 9d5cf348f5dda32f8889d5160bb1fe34a4e07f8c: Do not log graph errors to WMF servers (T274557) (duration: 01m 36s)

25 February 2021

  • curprev 00:2900:29, 25 February 2021imported>Stashbot 764,026 bytes +22,501 ryankemper: T274204 Restored service health on `elastic106[0,4,5]` via `sudo apt-get remove --purge wmf-elasticsearch-search-plugins --yes && sudo dpkg -i /var/cache/apt/archives/wmf-elasticsearch-search-plugins_6.5.4-4~stretch_all.deb && sudo puppet agent -tv`. There's some sort of issue with `6.5.4-5~stretch` that we will need to circle back and investigate; for now the fleet is staying on `6.5.4-4~stretch`

24 February 2021

  • curprev 00:5800:58, 24 February 2021imported>Stashbot 741,525 bytes +17,976 volker-e@deploy1001: Finished deploy [design/style-guide@a66b5b6]: Deploy design/style-guide: a66b5b6 “Components”: Add “Dialogs” (#430) (duration: 00m 06s)

23 February 2021

  • curprev 00:4600:46, 23 February 2021imported>Stashbot 723,549 bytes +23,528 eileen: civicrm revision changed from c535ac603a to 5e042e6e57, config revision is ef64f705bb

21 February 2021

  • curprev 16:0316:03, 21 February 2021imported>Stashbot 700,021 bytes +990 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1162 - crashed', diff saved to https://phabricator.wikimedia.org/P14424 and previous config saved to /var/cache/conftool/dbconfig/20210221-160258-marostegui.json

20 February 2021

  • curprev 00:1700:17, 20 February 2021imported>Stashbot 699,031 bytes +13,319 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1317.eqiad.wmnet
(newest | oldest) View (newer 500 | ) (20 | 50 | 100 | 250 | 500)