You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

19 May 2022

  • curprev 00:5800:58, 19 May 2022imported>Stashbot 538,445 bytes +60,753 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance

18 May 2022

  • curprev 01:0501:05, 18 May 2022imported>Stashbot 477,692 bytes +34,747 ejegg: updated fundraising CiviCRM from d45afdfc to b8b8c177

16 May 2022

  • curprev 22:1422:14, 16 May 2022imported>Stashbot 442,945 bytes +16,328 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx2001.wikimedia.org with reason: exim debugging

15 May 2022

  • curprev 21:4721:47, 15 May 2022imported>Stashbot 426,617 bytes +1,183 aqu@deploy1002: Finished deploy [airflow-dags/analytics_test@378e7ca]: (no justification provided) (duration: 00m 07s)

14 May 2022

  • curprev 08:3408:34, 14 May 2022imported>Stashbot 425,434 bytes +205 jynus@cumin1001: dbctl commit (dc=all): 'Depool db1172', diff saved to https://phabricator.wikimedia.org/P27830 and previous config saved to /var/cache/conftool/dbconfig/20220514-083421-jynus.json
  • curprev 00:5300:53, 14 May 2022imported>Stashbot 425,229 bytes +4,537 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on an-tool1005.eqiad.wmnet with reason: Server need to be downgraded to stretch, on monday

12 May 2022

  • curprev 21:5621:56, 12 May 2022imported>Stashbot 420,692 bytes +26,145 razzi@deploy1002: Finished deploy [analytics/turnilo/deploy@a2bdc3e]: (no justification provided) (duration: 02m 08s)

11 May 2022

  • curprev 22:2822:28, 11 May 2022imported>Stashbot 394,547 bytes +16,527 robh: cp305[67] returned to service and all green in icinga, cp305[89] depooling for firmware update T243167
  • curprev 01:4101:41, 11 May 2022imported>Stashbot 378,020 bytes +25,757 mutante: gitlab2001 - starting backup-restore service that had failed on previous automatic run

9 May 2022

  • curprev 21:5821:58, 9 May 2022imported>Stashbot 352,263 bytes +25,329 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx2001.wikimedia.org with reason: new kernel round deux

8 May 2022

  • curprev 07:1607:16, 8 May 2022imported>Stashbot 326,934 bytes +81 godog: silence probedown for thumbor:8800 until monday

7 May 2022

  • curprev 21:2921:29, 7 May 2022imported>Stashbot 326,853 bytes +2,312 andrew@deploy1002: Finished deploy [horizon/deploy@9d02cd6]: seeking consistency between codfw1dev and eqiad1 (duration: 04m 04s)

6 May 2022

  • curprev 19:1619:16, 6 May 2022imported>Stashbot 324,541 bytes +16,729 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-airflow1002.eqiad.wmnet
  • curprev 00:4600:46, 6 May 2022imported>Stashbot 307,812 bytes +83,868 rook@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudvirt1016.eqiad.wmnet

5 May 2022

  • curprev 01:4201:42, 5 May 2022imported>Stashbot 223,944 bytes +94,859 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P27586 and previous config saved to /var/cache/conftool/dbconfig/20220505-014205-ladsgroup.json

4 May 2022

  • curprev 00:5000:50, 4 May 2022imported>Stashbot 129,085 bytes +36,153 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1100.eqiad.wmnet with reason: Maintenance

2 May 2022

  • curprev 23:1523:15, 2 May 2022imported>Stashbot 92,932 bytes +77,432 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host krb2002.codfw.wmnet with OS bullseye
  • curprev 00:5900:59, 2 May 2022imported>Stashbot 15,500 bytes +15,385 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P27203 and previous config saved to /var/cache/conftool/dbconfig/20220502-005940-ladsgroup.json

1 May 2022

29 April 2022

  • curprev 23:1123:11, 29 April 2022imported>Stashbot 1,095,940 bytes +87,748 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T306560)', diff saved to https://phabricator.wikimedia.org/P27163 and previous config saved to /var/cache/conftool/dbconfig/20220429-231136-ladsgroup.json
  • curprev 00:5700:57, 29 April 2022imported>Stashbot 1,008,192 bytes +87,341 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P26967 and previous config saved to /var/cache/conftool/dbconfig/20220429-005702-ladsgroup.json

28 April 2022

  • curprev 01:4701:47, 28 April 2022imported>Stashbot 920,851 bytes +89,525 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1122', diff saved to https://phabricator.wikimedia.org/P26857 and previous config saved to /var/cache/conftool/dbconfig/20220428-014723-ladsgroup.json

27 April 2022

  • curprev 01:4301:43, 27 April 2022imported>Stashbot 831,326 bytes +86,215 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26663 and previous config saved to /var/cache/conftool/dbconfig/20220427-014355-ladsgroup.json

25 April 2022

  • curprev 23:0523:05, 25 April 2022imported>Stashbot 745,111 bytes +49,942 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 00:5400:54, 25 April 2022imported>Stashbot 695,169 bytes +25,636 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P26408 and previous config saved to /var/cache/conftool/dbconfig/20220425-005432-ladsgroup.json

24 April 2022

  • curprev 01:2801:28, 24 April 2022imported>Stashbot 669,533 bytes +28,655 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 12 hosts with reason: Maintenance

23 April 2022

  • curprev 01:3401:34, 23 April 2022imported>Stashbot 640,878 bytes +45,596 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T306560)', diff saved to https://phabricator.wikimedia.org/P26246 and previous config saved to /var/cache/conftool/dbconfig/20220423-013450-ladsgroup.json

22 April 2022

  • curprev 01:4701:47, 22 April 2022imported>Stashbot 595,282 bytes −356,134 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance

21 April 2022

  • curprev 00:5200:52, 21 April 2022imported>Stashbot 951,416 bytes +154,237 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P25837 and previous config saved to /var/cache/conftool/dbconfig/20220421-005225-ladsgroup.json

20 April 2022

  • curprev 01:3101:31, 20 April 2022imported>Stashbot 797,179 bytes +136,676 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

19 April 2022

  • curprev 00:5300:53, 19 April 2022imported>Stashbot 660,503 bytes +82,833 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1138 (T298565)', diff saved to https://phabricator.wikimedia.org/P25214 and previous config saved to /var/cache/conftool/dbconfig/20220419-005334-ladsgroup.json

18 April 2022

  • curprev 01:4001:40, 18 April 2022imported>Stashbot 577,670 bytes +74,955 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P24982 and previous config saved to /var/cache/conftool/dbconfig/20220418-014003-ladsgroup.json

17 April 2022

  • curprev 00:5100:51, 17 April 2022imported>Stashbot 502,715 bytes +28,006 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T298565)', diff saved to https://phabricator.wikimedia.org/P24761 and previous config saved to /var/cache/conftool/dbconfig/20220417-005150-ladsgroup.json

16 April 2022

  • curprev 00:3500:35, 16 April 2022imported>Stashbot 474,709 bytes +12,679 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P24681 and previous config saved to /var/cache/conftool/dbconfig/20220416-003538-ladsgroup.json

14 April 2022

  • curprev 22:2822:28, 14 April 2022imported>Stashbot 462,030 bytes +16,537 mutante: gitlab - deleting runner-1018, runner-1019, creating runner-1029, runner-1030 T297659
  • curprev 00:3700:37, 14 April 2022imported>Stashbot 445,493 bytes +50,258 ladsgroup@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance

13 April 2022

  • curprev 01:4201:42, 13 April 2022imported>Stashbot 395,235 bytes +39,686 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P24545 and previous config saved to /var/cache/conftool/dbconfig/20220413-014214-ladsgroup.json

12 April 2022

  • curprev 00:4900:49, 12 April 2022imported>Stashbot 355,549 bytes +52,260 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P24477 and previous config saved to /var/cache/conftool/dbconfig/20220412-004933-ladsgroup.json

11 April 2022

  • curprev 01:4301:43, 11 April 2022imported>Stashbot 303,289 bytes +5,989 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T298565)', diff saved to https://phabricator.wikimedia.org/P24355 and previous config saved to /var/cache/conftool/dbconfig/20220411-014316-ladsgroup.json

9 April 2022

  • curprev 12:3912:39, 9 April 2022imported>Stashbot 297,300 bytes +1,710 godog: bounce prometheus@ops on prometheus5001
  • curprev 00:5300:53, 9 April 2022imported>Stashbot 295,590 bytes +31,885 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1106 (T298565)', diff saved to https://phabricator.wikimedia.org/P24333 and previous config saved to /var/cache/conftool/dbconfig/20220409-005351-ladsgroup.json

7 April 2022

  • curprev 22:1822:18, 7 April 2022imported>Stashbot 263,705 bytes +77,590 ejegg: restarted fundraising scheduled jobs
  • curprev 00:5800:58, 7 April 2022imported>Stashbot 186,115 bytes +64,418 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T297189)', diff saved to https://phabricator.wikimedia.org/P24195 and previous config saved to /var/cache/conftool/dbconfig/20220407-005817-marostegui.json

6 April 2022

  • curprev 01:3401:34, 6 April 2022imported>Stashbot 121,697 bytes +61,769 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P24142 and previous config saved to /var/cache/conftool/dbconfig/20220406-013420-ladsgroup.json

5 April 2022

  • curprev 00:5800:58, 5 April 2022imported>Stashbot 59,928 bytes +52,310 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4034.ulsfo.wmnet

2 April 2022

  • curprev 11:2611:26, 2 April 2022imported>Stashbot 7,618 bytes +272 akosiaris: disable zotero paging until T291707 is resolved.

1 April 2022

  • curprev 23:2523:25, 1 April 2022imported>Stashbot 7,346 bytes −1,236,074 mutante: DNS - new project language 'kcg'. 'Tyap is a regionally important dialect cluster of Plateau languages in Nigeria's Middle Belt, named after its prestige dialect. It is also known by its Hausa exonym as Katab or Kataf.' T305279

31 March 2022

  • curprev 23:4523:45, 31 March 2022imported>Stashbot 1,243,420 bytes +66,130 mutante: gitlab2001 - fdisk /dev/vdb (g, w) (create partition table), (n, w) (create partition) ; mkfs.ext4 /dev/vdb1 (create filesystem); systemctl reset-failed (fix Icinga alert); mkdir /mnt/gitlab-backup; mount /dev/vdb1 /mnt/gitlab-backup ; blkid (get UUID); edit /etc/fstab and insert "UUID=c5235682-ac21-46a9-85ee-9603f694a6a4 /mnt/gitlab-backup ext4 errors=remount-ro 0 2" T274463
  • curprev 01:4401:44, 31 March 2022imported>Stashbot 1,177,290 bytes +118,543 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P23948 and previous config saved to /var/cache/conftool/dbconfig/20220331-014403-ladsgroup.json

30 March 2022

  • curprev 01:4601:46, 30 March 2022imported>Stashbot 1,058,747 bytes +98,930 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23664 and previous config saved to /var/cache/conftool/dbconfig/20220330-014621-ladsgroup.json

28 March 2022

  • curprev 23:1523:15, 28 March 2022imported>Stashbot 959,817 bytes +54,742 eileen: civicrm revision 15d22bd1 -> 1c5d10e1
  • curprev 00:5500:55, 28 March 2022imported>Stashbot 905,075 bytes +30,248 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23315 and previous config saved to /var/cache/conftool/dbconfig/20220328-005533-ladsgroup.json

27 March 2022

  • curprev 00:5000:50, 27 March 2022imported>Stashbot 874,827 bytes +29,440 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P23228 and previous config saved to /var/cache/conftool/dbconfig/20220327-005010-ladsgroup.json

26 March 2022

  • curprev 01:1201:12, 26 March 2022imported>Stashbot 845,387 bytes +31,577 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P23147 and previous config saved to /var/cache/conftool/dbconfig/20220326-011216-ladsgroup.json

25 March 2022

  • curprev 00:3900:39, 25 March 2022imported>Stashbot 813,810 bytes +36,424 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase2027.codfw.wmnet with OS buster

24 March 2022

  • curprev 00:3300:33, 24 March 2022imported>Stashbot 777,386 bytes +38,343 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1046.eqiad.wmnet with OS bullseye

23 March 2022

  • curprev 01:2001:20, 23 March 2022imported>Stashbot 739,043 bytes +42,681 ejegg: updated payments-wiki from 3048f0aa to 28e24856

22 March 2022

  • curprev 01:3501:35, 22 March 2022imported>Stashbot 696,362 bytes +39,479 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1047.eqiad.wmnet with OS bullseye

20 March 2022

  • curprev 23:4423:44, 20 March 2022imported>Stashbot 656,883 bytes +3,079 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T300775)', diff saved to https://phabricator.wikimedia.org/P22857 and previous config saved to /var/cache/conftool/dbconfig/20220320-234358-marostegui.json

19 March 2022

  • curprev 17:1817:18, 19 March 2022imported>Stashbot 653,804 bytes +4,978 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T300775)', diff saved to https://phabricator.wikimedia.org/P22845 and previous config saved to /var/cache/conftool/dbconfig/20220319-171757-marostegui.json
  • curprev 01:4601:46, 19 March 2022imported>Stashbot 648,826 bytes +15,864 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1016.eqiad.wmnet with reason: host reimage

17 March 2022

  • curprev 22:5522:55, 17 March 2022imported>Stashbot 632,962 bytes +44,817 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 01:1101:11, 17 March 2022imported>Stashbot 588,145 bytes +54,021 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1016.eqiad.wmnet with OS bullseye

16 March 2022

  • curprev 00:3600:36, 16 March 2022imported>Stashbot 534,124 bytes +72,992 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6011.drmrs.wmnet with reason: host reimage

15 March 2022

  • curprev 01:3001:30, 15 March 2022imported>Stashbot 461,132 bytes +46,179 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T300775)', diff saved to https://phabricator.wikimedia.org/P22465 and previous config saved to /var/cache/conftool/dbconfig/20220315-013013-marostegui.json

11 March 2022

  • curprev 15:5615:56, 11 March 2022imported>Stashbot 414,953 bytes +11,582 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2014.codfw.wmnet with OS bullseye
  • curprev 00:3300:33, 11 March 2022imported>Stashbot 403,371 bytes +68,747 TimStarling: on mwmaint1002 running populateGlobalEditCount.php

10 March 2022

  • curprev 00:2600:26, 10 March 2022imported>Stashbot 334,624 bytes +40,477 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@7975c27]: (no justification provided) (duration: 00m 08s)

9 March 2022

  • curprev 01:3201:32, 9 March 2022imported>Stashbot 294,147 bytes +75,530 marostegui@cumin2002: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P22170 and previous config saved to /var/cache/conftool/dbconfig/20220309-013256-marostegui.json

8 March 2022

  • curprev 00:3400:34, 8 March 2022imported>Stashbot 218,617 bytes +83,815 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@c8a753b]: (no justification provided) (duration: 00m 07s)

4 March 2022

  • curprev 17:5917:59, 4 March 2022imported>Stashbot 134,802 bytes +23,275 btullis@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:3501:35, 4 March 2022imported>Stashbot 111,527 bytes +40,357 rzl@deploy1002: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply

3 March 2022

  • curprev 01:4201:42, 3 March 2022imported>Stashbot 71,170 bytes +51,552 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on datahubsearch[1001-1003].eqiad.wmnet with reason: Still having errors setting up opensearch

2 March 2022

  • curprev 00:1500:15, 2 March 2022imported>Stashbot 19,618 bytes +18,689 topranks: Re-enabling Lumen AS3356 BGP session over IPv4 on cr3-ulsfo to assess affect on currently broken routing to ulsfo.

1 March 2022

  • curprev 01:1401:14, 1 March 2022imported>Stashbot 929 bytes −955,327 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1104 (T302185)', diff saved to https://phabricator.wikimedia.org/P21614 and previous config saved to /var/cache/conftool/dbconfig/20220301-011404-ladsgroup.json

27 February 2022

25 February 2022

  • curprev 23:3223:32, 25 February 2022imported>Stashbot 956,175 bytes +19,462 dzahn@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply

24 February 2022

  • curprev 23:3523:35, 24 February 2022imported>Stashbot 936,713 bytes +51,509 ryankemper: T302526 Deployed https://gerrit.wikimedia.org/r/765652 and ran puppet across wcqs*
  • curprev 00:5900:59, 24 February 2022imported>Stashbot 885,204 bytes +60,442 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2074.codfw.wmnet with OS bullseye

23 February 2022

  • curprev 01:4101:41, 23 February 2022imported>Stashbot 824,762 bytes +60,474 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2068.codfw.wmnet with reason: host reimage

21 February 2022

  • curprev 22:3022:30, 21 February 2022imported>Stashbot 764,288 bytes +74,792 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T300381)', diff saved to https://phabricator.wikimedia.org/P21231 and previous config saved to /var/cache/conftool/dbconfig/20220221-223015-marostegui.json
  • curprev 01:3901:39, 21 February 2022imported>Stashbot 689,496 bytes +3,448 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db2152.codfw.wmnet with OS bullseye

19 February 2022

  • curprev 16:5016:50, 19 February 2022imported>Stashbot 686,048 bytes +5,104 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 00:5900:59, 19 February 2022imported>Stashbot 680,944 bytes +28,538 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2020.codfw.wmnet with OS bullseye

17 February 2022

  • curprev 22:2822:28, 17 February 2022imported>Stashbot 652,406 bytes +28,397 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:3601:36, 17 February 2022imported>Stashbot 624,009 bytes +52,457 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T300381)', diff saved to https://phabricator.wikimedia.org/P20954 and previous config saved to /var/cache/conftool/dbconfig/20220217-013607-marostegui.json

15 February 2022

  • curprev 23:4723:47, 15 February 2022imported>Stashbot 571,552 bytes +59,268 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host restbase-dev2003.mgmt.codfw.wmnet with reboot policy FORCED

14 February 2022

  • curprev 22:0422:04, 14 February 2022imported>Stashbot 512,284 bytes +52,126 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

13 February 2022

  • curprev 23:1723:17, 13 February 2022imported>Stashbot 460,158 bytes +3,305 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T300775)', diff saved to https://phabricator.wikimedia.org/P20627 and previous config saved to /var/cache/conftool/dbconfig/20220213-231742-marostegui.json

12 February 2022

  • curprev 22:5822:58, 12 February 2022imported>Stashbot 456,853 bytes +2,897 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T300775)', diff saved to https://phabricator.wikimedia.org/P20617 and previous config saved to /var/cache/conftool/dbconfig/20220212-225806-marostegui.json

11 February 2022

10 February 2022

  • curprev 00:4200:42, 10 February 2022imported>Stashbot 362,781 bytes +25,944 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

8 February 2022

  • curprev 23:5223:52, 8 February 2022imported>Stashbot 336,837 bytes +73,681 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2055.codfw.wmnet with OS buster
  • curprev 00:1200:12, 8 February 2022imported>Stashbot 263,156 bytes +33,177 ryankemper: T294805 Re-enabling puppet across eqiad elastic fleet: `ryankemper@cumin1001:~$ sudo cumin -b 8 'elastic1*' 'sudo enable-puppet "Add new eqiad replacement hosts elastic10[68-83] - T294805 - root" && sudo run-puppet-agent'` tmux session `elastic`

5 February 2022

  • curprev 22:1022:10, 5 February 2022imported>Stashbot 229,979 bytes +1,284 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2003-dev.codfw.wmnet with OS bullseye

4 February 2022

  • curprev 23:4323:43, 4 February 2022imported>Stashbot 228,695 bytes +5,568 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mirror1001.wikimedia.org with reason: new kernel
  • curprev 01:0801:08, 4 February 2022imported>Stashbot 223,127 bytes +72,959 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

3 February 2022

2 February 2022

  • curprev 00:5300:53, 2 February 2022imported>Stashbot 90,142 bytes −738,181 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

1 February 2022

  • curprev 00:3100:31, 1 February 2022imported>Stashbot 828,323 bytes +72,250 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

29 January 2022

  • curprev 21:0821:08, 29 January 2022imported>Stashbot 756,073 bytes +1,014 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices2003-dev.wikimedia.org with OS bullseye
  • curprev 00:1400:14, 29 January 2022imported>Stashbot 755,059 bytes +14,112 ebernhardson: restart elasticsearch_6@production-search-psi-eqiad on elastic1049 to address CirrusSearchJVMGCOldPoolFlatlined alert

28 January 2022

  • curprev 01:4701:47, 28 January 2022imported>Stashbot 740,947 bytes +73,300 andrew@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcontrol2001-dev.wikimedia.org with OS bullseye

27 January 2022

26 January 2022

25 January 2022

  • curprev 00:3100:31, 25 January 2022imported>Stashbot 525,956 bytes +53,597 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

23 January 2022

  • curprev 22:0222:02, 23 January 2022imported>Stashbot 472,359 bytes +500 ebysans@deploy1002: Finished deploy [airflow-dags/analytics-test@37937f6]: (no justification provided) (duration: 00m 08s)

22 January 2022

  • curprev 22:3822:38, 22 January 2022imported>Stashbot 471,859 bytes +812 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx1001.wikimedia.org with reason: kernel testing
  • curprev 01:3001:30, 22 January 2022imported>Stashbot 471,047 bytes +18,324 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on mx1001.wikimedia.org with reason: kernel testing

20 January 2022

  • curprev 22:4022:40, 20 January 2022imported>Stashbot 452,723 bytes +39,372 inflatador: running puppet-merge for https://gerrit.wikimedia.org/r/755810

19 January 2022

17 January 2022

  • curprev 23:2723:27, 17 January 2022imported>Stashbot 327,176 bytes +12,624 jynus: forced session revocation on phab for a user T299315

16 January 2022

  • curprev 08:2108:21, 16 January 2022imported>Stashbot 314,552 bytes +684 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: sync on production

15 January 2022

  • curprev 08:5508:55, 15 January 2022imported>Stashbot 313,868 bytes +1,296 legoktm: finished running recountCategories on s4 wikis (T299244)
  • curprev 01:2201:22, 15 January 2022imported>Stashbot 312,572 bytes +10,517 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn

14 January 2022

  • curprev 00:3600:36, 14 January 2022imported>Stashbot 302,055 bytes +32,093 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

13 January 2022

  • curprev 00:3500:35, 13 January 2022imported>Stashbot 269,962 bytes +64,421 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

12 January 2022

  • curprev 00:5500:55, 12 January 2022imported>Stashbot 205,541 bytes +59,425 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

11 January 2022

8 January 2022

  • curprev 10:5110:51, 8 January 2022imported>Stashbot 107,107 bytes +180 elukey: restart hive daemons on an-coord1002 (after my last upgrade/rollback of packages the prometheus agent settings were not picked up, so no metrics)

7 January 2022

6 January 2022

5 January 2022

  • curprev 00:5900:59, 5 January 2022imported>Stashbot 64,897 bytes +32,691 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

4 January 2022

  • curprev 00:5400:54, 4 January 2022imported>Stashbot 32,206 bytes +32,091 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P18329 and previous config saved to /var/cache/conftool/dbconfig/20220104-005456-marostegui.json

1 January 2022

29 December 2021

  • curprev 10:3010:30, 29 December 2021imported>Stashbot 664,919 bytes +126 elukey: kill tcpdump process on kubestagemaster1001 (kept a big pcap file opened that kept growing)

28 December 2021

24 December 2021

  • curprev 20:0820:08, 24 December 2021imported>Stashbot 663,863 bytes +325 mforns@deploy1002: Finished deploy [airflow-dags/analytics@e282d2d]: (no justification provided) (duration: 00m 06s)
  • curprev 00:5700:57, 24 December 2021imported>Stashbot 663,538 bytes +3,353 ejegg: updated fundraising CiviCRM from 47dd67f2 to aaceb4ab

23 December 2021

  • curprev 00:0400:04, 23 December 2021imported>Stashbot 660,185 bytes +4,302 bking@cumin1001: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T297986

21 December 2021

19 December 2021

18 December 2021

  • curprev 13:5713:57, 18 December 2021imported>Stashbot 633,583 bytes +93 dcausse: restarting blazegraph on wdqs1013 (jvm stuck for 10hours)

17 December 2021

16 December 2021

  • curprev 00:3700:37, 16 December 2021imported>Stashbot 600,611 bytes +14,349 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

15 December 2021

14 December 2021

  • curprev 01:4201:42, 14 December 2021imported>Stashbot 545,410 bytes +41,246 ryankemper: T297468 `sudo cookbook sre.elasticsearch.rolling-operation search_eqiad "eqiad rolling restart" --nodes-per-run 3 --start-datetime 2021-12-14T01:27:58 --task-id T297468` on `ryankemper@cumin1001` tmux `elastic_restarts`

12 December 2021

  • curprev 14:3514:35, 12 December 2021imported>Stashbot 504,164 bytes +844 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host graphite1004.eqiad.wmnet

11 December 2021

  • curprev 19:0419:04, 11 December 2021imported>Stashbot 503,320 bytes +131 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster
  • curprev 00:0400:04, 11 December 2021imported>Stashbot 503,189 bytes +18,770 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

10 December 2021

9 December 2021

  • curprev 00:2600:26, 9 December 2021imported>Stashbot 469,358 bytes +7,967 rzl: graphite1004.mgmt: /admin1-> racadm serveraction powercycle (T297265)

8 December 2021

  • curprev 00:5100:51, 8 December 2021imported>Stashbot 461,391 bytes +29,464 ebernhardson@deploy1002: Synchronized php-1.38.0-wmf.12/extensions/GrowthExperiments/includes/NewcomerTasks/AddImage/AddImageSubmissionHandler.php: backport window for 744896 (duration: 01m 05s)

7 December 2021

4 December 2021

  • curprev 01:1401:14, 4 December 2021imported>Stashbot 424,523 bytes +12,137 mutante: mx2001 - did not come back from reboot, did not get IP on interface, could not start ferm, logged in via console with root password, in /etc/network/interfaces replaced all "ens5" with "ens13", rebooted again, selected previous kernel version

3 December 2021

2 December 2021

  • curprev 01:2101:21, 2 December 2021imported>Stashbot 394,618 bytes +24,122 ryankemper: T280001 Rolling restart of low-traffic pybal hosts complete. All of `wcqs` is pooled and the pybal / ipvs related alerts have cleared

1 December 2021

  • curprev 00:3500:35, 1 December 2021imported>Stashbot 370,496 bytes +9,705 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

30 November 2021

  • curprev 00:2200:22, 30 November 2021imported>Stashbot 360,791 bytes +17,004 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

28 November 2021

  • curprev 17:1417:14, 28 November 2021imported>Stashbot 343,787 bytes +293 elukey@deploy1002: Finished deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 (duration: 02m 11s)

27 November 2021

  • curprev 19:5519:55, 27 November 2021imported>Stashbot 343,494 bytes +1,043 andrew@deploy1002: Finished deploy [horizon/deploy@6115b3b]: network UI updates for T296548 (duration: 04m 14s)

26 November 2021

  • curprev 16:1116:11, 26 November 2021imported>Stashbot 342,451 bytes +4,657 arnoldokoth: drain kubestage1002 node in prep for decommissioning

25 November 2021

  • curprev 20:4420:44, 25 November 2021imported>Stashbot 337,794 bytes +25,467 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17872 and previous config saved to /var/cache/conftool/dbconfig/20211125-204357-ladsgroup.json
  • curprev 00:3700:37, 25 November 2021imported>Stashbot 312,327 bytes +56,025 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2066.codfw.wmnet with OS buster

24 November 2021

  • curprev 00:2500:25, 24 November 2021imported>Stashbot 256,302 bytes +24,082 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS buster

23 November 2021

  • curprev 00:5100:51, 23 November 2021imported>Stashbot 232,220 bytes +27,289 urbanecm@deploy1002: Started scap: 69aa4a7: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 4/4)

21 November 2021

20 November 2021

  • curprev 01:0201:02, 20 November 2021imported>Stashbot 204,605 bytes +91 mutante: lists1001 - restarted apache, icinga alerts for the web UI, but recovered
  • curprev 00:2700:27, 20 November 2021imported>Stashbot 204,514 bytes +9,398 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)

19 November 2021

  • curprev 01:4501:45, 19 November 2021imported>Stashbot 195,116 bytes +21,872 mutante: I think git-ssh6_22 is down (see alerts lvs2008/2009) due to the v6 issue from ongoing lvs maintenance. depooled in conftool

18 November 2021

  • curprev 01:4701:47, 18 November 2021imported>Stashbot 173,244 bytes +19,375 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2005.codfw.wmnet

17 November 2021

  • curprev 00:1900:19, 17 November 2021imported>Stashbot 153,869 bytes +12,582 ryankemper: T276198 `ryankemper@cumin1001:~$ sudo cumin -b 3 '*elastic*' 'sudo run-puppet-agent --force'` Change looks good (no complaints from systemd), rolling out to rest of fleet / reenabling puppet

16 November 2021

14 November 2021

  • curprev 11:4811:48, 14 November 2021imported>Stashbot 125,456 bytes +111 paravoid: disable cr1-eqiad:xe-3/0/6 (IXP port) to mitigate T295650

13 November 2021

  • curprev 18:4318:43, 13 November 2021imported>Stashbot 125,345 bytes +836 AndyRussG: Enabled debug logging for PayPal IPN listener (updated SmashPig config a9e30591 -> 9567cc4a on frpig1001)

12 November 2021

  • curprev 21:0021:00, 12 November 2021imported>Stashbot 124,509 bytes +5,102 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:1900:19, 12 November 2021imported>Stashbot 119,407 bytes +11,807 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

11 November 2021

  • curprev 00:2200:22, 11 November 2021imported>Stashbot 107,600 bytes +12,063 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

10 November 2021

  • curprev 01:0701:07, 10 November 2021imported>Stashbot 95,537 bytes +10,338 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

8 November 2021

  • curprev 23:3923:39, 8 November 2021imported>Stashbot 85,199 bytes +11,520 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

6 November 2021

5 November 2021

  • curprev 00:1600:16, 5 November 2021imported>Stashbot 66,667 bytes +20,851 mutante: phab1001 - sudo systemctl start phabricator_clean_tmp_files.service because Icinga alerted it had failed... worked fine

4 November 2021

2 November 2021

  • curprev 23:4723:47, 2 November 2021imported>Stashbot 22,959 bytes +11,789 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:2101:21, 2 November 2021imported>Stashbot 11,170 bytes −844,596 ejegg: updated SmashPig standalone deploy from dd3a81c7c2 to be68299b92

31 October 2021

  • curprev 21:4921:49, 31 October 2021imported>Stashbot 855,766 bytes +371 urbanecm: urbanecm@mwmaint1002:~$ mwscript userOptions.php --wiki=dewiki --nowarn --touserid 3802752 --old 'linkrecommendation' --new 'control' 'growthexperiments-homepage-variant' # T294712

30 October 2021

29 October 2021

  • curprev 22:5722:57, 29 October 2021imported>Stashbot 855,227 bytes +3,905 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=foundationwiki --userlist users.txt # T205347, users.txt is at P17641

28 October 2021

  • curprev 23:5023:50, 28 October 2021imported>Stashbot 851,322 bytes +21,486 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4033.ulsfo.wmnet with OS buster

27 October 2021

  • curprev 23:5523:55, 27 October 2021imported>Stashbot 829,836 bytes +10,201 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

26 October 2021

  • curprev 22:5922:59, 26 October 2021imported>Stashbot 819,635 bytes +12,871 legoktm: uploaded python-logstash to buster-wikimedia for T294393

25 October 2021

  • curprev 23:1223:12, 25 October 2021imported>Stashbot 806,764 bytes +11,982 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Create alias for Appendix and Appendix_talk namespaces on mywiktionary (T291146) (duration: 00m 55s)

23 October 2021

  • curprev 16:4016:40, 23 October 2021imported>Stashbot 794,782 bytes +253 dcausse: restarting blazegraph on wdqs1004 and wdqs1006 (free allocators alert)

22 October 2021

  • curprev 23:1723:17, 22 October 2021imported>Stashbot 794,529 bytes +3,761 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

21 October 2021

  • curprev 23:4023:40, 21 October 2021imported>Stashbot 790,768 bytes +17,384 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:0600:06, 21 October 2021imported>Stashbot 773,384 bytes +14,742 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

20 October 2021

  • curprev 01:3101:31, 20 October 2021imported>Stashbot 758,642 bytes +27,590 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

19 October 2021

  • curprev 00:3800:38, 19 October 2021imported>Stashbot 731,052 bytes +11,211 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer

16 October 2021

  • curprev 03:5603:56, 16 October 2021imported>Stashbot 719,841 bytes +266 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

15 October 2021

  • curprev 23:4823:48, 15 October 2021imported>Stashbot 719,575 bytes +5,052 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
  • curprev 00:0900:09, 15 October 2021imported>Stashbot 714,523 bytes +19,109 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

14 October 2021

  • curprev 01:4101:41, 14 October 2021imported>Stashbot 695,414 bytes +17,092 ejegg: updated payments-wiki from b329d2dea2 to 19d18c1852

13 October 2021

  • curprev 00:3800:38, 13 October 2021imported>Stashbot 678,322 bytes +15,911 ejegg: updated payments-wiki from 030b11da1a to b329d2dea2

12 October 2021

  • curprev 00:1100:11, 12 October 2021imported>Stashbot 662,411 bytes +3,728 eileen: civicrm revision changed from 598b59b0ee to 96090e4bd2, config revision is 85277466ed

9 October 2021

  • curprev 05:0105:01, 9 October 2021imported>Stashbot 658,683 bytes +224 jiji@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:3201:32, 9 October 2021imported>Stashbot 658,459 bytes +8,808 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

8 October 2021

7 October 2021

  • curprev 00:1100:11, 7 October 2021imported>Stashbot 637,764 bytes +11,864 mutante: [grafana2001:~] $ sudo systemctl start rsync-var-lib-grafana because of "PROBLEM - Check systemd state on grafana2001 is CRITICAL: CRITICAL - degraded" because of some race condition where a file vanished during sync

6 October 2021

  • curprev 01:3901:39, 6 October 2021imported>Stashbot 625,900 bytes +10,381 legoktm: legoktm@mwmaint1002:~$ echo "https://en.wikiversity.org/static/images/mobile/copyright/wikiversity.svg" |mwscript purgeList.php

4 October 2021

  • curprev 23:3023:30, 4 October 2021imported>Stashbot 615,519 bytes +9,952 foks: resetting some emails used for abuse by a globally-banned user

3 October 2021

2 October 2021

  • curprev 17:2817:28, 2 October 2021imported>Stashbot 605,040 bytes +230 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

1 October 2021

  • curprev 23:1923:19, 1 October 2021imported>Stashbot 604,810 bytes +38,661 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

29 September 2021

  • curprev 23:2023:20, 29 September 2021imported>Stashbot 566,149 bytes +14,880 bd808@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 00:5200:52, 29 September 2021imported>Stashbot 551,269 bytes +27,094 eileen: civicrm revision changed from a1929b3dfd to a480bf03c9, config revision is 77cb7ec866

27 September 2021

  • curprev 23:4223:42, 27 September 2021imported>Stashbot 524,175 bytes +27,476 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

26 September 2021

  • curprev 14:5114:51, 26 September 2021imported>Stashbot 496,699 bytes +432 volker-e@deploy1002: Finished deploy [design/style-guide@aac0ae9]: Deploy design/style-guide: aac0ae9 “Apps”: Fix image path (#490) (duration: 00m 06s)

25 September 2021

  • curprev 02:0002:00, 25 September 2021imported>Stashbot 496,267 bytes +374 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .

24 September 2021

  • curprev 20:0020:00, 24 September 2021imported>Stashbot 495,893 bytes +10,338 volker-e@deploy1002: Finished deploy [design/style-guide@362c6b1]: Deploy design/style-guide: 362c6b1 “Components”: Fix index link (#489) (duration: 00m 06s)
  • curprev 00:3900:39, 24 September 2021imported>Stashbot 485,555 bytes +19,157 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

23 September 2021

  • curprev 00:3500:35, 23 September 2021imported>Stashbot 466,398 bytes +12,744 catrope@deploy1002: Synchronized php-1.38.0-wmf.1/extensions/MediaSearch/: Use text() instead of parse() for MediaSearch UI messages (T291590) (duration: 01m 08s)

21 September 2021

  • curprev 23:2023:20, 21 September 2021imported>Stashbot 453,654 bytes +9,016 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:1600:16, 21 September 2021imported>Stashbot 444,638 bytes +11,548 tgr: Evening deploys done

18 September 2021

  • curprev 01:4701:47, 18 September 2021imported>Stashbot 433,090 bytes +322 ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.23/includes/libs/rdbms/database/Database.php: (no justification provided) (duration: 00m 57s)

17 September 2021

  • curprev 21:2821:28, 17 September 2021imported>Stashbot 432,768 bytes +3,000 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:0400:04, 17 September 2021imported>Stashbot 429,768 bytes +13,476 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .

15 September 2021

14 September 2021

  • curprev 23:0123:01, 14 September 2021imported>Stashbot 406,930 bytes +15,959 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Re-enable VipsScaler (2 of 2) (duration: 01m 04s)
  • curprev 00:0400:04, 14 September 2021imported>Stashbot 390,971 bytes +8,994 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

12 September 2021

11 September 2021

  • curprev 19:0219:02, 11 September 2021imported>Stashbot 381,555 bytes +650 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 27814b8eaacb5ba2fee1b6167a36ea14356a1ecf: testwiki: Fully remove securepoll-related groups (T290808) (duration: 00m 57s)

10 September 2021

  • curprev 21:2921:29, 10 September 2021imported>Stashbot 380,905 bytes +5,151 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • curprev 00:3500:35, 10 September 2021imported>Stashbot 375,754 bytes +15,347 tgr: Deployed patch for T290692

9 September 2021

  • curprev 00:5000:50, 9 September 2021imported>Stashbot 360,407 bytes +8,465 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Don't set default to Score (try #2) (duration: 00m 58s)

8 September 2021

  • curprev 00:0000:00, 8 September 2021imported>Stashbot 351,942 bytes +12,900 legoktm: legoktm@lists1001:~$ sudo rm -rf /etc/mailman # cleanup as part of 4869d91b0be / T282303

6 September 2021

  • curprev 23:5223:52, 6 September 2021imported>Stashbot 339,042 bytes +8,019 tstarling@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/SecurePoll/includes/Talliers/STVTallier.php: T290000 (duration: 00m 58s)

5 September 2021

  • curprev 18:5418:54, 5 September 2021imported>Stashbot 331,023 bytes +201 urbanecm: wikiadmin@10.192.0.119(ptwiki)> update protected_titles set pt_create_perm='editautoreviewprotected' where pt_create_perm='autoreviewer'; # T290396

4 September 2021

  • curprev 13:3513:35, 4 September 2021imported>Stashbot 330,822 bytes +1,989 marostegui@cumin1001: dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: Slowly repool T290374', diff saved to https://phabricator.wikimedia.org/P17217 and previous config saved to /var/cache/conftool/dbconfig/20210904-133532-root.json

3 September 2021

1 September 2021

  • curprev 23:5023:50, 1 September 2021imported>Stashbot 297,294 bytes +11,478 Amir1: mwscript createAndPromote.php --wiki=test2wiki --sysop --force Ladsgroup
  • curprev 00:5300:53, 1 September 2021imported>Stashbot 285,816 bytes +13,363 eileen: civicrm revision changed from e567b4c289 to 7da3eba4f9, config revision is 5f004d94d7

30 August 2021

  • curprev 23:1423:14, 30 August 2021imported>Stashbot 272,453 bytes +10,249 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

29 August 2021

  • curprev 00:1500:15, 29 August 2021imported>Stashbot 262,204 bytes +714 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

27 August 2021

  • curprev 16:4616:46, 27 August 2021imported>Stashbot 261,490 bytes +3,953 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps1005.eqiad.wmnet with reason: Resyncing from master
  • curprev 00:4600:46, 27 August 2021imported>Stashbot 257,537 bytes +9,223 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

25 August 2021

  • curprev 23:2323:23, 25 August 2021imported>Stashbot 248,314 bytes +23,982 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:5300:53, 25 August 2021imported>Stashbot 224,332 bytes +6,127 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .

24 August 2021

  • curprev 00:1700:17, 24 August 2021imported>Stashbot 218,205 bytes +10,888 ryankemper: [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'`

22 August 2021

21 August 2021

  • curprev 15:1215:12, 21 August 2021imported>Stashbot 1,011,788 bytes +262 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

20 August 2021

  • curprev 23:1823:18, 20 August 2021imported>Stashbot 1,011,526 bytes +4,817 legoktm: deployed patch for T289385
  • curprev 00:1800:18, 20 August 2021imported>Stashbot 1,006,709 bytes +10,542 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

19 August 2021

  • curprev 01:1301:13, 19 August 2021imported>Stashbot 996,167 bytes +6,256 ejegg: updated fundraising CiviCRM from 73f6ec9190 to 8ed303f2d1

18 August 2021

  • curprev 00:3900:39, 18 August 2021imported>Stashbot 989,911 bytes +13,880 dpifke@deploy1002: Finished deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243) (duration: 00m 05s)
(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)