You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

28 March 2022

  • curprev 23:1523:15, 28 March 2022imported>Stashbot 959,817 bytes +54,742 eileen: civicrm revision 15d22bd1 -> 1c5d10e1
  • curprev 00:5500:55, 28 March 2022imported>Stashbot 905,075 bytes +30,248 ladsgroup@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23315 and previous config saved to /var/cache/conftool/dbconfig/20220328-005533-ladsgroup.json

27 March 2022

  • curprev 00:5000:50, 27 March 2022imported>Stashbot 874,827 bytes +29,440 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T298565)', diff saved to https://phabricator.wikimedia.org/P23228 and previous config saved to /var/cache/conftool/dbconfig/20220327-005010-ladsgroup.json

26 March 2022

  • curprev 01:1201:12, 26 March 2022imported>Stashbot 845,387 bytes +31,577 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P23147 and previous config saved to /var/cache/conftool/dbconfig/20220326-011216-ladsgroup.json

25 March 2022

  • curprev 00:3900:39, 25 March 2022imported>Stashbot 813,810 bytes +36,424 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase2027.codfw.wmnet with OS buster

24 March 2022

  • curprev 00:3300:33, 24 March 2022imported>Stashbot 777,386 bytes +38,343 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1046.eqiad.wmnet with OS bullseye

23 March 2022

  • curprev 01:2001:20, 23 March 2022imported>Stashbot 739,043 bytes +42,681 ejegg: updated payments-wiki from 3048f0aa to 28e24856

22 March 2022

  • curprev 01:3501:35, 22 March 2022imported>Stashbot 696,362 bytes +39,479 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1047.eqiad.wmnet with OS bullseye

20 March 2022

  • curprev 23:4423:44, 20 March 2022imported>Stashbot 656,883 bytes +3,079 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T300775)', diff saved to https://phabricator.wikimedia.org/P22857 and previous config saved to /var/cache/conftool/dbconfig/20220320-234358-marostegui.json

19 March 2022

  • curprev 17:1817:18, 19 March 2022imported>Stashbot 653,804 bytes +4,978 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T300775)', diff saved to https://phabricator.wikimedia.org/P22845 and previous config saved to /var/cache/conftool/dbconfig/20220319-171757-marostegui.json
  • curprev 01:4601:46, 19 March 2022imported>Stashbot 648,826 bytes +15,864 andrew@cumin1001: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1016.eqiad.wmnet with reason: host reimage

17 March 2022

  • curprev 22:5522:55, 17 March 2022imported>Stashbot 632,962 bytes +44,817 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 01:1101:11, 17 March 2022imported>Stashbot 588,145 bytes +54,021 andrew@cumin1001: START - Cookbook sre.hosts.reimage for host cloudvirt1016.eqiad.wmnet with OS bullseye

16 March 2022

  • curprev 00:3600:36, 16 March 2022imported>Stashbot 534,124 bytes +72,992 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp6011.drmrs.wmnet with reason: host reimage

15 March 2022

  • curprev 01:3001:30, 15 March 2022imported>Stashbot 461,132 bytes +46,179 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1156 (T300775)', diff saved to https://phabricator.wikimedia.org/P22465 and previous config saved to /var/cache/conftool/dbconfig/20220315-013013-marostegui.json

11 March 2022

  • curprev 15:5615:56, 11 March 2022imported>Stashbot 414,953 bytes +11,582 elukey@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2014.codfw.wmnet with OS bullseye
  • curprev 00:3300:33, 11 March 2022imported>Stashbot 403,371 bytes +68,747 TimStarling: on mwmaint1002 running populateGlobalEditCount.php

10 March 2022

  • curprev 00:2600:26, 10 March 2022imported>Stashbot 334,624 bytes +40,477 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@7975c27]: (no justification provided) (duration: 00m 08s)

9 March 2022

  • curprev 01:3201:32, 9 March 2022imported>Stashbot 294,147 bytes +75,530 marostegui@cumin2002: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P22170 and previous config saved to /var/cache/conftool/dbconfig/20220309-013256-marostegui.json

8 March 2022

  • curprev 00:3400:34, 8 March 2022imported>Stashbot 218,617 bytes +83,815 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@c8a753b]: (no justification provided) (duration: 00m 07s)

4 March 2022

  • curprev 17:5917:59, 4 March 2022imported>Stashbot 134,802 bytes +23,275 btullis@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:3501:35, 4 March 2022imported>Stashbot 111,527 bytes +40,357 rzl@deploy1002: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply

3 March 2022

  • curprev 01:4201:42, 3 March 2022imported>Stashbot 71,170 bytes +51,552 razzi@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on datahubsearch[1001-1003].eqiad.wmnet with reason: Still having errors setting up opensearch

2 March 2022

  • curprev 00:1500:15, 2 March 2022imported>Stashbot 19,618 bytes +18,689 topranks: Re-enabling Lumen AS3356 BGP session over IPv4 on cr3-ulsfo to assess affect on currently broken routing to ulsfo.

1 March 2022

  • curprev 01:1401:14, 1 March 2022imported>Stashbot 929 bytes −955,327 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1104 (T302185)', diff saved to https://phabricator.wikimedia.org/P21614 and previous config saved to /var/cache/conftool/dbconfig/20220301-011404-ladsgroup.json

27 February 2022

25 February 2022

  • curprev 23:3223:32, 25 February 2022imported>Stashbot 956,175 bytes +19,462 dzahn@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply

24 February 2022

  • curprev 23:3523:35, 24 February 2022imported>Stashbot 936,713 bytes +51,509 ryankemper: T302526 Deployed https://gerrit.wikimedia.org/r/765652 and ran puppet across wcqs*
  • curprev 00:5900:59, 24 February 2022imported>Stashbot 885,204 bytes +60,442 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2074.codfw.wmnet with OS bullseye

23 February 2022

  • curprev 01:4101:41, 23 February 2022imported>Stashbot 824,762 bytes +60,474 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ms-be2068.codfw.wmnet with reason: host reimage

21 February 2022

  • curprev 22:3022:30, 21 February 2022imported>Stashbot 764,288 bytes +74,792 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1164 (T300381)', diff saved to https://phabricator.wikimedia.org/P21231 and previous config saved to /var/cache/conftool/dbconfig/20220221-223015-marostegui.json
  • curprev 01:3901:39, 21 February 2022imported>Stashbot 689,496 bytes +3,448 ladsgroup@cumin1001: START - Cookbook sre.hosts.reimage for host db2152.codfw.wmnet with OS bullseye

19 February 2022

  • curprev 16:5016:50, 19 February 2022imported>Stashbot 686,048 bytes +5,104 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: apply
  • curprev 00:5900:59, 19 February 2022imported>Stashbot 680,944 bytes +28,538 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2020.codfw.wmnet with OS bullseye

17 February 2022

  • curprev 22:2822:28, 17 February 2022imported>Stashbot 652,406 bytes +28,397 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:3601:36, 17 February 2022imported>Stashbot 624,009 bytes +52,457 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T300381)', diff saved to https://phabricator.wikimedia.org/P20954 and previous config saved to /var/cache/conftool/dbconfig/20220217-013607-marostegui.json

15 February 2022

  • curprev 23:4723:47, 15 February 2022imported>Stashbot 571,552 bytes +59,268 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host restbase-dev2003.mgmt.codfw.wmnet with reboot policy FORCED

14 February 2022

  • curprev 22:0422:04, 14 February 2022imported>Stashbot 512,284 bytes +52,126 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

13 February 2022

  • curprev 23:1723:17, 13 February 2022imported>Stashbot 460,158 bytes +3,305 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3315 (T300775)', diff saved to https://phabricator.wikimedia.org/P20627 and previous config saved to /var/cache/conftool/dbconfig/20220213-231742-marostegui.json

12 February 2022

  • curprev 22:5822:58, 12 February 2022imported>Stashbot 456,853 bytes +2,897 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1161 (T300775)', diff saved to https://phabricator.wikimedia.org/P20617 and previous config saved to /var/cache/conftool/dbconfig/20220212-225806-marostegui.json

11 February 2022

10 February 2022

  • curprev 00:4200:42, 10 February 2022imported>Stashbot 362,781 bytes +25,944 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

8 February 2022

  • curprev 23:5223:52, 8 February 2022imported>Stashbot 336,837 bytes +73,681 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc2055.codfw.wmnet with OS buster
  • curprev 00:1200:12, 8 February 2022imported>Stashbot 263,156 bytes +33,177 ryankemper: T294805 Re-enabling puppet across eqiad elastic fleet: `ryankemper@cumin1001:~$ sudo cumin -b 8 'elastic1*' 'sudo enable-puppet "Add new eqiad replacement hosts elastic10[68-83] - T294805 - root" && sudo run-puppet-agent'` tmux session `elastic`

5 February 2022

  • curprev 22:1022:10, 5 February 2022imported>Stashbot 229,979 bytes +1,284 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2003-dev.codfw.wmnet with OS bullseye

4 February 2022

  • curprev 23:4323:43, 4 February 2022imported>Stashbot 228,695 bytes +5,568 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mirror1001.wikimedia.org with reason: new kernel
  • curprev 01:0801:08, 4 February 2022imported>Stashbot 223,127 bytes +72,959 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

3 February 2022

2 February 2022

  • curprev 00:5300:53, 2 February 2022imported>Stashbot 90,142 bytes −738,181 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

1 February 2022

  • curprev 00:3100:31, 1 February 2022imported>Stashbot 828,323 bytes +72,250 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

29 January 2022

  • curprev 21:0821:08, 29 January 2022imported>Stashbot 756,073 bytes +1,014 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudservices2003-dev.wikimedia.org with OS bullseye
  • curprev 00:1400:14, 29 January 2022imported>Stashbot 755,059 bytes +14,112 ebernhardson: restart elasticsearch_6@production-search-psi-eqiad on elastic1049 to address CirrusSearchJVMGCOldPoolFlatlined alert

28 January 2022

  • curprev 01:4701:47, 28 January 2022imported>Stashbot 740,947 bytes +73,300 andrew@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudcontrol2001-dev.wikimedia.org with OS bullseye

27 January 2022

26 January 2022

25 January 2022

  • curprev 00:3100:31, 25 January 2022imported>Stashbot 525,956 bytes +53,597 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

23 January 2022

  • curprev 22:0222:02, 23 January 2022imported>Stashbot 472,359 bytes +500 ebysans@deploy1002: Finished deploy [airflow-dags/analytics-test@37937f6]: (no justification provided) (duration: 00m 08s)

22 January 2022

  • curprev 22:3822:38, 22 January 2022imported>Stashbot 471,859 bytes +812 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx1001.wikimedia.org with reason: kernel testing
  • curprev 01:3001:30, 22 January 2022imported>Stashbot 471,047 bytes +18,324 jhathaway@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on mx1001.wikimedia.org with reason: kernel testing

20 January 2022

  • curprev 22:4022:40, 20 January 2022imported>Stashbot 452,723 bytes +39,372 inflatador: running puppet-merge for https://gerrit.wikimedia.org/r/755810

19 January 2022

17 January 2022

  • curprev 23:2723:27, 17 January 2022imported>Stashbot 327,176 bytes +12,624 jynus: forced session revocation on phab for a user T299315

16 January 2022

  • curprev 08:2108:21, 16 January 2022imported>Stashbot 314,552 bytes +684 oblivian@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: sync on production

15 January 2022

  • curprev 08:5508:55, 15 January 2022imported>Stashbot 313,868 bytes +1,296 legoktm: finished running recountCategories on s4 wikis (T299244)
  • curprev 01:2201:22, 15 January 2022imported>Stashbot 312,572 bytes +10,517 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn

14 January 2022

  • curprev 00:3600:36, 14 January 2022imported>Stashbot 302,055 bytes +32,093 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

13 January 2022

  • curprev 00:3500:35, 13 January 2022imported>Stashbot 269,962 bytes +64,421 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

12 January 2022

  • curprev 00:5500:55, 12 January 2022imported>Stashbot 205,541 bytes +59,425 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

11 January 2022

8 January 2022

  • curprev 10:5110:51, 8 January 2022imported>Stashbot 107,107 bytes +180 elukey: restart hive daemons on an-coord1002 (after my last upgrade/rollback of packages the prometheus agent settings were not picked up, so no metrics)

7 January 2022

6 January 2022

5 January 2022

  • curprev 00:5900:59, 5 January 2022imported>Stashbot 64,897 bytes +32,691 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn

4 January 2022

  • curprev 00:5400:54, 4 January 2022imported>Stashbot 32,206 bytes +32,091 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P18329 and previous config saved to /var/cache/conftool/dbconfig/20220104-005456-marostegui.json

1 January 2022

29 December 2021

  • curprev 10:3010:30, 29 December 2021imported>Stashbot 664,919 bytes +126 elukey: kill tcpdump process on kubestagemaster1001 (kept a big pcap file opened that kept growing)

28 December 2021

24 December 2021

  • curprev 20:0820:08, 24 December 2021imported>Stashbot 663,863 bytes +325 mforns@deploy1002: Finished deploy [airflow-dags/analytics@e282d2d]: (no justification provided) (duration: 00m 06s)
  • curprev 00:5700:57, 24 December 2021imported>Stashbot 663,538 bytes +3,353 ejegg: updated fundraising CiviCRM from 47dd67f2 to aaceb4ab

23 December 2021

  • curprev 00:0400:04, 23 December 2021imported>Stashbot 660,185 bytes +4,302 bking@cumin1001: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) restart without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin1001 - T297986

21 December 2021

19 December 2021

18 December 2021

  • curprev 13:5713:57, 18 December 2021imported>Stashbot 633,583 bytes +93 dcausse: restarting blazegraph on wdqs1013 (jvm stuck for 10hours)

17 December 2021

16 December 2021

  • curprev 00:3700:37, 16 December 2021imported>Stashbot 600,611 bytes +14,349 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

15 December 2021

14 December 2021

  • curprev 01:4201:42, 14 December 2021imported>Stashbot 545,410 bytes +41,246 ryankemper: T297468 `sudo cookbook sre.elasticsearch.rolling-operation search_eqiad "eqiad rolling restart" --nodes-per-run 3 --start-datetime 2021-12-14T01:27:58 --task-id T297468` on `ryankemper@cumin1001` tmux `elastic_restarts`

12 December 2021

  • curprev 14:3514:35, 12 December 2021imported>Stashbot 504,164 bytes +844 filippo@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host graphite1004.eqiad.wmnet

11 December 2021

  • curprev 19:0419:04, 11 December 2021imported>Stashbot 503,320 bytes +131 andrew@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster
  • curprev 00:0400:04, 11 December 2021imported>Stashbot 503,189 bytes +18,770 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

10 December 2021

9 December 2021

  • curprev 00:2600:26, 9 December 2021imported>Stashbot 469,358 bytes +7,967 rzl: graphite1004.mgmt: /admin1-> racadm serveraction powercycle (T297265)

8 December 2021

  • curprev 00:5100:51, 8 December 2021imported>Stashbot 461,391 bytes +29,464 ebernhardson@deploy1002: Synchronized php-1.38.0-wmf.12/extensions/GrowthExperiments/includes/NewcomerTasks/AddImage/AddImageSubmissionHandler.php: backport window for 744896 (duration: 01m 05s)

7 December 2021

4 December 2021

  • curprev 01:1401:14, 4 December 2021imported>Stashbot 424,523 bytes +12,137 mutante: mx2001 - did not come back from reboot, did not get IP on interface, could not start ferm, logged in via console with root password, in /etc/network/interfaces replaced all "ens5" with "ens13", rebooted again, selected previous kernel version

3 December 2021

2 December 2021

  • curprev 01:2101:21, 2 December 2021imported>Stashbot 394,618 bytes +24,122 ryankemper: T280001 Rolling restart of low-traffic pybal hosts complete. All of `wcqs` is pooled and the pybal / ipvs related alerts have cleared

1 December 2021

  • curprev 00:3500:35, 1 December 2021imported>Stashbot 370,496 bytes +9,705 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

30 November 2021

  • curprev 00:2200:22, 30 November 2021imported>Stashbot 360,791 bytes +17,004 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

28 November 2021

  • curprev 17:1417:14, 28 November 2021imported>Stashbot 343,787 bytes +293 elukey@deploy1002: Finished deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 (duration: 02m 11s)

27 November 2021

  • curprev 19:5519:55, 27 November 2021imported>Stashbot 343,494 bytes +1,043 andrew@deploy1002: Finished deploy [horizon/deploy@6115b3b]: network UI updates for T296548 (duration: 04m 14s)

26 November 2021

  • curprev 16:1116:11, 26 November 2021imported>Stashbot 342,451 bytes +4,657 arnoldokoth: drain kubestage1002 node in prep for decommissioning

25 November 2021

  • curprev 20:4420:44, 25 November 2021imported>Stashbot 337,794 bytes +25,467 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17872 and previous config saved to /var/cache/conftool/dbconfig/20211125-204357-ladsgroup.json
  • curprev 00:3700:37, 25 November 2021imported>Stashbot 312,327 bytes +56,025 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2066.codfw.wmnet with OS buster

24 November 2021

  • curprev 00:2500:25, 24 November 2021imported>Stashbot 256,302 bytes +24,082 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS buster

23 November 2021

  • curprev 00:5100:51, 23 November 2021imported>Stashbot 232,220 bytes +27,289 urbanecm@deploy1002: Started scap: 69aa4a7: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 4/4)

21 November 2021

20 November 2021

  • curprev 01:0201:02, 20 November 2021imported>Stashbot 204,605 bytes +91 mutante: lists1001 - restarted apache, icinga alerts for the web UI, but recovered
  • curprev 00:2700:27, 20 November 2021imported>Stashbot 204,514 bytes +9,398 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)

19 November 2021

  • curprev 01:4501:45, 19 November 2021imported>Stashbot 195,116 bytes +21,872 mutante: I think git-ssh6_22 is down (see alerts lvs2008/2009) due to the v6 issue from ongoing lvs maintenance. depooled in conftool

18 November 2021

  • curprev 01:4701:47, 18 November 2021imported>Stashbot 173,244 bytes +19,375 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2005.codfw.wmnet

17 November 2021

  • curprev 00:1900:19, 17 November 2021imported>Stashbot 153,869 bytes +12,582 ryankemper: T276198 `ryankemper@cumin1001:~$ sudo cumin -b 3 '*elastic*' 'sudo run-puppet-agent --force'` Change looks good (no complaints from systemd), rolling out to rest of fleet / reenabling puppet

16 November 2021

14 November 2021

  • curprev 11:4811:48, 14 November 2021imported>Stashbot 125,456 bytes +111 paravoid: disable cr1-eqiad:xe-3/0/6 (IXP port) to mitigate T295650

13 November 2021

  • curprev 18:4318:43, 13 November 2021imported>Stashbot 125,345 bytes +836 AndyRussG: Enabled debug logging for PayPal IPN listener (updated SmashPig config a9e30591 -> 9567cc4a on frpig1001)

12 November 2021

  • curprev 21:0021:00, 12 November 2021imported>Stashbot 124,509 bytes +5,102 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:1900:19, 12 November 2021imported>Stashbot 119,407 bytes +11,807 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

11 November 2021

  • curprev 00:2200:22, 11 November 2021imported>Stashbot 107,600 bytes +12,063 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

10 November 2021

  • curprev 01:0701:07, 10 November 2021imported>Stashbot 95,537 bytes +10,338 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

8 November 2021

  • curprev 23:3923:39, 8 November 2021imported>Stashbot 85,199 bytes +11,520 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

6 November 2021

5 November 2021

  • curprev 00:1600:16, 5 November 2021imported>Stashbot 66,667 bytes +20,851 mutante: phab1001 - sudo systemctl start phabricator_clean_tmp_files.service because Icinga alerted it had failed... worked fine

4 November 2021

2 November 2021

  • curprev 23:4723:47, 2 November 2021imported>Stashbot 22,959 bytes +11,789 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:2101:21, 2 November 2021imported>Stashbot 11,170 bytes −844,596 ejegg: updated SmashPig standalone deploy from dd3a81c7c2 to be68299b92

31 October 2021

  • curprev 21:4921:49, 31 October 2021imported>Stashbot 855,766 bytes +371 urbanecm: urbanecm@mwmaint1002:~$ mwscript userOptions.php --wiki=dewiki --nowarn --touserid 3802752 --old 'linkrecommendation' --new 'control' 'growthexperiments-homepage-variant' # T294712

30 October 2021

29 October 2021

  • curprev 22:5722:57, 29 October 2021imported>Stashbot 855,227 bytes +3,905 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=foundationwiki --userlist users.txt # T205347, users.txt is at P17641

28 October 2021

  • curprev 23:5023:50, 28 October 2021imported>Stashbot 851,322 bytes +21,486 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4033.ulsfo.wmnet with OS buster

27 October 2021

  • curprev 23:5523:55, 27 October 2021imported>Stashbot 829,836 bytes +10,201 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

26 October 2021

  • curprev 22:5922:59, 26 October 2021imported>Stashbot 819,635 bytes +12,871 legoktm: uploaded python-logstash to buster-wikimedia for T294393

25 October 2021

  • curprev 23:1223:12, 25 October 2021imported>Stashbot 806,764 bytes +11,982 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Create alias for Appendix and Appendix_talk namespaces on mywiktionary (T291146) (duration: 00m 55s)

23 October 2021

  • curprev 16:4016:40, 23 October 2021imported>Stashbot 794,782 bytes +253 dcausse: restarting blazegraph on wdqs1004 and wdqs1006 (free allocators alert)

22 October 2021

  • curprev 23:1723:17, 22 October 2021imported>Stashbot 794,529 bytes +3,761 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

21 October 2021

  • curprev 23:4023:40, 21 October 2021imported>Stashbot 790,768 bytes +17,384 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:0600:06, 21 October 2021imported>Stashbot 773,384 bytes +14,742 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

20 October 2021

  • curprev 01:3101:31, 20 October 2021imported>Stashbot 758,642 bytes +27,590 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

19 October 2021

  • curprev 00:3800:38, 19 October 2021imported>Stashbot 731,052 bytes +11,211 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer

16 October 2021

  • curprev 03:5603:56, 16 October 2021imported>Stashbot 719,841 bytes +266 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

15 October 2021

  • curprev 23:4823:48, 15 October 2021imported>Stashbot 719,575 bytes +5,052 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
  • curprev 00:0900:09, 15 October 2021imported>Stashbot 714,523 bytes +19,109 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

14 October 2021

  • curprev 01:4101:41, 14 October 2021imported>Stashbot 695,414 bytes +17,092 ejegg: updated payments-wiki from b329d2dea2 to 19d18c1852

13 October 2021

  • curprev 00:3800:38, 13 October 2021imported>Stashbot 678,322 bytes +15,911 ejegg: updated payments-wiki from 030b11da1a to b329d2dea2

12 October 2021

  • curprev 00:1100:11, 12 October 2021imported>Stashbot 662,411 bytes +3,728 eileen: civicrm revision changed from 598b59b0ee to 96090e4bd2, config revision is 85277466ed

9 October 2021

  • curprev 05:0105:01, 9 October 2021imported>Stashbot 658,683 bytes +224 jiji@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:3201:32, 9 October 2021imported>Stashbot 658,459 bytes +8,808 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

8 October 2021

7 October 2021

  • curprev 00:1100:11, 7 October 2021imported>Stashbot 637,764 bytes +11,864 mutante: [grafana2001:~] $ sudo systemctl start rsync-var-lib-grafana because of "PROBLEM - Check systemd state on grafana2001 is CRITICAL: CRITICAL - degraded" because of some race condition where a file vanished during sync

6 October 2021

  • curprev 01:3901:39, 6 October 2021imported>Stashbot 625,900 bytes +10,381 legoktm: legoktm@mwmaint1002:~$ echo "https://en.wikiversity.org/static/images/mobile/copyright/wikiversity.svg" |mwscript purgeList.php

4 October 2021

  • curprev 23:3023:30, 4 October 2021imported>Stashbot 615,519 bytes +9,952 foks: resetting some emails used for abuse by a globally-banned user

3 October 2021

2 October 2021

  • curprev 17:2817:28, 2 October 2021imported>Stashbot 605,040 bytes +230 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

1 October 2021

  • curprev 23:1923:19, 1 October 2021imported>Stashbot 604,810 bytes +38,661 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

29 September 2021

  • curprev 23:2023:20, 29 September 2021imported>Stashbot 566,149 bytes +14,880 bd808@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 00:5200:52, 29 September 2021imported>Stashbot 551,269 bytes +27,094 eileen: civicrm revision changed from a1929b3dfd to a480bf03c9, config revision is 77cb7ec866

27 September 2021

  • curprev 23:4223:42, 27 September 2021imported>Stashbot 524,175 bytes +27,476 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

26 September 2021

  • curprev 14:5114:51, 26 September 2021imported>Stashbot 496,699 bytes +432 volker-e@deploy1002: Finished deploy [design/style-guide@aac0ae9]: Deploy design/style-guide: aac0ae9 “Apps”: Fix image path (#490) (duration: 00m 06s)

25 September 2021

  • curprev 02:0002:00, 25 September 2021imported>Stashbot 496,267 bytes +374 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .

24 September 2021

  • curprev 20:0020:00, 24 September 2021imported>Stashbot 495,893 bytes +10,338 volker-e@deploy1002: Finished deploy [design/style-guide@362c6b1]: Deploy design/style-guide: 362c6b1 “Components”: Fix index link (#489) (duration: 00m 06s)
  • curprev 00:3900:39, 24 September 2021imported>Stashbot 485,555 bytes +19,157 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

23 September 2021

  • curprev 00:3500:35, 23 September 2021imported>Stashbot 466,398 bytes +12,744 catrope@deploy1002: Synchronized php-1.38.0-wmf.1/extensions/MediaSearch/: Use text() instead of parse() for MediaSearch UI messages (T291590) (duration: 01m 08s)

21 September 2021

  • curprev 23:2023:20, 21 September 2021imported>Stashbot 453,654 bytes +9,016 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:1600:16, 21 September 2021imported>Stashbot 444,638 bytes +11,548 tgr: Evening deploys done

18 September 2021

  • curprev 01:4701:47, 18 September 2021imported>Stashbot 433,090 bytes +322 ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.23/includes/libs/rdbms/database/Database.php: (no justification provided) (duration: 00m 57s)

17 September 2021

  • curprev 21:2821:28, 17 September 2021imported>Stashbot 432,768 bytes +3,000 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:0400:04, 17 September 2021imported>Stashbot 429,768 bytes +13,476 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .

15 September 2021

14 September 2021

  • curprev 23:0123:01, 14 September 2021imported>Stashbot 406,930 bytes +15,959 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Re-enable VipsScaler (2 of 2) (duration: 01m 04s)
  • curprev 00:0400:04, 14 September 2021imported>Stashbot 390,971 bytes +8,994 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

12 September 2021

11 September 2021

  • curprev 19:0219:02, 11 September 2021imported>Stashbot 381,555 bytes +650 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 27814b8eaacb5ba2fee1b6167a36ea14356a1ecf: testwiki: Fully remove securepoll-related groups (T290808) (duration: 00m 57s)

10 September 2021

  • curprev 21:2921:29, 10 September 2021imported>Stashbot 380,905 bytes +5,151 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • curprev 00:3500:35, 10 September 2021imported>Stashbot 375,754 bytes +15,347 tgr: Deployed patch for T290692

9 September 2021

  • curprev 00:5000:50, 9 September 2021imported>Stashbot 360,407 bytes +8,465 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Don't set default to Score (try #2) (duration: 00m 58s)

8 September 2021

  • curprev 00:0000:00, 8 September 2021imported>Stashbot 351,942 bytes +12,900 legoktm: legoktm@lists1001:~$ sudo rm -rf /etc/mailman # cleanup as part of 4869d91b0be / T282303

6 September 2021

  • curprev 23:5223:52, 6 September 2021imported>Stashbot 339,042 bytes +8,019 tstarling@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/SecurePoll/includes/Talliers/STVTallier.php: T290000 (duration: 00m 58s)

5 September 2021

  • curprev 18:5418:54, 5 September 2021imported>Stashbot 331,023 bytes +201 urbanecm: wikiadmin@10.192.0.119(ptwiki)> update protected_titles set pt_create_perm='editautoreviewprotected' where pt_create_perm='autoreviewer'; # T290396

4 September 2021

  • curprev 13:3513:35, 4 September 2021imported>Stashbot 330,822 bytes +1,989 marostegui@cumin1001: dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: Slowly repool T290374', diff saved to https://phabricator.wikimedia.org/P17217 and previous config saved to /var/cache/conftool/dbconfig/20210904-133532-root.json

3 September 2021

1 September 2021

  • curprev 23:5023:50, 1 September 2021imported>Stashbot 297,294 bytes +11,478 Amir1: mwscript createAndPromote.php --wiki=test2wiki --sysop --force Ladsgroup
  • curprev 00:5300:53, 1 September 2021imported>Stashbot 285,816 bytes +13,363 eileen: civicrm revision changed from e567b4c289 to 7da3eba4f9, config revision is 5f004d94d7

30 August 2021

  • curprev 23:1423:14, 30 August 2021imported>Stashbot 272,453 bytes +10,249 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

29 August 2021

  • curprev 00:1500:15, 29 August 2021imported>Stashbot 262,204 bytes +714 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

27 August 2021

  • curprev 16:4616:46, 27 August 2021imported>Stashbot 261,490 bytes +3,953 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps1005.eqiad.wmnet with reason: Resyncing from master
  • curprev 00:4600:46, 27 August 2021imported>Stashbot 257,537 bytes +9,223 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

25 August 2021

  • curprev 23:2323:23, 25 August 2021imported>Stashbot 248,314 bytes +23,982 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:5300:53, 25 August 2021imported>Stashbot 224,332 bytes +6,127 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .

24 August 2021

  • curprev 00:1700:17, 24 August 2021imported>Stashbot 218,205 bytes +10,888 ryankemper: [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'`

22 August 2021

21 August 2021

  • curprev 15:1215:12, 21 August 2021imported>Stashbot 1,011,788 bytes +262 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

20 August 2021

  • curprev 23:1823:18, 20 August 2021imported>Stashbot 1,011,526 bytes +4,817 legoktm: deployed patch for T289385
  • curprev 00:1800:18, 20 August 2021imported>Stashbot 1,006,709 bytes +10,542 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

19 August 2021

  • curprev 01:1301:13, 19 August 2021imported>Stashbot 996,167 bytes +6,256 ejegg: updated fundraising CiviCRM from 73f6ec9190 to 8ed303f2d1

18 August 2021

  • curprev 00:3900:39, 18 August 2021imported>Stashbot 989,911 bytes +13,880 dpifke@deploy1002: Finished deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243) (duration: 00m 05s)

17 August 2021

  • curprev 00:4500:45, 17 August 2021imported>Stashbot 976,031 bytes +12,823 eileen: civicrm revision changed from ba0c7705bb to 175a3101f7, config revision is 7bdc78073d

15 August 2021

14 August 2021

  • curprev 03:5403:54, 14 August 2021imported>Stashbot 962,909 bytes +121 legoktm[m]: restarting mailman3 on lists1001, bounce runner crashed (T288880)

13 August 2021

  • curprev 18:4318:43, 13 August 2021imported>Stashbot 962,788 bytes +9,163 bblack: reprepro: uploaded gdnsd-3.8.0-1~wmf1 to buster-wikimedia - T252132

12 August 2021

11 August 2021

  • curprev 23:3023:30, 11 August 2021imported>Stashbot 931,568 bytes +15,885 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:4701:47, 11 August 2021imported>Stashbot 915,683 bytes +19,823 dpifke@deploy1002: Finished deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 06s)

9 August 2021

  • curprev 16:1216:12, 9 August 2021imported>Stashbot 895,860 bytes +14,288 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .

6 August 2021

  • curprev 19:1719:17, 6 August 2021imported>Stashbot 881,572 bytes +8,445 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:0101:01, 6 August 2021imported>Stashbot 873,127 bytes +23,598 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be2065.codfw.wmnet with reason: REIMAGE

5 August 2021

  • curprev 01:2601:26, 5 August 2021imported>Stashbot 849,529 bytes +19,135 Krinkle: krinkle@mwmaint1002 Temporarily grant myself `translationadmin` on wikimania2016wiki in order to approve an edit given FlaggedRevs-like nature of Translate

3 August 2021

  • curprev 23:3423:34, 3 August 2021imported>Stashbot 830,394 bytes +13,656 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:4400:44, 3 August 2021imported>Stashbot 816,738 bytes +12,154 reedy@deploy1002: Finished deploy [integration/docroot@f9d225d]: with less gref (duration: 00m 05s)

31 July 2021

  • curprev 12:4012:40, 31 July 2021imported>Stashbot 804,584 bytes +151 reedy@deploy1002: Synchronized php-1.37.0-wmf.16/extensions/SecurePoll/: T287780 T287782 (duration: 00m 58s)
  • curprev 00:0100:01, 31 July 2021imported>Stashbot 804,433 bytes +8,409 eileen: civicrm revision changed from 158ed65e00 to d6baf291f4, config revision is 6011d9c471

29 July 2021

28 July 2021

27 July 2021

26 July 2021

  • curprev 23:3723:37, 26 July 2021imported>Stashbot 765,495 bytes +4,200 legoktm@deploy1002: Synchronized php-1.37.0-wmf.15/extensions/Score/includes/Score.php: Increase lilypond version cache TTL to 1 hour (duration: 00m 57s)

24 July 2021

  • curprev 11:0411:04, 24 July 2021imported>Stashbot 761,295 bytes +338 urbanecm: [urbanecm@mwmaint2002 ~]$ mwscript extensions/Translate/scripts/moveTranslatablePage.php --wiki=commonswiki --reason='OTRS -> VRTS renaming process; see Phab:T280392 and Phab:T280397' --move-subpages 'Commons:OTRS' 'Commons:Volunteer Response Team' 'Martin Urbanec' # T287321

23 July 2021

  • curprev 19:1119:11, 23 July 2021imported>Stashbot 760,957 bytes +5,409 topranks: Successfully re-pooled eqiad - reversed change from yesterday after successful line card replacement in cr2-codfw - T287110

22 July 2021

20 July 2021

  • curprev 20:5320:53, 20 July 2021imported>Stashbot 730,828 bytes +12,637 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: caa5a076f39b051b01622aa3e4c9d716a8643eef: Set wgGEMentorDashboardBackendEnabled properly (T285811) (duration: 00m 57s)

19 July 2021

16 July 2021

  • curprev 19:5019:50, 16 July 2021imported>Stashbot 704,075 bytes +7,705 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on copernicium.wikimedia.org with reason: REIMAGE
  • curprev 00:0600:06, 16 July 2021imported>Stashbot 696,370 bytes +21,594 hoo: Updated the Wikidata property suggester with data from the 2021-07-12 JSON dump (with pre-applied T132839 workarounds)

15 July 2021

14 July 2021

  • curprev 00:5800:58, 14 July 2021imported>Stashbot 662,656 bytes +13,088 eileen: process control updated to c291b3c6890364281d

12 July 2021

  • curprev 23:5723:57, 12 July 2021imported>Stashbot 649,568 bytes +13,413 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 1896efc27f3de39659673091bc4c43ad874da0c5: Add sayahna.org to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T286163) (duration: 00m 56s)

9 July 2021

  • curprev 23:2823:28, 9 July 2021imported>Stashbot 636,155 bytes +1,014 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • curprev 00:4700:47, 9 July 2021imported>Stashbot 635,141 bytes +9,116 legoktm: zotero rolling restart didn't help, filed T286360 for DNS issues

7 July 2021

  • curprev 20:2220:22, 7 July 2021imported>Stashbot 626,025 bytes +5,635 legoktm: repooling eqiad - https://gerrit.wikimedia.org/r/703561

6 July 2021

  • curprev 18:3418:34, 6 July 2021imported>Stashbot 620,390 bytes +5,171 otto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
  • curprev 00:5600:56, 6 July 2021imported>Stashbot 615,219 bytes +1,686 eileen: process-control config revision is 8d46b52ed4

5 July 2021

  • curprev 00:5300:53, 5 July 2021imported>Stashbot 613,533 bytes +528 eileen: process-control config revision is a1717c7fde

3 July 2021

  • curprev 17:4617:46, 3 July 2021imported>Stashbot 613,005 bytes +322 elukey: depool eqsin due to loss of power redundancy (equinix maintenance) - T286113

2 July 2021

  • curprev 22:0622:06, 2 July 2021imported>Stashbot 612,683 bytes +8,804 foks: removing three files for legal compliance
  • curprev 01:4701:47, 2 July 2021imported>Stashbot 603,879 bytes +9,552 eileen: civicrm revision changed from e07c2be1a7 to bb62188ec6, config revision is 1739c53fcb

30 June 2021

  • curprev 23:2823:28, 30 June 2021imported>Stashbot 594,327 bytes +11,040 urbanecm: Evening B&C window finished
  • curprev 00:3600:36, 30 June 2021imported>Stashbot 583,287 bytes +15,990 tstarling@deploy1002: Synchronized wmf-config/db-labs.php: gerrit 701995 SQL query log (duration: 01m 05s)

29 June 2021

  • curprev 00:2600:26, 29 June 2021imported>Stashbot 567,297 bytes +16,261 Krinkle: krinkle@mwmaint1002: purgeParserCache.php --tag pc1, ref T282761

27 June 2021

  • curprev 09:1909:19, 27 June 2021imported>Elukey 551,036 bytes +741 Add missing restart entry (logged on a different chan)

26 June 2021

  • curprev 21:2821:28, 26 June 2021imported>Stashbot 550,295 bytes +1,342 volans: upgraded spicerack to v0.0.56 on the cumin hosts (includes only bug fixes for the switchdc)

25 June 2021

  • curprev 21:3721:37, 25 June 2021imported>Stashbot 548,953 bytes +6,188 ebernhardson@deploy1002: Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s)

24 June 2021

  • curprev 23:0223:02, 24 June 2021imported>Stashbot 542,765 bytes +11,139 legoktm: reverted cumin1001 spicerack live hacks
  • curprev 00:2500:25, 24 June 2021imported>Stashbot 531,626 bytes +19,731 eileen: civicrm revision changed from bd906975f0 to 6d3dd6e5a5, config revision is 821e5889f7

23 June 2021

  • curprev 00:5000:50, 23 June 2021imported>Stashbot 511,895 bytes +14,361 eileen: civicrm revision changed from c745d4f075 to 03bead707d, config revision is 4ab72c1033

22 June 2021

  • curprev 01:0201:02, 22 June 2021imported>Stashbot 497,534 bytes +26,215 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE

18 June 2021

  • curprev 20:5520:55, 18 June 2021imported>Stashbot 471,319 bytes +9,077 Krinkle: Remove doc1001:/srv/doc/mediawiki-core/wmf-1.36.0-wmf.31-testing
(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)