You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

11 September 2021

  • curprev 19:0219:02, 11 September 2021imported>Stashbot 381,555 bytes +650 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 27814b8eaacb5ba2fee1b6167a36ea14356a1ecf: testwiki: Fully remove securepoll-related groups (T290808) (duration: 00m 57s)

10 September 2021

  • curprev 21:2921:29, 10 September 2021imported>Stashbot 380,905 bytes +5,151 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • curprev 00:3500:35, 10 September 2021imported>Stashbot 375,754 bytes +15,347 tgr: Deployed patch for T290692

9 September 2021

  • curprev 00:5000:50, 9 September 2021imported>Stashbot 360,407 bytes +8,465 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Don't set default to Score (try #2) (duration: 00m 58s)

8 September 2021

  • curprev 00:0000:00, 8 September 2021imported>Stashbot 351,942 bytes +12,900 legoktm: legoktm@lists1001:~$ sudo rm -rf /etc/mailman # cleanup as part of 4869d91b0be / T282303

6 September 2021

  • curprev 23:5223:52, 6 September 2021imported>Stashbot 339,042 bytes +8,019 tstarling@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/SecurePoll/includes/Talliers/STVTallier.php: T290000 (duration: 00m 58s)

5 September 2021

  • curprev 18:5418:54, 5 September 2021imported>Stashbot 331,023 bytes +201 urbanecm: wikiadmin@10.192.0.119(ptwiki)> update protected_titles set pt_create_perm='editautoreviewprotected' where pt_create_perm='autoreviewer'; # T290396

4 September 2021

  • curprev 13:3513:35, 4 September 2021imported>Stashbot 330,822 bytes +1,989 marostegui@cumin1001: dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: Slowly repool T290374', diff saved to https://phabricator.wikimedia.org/P17217 and previous config saved to /var/cache/conftool/dbconfig/20210904-133532-root.json

3 September 2021

1 September 2021

  • curprev 23:5023:50, 1 September 2021imported>Stashbot 297,294 bytes +11,478 Amir1: mwscript createAndPromote.php --wiki=test2wiki --sysop --force Ladsgroup
  • curprev 00:5300:53, 1 September 2021imported>Stashbot 285,816 bytes +13,363 eileen: civicrm revision changed from e567b4c289 to 7da3eba4f9, config revision is 5f004d94d7

30 August 2021

  • curprev 23:1423:14, 30 August 2021imported>Stashbot 272,453 bytes +10,249 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

29 August 2021

  • curprev 00:1500:15, 29 August 2021imported>Stashbot 262,204 bytes +714 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

27 August 2021

  • curprev 16:4616:46, 27 August 2021imported>Stashbot 261,490 bytes +3,953 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps1005.eqiad.wmnet with reason: Resyncing from master
  • curprev 00:4600:46, 27 August 2021imported>Stashbot 257,537 bytes +9,223 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

25 August 2021

  • curprev 23:2323:23, 25 August 2021imported>Stashbot 248,314 bytes +23,982 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:5300:53, 25 August 2021imported>Stashbot 224,332 bytes +6,127 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .

24 August 2021

  • curprev 00:1700:17, 24 August 2021imported>Stashbot 218,205 bytes +10,888 ryankemper: [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'`

22 August 2021

21 August 2021

  • curprev 15:1215:12, 21 August 2021imported>Stashbot 1,011,788 bytes +262 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

20 August 2021

  • curprev 23:1823:18, 20 August 2021imported>Stashbot 1,011,526 bytes +4,817 legoktm: deployed patch for T289385
  • curprev 00:1800:18, 20 August 2021imported>Stashbot 1,006,709 bytes +10,542 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

19 August 2021

  • curprev 01:1301:13, 19 August 2021imported>Stashbot 996,167 bytes +6,256 ejegg: updated fundraising CiviCRM from 73f6ec9190 to 8ed303f2d1

18 August 2021

  • curprev 00:3900:39, 18 August 2021imported>Stashbot 989,911 bytes +13,880 dpifke@deploy1002: Finished deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243) (duration: 00m 05s)

17 August 2021

  • curprev 00:4500:45, 17 August 2021imported>Stashbot 976,031 bytes +12,823 eileen: civicrm revision changed from ba0c7705bb to 175a3101f7, config revision is 7bdc78073d

15 August 2021

14 August 2021

  • curprev 03:5403:54, 14 August 2021imported>Stashbot 962,909 bytes +121 legoktm[m]: restarting mailman3 on lists1001, bounce runner crashed (T288880)

13 August 2021

  • curprev 18:4318:43, 13 August 2021imported>Stashbot 962,788 bytes +9,163 bblack: reprepro: uploaded gdnsd-3.8.0-1~wmf1 to buster-wikimedia - T252132

12 August 2021

11 August 2021

  • curprev 23:3023:30, 11 August 2021imported>Stashbot 931,568 bytes +15,885 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:4701:47, 11 August 2021imported>Stashbot 915,683 bytes +19,823 dpifke@deploy1002: Finished deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 06s)

9 August 2021

  • curprev 16:1216:12, 9 August 2021imported>Stashbot 895,860 bytes +14,288 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .

6 August 2021

  • curprev 19:1719:17, 6 August 2021imported>Stashbot 881,572 bytes +8,445 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:0101:01, 6 August 2021imported>Stashbot 873,127 bytes +23,598 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be2065.codfw.wmnet with reason: REIMAGE

5 August 2021

  • curprev 01:2601:26, 5 August 2021imported>Stashbot 849,529 bytes +19,135 Krinkle: krinkle@mwmaint1002 Temporarily grant myself `translationadmin` on wikimania2016wiki in order to approve an edit given FlaggedRevs-like nature of Translate

3 August 2021

  • curprev 23:3423:34, 3 August 2021imported>Stashbot 830,394 bytes +13,656 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:4400:44, 3 August 2021imported>Stashbot 816,738 bytes +12,154 reedy@deploy1002: Finished deploy [integration/docroot@f9d225d]: with less gref (duration: 00m 05s)

31 July 2021

  • curprev 12:4012:40, 31 July 2021imported>Stashbot 804,584 bytes +151 reedy@deploy1002: Synchronized php-1.37.0-wmf.16/extensions/SecurePoll/: T287780 T287782 (duration: 00m 58s)
  • curprev 00:0100:01, 31 July 2021imported>Stashbot 804,433 bytes +8,409 eileen: civicrm revision changed from 158ed65e00 to d6baf291f4, config revision is 6011d9c471

29 July 2021

28 July 2021

27 July 2021

26 July 2021

  • curprev 23:3723:37, 26 July 2021imported>Stashbot 765,495 bytes +4,200 legoktm@deploy1002: Synchronized php-1.37.0-wmf.15/extensions/Score/includes/Score.php: Increase lilypond version cache TTL to 1 hour (duration: 00m 57s)

24 July 2021

  • curprev 11:0411:04, 24 July 2021imported>Stashbot 761,295 bytes +338 urbanecm: [urbanecm@mwmaint2002 ~]$ mwscript extensions/Translate/scripts/moveTranslatablePage.php --wiki=commonswiki --reason='OTRS -> VRTS renaming process; see Phab:T280392 and Phab:T280397' --move-subpages 'Commons:OTRS' 'Commons:Volunteer Response Team' 'Martin Urbanec' # T287321

23 July 2021

  • curprev 19:1119:11, 23 July 2021imported>Stashbot 760,957 bytes +5,409 topranks: Successfully re-pooled eqiad - reversed change from yesterday after successful line card replacement in cr2-codfw - T287110

22 July 2021

20 July 2021

  • curprev 20:5320:53, 20 July 2021imported>Stashbot 730,828 bytes +12,637 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: caa5a076f39b051b01622aa3e4c9d716a8643eef: Set wgGEMentorDashboardBackendEnabled properly (T285811) (duration: 00m 57s)

19 July 2021

16 July 2021

  • curprev 19:5019:50, 16 July 2021imported>Stashbot 704,075 bytes +7,705 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on copernicium.wikimedia.org with reason: REIMAGE
  • curprev 00:0600:06, 16 July 2021imported>Stashbot 696,370 bytes +21,594 hoo: Updated the Wikidata property suggester with data from the 2021-07-12 JSON dump (with pre-applied T132839 workarounds)

15 July 2021

14 July 2021

  • curprev 00:5800:58, 14 July 2021imported>Stashbot 662,656 bytes +13,088 eileen: process control updated to c291b3c6890364281d

12 July 2021

  • curprev 23:5723:57, 12 July 2021imported>Stashbot 649,568 bytes +13,413 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 1896efc27f3de39659673091bc4c43ad874da0c5: Add sayahna.org to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T286163) (duration: 00m 56s)

9 July 2021

  • curprev 23:2823:28, 9 July 2021imported>Stashbot 636,155 bytes +1,014 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • curprev 00:4700:47, 9 July 2021imported>Stashbot 635,141 bytes +9,116 legoktm: zotero rolling restart didn't help, filed T286360 for DNS issues

7 July 2021

  • curprev 20:2220:22, 7 July 2021imported>Stashbot 626,025 bytes +5,635 legoktm: repooling eqiad - https://gerrit.wikimedia.org/r/703561

6 July 2021

  • curprev 18:3418:34, 6 July 2021imported>Stashbot 620,390 bytes +5,171 otto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
  • curprev 00:5600:56, 6 July 2021imported>Stashbot 615,219 bytes +1,686 eileen: process-control config revision is 8d46b52ed4

5 July 2021

  • curprev 00:5300:53, 5 July 2021imported>Stashbot 613,533 bytes +528 eileen: process-control config revision is a1717c7fde

3 July 2021

  • curprev 17:4617:46, 3 July 2021imported>Stashbot 613,005 bytes +322 elukey: depool eqsin due to loss of power redundancy (equinix maintenance) - T286113

2 July 2021

  • curprev 22:0622:06, 2 July 2021imported>Stashbot 612,683 bytes +8,804 foks: removing three files for legal compliance
  • curprev 01:4701:47, 2 July 2021imported>Stashbot 603,879 bytes +9,552 eileen: civicrm revision changed from e07c2be1a7 to bb62188ec6, config revision is 1739c53fcb

30 June 2021

  • curprev 23:2823:28, 30 June 2021imported>Stashbot 594,327 bytes +11,040 urbanecm: Evening B&C window finished
  • curprev 00:3600:36, 30 June 2021imported>Stashbot 583,287 bytes +15,990 tstarling@deploy1002: Synchronized wmf-config/db-labs.php: gerrit 701995 SQL query log (duration: 01m 05s)

29 June 2021

  • curprev 00:2600:26, 29 June 2021imported>Stashbot 567,297 bytes +16,261 Krinkle: krinkle@mwmaint1002: purgeParserCache.php --tag pc1, ref T282761

27 June 2021

  • curprev 09:1909:19, 27 June 2021imported>Elukey 551,036 bytes +741 Add missing restart entry (logged on a different chan)

26 June 2021

  • curprev 21:2821:28, 26 June 2021imported>Stashbot 550,295 bytes +1,342 volans: upgraded spicerack to v0.0.56 on the cumin hosts (includes only bug fixes for the switchdc)

25 June 2021

  • curprev 21:3721:37, 25 June 2021imported>Stashbot 548,953 bytes +6,188 ebernhardson@deploy1002: Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s)

24 June 2021

  • curprev 23:0223:02, 24 June 2021imported>Stashbot 542,765 bytes +11,139 legoktm: reverted cumin1001 spicerack live hacks
  • curprev 00:2500:25, 24 June 2021imported>Stashbot 531,626 bytes +19,731 eileen: civicrm revision changed from bd906975f0 to 6d3dd6e5a5, config revision is 821e5889f7

23 June 2021

  • curprev 00:5000:50, 23 June 2021imported>Stashbot 511,895 bytes +14,361 eileen: civicrm revision changed from c745d4f075 to 03bead707d, config revision is 4ab72c1033

22 June 2021

  • curprev 01:0201:02, 22 June 2021imported>Stashbot 497,534 bytes +26,215 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE

18 June 2021

  • curprev 20:5520:55, 18 June 2021imported>Stashbot 471,319 bytes +9,077 Krinkle: Remove doc1001:/srv/doc/mediawiki-core/wmf-1.36.0-wmf.31-testing

17 June 2021

  • curprev 21:4921:49, 17 June 2021imported>Stashbot 462,242 bytes +15,302 legoktm: regenerating pipermail redirects to skip those with duplicate message-ids (T280731)

16 June 2021

  • curprev 21:3521:35, 16 June 2021imported>Stashbot 446,940 bytes +4,876 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .

15 June 2021

  • curprev 17:5417:54, 15 June 2021imported>Stashbot 442,064 bytes +4,022 dancy: testing upcoming Scap release on beta
  • curprev 00:3700:37, 15 June 2021imported>Stashbot 438,042 bytes +26,468 eileen: civicrm revision changed from 31d07115a0 to 28ace1b86f, config revision is 2aed6ff89b

12 June 2021

  • curprev 13:4913:49, 12 June 2021imported>Stashbot 411,574 bytes +317 rzl@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: alert noise, no impact, x2 is unused

11 June 2021

  • curprev 23:3723:37, 11 June 2021imported>Stashbot 411,257 bytes +6,591 mutante: removing firewall hole for mgmt networks to install* because it turned out it cant be used for firmware upgrades
  • curprev 01:1001:10, 11 June 2021imported>Stashbot 404,666 bytes +17,032 eileen: process-control config revision is 2aed6ff89b

9 June 2021

7 June 2021

  • curprev 21:2621:26, 7 June 2021imported>Stashbot 350,845 bytes +11,147 otto@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0)

5 June 2021

  • curprev 16:1616:16, 5 June 2021imported>Stashbot 339,698 bytes +307 Amir1: deleting all private archives of mm2. All are inaccessible now (T282303)
  • curprev 00:2100:21, 5 June 2021imported>Stashbot 339,391 bytes +11,476 mutante: backup1001 - systemctl baclua-dir works again (restoring backup for non-existing host)

4 June 2021

  • curprev 00:1200:12, 4 June 2021imported>Stashbot 327,915 bytes +24,727 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

3 June 2021

1 June 2021

  • curprev 21:0921:09, 1 June 2021imported>Stashbot 287,834 bytes +5,348 andrewbogott: dropping a bunch of tables from the labswiki db as per T284108
  • curprev 00:4600:46, 1 June 2021imported>Stashbot 282,486 bytes +974 legoktm@deploy1002: Synchronized logos/config.yaml: Revert "Use eswiki 20th anniversary logos" (T280908) (duration: 01m 07s)

29 May 2021

  • curprev 14:4414:44, 29 May 2021imported>Stashbot 281,512 bytes +194 elukey: execute apt-get clean on an-airflow1001 to free space

28 May 2021

  • curprev 08:0608:06, 28 May 2021imported>Stashbot 281,318 bytes +368 oblivian@cumin1001: conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet,dc=eqiad

27 May 2021

  • curprev 23:5623:56, 27 May 2021imported>Stashbot 280,950 bytes +10,488 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab1004.eqiad.wmnet with reason: REIMAGE

26 May 2021

25 May 2021

  • curprev 00:4800:48, 25 May 2021imported>Stashbot 245,293 bytes +11,934 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

23 May 2021

  • curprev 14:2514:25, 23 May 2021imported>Stashbot 233,359 bytes +262 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: EMERGENCY: f752f8b80c57a5b7e41b91873b3eac388535ac46: enwiktionary: Raise AF emergency disable treshold+count (T283460) (duration: 00m 57s)

22 May 2021

  • curprev 22:1322:13, 22 May 2021imported>Stashbot 233,097 bytes +265 legoktm: reset 2FA for User:Yuvipanda on wikitech

21 May 2021

  • curprev 22:3222:32, 21 May 2021imported>Stashbot 232,832 bytes +12,510 bstorm: upload nfsd-ldap: 1.2+deb10u1 to buster-wikimedia T283385

20 May 2021

  • curprev 21:4521:45, 20 May 2021imported>Stashbot 220,322 bytes +13,158 herron@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:0101:01, 20 May 2021imported>Stashbot 207,164 bytes +12,243 mutante: signing puppet certs for doh2001 and doh2002.wikimedia.org (T283192)

18 May 2021

  • curprev 18:4018:40, 18 May 2021imported>Stashbot 194,921 bytes +14,463 razzi@deploy1002: Finished deploy [analytics/refinery@9392f1d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@9392f1db6e66975304c8e9b2b7031acd3ed87fa7] (duration: 05m 16s)
  • curprev 00:5600:56, 18 May 2021imported>Stashbot 180,458 bytes +19,706 eileen: civicrm revision changed from 38ac15233f to b3fb3c9cb0, config revision is 1f8d0a6bfa

17 May 2021

16 May 2021

  • curprev 00:5300:53, 16 May 2021imported>Stashbot 160,344 bytes +324 legoktm: restarted mailman3-web on lists1001, uwsgi looked like it got stuck, consuming all CPU/memory

14 May 2021

  • curprev 20:4220:42, 14 May 2021imported>Stashbot 160,020 bytes +3,428 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people1002.eqiad.wmnet
  • curprev 00:3900:39, 14 May 2021imported>Stashbot 156,592 bytes +8,035 ryankemper: T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2003.codfw.wmnet` on `ryankemper@cumin2001` tmux session `wdqs_reimage`

12 May 2021

  • curprev 23:4823:48, 12 May 2021imported>Stashbot 148,557 bytes +16,709 urbanecm@deploy1002: Synchronized php-1.37.0-wmf.4/extensions/WikiEditor/includes/WikiEditorHooks.php: 2f6af514c49d47bbec5ce51f9f7263015e039003? PHP VisualEditorFeatureUse logging: properly record session id (T281409) (duration: 01m 07s)
  • curprev 01:4201:42, 12 May 2021imported>Stashbot 131,848 bytes +17,502 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host

10 May 2021

  • curprev 23:3823:38, 10 May 2021imported>Stashbot 114,346 bytes +15,358 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: 779fb53bfd7a4d9b11f865df14f8a72adb97f33b: Update messages used for tech CoC (T280886) (duration: 00m 56s)

9 May 2021

  • curprev 21:4421:44, 9 May 2021imported>Stashbot 98,988 bytes −1,654,362 legoktm: restarted mailman3 again (T282348) pymysql.err.InternalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

8 May 2021

  • curprev 17:1817:18, 8 May 2021imported>Stashbot 1,753,350 bytes +105 Amir1: starting upgrade of batch G of mailing lists (T280322)

7 May 2021

  • curprev 21:4021:40, 7 May 2021imported>Stashbot 1,753,245 bytes +7,667 legoktm: deleted education@ from MM3, didn't import properly

6 May 2021

  • curprev 23:5023:50, 6 May 2021imported>Stashbot 1,745,578 bytes +25,003 brennen@deploy1002: rebuilt and synchronized wikiversions files: Rollback group1 and group2 to 1.37.0-wmf.3 (T282193)
  • curprev 00:3500:35, 6 May 2021imported>Stashbot 1,720,575 bytes +28,723 Amir1: sudo service mailman3-web restart

5 May 2021

  • curprev 01:4501:45, 5 May 2021imported>Stashbot 1,691,852 bytes +24,478 ryankemper: T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1011.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph` on `ryankemper@cumin1001` tmux session `reimage`

3 May 2021

  • curprev 23:1823:18, 3 May 2021imported>Stashbot 1,667,374 bytes +11,155 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 230ef5716b34ca83348667f289180313b76ce8a3: Prepare for new configuration option (T277951) (duration: 00m 57s)

2 May 2021

  • curprev 13:4013:40, 2 May 2021imported>Stashbot 1,656,219 bytes +311 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cloudmetrics1002.eqiad.wmnet with reason: Flaky host

1 May 2021

  • curprev 19:1219:12, 1 May 2021imported>Stashbot 1,655,908 bytes +557 Urbanecm: Invalidate password for MaraBot@SUL (T281586)

30 April 2021

  • curprev 21:5521:55, 30 April 2021imported>Stashbot 1,655,351 bytes +6,117 mutante: people1003 - rsycncing /home from peopel1002
  • curprev 01:0801:08, 30 April 2021imported>Stashbot 1,649,234 bytes +23,564 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw reboot - ryankemper@cumin1001 - T280563

29 April 2021

  • curprev 00:4100:41, 29 April 2021imported>Stashbot 1,625,670 bytes +24,040 tstarling@deploy1002: Synchronized php-1.37.0-wmf.3/includes/specials/pagers/ImageListPager.php: T281405 (duration: 01m 08s)

28 April 2021

  • curprev 01:0601:06, 28 April 2021imported>Stashbot 1,601,630 bytes +35,800 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

26 April 2021

  • curprev 23:2823:28, 26 April 2021imported>Stashbot 1,565,830 bytes +16,596 mutante: renewing TLS cert for peopleweb.discovery.wmnet, adding *3 hosts

25 April 2021

  • curprev 15:2315:23, 25 April 2021imported>Stashbot 1,549,234 bytes +138 Amir1: sudo -u list /var/lib/mailman/bin/change_pw -l wikica-l -p $(pwgen -c1 -s 12) (T281066)

24 April 2021

23 April 2021

22 April 2021

  • curprev 17:2617:26, 22 April 2021imported>Stashbot 1,545,052 bytes +1,074 marostegui: Stop mysql on tendril/dbtree database
  • curprev 01:3401:34, 22 April 2021imported>Stashbot 1,543,978 bytes +14,300 eileen: civicrm revision changed from 35a8dd33ba to 42ca3cf65a, config revision is cf07e7ba0b

21 April 2021

  • curprev 00:3800:38, 21 April 2021imported>Stashbot 1,529,678 bytes +27,920 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE

19 April 2021

  • curprev 22:5622:56, 19 April 2021imported>Stashbot 1,501,758 bytes +29,254 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE

18 April 2021

  • curprev 06:4006:40, 18 April 2021imported>Stashbot 1,472,504 bytes +106 Amir1: cleaning watchlist of User:Mr._Ibrahem in wikidatawiki (in main ns only)

17 April 2021

  • curprev 16:1616:16, 17 April 2021imported>Stashbot 1,472,398 bytes +61 Amir1: cleaning SuccuBot's watchlist in wikidatawiki
  • curprev 00:5300:53, 17 April 2021imported>Stashbot 1,472,337 bytes +38,446 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet

15 April 2021

  • curprev 01:1901:19, 15 April 2021imported>Stashbot 1,433,891 bytes +21,435 Amir1: mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P8671 --new-data-type external-id (T278427)

14 April 2021

  • curprev 00:1100:11, 14 April 2021imported>Stashbot 1,412,456 bytes +25,158 legoktm@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw2411.codfw.wmnet

12 April 2021

  • curprev 23:2523:25, 12 April 2021imported>Stashbot 1,387,298 bytes +4,421 krinkle@deploy1002: Synchronized wmf-config/mc.php: I390b4726d01037107 (duration: 00m 58s)

10 April 2021

  • curprev 14:2114:21, 10 April 2021imported>Stashbot 1,382,877 bytes +815 andrew@deploy1002: Finished deploy [horizon/deploy@ee1be56]: fix for T279699 (duration: 04m 12s)

9 April 2021

  • curprev 14:0714:07, 9 April 2021imported>Stashbot 1,382,062 bytes +59 jynus: retry es4 backup dump on eqiad (backup1002)
  • curprev 01:2501:25, 9 April 2021imported>Stashbot 1,382,003 bytes +23,659 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on moss-be2002.codfw.wmnet with reason: REIMAGE

7 April 2021

  • curprev 23:3823:38, 7 April 2021imported>Stashbot 1,358,344 bytes +22,437 ejegg: updated payments-wiki from b06009c099 to 70f5163816,
  • curprev 01:4601:46, 7 April 2021imported>Stashbot 1,335,907 bytes +16,883 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: REIMAGE

5 April 2021

  • curprev 23:1723:17, 5 April 2021imported>Stashbot 1,319,024 bytes +12,454 AaronSchulz: Running importMissingLocalNames.php on mwmaint1002 in a screen

4 April 2021

  • curprev 14:4714:47, 4 April 2021imported>Stashbot 1,306,570 bytes +265 andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 01m 36s)

3 April 2021

  • curprev 19:2019:20, 3 April 2021imported>Stashbot 1,306,305 bytes +1,408 andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 02m 11s)

2 April 2021

  • curprev 22:3122:31, 2 April 2021imported>Stashbot 1,304,897 bytes +10,597 bstorm@cumin1001: END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0)

1 April 2021

  • curprev 23:3223:32, 1 April 2021imported>Stashbot 1,294,300 bytes +18,843 thcipriani@deploy1002: Synchronized php-1.36.0-wmf.37/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: Backport: Revert "Turn on glent m1 AB test" T262612 (duration: 00m 58s)
  • curprev 00:5600:56, 1 April 2021imported>Stashbot 1,275,457 bytes +7,735 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2381.codfw.wmnet with reason: REIMAGE

30 March 2021

  • curprev 23:5923:59, 30 March 2021imported>Stashbot 1,267,722 bytes +13,942 Trey314159: reindexing English wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T274200)

29 March 2021

  • curprev 19:0619:06, 29 March 2021imported>Stashbot 1,253,780 bytes +3,056 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=aqs1004.eqiad.wmnet

27 March 2021

26 March 2021

25 March 2021

  • curprev 23:4723:47, 25 March 2021imported>Stashbot 1,243,504 bytes +12,592 thcipriani@deploy1002: Synchronized php-1.36.0-wmf.36/extensions/3D/package.json: No-op demo sync (duration: 01m 07s)
  • curprev 01:3301:33, 25 March 2021imported>Stashbot 1,230,912 bytes +22,060 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)

24 March 2021

  • curprev 00:2200:22, 24 March 2021imported>Stashbot 1,208,852 bytes +46,207 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2378.codfw.wmnet with reason: REIMAGE

23 March 2021

  • curprev 00:0800:08, 23 March 2021imported>Stashbot 1,162,645 bytes +17,284 tstarling@deploy1002: Synchronized wmf-config: use RequestTimeout library step 3: clean up (duration: 00m 58s)

21 March 2021

  • curprev 10:2510:25, 21 March 2021imported>Stashbot 1,145,361 bytes +368 _joe_: restarting gerrit on gerrit1001, using 45G of reserved memory

20 March 2021

  • curprev 00:2200:22, 20 March 2021imported>Stashbot 1,144,993 bytes +6,348 tzatziki: altering emails for STei (WMF) and SGrabarczuk (WMF)

19 March 2021

  • curprev 00:5100:51, 19 March 2021imported>Stashbot 1,138,645 bytes +9,921 dancy@deploy1002: Synchronized php-1.36.0-wmf.35/extensions/LiquidThreads/classes/Thread.php: T277772 (duration: 00m 58s)

18 March 2021

  • curprev 00:0500:05, 18 March 2021imported>Stashbot 1,128,724 bytes +22,211 eileen: tools revision changed from b7b4060c30 to ef54260b0d

16 March 2021

  • curprev 23:5623:56, 16 March 2021imported>Stashbot 1,106,513 bytes +44,018 krinkle@deploy1002: Synchronized php-1.36.0-wmf.35/includes/Revision/: I8619ab9e92b, T277362, T275531 (duration: 00m 58s)

15 March 2021

  • curprev 23:3123:31, 15 March 2021imported>Stashbot 1,062,495 bytes +21,243 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Remove back-compat from when IRC feed servers was a string (T224579) (duration: 00m 59s)

14 March 2021

  • curprev 17:5717:57, 14 March 2021imported>Stashbot 1,041,252 bytes +1,114 marostegui@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Repool db1146:3314', diff saved to https://phabricator.wikimedia.org/P14827 and previous config saved to /var/cache/conftool/dbconfig/20210314-175751-root.json

13 March 2021

  • curprev 19:0219:02, 13 March 2021imported>Stashbot 1,040,138 bytes +699 Amir1: change default charset of all core tables in labstestwiki to binary (T269348)
  • curprev 00:5500:55, 13 March 2021imported>Stashbot 1,039,439 bytes +10,754 mutante: [wdqs1009:/etc/envoy] $ sudo /usr/local/sbin/build-envoy-config -c /etc/envoy/

12 March 2021

  • curprev 01:3001:30, 12 March 2021imported>Stashbot 1,028,685 bytes +31,013 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2215.codfw.wmnet

11 March 2021

  • curprev 00:5000:50, 11 March 2021imported>Stashbot 997,672 bytes +37,248 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2219.codfw.wmnet

10 March 2021

  • curprev 01:0801:08, 10 March 2021imported>Stashbot 960,424 bytes +27,385 krinkle@deploy1002: Synchronized php-1.36.0-wmf.34/extensions/NavigationTiming/modules/ext.navigationTiming.js: T276826 Ibd9ddf14d64 (duration: 01m 14s)

9 March 2021

  • curprev 00:5800:58, 9 March 2021imported>Stashbot 933,039 bytes +16,107 Krinkle: krinkle@mwmaint1002 Ran invalidateUserSesssions.php for one user

7 March 2021

  • curprev 08:0108:01, 7 March 2021imported>Stashbot 916,932 bytes +170 elukey: "megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll" on analytics1066 - BBU looks fine, but the raid controller was using WriteThrough

5 March 2021

  • curprev 23:1623:16, 5 March 2021imported>Stashbot 916,762 bytes +8,775 legoktm: imported pygments 2.8.0+dfsg-1 to apt.wm.o buster-wikimedia component/pygments (T276298)
  • curprev 00:5900:59, 5 March 2021imported>Stashbot 907,987 bytes +16,215 legoktm: depooled registry1001/registry1002 (old stretch VMs) - T272550

4 March 2021

3 March 2021

  • curprev 00:4200:42, 3 March 2021imported>Stashbot 861,710 bytes +24,132 Urbanecm: Finished deployment in Evening B&C window; logmsgbot is currently down, and a simple restart did not bring it back up

2 March 2021

  • curprev 00:5900:59, 2 March 2021imported>Stashbot 837,578 bytes +21,612 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 0f08e8bbe1f74220a7a01a7606b67e0f75734a53: Update the Persian Wikipedia logos (T261033; 2/2) (duration: 00m 56s)

28 February 2021

27 February 2021

  • curprev 21:1921:19, 27 February 2021imported>Stashbot 815,895 bytes +260 dwisehaupt: ran the following on frdb2002 to allow replication to continue after conversion to utf8mb4 charset: set global slave_type_conversions = ALL_NON_LOSSY;
  • curprev 00:0800:08, 27 February 2021imported>Stashbot 815,635 bytes +26,358 mutante: deploy1002 - rsyncing home dirs from deploy1001

26 February 2021

  • curprev 00:1400:14, 26 February 2021imported>Stashbot 789,277 bytes +25,251 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.32/extensions/Graph/: 9d5cf348f5dda32f8889d5160bb1fe34a4e07f8c: Do not log graph errors to WMF servers (T274557) (duration: 01m 36s)

25 February 2021

  • curprev 00:2900:29, 25 February 2021imported>Stashbot 764,026 bytes +22,501 ryankemper: T274204 Restored service health on `elastic106[0,4,5]` via `sudo apt-get remove --purge wmf-elasticsearch-search-plugins --yes && sudo dpkg -i /var/cache/apt/archives/wmf-elasticsearch-search-plugins_6.5.4-4~stretch_all.deb && sudo puppet agent -tv`. There's some sort of issue with `6.5.4-5~stretch` that we will need to circle back and investigate; for now the fleet is staying on `6.5.4-4~stretch`

24 February 2021

  • curprev 00:5800:58, 24 February 2021imported>Stashbot 741,525 bytes +17,976 volker-e@deploy1001: Finished deploy [design/style-guide@a66b5b6]: Deploy design/style-guide: a66b5b6 “Components”: Add “Dialogs” (#430) (duration: 00m 06s)

23 February 2021

  • curprev 00:4600:46, 23 February 2021imported>Stashbot 723,549 bytes +23,528 eileen: civicrm revision changed from c535ac603a to 5e042e6e57, config revision is ef64f705bb

21 February 2021

  • curprev 16:0316:03, 21 February 2021imported>Stashbot 700,021 bytes +990 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1162 - crashed', diff saved to https://phabricator.wikimedia.org/P14424 and previous config saved to /var/cache/conftool/dbconfig/20210221-160258-marostegui.json

20 February 2021

  • curprev 00:1700:17, 20 February 2021imported>Stashbot 699,031 bytes +13,319 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1317.eqiad.wmnet

19 February 2021

18 February 2021

  • curprev 01:3801:38, 18 February 2021imported>Stashbot 677,805 bytes +16,614 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

17 February 2021

  • curprev 01:0001:00, 17 February 2021imported>Stashbot 661,191 bytes +18,673 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mwdebug1001.eqiad.wmnet

15 February 2021

  • curprev 21:3321:33, 15 February 2021imported>Stashbot 642,518 bytes +7,693 eileen: civicrm revision changed from dfbb8f41bc to c535ac603a, config revision is ba9b2380b1

14 February 2021

  • curprev 13:1313:13, 14 February 2021imported>Stashbot 634,825 bytes +806 akosiaris: sudo cumin -b 1 -s 120 'cp500[2,3,5,6].eqsin.wmnet' 'systemctl restart varnish-frontend.service'

13 February 2021

  • curprev 03:2403:24, 13 February 2021imported>Stashbot 634,019 bytes +232 ryankemper: Depooled `wdqs1006` to catch up on lag
  • curprev 01:0001:00, 13 February 2021imported>Stashbot 633,787 bytes +21,925 crusnov@cumin1001: START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet

12 February 2021

  • curprev 01:2101:21, 12 February 2021imported>Stashbot 611,862 bytes +30,779 legoktm@cumin1001: conftool action : set/pooled=yes; selector: name=mw1329.eqiad.wmnet

11 February 2021

  • curprev 00:2600:26, 11 February 2021imported>Stashbot 581,083 bytes +30,413 legoktm@deploy1001: Synchronized wmf-config/profiler.php: Revert "profiler: Send data to excimer-buster pipeline" (duration: 02m 00s)

10 February 2021

9 February 2021

  • curprev 00:4800:48, 9 February 2021imported>Stashbot 514,235 bytes +25,431 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw1302.eqiad.wmnet

7 February 2021

6 February 2021

  • curprev 08:5908:59, 6 February 2021imported>Stashbot 488,709 bytes +1,136 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro-from-cdh-clients (exit_code=0) for Hadoop test cluster: Change Hadoop distribution - elukey@cumin1001
  • curprev 01:0001:00, 6 February 2021imported>Stashbot 487,573 bytes +17,493 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2265.codfw.wmnet

5 February 2021

  • curprev 00:5900:59, 5 February 2021imported>Stashbot 470,080 bytes +27,954 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@85713c1]: restore data range specifier in extract job partition spec

4 February 2021

  • curprev 01:1501:15, 4 February 2021imported>Stashbot 442,126 bytes +22,570 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@4b4872d]: transfer_to_es: Increase timeout waiting for source data to three hours (duration: 01m 16s)

3 February 2021

  • curprev 00:1600:16, 3 February 2021imported>Stashbot 419,556 bytes +21,008 jhuneidi@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .

1 February 2021

  • curprev 23:5423:54, 1 February 2021imported>Stashbot 398,548 bytes +28,951 legoktm@deploy1001: Synchronized wmf-config/profiler.php: profiler: Send data to excimer-buster pipeline (T273312) (duration: 00m 57s)

29 January 2021

  • curprev 23:2623:26, 29 January 2021imported>Stashbot 369,597 bytes +9,826 razzi@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001
  • curprev 00:1900:19, 29 January 2021imported>Stashbot 359,771 bytes +37,885 legoktm@cumin1001: conftool action : set/pooled=yes; selector: name=mw2261.codfw.wmnet

28 January 2021

  • curprev 00:5600:56, 28 January 2021imported>Stashbot 321,886 bytes +25,610 legoktm@cumin1001: conftool action : set/pooled=no; selector: name=mw2294.codfw.wmnet

27 January 2021

  • curprev 00:5400:54, 27 January 2021imported>Stashbot 296,276 bytes +26,066 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2296.codfw.wmnet with reason: REIMAGE

26 January 2021

  • curprev 00:4800:48, 26 January 2021imported>Stashbot 270,210 bytes +18,727 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2331.codfw.wmnet

23 January 2021

  • curprev 22:2122:21, 23 January 2021imported>Stashbot 251,483 bytes +1,766 volker-e@deploy1001: Finished deploy [design/style-guide@63e39e7]: Deploy design/style-guide: 63e39e7 “Components”: Amend button groups states SVG font stack (#427) (duration: 00m 06s)
  • curprev 00:4600:46, 23 January 2021imported>Stashbot 249,717 bytes +27,911 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch enwiki to use enwiki20 "Option A" logo variant (T272526) (duration: 00m 57s)

22 January 2021

21 January 2021

  • curprev 01:2701:27, 21 January 2021imported>Stashbot 205,931 bytes +34,902 ryankemper: [WDQS Deploy] Rollback complete, service health of `wdqs1003` is restored. Need to investigate source of 404 (possibly related to some recent changes we made in the `gui` repo)

20 January 2021

18 January 2021

  • curprev 21:3321:33, 18 January 2021imported>Stashbot 147,203 bytes +19,595 eileen: civicrm revision changed from 4220fc8177 to a4caad22b1, config revision is f08249ecf9

16 January 2021

  • curprev 12:1812:18, 16 January 2021imported>Stashbot 127,608 bytes +369 elukey: elukey@cumin1001:~$ sudo cumin 'A:mw-app-canary and A:mw-eqiad' 'run-puppet-agent' -b 10 - T272215

15 January 2021

  • curprev 23:5923:59, 15 January 2021imported>Stashbot 127,239 bytes +6,731 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE

14 January 2021

  • curprev 23:3523:35, 14 January 2021imported>Stashbot 120,508 bytes +16,149 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2236.codfw.wmnet
  • curprev 00:2200:22, 14 January 2021imported>Stashbot 104,359 bytes +24,462 ryankemper: T266492 Restart of `relforge` successful

12 January 2021

  • curprev 22:5522:55, 12 January 2021imported>Stashbot 79,897 bytes +10,636 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2225.codfw.wmnet

11 January 2021

  • curprev 22:1622:16, 11 January 2021imported>Stashbot 69,261 bytes +16,348 eileen: process-control config revision is f08249ecf9 eoy jobs disabled

10 January 2021

9 January 2021

  • curprev 00:1100:11, 9 January 2021imported>Stashbot 208,808 bytes +13,848 mutante: puppetmaster2003 - restarted apache after spweing 500s

8 January 2021

  • curprev 01:2901:29, 8 January 2021imported>Stashbot 194,960 bytes +13,843 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1267.eqiad.wmnet

7 January 2021

  • curprev 00:5000:50, 7 January 2021imported>Stashbot 181,117 bytes +8,175 eileen: civicrm revision changed from 95fefcb421 to 2df572bdcd, config revision is 603c0ff4a0

6 January 2021

  • curprev 01:2201:22, 6 January 2021imported>Stashbot 172,942 bytes +9,241 mutante: testreduce1001 rm -rf /srv/deployment/parsoid/deploy

5 January 2021

  • curprev 00:4000:40, 5 January 2021imported>Stashbot 163,701 bytes +5,675 jgleeson: updated civicrm revision changed from bb8baac617 to 6be8a130df

3 January 2021

  • curprev 16:1716:17, 3 January 2021imported>Stashbot 158,026 bytes +627 arturo: merged change to TLS cert used by slapd/openldap servers https://gerrit.wikimedia.org/r/c/operations/puppet/+/653871

2 January 2021

1 January 2021

  • curprev 14:4914:49, 1 January 2021imported>Stashbot 157,325 bytes +1,317 milimetric@deploy1001: Finished deploy [analytics/refinery@f9281dd] (thin): [SAFE, IGNORE] Simple hotfix for a python bug, analytics refinery only, not urgent (duration: 00m 07s)

29 December 2020

  • curprev 15:5215:52, 29 December 2020imported>Stashbot 156,008 bytes +210 vgutierrez: reloading nginx on cloudelastic1005 and cloudelastic1006

28 December 2020

  • curprev 09:5409:54, 28 December 2020imported>Stashbot 155,798 bytes +134 elukey: reboot an-coord1002 (puppet in D state after issues with broken disk - host in standby, no traffic)

24 December 2020

22 December 2020

  • curprev 21:5721:57, 22 December 2020imported>Stashbot 154,236 bytes +2,451 mutante: apt1001 - sudo systemctl status rsync-aptrepo-apt2001.wikimedia.org.service - confirmed timer job is working like the cron before
  • curprev 00:2000:20, 22 December 2020imported>Stashbot 151,785 bytes +6,451 crusnov@deploy1001: Finished deploy [netbox/deploy@b17db99]: Redeploy of 2.9.10 to netbox-dev for dep test (duration: 00m 54s)

18 December 2020

17 December 2020

15 December 2020

  • curprev 23:4723:47, 15 December 2020imported>Stashbot 109,966 bytes +14,076 dduvall@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
  • curprev 00:1300:13, 15 December 2020imported>Stashbot 95,890 bytes +6,424 ejegg: updated payments-wiki from 63ae7413a8 to 3d3055c478

11 December 2020

  • curprev 22:0522:05, 11 December 2020imported>Stashbot 89,466 bytes +7,632 dduvall@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .

10 December 2020

  • curprev 23:3523:35, 10 December 2020imported>Stashbot 81,834 bytes +10,617 Urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript resetAuthenticationThrottle.php --wiki=enwiki --login --ip 'REDACTED' --user 'WP 1.0 bot' # T269898
  • curprev 00:4200:42, 10 December 2020imported>Stashbot 71,217 bytes −965,466 ejegg: updated payments-wiki from 756c2f7ce0 to df80a99b40

9 December 2020

  • curprev 00:5300:53, 9 December 2020imported>Stashbot 1,036,683 bytes +9,549 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.20/vendor/: 3278ffd107888757c4620383160a6d5fa67d05b5: Bump wikimedia/parsoid to v0.13.0-a19 (T269685) (duration: 01m 16s)

8 December 2020

  • curprev 01:4401:44, 8 December 2020imported>Stashbot 1,027,134 bytes +11,262 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

5 December 2020

4 December 2020

  • curprev 01:4301:43, 4 December 2020imported>Stashbot 1,009,666 bytes +11,577 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

2 December 2020

  • curprev 23:4223:42, 2 December 2020imported>Stashbot 998,089 bytes +10,158 reedy@deploy1001: Synchronized php-1.36.0-wmf.20/includes/debug/logger/monolog/LogstashFormatter.php: T269286 (duration: 01m 07s)
  • curprev 00:5800:58, 2 December 2020imported>Stashbot 987,931 bytes +14,689 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-reload (exit_code=0)

1 December 2020

  • curprev 01:1801:18, 1 December 2020imported>Stashbot 973,242 bytes +18,284 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

27 November 2020

  • curprev 17:3017:30, 27 November 2020imported>Stashbot 954,958 bytes +2,219 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

26 November 2020

25 November 2020

24 November 2020

  • curprev 00:4200:42, 24 November 2020imported>Stashbot 912,325 bytes +9,753 reedy@deploy1001: Synchronized php-1.36.0-wmf.16/extensions/SecurePoll/includes/Crypt/GpgCrypt.php: Unbreak gpg encrypted polls T268583 (duration: 01m 05s)

21 November 2020

  • curprev 09:1809:18, 21 November 2020imported>Stashbot 902,572 bytes +604 joal: Drop historical logs of 'Wikidata Concepts Monitor ETL' on HDFS keeping one example - freeing 60Tb

20 November 2020

  • curprev 23:3823:38, 20 November 2020imported>Stashbot 901,968 bytes +7,656 mutante: synced puppet-compiler facts - new hosts should be usable in compiler

19 November 2020

  • curprev 23:5923:59, 19 November 2020imported>Stashbot 894,312 bytes +11,241 razzi@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)

18 November 2020

16 November 2020

  • curprev 23:2823:28, 16 November 2020imported>Stashbot 839,536 bytes +8,318 mutante: cumin1001 - sudo systemctl start cumin-check-aliases (to confirm switching cron to timer worked) T265138
  • curprev 00:5500:55, 16 November 2020imported>Stashbot 831,218 bytes +2,135 shdubsh: re-applied mask to kafka and kafka-mirror-main-eqiad_to_main-codfw@0 on kafka-main2003 and disabled puppet to prevent restart - T267865

13 November 2020

  • curprev 22:0622:06, 13 November 2020imported>Stashbot 829,083 bytes +1,249 Urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=myvwiki autopatrolled # T105570
  • curprev 01:3001:30, 13 November 2020imported>Stashbot 827,834 bytes +2,596 TimStarling: on mwmaint1002 running fixT260485.php unmerged fixup script from https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMaintenance/+/640348

11 November 2020

  • curprev 16:4416:44, 11 November 2020imported>Stashbot 825,238 bytes +1,480 XioNoX: Revert "temporarily route Italy to codfw"
  • curprev 00:4800:48, 11 November 2020imported>Stashbot 823,758 bytes +6,737 ryankemper: Restarting `wdqs-categories` one host at a time across all wdqs production instances: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 60 && systemctl restart wdqs-categories && sleep 30 && pool'`

9 November 2020

  • curprev 22:2622:26, 9 November 2020imported>Stashbot 817,021 bytes +7,948 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

8 November 2020

  • curprev 23:0823:08, 8 November 2020imported>Stashbot 809,073 bytes +654 tstarling@deploy1001: Synchronized php-1.36.0-wmf.16/resources/src/mediawiki.api/upload.js: fixing UBN T266903 (duration: 01m 06s)

6 November 2020

  • curprev 23:3823:38, 6 November 2020imported>Stashbot 808,419 bytes +5,409 dwisehaupt: frdata1001 upgraded to buster
  • curprev 00:5300:53, 6 November 2020imported>Stashbot 803,010 bytes +18,393 brennen@deploy1001: Finished scap: Synchronizing to pick up i18n for gerrit:639505. Will resume moving train to group1 on Monday morning (US) (T263182) (duration: 69m 02s)

5 November 2020

  • curprev 01:4701:47, 5 November 2020imported>Stashbot 784,617 bytes +19,362 milimetric@deploy1001: Started deploy [analytics/refinery@6913407]: Regular analytics weekly train [analytics/refinery@6913407]

3 November 2020

1 November 2020

  • curprev 22:4122:41, 1 November 2020imported>Stashbot 747,823 bytes +627 Urbanecm: mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=metawiki Turkmen # T266976
  • curprev 01:2601:26, 1 November 2020imported>Stashbot 747,196 bytes +356 rzl@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

31 October 2020

  • curprev 00:1200:12, 31 October 2020imported>Stashbot 746,840 bytes +5,375 mutante: removed Nuria from wmf group, she is already in nda group (T266086)

30 October 2020

29 October 2020

  • curprev 01:4101:41, 29 October 2020imported>Stashbot 726,980 bytes +9,799 mutante: scandium reimaged a second time after making puppet changes to ensure nodejs/npm is NOT installed anymore (T257906)

28 October 2020

  • curprev 00:4000:40, 28 October 2020imported>Stashbot 717,181 bytes +10,760 eileen: civicrm revision changed from 4fdfb8408b to e1d65b0f3a, config revision is f16003ab62

26 October 2020

  • curprev 23:1223:12, 26 October 2020imported>Stashbot 706,421 bytes +13,445 catrope@deploy1001: Synchronized php-1.36.0-wmf.14/extensions/GrowthExperiments/: Fix JS error when no topics set (T266501) (duration: 01m 00s)

25 October 2020

23 October 2020

  • curprev 22:5622:56, 23 October 2020imported>Stashbot 692,840 bytes +1,858 mutante: added Nuria to "nda" LDAP group - leaving her in "wmf" until the actual last day - shell account remains so no puppet change needed in ldap_only_admins (T266086)

22 October 2020

  • curprev 22:4222:42, 22 October 2020imported>Stashbot 690,982 bytes +6,223 mutante: ganeti1001 - adding 2 more vcpus to VM testreduce1001 - T257940

21 October 2020

  • curprev 23:1623:16, 21 October 2020imported>Stashbot 684,759 bytes +8,846 catrope@deploy1001: Synchronized php-1.36.0-wmf.14/extensions/GrowthExperiments/: T266033 (duration: 01m 05s)

20 October 2020

  • curprev 22:1022:10, 20 October 2020imported>Stashbot 675,913 bytes +8,976 dwisehaupt: frmon2001 upgraded to buster with grafana 7.2.1

19 October 2020

  • curprev 23:5723:57, 19 October 2020imported>Stashbot 666,937 bytes +14,227 gehel@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)

17 October 2020

  • curprev 13:2213:22, 17 October 2020imported>Stashbot 652,710 bytes +173 Urbanecm: [urbanecm@mwmaint2001 ~/uploads]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Fæ . # T264529

16 October 2020

  • curprev 21:4621:46, 16 October 2020imported>Stashbot 652,537 bytes +3,177 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:0101:01, 16 October 2020imported>Stashbot 649,360 bytes +9,475 ryankemper: Cleaning up a dangling no-longer-puppet-managed udev elasticsearch-readahead rule across all cirrus instances: `sudo cumin -b 36 C:profile::elasticsearch::cirrus 'sudo rm -fv /etc/udev/rules.d/elasticsearch-readahead.rules && sudo /sbin/udevadm control --reload && sudo /sbin/udevadm trigger'`

15 October 2020

  • curprev 00:3600:36, 15 October 2020imported>Stashbot 639,885 bytes +8,944 ryankemper: Beginning rolling upgrade for cirrussearch `eqiad`. Cookbook will restart elasticsearch on 36 nodes total, 3 nodes at a time

13 October 2020

  • curprev 23:2223:22, 13 October 2020imported>Stashbot 630,941 bytes +8,939 catrope@deploy1001: Synchronized php-1.36.0-wmf.13/extensions/GrowthExperiments/: Revert removal of variant A (T265372) (duration: 01m 04s)

12 October 2020

  • curprev 17:0317:03, 12 October 2020imported>Stashbot 622,002 bytes +1,861 jayme: fixed /var/lock/ permission (1777) on ms-be2036 - T265208

10 October 2020

9 October 2020

  • curprev 00:1600:16, 9 October 2020imported>Stashbot 611,953 bytes +10,880 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)

8 October 2020

7 October 2020

  • curprev 01:0001:00, 7 October 2020imported>Stashbot 591,948 bytes +13,893 cdanis: repool esams; cr2-esams router upgrade complete

5 October 2020

  • curprev 23:1123:11, 5 October 2020imported>Stashbot 578,055 bytes +8,516 ejegg: updated payments staging from 52704ffe24 to db03677b2d

3 October 2020

  • curprev 15:5215:52, 3 October 2020imported>Stashbot 569,539 bytes +246 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: emergency: 840545f1d9115ea6b672cecce1762d850d8b1f54: Restrict flow-hide right to autoconfirmed users on zhwiki (T264489) (duration: 01m 17s)
  • curprev 00:0800:08, 3 October 2020imported>Stashbot 569,293 bytes +9,618 ejegg: updated fundraising CiviCRM from 256adda03c to a30da7f92a

1 October 2020

  • curprev 23:3823:38, 1 October 2020imported>Stashbot 559,675 bytes +14,499 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% (duration: 00m 34s)

30 September 2020

29 September 2020

27 September 2020

26 September 2020

24 September 2020

  • curprev 23:3923:39, 24 September 2020imported>Stashbot 493,213 bytes +19,136 andrew@deploy1001: Finished deploy [horizon/deploy@7b61460]: (no justification provided) (duration: 01m 58s)

23 September 2020

  • curprev 23:5223:52, 23 September 2020imported>Stashbot 474,077 bytes +20,309 mutante: alert1001 - systemctl restar ircecho because icinga-wm left the chat

22 September 2020

  • curprev 23:2723:27, 22 September 2020imported>Stashbot 453,768 bytes +20,420 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: clientError: enable on ja,es,de,ru,it,zh,pt wikipedias (T255585) (duration: 01m 04s)

21 September 2020

  • curprev 23:4223:42, 21 September 2020imported>Stashbot 433,348 bytes +16,298 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' .

20 September 2020

  • curprev 08:4608:46, 20 September 2020imported>Stashbot 417,050 bytes +345 Urbanecm: [urbanecm@mwmaint2001 ~]$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Tepig10102020' 'Davidfromtheworld' # T263317

19 September 2020

  • curprev 19:0319:03, 19 September 2020imported>Stashbot 416,705 bytes +407 ariel@deploy1001: Finished deploy [dumps/dumps@14ba6e9]: defer getting db creds until really needed (duration: 00m 04s)

18 September 2020

17 September 2020

16 September 2020

  • curprev 23:4123:41, 16 September 2020imported>Stashbot 389,397 bytes +13,843 catrope@deploy1001: Synchronized php-1.36.0-wmf.8/extensions/FlaggedRevs: T262970 (duration: 01m 06s)

15 September 2020

  • curprev 23:2023:20, 15 September 2020imported>Stashbot 375,554 bytes +10,278 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.9/extensions/FlaggedRevs/backend/FlaggedRevsHooks.php: 1c0b0d161fe1024d6d08a27bbacf5b62c56c9c01: Fix APCOND_FR_NEVERBLOCKED handling (T262970) (duration: 00m 56s)

14 September 2020

  • curprev 22:5922:59, 14 September 2020imported>Stashbot 365,276 bytes +15,318 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' .

13 September 2020

  • curprev 23:4723:47, 13 September 2020imported>Stashbot 349,958 bytes +218 Urbanecm: Change email address of User:Oversight@enwiki to oversight-l@lists.wikimedia.org as part of OTRS downtime preparation (T262733)

12 September 2020

11 September 2020

  • curprev 00:4100:41, 11 September 2020imported>Stashbot 345,704 bytes +10,447 milimetric@deploy1001: Finished deploy [analytics/refinery@7f5a6ca] (thin): Regular analytics weekly train THIN [analytics/refinery@7f5a6ca] (duration: 00m 08s)

10 September 2020

  • curprev 00:3100:31, 10 September 2020imported>Stashbot 335,257 bytes +14,554 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@00b0e20]: Update to current master (duration: 06m 42s)

8 September 2020

  • curprev 23:4723:47, 8 September 2020imported>Stashbot 320,703 bytes +17,255 eileen: civicrm revision is 4e40a59d42, config revision is d26334fa36

7 September 2020

6 September 2020

  • curprev 19:4519:45, 6 September 2020imported>Stashbot 294,441 bytes +598 marostegui@cumin1001: dbctl commit (dc=all): 'Decrease db2127's weight a bit', diff saved to https://phabricator.wikimedia.org/P12496 and previous config saved to /var/cache/conftool/dbconfig/20200906-194512-marostegui.json

5 September 2020

4 September 2020

  • curprev 01:3501:35, 4 September 2020imported>Stashbot 289,064 bytes +20,066 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

2 September 2020

1 September 2020

  • curprev 22:3922:39, 1 September 2020imported>Stashbot 257,665 bytes +5,601 Urbanecm: [urbanecm@mwmaint2001 ~]$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=sysop_itwiki Pierpao (T261722)
  • curprev 00:0200:02, 1 September 2020imported>Stashbot 252,064 bytes +8,872 mutante: wb2-grrrri was not running and wikibugs had no more Gerrit updates since a while

30 August 2020

29 August 2020

  • curprev 18:0518:05, 29 August 2020imported>Stashbot 243,128 bytes +402 Amir1: end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T261451)

28 August 2020

  • curprev 21:5321:53, 28 August 2020imported>Stashbot 242,726 bytes +5,179 ryankemper: `sudo systemctl reload nginx.service` on `cloudelastic100[5,6].wikimedia.org` to try to resolve certificate warning issues
  • curprev 00:0600:06, 28 August 2020imported>Stashbot 237,547 bytes +27,285 eileen: process-control config revision is dd541a25dc

26 August 2020

  • curprev 23:3523:35, 26 August 2020imported>Stashbot 210,262 bytes +10,539 eileen: civicrm revision changed from d2e80f7522 to 481ab742db, config revision is e2ab4d7c1f

25 August 2020

  • curprev 21:5121:51, 25 August 2020imported>Stashbot 199,723 bytes +10,845 mutante: xhgui1001/xhgui2001 - Unpacking xhgui (0.12.0-2-wmf1) over (0.9.0-1-wmf1) (T260397)
  • curprev 00:5700:57, 25 August 2020imported>Stashbot 188,878 bytes +12,712 cdanis: T259621 ❌cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 homer 'cr*' commit 'drain cr2-esams transport link'

23 August 2020

  • curprev 20:3820:38, 23 August 2020imported>Stashbot 176,166 bytes +518 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

22 August 2020

  • curprev 19:3319:33, 22 August 2020imported>Stashbot 175,648 bytes +367 ryankemper: depooled wdqs1006 (still has 2.5 hours to catch up on)

21 August 2020

  • curprev 17:4317:43, 21 August 2020imported>Stashbot 175,281 bytes +1,663 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

20 August 2020

  • curprev 22:3122:31, 20 August 2020imported>Stashbot 173,618 bytes +10,495 eileen: civicrm revision changed from 27d5900f7d to ce28723709, config revision is 706cf3c898
  • curprev 00:5100:51, 20 August 2020imported>Stashbot 163,123 bytes +12,052 mutante: ms-be1039 - started failed ferm service

19 August 2020

  • curprev 01:0501:05, 19 August 2020imported>Stashbot 151,071 bytes +12,992 dpifke@deploy1001: Synchronized wmf-config/profiler.php: Deploying https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/620139 (duration: 01m 18s)

18 August 2020

  • curprev 00:5300:53, 18 August 2020imported>Stashbot 138,079 bytes +29,239 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)

16 August 2020

15 August 2020

  • curprev 21:1821:18, 15 August 2020imported>Stashbot 108,768 bytes +123 gehel: depooling wdqs1004 and restarting services, will wait to catch up on lag before repooling

14 August 2020

13 August 2020

  • curprev 23:3923:39, 13 August 2020imported>Stashbot 107,903 bytes +7,860 tzatziki: removing 3 files for legal compliance
  • curprev 00:3700:37, 13 August 2020imported>Stashbot 100,043 bytes +31,681 mutante: removing jenkins_service_running checks from secondary servers where it's stopped, manually from icinga config, running puppet on icinga

11 August 2020

  • curprev 23:4123:41, 11 August 2020imported>Stashbot 68,362 bytes +9,831 Urbanecm: Evening B&C window completed
  • curprev 00:3300:33, 11 August 2020imported>Stashbot 58,531 bytes +11,201 dpifke@deploy1001: Finished deploy [performance/arc-lamp@fc5f1c6]: Deploying latest attempt to fix T259167 (duration: 01m 03s)

9 August 2020

  • curprev 21:5821:58, 9 August 2020imported>Stashbot 47,330 bytes +195 ejegg: updated payments-wiki from cd012f37f1 to 932aacde54

8 August 2020

  • curprev 02:2302:23, 8 August 2020imported>Stashbot 47,135 bytes +243 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-reload

7 August 2020

  • curprev 16:4216:42, 7 August 2020imported>Stashbot 46,892 bytes +2,081 jforrester@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/DiscussionTools/: T259855 Revert new reply API (duration: 01m 06s)

6 August 2020

  • curprev 23:2123:21, 6 August 2020imported>Stashbot 44,811 bytes +12,330 catrope@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/GrowthExperiments/: Fixes for WelcomeSurvey language question (T232410) (duration: 00m 59s)
  • curprev 00:3500:35, 6 August 2020imported>Stashbot 32,481 bytes +9,981 mutante: wtp2019 - reimaging - parsoid service does not work, unlike on all other wtp*, making sure it's clean

4 August 2020

  • curprev 22:4122:41, 4 August 2020imported>Stashbot 22,500 bytes +12,987 brennen: restarting php7.2-fpm on mw1404 for opcache issues

3 August 2020

  • curprev 23:4323:43, 3 August 2020imported>Stashbot 9,513 bytes +9,109 mutante: mwdebug1001 - temp installing apt-file for debugging an issue on mwmaint

1 August 2020

  • curprev 16:3016:30, 1 August 2020imported>Stashbot 404 bytes −726,353 Amir1: wikiadmin@10.64.32.197(avkwiki)> delete from site_identifiers; (T259122)

31 July 2020

  • curprev 23:4823:48, 31 July 2020imported>Stashbot 726,757 bytes +6,358 ejegg: updated payments-wiki from c365c136d2 to cd012f37f1
  • curprev 00:4700:47, 31 July 2020imported>Stashbot 720,399 bytes +10,664 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: disable lilypond execution again (duration: 01m 10s)

30 July 2020

  • curprev 00:5200:52, 30 July 2020imported>Stashbot 709,735 bytes +18,579 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

29 July 2020

  • curprev 00:5300:53, 29 July 2020imported>Stashbot 691,156 bytes +25,595 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1046.eqiad.wmnet

28 July 2020

  • curprev 00:1900:19, 28 July 2020imported>Stashbot 665,561 bytes +13,224 andrew@cumin1001: conftool action : set/pooled=inactive; selector: name=cloudcephmon1003.eqiad.wmnet

25 July 2020

  • curprev 12:4112:41, 25 July 2020imported>Stashbot 652,337 bytes +620 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1096:3315 into s5 api afte db1082 crashed T258336', diff saved to https://phabricator.wikimedia.org/P12041 and previous config saved to /var/cache/conftool/dbconfig/20200725-124104-marostegui.json
  • curprev 00:4700:47, 25 July 2020imported>Stashbot 651,717 bytes +3,102 mutante: ganeti - removing disk 3 (20G) from webperf1002. the disks are 0-indexed, so the ones actually mounted are 0 (50G) and 1 (300G) (T257931)

24 July 2020

  • curprev 01:1901:19, 24 July 2020imported>Stashbot 648,615 bytes +12,821 ejegg: updated payments-wiki from 31a3de1130 to c365c136d2

23 July 2020

  • curprev 00:2400:24, 23 July 2020imported>Stashbot 635,794 bytes +17,409 legoktm@deploy1001: Synchronized php-1.35.0-wmf.41/includes/: T258664: Revert "Add a new type of database to the installer from extension" (2/2) (duration: 01m 08s)

21 July 2020

  • curprev 23:3723:37, 21 July 2020imported>Stashbot 618,385 bytes +16,371 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump cirrus MLR models to latest (duration: 01m 06s)
  • curprev 00:0200:02, 21 July 2020imported>Stashbot 602,014 bytes +20,177 ryankemper: Began Elasticsearch reindex job on index `dewiki_content` across [`eqiad`, `codfw`, `cloudelastic`], on `rkemper@mwmaint1002` under tmux session `reindex`. Should complete in <24 hours

19 July 2020

18 July 2020

  • curprev 21:4121:41, 18 July 2020imported>Stashbot 581,291 bytes +539 shdubsh: restart logstash on logstash200[456]
  • curprev 00:1300:13, 18 July 2020imported>Stashbot 580,752 bytes +4,067 dpifke: Performing one-time expiration of ArcLamp files older than 40 days (normal retention is 45 days), to solve disk space issue until either Ganeti issue is solved or compressed logfile support is merged.

16 July 2020

  • curprev 22:1622:16, 16 July 2020imported>Stashbot 576,685 bytes +13,808 mutante: testreduce1001 manually git clone 'scandium' branch of integration/visualdiff into /srv/visualdiff (T257906)

15 July 2020

  • curprev 23:5423:54, 15 July 2020imported>Stashbot 562,877 bytes +11,781 eileen: tools revision changed from 7b6018a16e to 711d671600
  • curprev 01:4501:45, 15 July 2020imported>Stashbot 551,096 bytes +15,235 eileen: tools revision changed from a9e7dc1559 to 7b6018a16e

14 July 2020

  • curprev 01:2301:23, 14 July 2020imported>Stashbot 535,861 bytes +9,041 ryankemper: Started long-running Elasticsearch reindex of `eqiad`, `codfw`, and `cloudelastic`. tmux session `reindex` under `ryankemper` on `mwmaint1002`

11 July 2020

  • curprev 19:1619:16, 11 July 2020imported>Stashbot 526,820 bytes +852 qchris: Restarting Gerrit on gerrit1001 to switch to new gerrit.war and zuul plugin

10 July 2020

  • curprev 21:5221:52, 10 July 2020imported>Stashbot 525,968 bytes +7,988 ryankemper: Started long-running reindex of Elasticsearch indices in `eqiad`, `codfw`, and `dewiki` on `mwmaint1002` under tmux session `reindex` for user `ryankemper`
  • curprev 01:4401:44, 10 July 2020imported>Stashbot 517,980 bytes +10,370 mutante: LDAP - adding coka to wmde and nda (T257038)

9 July 2020

  • curprev 00:5800:58, 9 July 2020imported>Stashbot 507,610 bytes +15,943 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6c1b646e T256395"'

7 July 2020

  • curprev 22:4122:41, 7 July 2020imported>Stashbot 491,667 bytes +21,022 mutante: new Wikimedia Annual Report 2019 now available on annual.wikimedia.org
  • curprev 00:2200:22, 7 July 2020imported>Stashbot 470,645 bytes +11,069 cstone: civicrm revision changed from a48caf0f37 to d73ee2e73f

5 July 2020

  • curprev 21:5021:50, 5 July 2020imported>Stashbot 459,576 bytes +1,640 qchris: Restarting gerrit on gerrit1001 to pick up new war and jars.

4 July 2020

  • curprev 19:2319:23, 4 July 2020imported>Stashbot 457,936 bytes +722 qchris@deploy1001: Finished deploy [gerrit/gerrit@b78914b]: Bump gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1002 (duration: 00m 08s)

3 July 2020

  • curprev 21:4921:49, 3 July 2020imported>Stashbot 457,214 bytes +4,336 reedy@deploy1001: Synchronized php-1.35.0-wmf.39/extensions/Score/: Sync maintenance script (duration: 00m 58s)

2 July 2020

  • curprev 23:2223:22, 2 July 2020imported>Stashbot 452,878 bytes +4,683 jhuneidi@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
  • curprev 00:5300:53, 2 July 2020imported>Stashbot 448,195 bytes +11,373 eileen: tools revision changed from 806e2b4412 to e38f7a83d4

1 July 2020

  • curprev 00:1400:14, 1 July 2020imported>Stashbot 436,822 bytes +12,688 krinkle@deploy1001: Synchronized private/PrivateSettings.php: T254795 - Set $wmgXhguiDBuser and $wmgXhguiDBpasswor (duration: 01m 06s)

29 June 2020

  • curprev 23:2823:28, 29 June 2020imported>Stashbot 424,134 bytes +9,692 eileen: civicrm revision changed from 52a32f2d66 to 391d0fdf75, config revision is f1b4bdb7b7

28 June 2020

  • curprev 21:4321:43, 28 June 2020imported>Stashbot 414,442 bytes +271 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: no-op I56eb4a802 (duration: 00m 58s)

27 June 2020

26 June 2020

  • curprev 18:4218:42, 26 June 2020imported>Stashbot 411,866 bytes +4,536 robh: all ulsfo onsite work completed as of 30 minutes ago
  • curprev 01:4101:41, 26 June 2020imported>Stashbot 407,330 bytes +15,922 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6cc5f3e6 T256395"'

25 June 2020

  • curprev 00:4800:48, 25 June 2020imported>Stashbot 391,408 bytes +12,937 twentyafterfour: restart php-fpm on phab1001 to fix T256343

24 June 2020

  • curprev 00:3500:35, 24 June 2020imported>Stashbot 378,471 bytes +8,585 ejegg: restarted fundraising jobs on main CiviCRM box

22 June 2020

  • curprev 23:4123:41, 22 June 2020imported>Stashbot 369,886 bytes +10,612 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: touch for T247330 (duration: 00m 56s)

20 June 2020

  • curprev 22:5622:56, 20 June 2020imported>Stashbot 359,274 bytes +414 cdanis@cumin2001: dbctl commit (dc=all): 'db1088 seems to have crashed', diff saved to https://phabricator.wikimedia.org/P11611 and previous config saved to /var/cache/conftool/dbconfig/20200620-225624-cdanis.json

19 June 2020

  • curprev 18:1018:10, 19 June 2020imported>Stashbot 358,860 bytes +6,598 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump eventlogging_Test schema version to 1.1.0 to pick up client_dt - T238230 (duration: 00m 59s)

18 June 2020

  • curprev 22:3022:30, 18 June 2020imported>Stashbot 352,262 bytes +9,559 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on all wikis - T249261 (duration: 00m 56s)

17 June 2020

  • curprev 23:2523:25, 17 June 2020imported>Stashbot 342,703 bytes +17,347 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 0e7079d: Install DiscussionTools on all wikis (attempt 2) (T252264; T253943) (duration: 00m 56s)
  • curprev 01:4301:43, 17 June 2020imported>Stashbot 325,356 bytes +20,465 shdubsh: restart elasticsearch on logstash1011

16 June 2020

  • curprev 00:2800:28, 16 June 2020imported>Stashbot 304,891 bytes +12,502 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: limit HTTP client timeout T245170 (duration: 00m 56s)

14 June 2020

  • curprev 13:5113:51, 14 June 2020imported>Stashbot 292,389 bytes +113 qchris: Disabling puppet on gerrit1002 (test instance) to do some more upgrade testing

13 June 2020

  • curprev 21:1221:12, 13 June 2020imported>Stashbot 292,276 bytes +335 qchris: Enabling puppet on gerrit1002 (test instance). Done with testing for today.

12 June 2020

  • curprev 17:4417:44, 12 June 2020imported>Stashbot 291,941 bytes +6,714 herron: restarting logstash1011 elasticsearch instance
  • curprev 01:0201:02, 12 June 2020imported>Stashbot 285,227 bytes +14,139 ejegg: updated payments-wiki from aceddff8b5 to 5fd4eb1519

10 June 2020

  • curprev 23:5523:55, 10 June 2020imported>Stashbot 271,088 bytes +14,140 catrope@deploy1001: Synchronized php-1.35.0-wmf.36/includes/skins/SkinTemplate.php: T255073 (duration: 01m 07s)

9 June 2020

  • curprev 23:1823:18, 9 June 2020imported>Stashbot 256,948 bytes +12,465 Reedy: run namespaceDupes.php --fix for hiwikibooks T254012
  • curprev 00:3900:39, 9 June 2020imported>Stashbot 244,483 bytes +10,746 ejegg: updated payments-wiki from c1d14a5db7 to aceddff8b5

5 June 2020

  • curprev 16:4516:45, 5 June 2020imported>Stashbot 233,737 bytes +4,799 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@f7e4f78]: Upgrade to 1.24.0 (duration: 00m 11s)

4 June 2020

  • curprev 23:4523:45, 4 June 2020imported>Stashbot 228,938 bytes +11,913 catrope@deploy1001: Synchronized wmf-config/mc.php: Set coalesceKeys=non-global for WANCache on enwiki (duration: 00m 59s)
  • curprev 00:3600:36, 4 June 2020imported>Stashbot 217,025 bytes +14,843 reedy@deploy1001: Synchronized php-1.35.0-wmf.35/includes/specials/SpecialUserrights.php: T254417 T251534 (duration: 01m 06s)

3 June 2020

  • curprev 01:0001:00, 3 June 2020imported>Stashbot 202,182 bytes +10,631 reedy@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: wgLocalVirtualHosts (duration: 01m 06s)

1 June 2020

  • curprev 20:1420:14, 1 June 2020imported>Stashbot 191,551 bytes +5,297 shdubsh: downgrade mtail to rc5 in ulsfo -- T254192

31 May 2020

  • curprev 09:5609:56, 31 May 2020imported>Stashbot 186,254 bytes +199 Urbanecm: mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Vox Golf' 'Colonel Chicken' (T254068)

29 May 2020

  • curprev 22:3222:32, 29 May 2020imported>Stashbot 186,055 bytes +4,498 bstorm_: updated views on labsdb1010 T252219

28 May 2020

  • curprev 23:4823:48, 28 May 2020imported>Stashbot 181,557 bytes +7,100 jforrester@deploy1001: Synchronized php-1.35.0-wmf.34/skins/Vector/resources/skins.vector.styles/Menu.less: T253912 Hotfix: Cannot rename emptyPortlet to empty-portlet yet (duration: 00m 59s)

27 May 2020

  • curprev 23:2023:20, 27 May 2020imported>Stashbot 174,457 bytes +16,759 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add autoreviewrestore right to rollbacker group on hiwiki (T252986) (duration: 01m 05s)

26 May 2020

  • curprev 21:3421:34, 26 May 2020imported>Stashbot 157,698 bytes +8,943 krinkle@deploy1001: Synchronized wmf-config/mc.php: I0fb124b3593 (duration: 01m 05s)

25 May 2020

  • curprev 23:3423:34, 25 May 2020imported>Stashbot 148,755 bytes +7,068 ejegg: re-enabled fundraising queue consumers and job runners, except audits, dedupe, and recurring

24 May 2020

  • curprev 17:3617:36, 24 May 2020imported>Stashbot 141,687 bytes +170 gehel: restarting elasticsearch psi on elastic1052

23 May 2020

  • curprev 19:0419:04, 23 May 2020imported>Stashbot 141,517 bytes +906 krinkle@deploy1001: Synchronized php-1.35.0-wmf.31/includes/filerepo/file/LocalFile.php: I0f7e885997d60 (duration: 01m 06s)

22 May 2020

  • curprev 22:4222:42, 22 May 2020imported>Stashbot 140,611 bytes +8,134 krinkle@deploy1001: Synchronized php-1.35.0-wmf.31/includes/filerepo/: Ie19613ef7643a (duration: 01m 06s)

21 May 2020

  • curprev 23:5823:58, 21 May 2020imported>Stashbot 132,477 bytes +8,419 ejegg: updated civicrm from b658fd8233 to 6b1d5902dd

20 May 2020

  • curprev 20:1620:16, 20 May 2020imported>Stashbot 124,058 bytes +10,406 herron: logstash1011:~# kafka-preferred-replica-election --zookeeper conf1004.eqiad.wmnet,conf1005.eqiad.wmnet,conf1006.eqiad.wmnet/kafka/logging-eqiad
  • curprev 00:0500:05, 20 May 2020imported>Stashbot 113,652 bytes +7,654 RoanKattouw: Ran namespaceDupes.php on tiwiki and tiwiktionary for T251287

18 May 2020

  • curprev 23:5023:50, 18 May 2020imported>Stashbot 105,998 bytes +9,614 pt1979@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)

16 May 2020

  • curprev 22:0422:04, 16 May 2020imported>Stashbot 96,384 bytes +1,903 Krinkle: krinkle@mc1022 Pruning the old `echo:seen:` Redis keys that didn't have a ttl yet
  • curprev 00:5500:55, 16 May 2020imported>Stashbot 94,481 bytes +6,235 krinkle@deploy1001: Synchronized wmf-config/logging.php: I046868190b472 (duration: 01m 13s)

14 May 2020

  • curprev 23:2423:24, 14 May 2020imported>Stashbot 88,246 bytes +6,005 catrope@deploy1001: Synchronized static/images/project-logos/: Revert temporary 20k logo for vecwiki (T252770) (duration: 01m 06s)
  • curprev 00:5400:54, 14 May 2020imported>Stashbot 82,241 bytes +9,400 foks: change password for "Python eggs"

13 May 2020

  • curprev 00:4300:43, 13 May 2020imported>Stashbot 72,841 bytes +10,801 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer

12 May 2020

  • curprev 00:3700:37, 12 May 2020imported>Stashbot 62,040 bytes +11,598 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)

10 May 2020

  • curprev 12:1812:18, 10 May 2020imported>Stashbot 50,442 bytes +926 marostegui: Start event scheduler on db1115 after a massive delete - T252324

8 May 2020

  • curprev 21:4521:45, 8 May 2020imported>Stashbot 49,516 bytes +4,853 bstorm_: cleaned up wb_terms_no_longer_updated view for testwikidatawiki and testcommonswiki on labsdb1010 T251598
  • curprev 00:3000:30, 8 May 2020imported>Stashbot 44,663 bytes +11,109 brennen@deploy1001: rebuilt and synchronized wikiversions files: Revert all wikis except test to 1.35.0-wmf.30 for T252179

7 May 2020

  • curprev 00:1200:12, 7 May 2020imported>Stashbot 33,554 bytes +5,907 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

5 May 2020

  • curprev 23:4423:44, 5 May 2020imported>Stashbot 27,647 bytes +10,916 catrope@deploy1001: Synchronized wmf-config/flaggedrevs.php: Restore the reviewer group on fawiki (T249643) (duration: 01m 06s)

4 May 2020

  • curprev 23:3823:38, 4 May 2020imported>Stashbot 16,731 bytes +12,007 mstyles@deploy1001: Finished deploy [wdqs/wdqs@6518a8d]: v.0.3.26 (duration: 14m 39s)

3 May 2020

  • curprev 22:5222:52, 3 May 2020imported>Stashbot 4,724 bytes +491 Krinkle: scap pull mwmaint1002 and mw2001 for noc.wm.o. – https://gerrit.wikimedia.org/r/593929

2 May 2020

  • curprev 07:4907:49, 2 May 2020imported>Stashbot 4,233 bytes +460 oblivian@cumin1001: conftool action : set/pooled=yes; selector: name=mw13(49|5[0-9]|6[0-2])\.eqiad\.wmnet

1 May 2020

  • curprev 19:5619:56, 1 May 2020imported>Stashbot 3,773 bytes −815,836 rzl@cumin1001: conftool action : set/pooled=no; selector: name=mw13(5[6-9]|6[0-2]).eqiad.wmnet
  • curprev 01:3601:36, 1 May 2020imported>Stashbot 819,609 bytes +7,834 bmansurov@deploy1001: Finished deploy [recommendation-api/deploy@5f47cd7]: Update the recommendation API service (duration: 04m 33s)

30 April 2020

29 April 2020

  • curprev 00:5600:56, 29 April 2020imported>Stashbot 802,927 bytes +15,206 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Growth Study QuickSurvey on enwiki (with sample size 0, for testing) (T248421) (duration: 01m 10s)

27 April 2020

  • curprev 23:2523:25, 27 April 2020imported>Stashbot 787,721 bytes +15,220 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Update logos for tiwiki and tiwiktionary (T150618, T249451) (duration: 00m 57s)

26 April 2020

  • curprev 18:0818:08, 26 April 2020imported>Stashbot 772,501 bytes +140 elukey: powercycle puppetmaster1001 - mgmt serial console not usable, no ssh, racadm getsel doesn't show anything

25 April 2020

  • curprev 10:2310:23, 25 April 2020imported>Stashbot 772,361 bytes +271 addshore: going to restart and probably depool for a short time wdqs1005 as it is in a deadlock T242453

24 April 2020

  • curprev 21:2521:25, 24 April 2020imported>Stashbot 772,090 bytes +562 cdanis@cumin1001: conftool action : set/pooled=true; selector: dnsdisc=wdqs,name=eqiad

23 April 2020

  • curprev 22:0622:06, 23 April 2020imported>Stashbot 771,528 bytes +300 Urbanecm: Perform timeouting rename at enwiki Wikipedia talk:Introduction --> Wikipedia talk:Introduction (historical) using moveBatch.php (request)

22 April 2020

  • curprev 08:5508:55, 22 April 2020imported>Stashbot 771,228 bytes +1,318 Urbanecm: Move User:Wikipedia:Introduction (historical) --> Wikipedia:Introduction (historical) at enwiki using moveBatch.php, on-wiki interface was time-outing
  • curprev 01:4601:46, 22 April 2020imported>Stashbot 769,910 bytes +5,717 milimetric@deploy1001: Started deploy [analytics/refinery@30facc4]: Analytics: another follow-up on the train, jar version bump

20 April 2020

  • curprev 23:2923:29, 20 April 2020imported>Stashbot 764,193 bytes +9,549 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Update project wordmarks and icons (T249047) (duration: 01m 01s)

19 April 2020

  • curprev 16:1916:19, 19 April 2020imported>Stashbot 754,644 bytes +302 reedy@deploy1001: Synchronized wmf-config/LabsServices.php: labs: Move RB traffic to new stretch host (duration: 01m 11s)

18 April 2020

17 April 2020

  • curprev 19:3319:33, 17 April 2020imported>Stashbot 753,607 bytes +5,333 Krinkle: Depool mw1407.eqiad.wmnet for opcache testing. Do not repool without first reverting https://gerrit.wikimedia.org/r/589674.

16 April 2020

  • curprev 22:3422:34, 16 April 2020imported>Stashbot 748,274 bytes +7,894 maryum: reindexing wikis that failed from previous reindex on mwmain1002

15 April 2020

  • curprev 22:1122:11, 15 April 2020imported>Stashbot 740,380 bytes +9,855 mholloway-shell@deploy1001: Synchronized php-1.35.0-wmf.28/extensions/MachineVision: Fix: Initialize categories array for initial images (T250321) (duration: 01m 07s)

14 April 2020

  • curprev 23:2423:24, 14 April 2020imported>Stashbot 730,525 bytes +9,174 AndyRussG: re-enabled thank-you, onimailing and new recurring charge jobs

13 April 2020

  • curprev 23:2423:24, 13 April 2020imported>Stashbot 721,351 bytes +6,995 mdholloway: re-ran extensions/MachineVision/maintenance/withholdImages.php on commonswiki

12 April 2020

  • curprev 11:1111:11, 12 April 2020imported>Stashbot 714,356 bytes +810 vgutierrez: restart ats-tls on cp5008.eqsin.wmnet - T249335

11 April 2020

  • curprev 19:5219:52, 11 April 2020imported>Stashbot 713,546 bytes +836 cdanis@cumin1001: dbctl commit (dc=all): 'slight deweight to db1111', diff saved to https://phabricator.wikimedia.org/P10960 and previous config saved to /var/cache/conftool/dbconfig/20200411-195235-cdanis.json

10 April 2020

  • curprev 21:1221:12, 10 April 2020imported>Stashbot 712,710 bytes +3,913 cdanis@cumin1001: dbctl commit (dc=all): 'db1111 seems overloaded', diff saved to https://phabricator.wikimedia.org/P10954 and previous config saved to /var/cache/conftool/dbconfig/20200410-211202-cdanis.json

9 April 2020

  • curprev 23:4423:44, 9 April 2020imported>Stashbot 708,797 bytes +5,617 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Touch and secondary sync of IS for cache-busting (duration: 00m 58s)

8 April 2020

  • curprev 21:2021:20, 8 April 2020imported>Stashbot 703,180 bytes +8,049 jforrester@deploy1001: Synchronized php-1.35.0-wmf.27/extensions/TemplateData/includes/TemplateDataHooks.php: Restore call to OutputPage::setupOOUI() (duration: 01m 07s)

7 April 2020

  • curprev 20:3920:39, 7 April 2020imported>Stashbot 695,131 bytes +13,653 andrewbogott: correction: briefly downtiming ldap-eqiad-replica0 and ldap-eqiad-replica1. I'm trying to investigate a possible split-brain so going to turn ldap off on one, and then the other, to see if behavior changes
  • curprev 00:0100:01, 7 April 2020imported>Stashbot 681,478 bytes +13,583 addshore@deploy1001: Synchronized php-1.35.0-wmf.26/extensions/Wikibase/repo/includes/Store/Sql/DatabaseSchemaUpdater.php: Do not try to drop things when theres no wb_terms table T208425 T249565 cache bust (duration: 01m 01s)

3 April 2020

  • curprev 21:1721:17, 3 April 2020imported>Stashbot 667,895 bytes +4,748 andrewbogott: ugpraded wikitech-static to 1.34.1
  • curprev 00:4200:42, 3 April 2020imported>Stashbot 663,147 bytes +9,745 catrope@deploy1001: Synchronized php-1.35.0-wmf.26/extensions/FlaggedRevs/: Fix logic for determining if pending edits were null (T249277) (duration: 01m 00s)

1 April 2020

  • curprev 22:4422:44, 1 April 2020imported>Stashbot 653,402 bytes +9,945 volker-e@deploy1001: Finished deploy [design/style-guide@4bfe647]: Deploy design/style-guide: (duration: 00m 08s)
  • curprev 00:3900:39, 1 April 2020imported>Stashbot 643,457 bytes +11,449 reedy@deploy1001: Synchronized docroot/mediawiki.org/xml/: Update http and prot rel links to https, fix link to sitelist in MW Core (duration: 01m 06s)

31 March 2020

  • curprev 00:2600:26, 31 March 2020imported>Stashbot 632,008 bytes +8,786 eileen: civicrm revision changed from cf2e2c11c3 to 524b162174, config revision is 708198a154

29 March 2020

  • curprev 08:2408:24, 29 March 2020imported>Stashbot 623,222 bytes +162 elukey: powercycle elastic1059 - mgmt/serial console stuck, no ssh - racadm getsel shows a lot of OEM errors occurred, nothing specific

28 March 2020

27 March 2020

  • curprev 20:5120:51, 27 March 2020imported>Stashbot 622,744 bytes +6,524 ejegg: updated payments-wiki from db618f429d to 1640f5e21e

26 March 2020

  • curprev 23:0723:07, 26 March 2020imported>Stashbot 616,220 bytes +12,328 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: ce63a4e: Enable wmgUseFooterContactLink for cswiki (T248584; take II) (duration: 00m 57s)
  • curprev 00:0600:06, 26 March 2020imported>Stashbot 603,892 bytes +15,568 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Special:Investigate on testwiki (T247645) (duration: 03m 14s)

24 March 2020

23 March 2020

  • curprev 21:5021:50, 23 March 2020imported>Stashbot 582,770 bytes +7,770 krinkle@deploy1001: Synchronized docroot/noc/css/vector.css: I627a0ddba5 (duration: 01m 02s)

22 March 2020

  • curprev 23:1923:19, 22 March 2020imported>Stashbot 575,000 bytes +225 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: T248274 (duration: 01m 19s)

20 March 2020

  • curprev 23:1623:16, 20 March 2020imported>Stashbot 574,775 bytes +5,506 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)

19 March 2020

  • curprev 22:1522:15, 19 March 2020imported>Stashbot 569,269 bytes +4,690 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@794f099]: Update mobileapps to 99869f45 (duration: 05m 13s)
  • curprev 00:3900:39, 19 March 2020imported>Stashbot 564,579 bytes +8,440 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikiws to 1.35.0-wmf.24 refs T233872

18 March 2020

  • curprev 00:3100:31, 18 March 2020imported>Stashbot 556,139 bytes +12,787 Amir1: foreachwikiindblist medium deleteEqualMessages.php --delete (T247562)

17 March 2020

  • curprev 00:5700:57, 17 March 2020imported>Stashbot 543,352 bytes +9,300 krinkle@deploy1001: Synchronized php-1.35.0-wmf.23/extensions/Wikibase/lib/includes/Formatters/: Ic77b2c6b33a, T247458 (duration: 01m 12s)

15 March 2020

14 March 2020

  • curprev 08:3308:33, 14 March 2020imported>Stashbot 533,856 bytes +358 elukey: run kafka preferred-replica-election on kafka-jumbo1001 - T247561

13 March 2020

  • curprev 23:1223:12, 13 March 2020imported>Stashbot 533,498 bytes +3,031 bstorm_: rebooting labstore1006 for upgrade to stretch T224583
  • curprev 00:2000:20, 13 March 2020imported>Stashbot 530,467 bytes +13,395 shdubsh: reload prometheus@ops on prometheus1003

12 March 2020

  • curprev 00:5600:56, 12 March 2020imported>Stashbot 517,072 bytes +10,878 ebernhardson@deploy1001: Synchronized php-1.35.0-wmf.22/extensions/CirrusSearch/includes/Maintenance/Reindexer.php: wait around for counts to match up in reindexer before giving up (duration: 01m 08s)

11 March 2020

  • curprev 00:2400:24, 11 March 2020imported>Stashbot 506,194 bytes +18,985 eileen: civicrm revision changed from 35651da117 to c4b81b19b0, config revision is 71c8cda115

9 March 2020

  • curprev 23:2123:21, 9 March 2020imported>Stashbot 487,209 bytes +7,801 catrope@deploy1001: Synchronized wmf-config/throttle.php: Remove expired throttle exemptions (duration: 01m 00s)

8 March 2020

  • curprev 17:5817:58, 8 March 2020imported>Stashbot 479,408 bytes +406 elukey: restart hadoop-yarn-nodemanger on an-worker1087

7 March 2020

  • curprev 12:4812:48, 7 March 2020imported>Stashbot 479,002 bytes +109 reedy@deploy1001: Synchronized wmf-config/throttle.php: T247149 (duration: 01m 07s)
  • curprev 01:3501:35, 7 March 2020imported>Stashbot 478,893 bytes +10,335 reedy@deploy1001: Synchronized wmf-config/throttle.php: tidy up (duration: 00m 56s)

6 March 2020

  • curprev 00:5800:58, 6 March 2020imported>Stashbot 468,558 bytes +21,223 dzahn@cumin1001: START - Cookbook sre.hosts.downtime

5 March 2020

  • curprev 00:5700:57, 5 March 2020imported>Stashbot 447,335 bytes +13,946 twentyafterfour: deploying phabricator-extensions tag release/2020-03-04/1 ( https://phabricator.wikimedia.org/source/phab-extensions/history/wmf%252Fstable/;release/2020-03-04/1 )

4 March 2020

  • curprev 00:2500:25, 4 March 2020imported>Stashbot 433,389 bytes +23,744 ebernhardson@deploy1001: Synchronized php-1.35.0-wmf.21/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: [cirrus] Match fallback config key with the one used in cirrus (duration: 01m 03s)
(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)