You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Revision history

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)

14 October 2021

  • curprev 01:4101:41, 14 October 2021imported>Stashbot 695,414 bytes +17,092 ejegg: updated payments-wiki from b329d2dea2 to 19d18c1852

13 October 2021

  • curprev 00:3800:38, 13 October 2021imported>Stashbot 678,322 bytes +15,911 ejegg: updated payments-wiki from 030b11da1a to b329d2dea2

12 October 2021

  • curprev 00:1100:11, 12 October 2021imported>Stashbot 662,411 bytes +3,728 eileen: civicrm revision changed from 598b59b0ee to 96090e4bd2, config revision is 85277466ed

9 October 2021

  • curprev 05:0105:01, 9 October 2021imported>Stashbot 658,683 bytes +224 jiji@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:3201:32, 9 October 2021imported>Stashbot 658,459 bytes +8,808 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814

8 October 2021

7 October 2021

  • curprev 00:1100:11, 7 October 2021imported>Stashbot 637,764 bytes +11,864 mutante: [grafana2001:~] $ sudo systemctl start rsync-var-lib-grafana because of "PROBLEM - Check systemd state on grafana2001 is CRITICAL: CRITICAL - degraded" because of some race condition where a file vanished during sync

6 October 2021

  • curprev 01:3901:39, 6 October 2021imported>Stashbot 625,900 bytes +10,381 legoktm: legoktm@mwmaint1002:~$ echo "https://en.wikiversity.org/static/images/mobile/copyright/wikiversity.svg" |mwscript purgeList.php

4 October 2021

  • curprev 23:3023:30, 4 October 2021imported>Stashbot 615,519 bytes +9,952 foks: resetting some emails used for abuse by a globally-banned user

3 October 2021

2 October 2021

  • curprev 17:2817:28, 2 October 2021imported>Stashbot 605,040 bytes +230 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

1 October 2021

  • curprev 23:1923:19, 1 October 2021imported>Stashbot 604,810 bytes +38,661 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

29 September 2021

  • curprev 23:2023:20, 29 September 2021imported>Stashbot 566,149 bytes +14,880 bd808@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 00:5200:52, 29 September 2021imported>Stashbot 551,269 bytes +27,094 eileen: civicrm revision changed from a1929b3dfd to a480bf03c9, config revision is 77cb7ec866

27 September 2021

  • curprev 23:4223:42, 27 September 2021imported>Stashbot 524,175 bytes +27,476 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

26 September 2021

  • curprev 14:5114:51, 26 September 2021imported>Stashbot 496,699 bytes +432 volker-e@deploy1002: Finished deploy [design/style-guide@aac0ae9]: Deploy design/style-guide: aac0ae9 “Apps”: Fix image path (#490) (duration: 00m 06s)

25 September 2021

  • curprev 02:0002:00, 25 September 2021imported>Stashbot 496,267 bytes +374 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .

24 September 2021

  • curprev 20:0020:00, 24 September 2021imported>Stashbot 495,893 bytes +10,338 volker-e@deploy1002: Finished deploy [design/style-guide@362c6b1]: Deploy design/style-guide: 362c6b1 “Components”: Fix index link (#489) (duration: 00m 06s)
  • curprev 00:3900:39, 24 September 2021imported>Stashbot 485,555 bytes +19,157 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

23 September 2021

  • curprev 00:3500:35, 23 September 2021imported>Stashbot 466,398 bytes +12,744 catrope@deploy1002: Synchronized php-1.38.0-wmf.1/extensions/MediaSearch/: Use text() instead of parse() for MediaSearch UI messages (T291590) (duration: 01m 08s)

21 September 2021

  • curprev 23:2023:20, 21 September 2021imported>Stashbot 453,654 bytes +9,016 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:1600:16, 21 September 2021imported>Stashbot 444,638 bytes +11,548 tgr: Evening deploys done

18 September 2021

  • curprev 01:4701:47, 18 September 2021imported>Stashbot 433,090 bytes +322 ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.23/includes/libs/rdbms/database/Database.php: (no justification provided) (duration: 00m 57s)

17 September 2021

  • curprev 21:2821:28, 17 September 2021imported>Stashbot 432,768 bytes +3,000 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:0400:04, 17 September 2021imported>Stashbot 429,768 bytes +13,476 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .

15 September 2021

14 September 2021

  • curprev 23:0123:01, 14 September 2021imported>Stashbot 406,930 bytes +15,959 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Re-enable VipsScaler (2 of 2) (duration: 01m 04s)
  • curprev 00:0400:04, 14 September 2021imported>Stashbot 390,971 bytes +8,994 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

12 September 2021

11 September 2021

  • curprev 19:0219:02, 11 September 2021imported>Stashbot 381,555 bytes +650 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 27814b8eaacb5ba2fee1b6167a36ea14356a1ecf: testwiki: Fully remove securepoll-related groups (T290808) (duration: 00m 57s)

10 September 2021

  • curprev 21:2921:29, 10 September 2021imported>Stashbot 380,905 bytes +5,151 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • curprev 00:3500:35, 10 September 2021imported>Stashbot 375,754 bytes +15,347 tgr: Deployed patch for T290692

9 September 2021

  • curprev 00:5000:50, 9 September 2021imported>Stashbot 360,407 bytes +8,465 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Don't set default to Score (try #2) (duration: 00m 58s)

8 September 2021

  • curprev 00:0000:00, 8 September 2021imported>Stashbot 351,942 bytes +12,900 legoktm: legoktm@lists1001:~$ sudo rm -rf /etc/mailman # cleanup as part of 4869d91b0be / T282303

6 September 2021

  • curprev 23:5223:52, 6 September 2021imported>Stashbot 339,042 bytes +8,019 tstarling@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/SecurePoll/includes/Talliers/STVTallier.php: T290000 (duration: 00m 58s)

5 September 2021

  • curprev 18:5418:54, 5 September 2021imported>Stashbot 331,023 bytes +201 urbanecm: wikiadmin@10.192.0.119(ptwiki)> update protected_titles set pt_create_perm='editautoreviewprotected' where pt_create_perm='autoreviewer'; # T290396

4 September 2021

  • curprev 13:3513:35, 4 September 2021imported>Stashbot 330,822 bytes +1,989 marostegui@cumin1001: dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: Slowly repool T290374', diff saved to https://phabricator.wikimedia.org/P17217 and previous config saved to /var/cache/conftool/dbconfig/20210904-133532-root.json

3 September 2021

1 September 2021

  • curprev 23:5023:50, 1 September 2021imported>Stashbot 297,294 bytes +11,478 Amir1: mwscript createAndPromote.php --wiki=test2wiki --sysop --force Ladsgroup
  • curprev 00:5300:53, 1 September 2021imported>Stashbot 285,816 bytes +13,363 eileen: civicrm revision changed from e567b4c289 to 7da3eba4f9, config revision is 5f004d94d7

30 August 2021

  • curprev 23:1423:14, 30 August 2021imported>Stashbot 272,453 bytes +10,249 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

29 August 2021

  • curprev 00:1500:15, 29 August 2021imported>Stashbot 262,204 bytes +714 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

27 August 2021

  • curprev 16:4616:46, 27 August 2021imported>Stashbot 261,490 bytes +3,953 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps1005.eqiad.wmnet with reason: Resyncing from master
  • curprev 00:4600:46, 27 August 2021imported>Stashbot 257,537 bytes +9,223 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

25 August 2021

  • curprev 23:2323:23, 25 August 2021imported>Stashbot 248,314 bytes +23,982 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:5300:53, 25 August 2021imported>Stashbot 224,332 bytes +6,127 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .

24 August 2021

  • curprev 00:1700:17, 24 August 2021imported>Stashbot 218,205 bytes +10,888 ryankemper: [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'`

22 August 2021

21 August 2021

  • curprev 15:1215:12, 21 August 2021imported>Stashbot 1,011,788 bytes +262 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

20 August 2021

  • curprev 23:1823:18, 20 August 2021imported>Stashbot 1,011,526 bytes +4,817 legoktm: deployed patch for T289385
  • curprev 00:1800:18, 20 August 2021imported>Stashbot 1,006,709 bytes +10,542 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

19 August 2021

  • curprev 01:1301:13, 19 August 2021imported>Stashbot 996,167 bytes +6,256 ejegg: updated fundraising CiviCRM from 73f6ec9190 to 8ed303f2d1

18 August 2021

  • curprev 00:3900:39, 18 August 2021imported>Stashbot 989,911 bytes +13,880 dpifke@deploy1002: Finished deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243) (duration: 00m 05s)

17 August 2021

  • curprev 00:4500:45, 17 August 2021imported>Stashbot 976,031 bytes +12,823 eileen: civicrm revision changed from ba0c7705bb to 175a3101f7, config revision is 7bdc78073d

15 August 2021

14 August 2021

  • curprev 03:5403:54, 14 August 2021imported>Stashbot 962,909 bytes +121 legoktm[m]: restarting mailman3 on lists1001, bounce runner crashed (T288880)

13 August 2021

  • curprev 18:4318:43, 13 August 2021imported>Stashbot 962,788 bytes +9,163 bblack: reprepro: uploaded gdnsd-3.8.0-1~wmf1 to buster-wikimedia - T252132

12 August 2021

11 August 2021

  • curprev 23:3023:30, 11 August 2021imported>Stashbot 931,568 bytes +15,885 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:4701:47, 11 August 2021imported>Stashbot 915,683 bytes +19,823 dpifke@deploy1002: Finished deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 06s)

9 August 2021

  • curprev 16:1216:12, 9 August 2021imported>Stashbot 895,860 bytes +14,288 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .

6 August 2021

  • curprev 19:1719:17, 6 August 2021imported>Stashbot 881,572 bytes +8,445 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:0101:01, 6 August 2021imported>Stashbot 873,127 bytes +23,598 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be2065.codfw.wmnet with reason: REIMAGE

5 August 2021

  • curprev 01:2601:26, 5 August 2021imported>Stashbot 849,529 bytes +19,135 Krinkle: krinkle@mwmaint1002 Temporarily grant myself `translationadmin` on wikimania2016wiki in order to approve an edit given FlaggedRevs-like nature of Translate

3 August 2021

  • curprev 23:3423:34, 3 August 2021imported>Stashbot 830,394 bytes +13,656 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:4400:44, 3 August 2021imported>Stashbot 816,738 bytes +12,154 reedy@deploy1002: Finished deploy [integration/docroot@f9d225d]: with less gref (duration: 00m 05s)

31 July 2021

  • curprev 12:4012:40, 31 July 2021imported>Stashbot 804,584 bytes +151 reedy@deploy1002: Synchronized php-1.37.0-wmf.16/extensions/SecurePoll/: T287780 T287782 (duration: 00m 58s)
  • curprev 00:0100:01, 31 July 2021imported>Stashbot 804,433 bytes +8,409 eileen: civicrm revision changed from 158ed65e00 to d6baf291f4, config revision is 6011d9c471

29 July 2021

28 July 2021

27 July 2021

26 July 2021

  • curprev 23:3723:37, 26 July 2021imported>Stashbot 765,495 bytes +4,200 legoktm@deploy1002: Synchronized php-1.37.0-wmf.15/extensions/Score/includes/Score.php: Increase lilypond version cache TTL to 1 hour (duration: 00m 57s)

24 July 2021

  • curprev 11:0411:04, 24 July 2021imported>Stashbot 761,295 bytes +338 urbanecm: [urbanecm@mwmaint2002 ~]$ mwscript extensions/Translate/scripts/moveTranslatablePage.php --wiki=commonswiki --reason='OTRS -> VRTS renaming process; see Phab:T280392 and Phab:T280397' --move-subpages 'Commons:OTRS' 'Commons:Volunteer Response Team' 'Martin Urbanec' # T287321

23 July 2021

  • curprev 19:1119:11, 23 July 2021imported>Stashbot 760,957 bytes +5,409 topranks: Successfully re-pooled eqiad - reversed change from yesterday after successful line card replacement in cr2-codfw - T287110

22 July 2021

20 July 2021

  • curprev 20:5320:53, 20 July 2021imported>Stashbot 730,828 bytes +12,637 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: caa5a076f39b051b01622aa3e4c9d716a8643eef: Set wgGEMentorDashboardBackendEnabled properly (T285811) (duration: 00m 57s)

19 July 2021

16 July 2021

  • curprev 19:5019:50, 16 July 2021imported>Stashbot 704,075 bytes +7,705 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on copernicium.wikimedia.org with reason: REIMAGE
  • curprev 00:0600:06, 16 July 2021imported>Stashbot 696,370 bytes +21,594 hoo: Updated the Wikidata property suggester with data from the 2021-07-12 JSON dump (with pre-applied T132839 workarounds)

15 July 2021

14 July 2021

  • curprev 00:5800:58, 14 July 2021imported>Stashbot 662,656 bytes +13,088 eileen: process control updated to c291b3c6890364281d

12 July 2021

  • curprev 23:5723:57, 12 July 2021imported>Stashbot 649,568 bytes +13,413 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 1896efc27f3de39659673091bc4c43ad874da0c5: Add sayahna.org to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T286163) (duration: 00m 56s)

9 July 2021

  • curprev 23:2823:28, 9 July 2021imported>Stashbot 636,155 bytes +1,014 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • curprev 00:4700:47, 9 July 2021imported>Stashbot 635,141 bytes +9,116 legoktm: zotero rolling restart didn't help, filed T286360 for DNS issues

7 July 2021

  • curprev 20:2220:22, 7 July 2021imported>Stashbot 626,025 bytes +5,635 legoktm: repooling eqiad - https://gerrit.wikimedia.org/r/703561

6 July 2021

  • curprev 18:3418:34, 6 July 2021imported>Stashbot 620,390 bytes +5,171 otto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
  • curprev 00:5600:56, 6 July 2021imported>Stashbot 615,219 bytes +1,686 eileen: process-control config revision is 8d46b52ed4

5 July 2021

  • curprev 00:5300:53, 5 July 2021imported>Stashbot 613,533 bytes +528 eileen: process-control config revision is a1717c7fde

3 July 2021

  • curprev 17:4617:46, 3 July 2021imported>Stashbot 613,005 bytes +322 elukey: depool eqsin due to loss of power redundancy (equinix maintenance) - T286113

2 July 2021

  • curprev 22:0622:06, 2 July 2021imported>Stashbot 612,683 bytes +8,804 foks: removing three files for legal compliance
  • curprev 01:4701:47, 2 July 2021imported>Stashbot 603,879 bytes +9,552 eileen: civicrm revision changed from e07c2be1a7 to bb62188ec6, config revision is 1739c53fcb

30 June 2021

  • curprev 23:2823:28, 30 June 2021imported>Stashbot 594,327 bytes +11,040 urbanecm: Evening B&C window finished
  • curprev 00:3600:36, 30 June 2021imported>Stashbot 583,287 bytes +15,990 tstarling@deploy1002: Synchronized wmf-config/db-labs.php: gerrit 701995 SQL query log (duration: 01m 05s)

29 June 2021

  • curprev 00:2600:26, 29 June 2021imported>Stashbot 567,297 bytes +16,261 Krinkle: krinkle@mwmaint1002: purgeParserCache.php --tag pc1, ref T282761

27 June 2021

  • curprev 09:1909:19, 27 June 2021imported>Elukey 551,036 bytes +741 Add missing restart entry (logged on a different chan)

26 June 2021

  • curprev 21:2821:28, 26 June 2021imported>Stashbot 550,295 bytes +1,342 volans: upgraded spicerack to v0.0.56 on the cumin hosts (includes only bug fixes for the switchdc)

25 June 2021

  • curprev 21:3721:37, 25 June 2021imported>Stashbot 548,953 bytes +6,188 ebernhardson@deploy1002: Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s)

24 June 2021

  • curprev 23:0223:02, 24 June 2021imported>Stashbot 542,765 bytes +11,139 legoktm: reverted cumin1001 spicerack live hacks
  • curprev 00:2500:25, 24 June 2021imported>Stashbot 531,626 bytes +19,731 eileen: civicrm revision changed from bd906975f0 to 6d3dd6e5a5, config revision is 821e5889f7

23 June 2021

  • curprev 00:5000:50, 23 June 2021imported>Stashbot 511,895 bytes +14,361 eileen: civicrm revision changed from c745d4f075 to 03bead707d, config revision is 4ab72c1033

22 June 2021

  • curprev 01:0201:02, 22 June 2021imported>Stashbot 497,534 bytes +26,215 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE

18 June 2021

  • curprev 20:5520:55, 18 June 2021imported>Stashbot 471,319 bytes +9,077 Krinkle: Remove doc1001:/srv/doc/mediawiki-core/wmf-1.36.0-wmf.31-testing

17 June 2021

  • curprev 21:4921:49, 17 June 2021imported>Stashbot 462,242 bytes +15,302 legoktm: regenerating pipermail redirects to skip those with duplicate message-ids (T280731)

16 June 2021

  • curprev 21:3521:35, 16 June 2021imported>Stashbot 446,940 bytes +4,876 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .

15 June 2021

  • curprev 17:5417:54, 15 June 2021imported>Stashbot 442,064 bytes +4,022 dancy: testing upcoming Scap release on beta
  • curprev 00:3700:37, 15 June 2021imported>Stashbot 438,042 bytes +26,468 eileen: civicrm revision changed from 31d07115a0 to 28ace1b86f, config revision is 2aed6ff89b

12 June 2021

  • curprev 13:4913:49, 12 June 2021imported>Stashbot 411,574 bytes +317 rzl@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: alert noise, no impact, x2 is unused

11 June 2021

  • curprev 23:3723:37, 11 June 2021imported>Stashbot 411,257 bytes +6,591 mutante: removing firewall hole for mgmt networks to install* because it turned out it cant be used for firmware upgrades
  • curprev 01:1001:10, 11 June 2021imported>Stashbot 404,666 bytes +17,032 eileen: process-control config revision is 2aed6ff89b

9 June 2021

7 June 2021

  • curprev 21:2621:26, 7 June 2021imported>Stashbot 350,845 bytes +11,147 otto@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0)

5 June 2021

  • curprev 16:1616:16, 5 June 2021imported>Stashbot 339,698 bytes +307 Amir1: deleting all private archives of mm2. All are inaccessible now (T282303)
  • curprev 00:2100:21, 5 June 2021imported>Stashbot 339,391 bytes +11,476 mutante: backup1001 - systemctl baclua-dir works again (restoring backup for non-existing host)

4 June 2021

  • curprev 00:1200:12, 4 June 2021imported>Stashbot 327,915 bytes +24,727 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

3 June 2021

1 June 2021

  • curprev 21:0921:09, 1 June 2021imported>Stashbot 287,834 bytes +5,348 andrewbogott: dropping a bunch of tables from the labswiki db as per T284108
  • curprev 00:4600:46, 1 June 2021imported>Stashbot 282,486 bytes +974 legoktm@deploy1002: Synchronized logos/config.yaml: Revert "Use eswiki 20th anniversary logos" (T280908) (duration: 01m 07s)

29 May 2021

  • curprev 14:4414:44, 29 May 2021imported>Stashbot 281,512 bytes +194 elukey: execute apt-get clean on an-airflow1001 to free space

28 May 2021

  • curprev 08:0608:06, 28 May 2021imported>Stashbot 281,318 bytes +368 oblivian@cumin1001: conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet,dc=eqiad

27 May 2021

  • curprev 23:5623:56, 27 May 2021imported>Stashbot 280,950 bytes +10,488 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab1004.eqiad.wmnet with reason: REIMAGE

26 May 2021

25 May 2021

  • curprev 00:4800:48, 25 May 2021imported>Stashbot 245,293 bytes +11,934 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

23 May 2021

  • curprev 14:2514:25, 23 May 2021imported>Stashbot 233,359 bytes +262 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: EMERGENCY: f752f8b80c57a5b7e41b91873b3eac388535ac46: enwiktionary: Raise AF emergency disable treshold+count (T283460) (duration: 00m 57s)

22 May 2021

  • curprev 22:1322:13, 22 May 2021imported>Stashbot 233,097 bytes +265 legoktm: reset 2FA for User:Yuvipanda on wikitech

21 May 2021

  • curprev 22:3222:32, 21 May 2021imported>Stashbot 232,832 bytes +12,510 bstorm: upload nfsd-ldap: 1.2+deb10u1 to buster-wikimedia T283385

20 May 2021

  • curprev 21:4521:45, 20 May 2021imported>Stashbot 220,322 bytes +13,158 herron@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:0101:01, 20 May 2021imported>Stashbot 207,164 bytes +12,243 mutante: signing puppet certs for doh2001 and doh2002.wikimedia.org (T283192)

18 May 2021

  • curprev 18:4018:40, 18 May 2021imported>Stashbot 194,921 bytes +14,463 razzi@deploy1002: Finished deploy [analytics/refinery@9392f1d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@9392f1db6e66975304c8e9b2b7031acd3ed87fa7] (duration: 05m 16s)
  • curprev 00:5600:56, 18 May 2021imported>Stashbot 180,458 bytes +19,706 eileen: civicrm revision changed from 38ac15233f to b3fb3c9cb0, config revision is 1f8d0a6bfa

17 May 2021

16 May 2021

  • curprev 00:5300:53, 16 May 2021imported>Stashbot 160,344 bytes +324 legoktm: restarted mailman3-web on lists1001, uwsgi looked like it got stuck, consuming all CPU/memory

14 May 2021

  • curprev 20:4220:42, 14 May 2021imported>Stashbot 160,020 bytes +3,428 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people1002.eqiad.wmnet
  • curprev 00:3900:39, 14 May 2021imported>Stashbot 156,592 bytes +8,035 ryankemper: T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2003.codfw.wmnet` on `ryankemper@cumin2001` tmux session `wdqs_reimage`

12 May 2021

  • curprev 23:4823:48, 12 May 2021imported>Stashbot 148,557 bytes +16,709 urbanecm@deploy1002: Synchronized php-1.37.0-wmf.4/extensions/WikiEditor/includes/WikiEditorHooks.php: 2f6af514c49d47bbec5ce51f9f7263015e039003? PHP VisualEditorFeatureUse logging: properly record session id (T281409) (duration: 01m 07s)
  • curprev 01:4201:42, 12 May 2021imported>Stashbot 131,848 bytes +17,502 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host

10 May 2021

  • curprev 23:3823:38, 10 May 2021imported>Stashbot 114,346 bytes +15,358 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: 779fb53bfd7a4d9b11f865df14f8a72adb97f33b: Update messages used for tech CoC (T280886) (duration: 00m 56s)

9 May 2021

  • curprev 21:4421:44, 9 May 2021imported>Stashbot 98,988 bytes −1,654,362 legoktm: restarted mailman3 again (T282348) pymysql.err.InternalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

8 May 2021

  • curprev 17:1817:18, 8 May 2021imported>Stashbot 1,753,350 bytes +105 Amir1: starting upgrade of batch G of mailing lists (T280322)

7 May 2021

  • curprev 21:4021:40, 7 May 2021imported>Stashbot 1,753,245 bytes +7,667 legoktm: deleted education@ from MM3, didn't import properly

6 May 2021

  • curprev 23:5023:50, 6 May 2021imported>Stashbot 1,745,578 bytes +25,003 brennen@deploy1002: rebuilt and synchronized wikiversions files: Rollback group1 and group2 to 1.37.0-wmf.3 (T282193)
  • curprev 00:3500:35, 6 May 2021imported>Stashbot 1,720,575 bytes +28,723 Amir1: sudo service mailman3-web restart

5 May 2021

  • curprev 01:4501:45, 5 May 2021imported>Stashbot 1,691,852 bytes +24,478 ryankemper: T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1011.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph` on `ryankemper@cumin1001` tmux session `reimage`

3 May 2021

  • curprev 23:1823:18, 3 May 2021imported>Stashbot 1,667,374 bytes +11,155 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 230ef5716b34ca83348667f289180313b76ce8a3: Prepare for new configuration option (T277951) (duration: 00m 57s)

2 May 2021

  • curprev 13:4013:40, 2 May 2021imported>Stashbot 1,656,219 bytes +311 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cloudmetrics1002.eqiad.wmnet with reason: Flaky host

1 May 2021

  • curprev 19:1219:12, 1 May 2021imported>Stashbot 1,655,908 bytes +557 Urbanecm: Invalidate password for MaraBot@SUL (T281586)

30 April 2021

  • curprev 21:5521:55, 30 April 2021imported>Stashbot 1,655,351 bytes +6,117 mutante: people1003 - rsycncing /home from peopel1002
  • curprev 01:0801:08, 30 April 2021imported>Stashbot 1,649,234 bytes +23,564 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw reboot - ryankemper@cumin1001 - T280563

29 April 2021

  • curprev 00:4100:41, 29 April 2021imported>Stashbot 1,625,670 bytes +24,040 tstarling@deploy1002: Synchronized php-1.37.0-wmf.3/includes/specials/pagers/ImageListPager.php: T281405 (duration: 01m 08s)

28 April 2021

  • curprev 01:0601:06, 28 April 2021imported>Stashbot 1,601,630 bytes +35,800 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

26 April 2021

  • curprev 23:2823:28, 26 April 2021imported>Stashbot 1,565,830 bytes +16,596 mutante: renewing TLS cert for peopleweb.discovery.wmnet, adding *3 hosts

25 April 2021

  • curprev 15:2315:23, 25 April 2021imported>Stashbot 1,549,234 bytes +138 Amir1: sudo -u list /var/lib/mailman/bin/change_pw -l wikica-l -p $(pwgen -c1 -s 12) (T281066)

24 April 2021

23 April 2021

22 April 2021

  • curprev 17:2617:26, 22 April 2021imported>Stashbot 1,545,052 bytes +1,074 marostegui: Stop mysql on tendril/dbtree database
  • curprev 01:3401:34, 22 April 2021imported>Stashbot 1,543,978 bytes +14,300 eileen: civicrm revision changed from 35a8dd33ba to 42ca3cf65a, config revision is cf07e7ba0b

21 April 2021

  • curprev 00:3800:38, 21 April 2021imported>Stashbot 1,529,678 bytes +27,920 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE

19 April 2021

  • curprev 22:5622:56, 19 April 2021imported>Stashbot 1,501,758 bytes +29,254 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE

18 April 2021

  • curprev 06:4006:40, 18 April 2021imported>Stashbot 1,472,504 bytes +106 Amir1: cleaning watchlist of User:Mr._Ibrahem in wikidatawiki (in main ns only)

17 April 2021

  • curprev 16:1616:16, 17 April 2021imported>Stashbot 1,472,398 bytes +61 Amir1: cleaning SuccuBot's watchlist in wikidatawiki
  • curprev 00:5300:53, 17 April 2021imported>Stashbot 1,472,337 bytes +38,446 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet

15 April 2021

  • curprev 01:1901:19, 15 April 2021imported>Stashbot 1,433,891 bytes +21,435 Amir1: mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P8671 --new-data-type external-id (T278427)

14 April 2021

  • curprev 00:1100:11, 14 April 2021imported>Stashbot 1,412,456 bytes +25,158 legoktm@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw2411.codfw.wmnet

12 April 2021

  • curprev 23:2523:25, 12 April 2021imported>Stashbot 1,387,298 bytes +4,421 krinkle@deploy1002: Synchronized wmf-config/mc.php: I390b4726d01037107 (duration: 00m 58s)

10 April 2021

  • curprev 14:2114:21, 10 April 2021imported>Stashbot 1,382,877 bytes +815 andrew@deploy1002: Finished deploy [horizon/deploy@ee1be56]: fix for T279699 (duration: 04m 12s)

9 April 2021

  • curprev 14:0714:07, 9 April 2021imported>Stashbot 1,382,062 bytes +59 jynus: retry es4 backup dump on eqiad (backup1002)
  • curprev 01:2501:25, 9 April 2021imported>Stashbot 1,382,003 bytes +23,659 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on moss-be2002.codfw.wmnet with reason: REIMAGE

7 April 2021

  • curprev 23:3823:38, 7 April 2021imported>Stashbot 1,358,344 bytes +22,437 ejegg: updated payments-wiki from b06009c099 to 70f5163816,
  • curprev 01:4601:46, 7 April 2021imported>Stashbot 1,335,907 bytes +16,883 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: REIMAGE

5 April 2021

  • curprev 23:1723:17, 5 April 2021imported>Stashbot 1,319,024 bytes +12,454 AaronSchulz: Running importMissingLocalNames.php on mwmaint1002 in a screen

4 April 2021

  • curprev 14:4714:47, 4 April 2021imported>Stashbot 1,306,570 bytes +265 andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 01m 36s)

3 April 2021

  • curprev 19:2019:20, 3 April 2021imported>Stashbot 1,306,305 bytes +1,408 andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 02m 11s)

2 April 2021

  • curprev 22:3122:31, 2 April 2021imported>Stashbot 1,304,897 bytes +10,597 bstorm@cumin1001: END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0)

1 April 2021

  • curprev 23:3223:32, 1 April 2021imported>Stashbot 1,294,300 bytes +18,843 thcipriani@deploy1002: Synchronized php-1.36.0-wmf.37/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: Backport: Revert "Turn on glent m1 AB test" T262612 (duration: 00m 58s)
  • curprev 00:5600:56, 1 April 2021imported>Stashbot 1,275,457 bytes +7,735 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2381.codfw.wmnet with reason: REIMAGE

30 March 2021

  • curprev 23:5923:59, 30 March 2021imported>Stashbot 1,267,722 bytes +13,942 Trey314159: reindexing English wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T274200)

29 March 2021

  • curprev 19:0619:06, 29 March 2021imported>Stashbot 1,253,780 bytes +3,056 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=aqs1004.eqiad.wmnet

27 March 2021

26 March 2021

25 March 2021

  • curprev 23:4723:47, 25 March 2021imported>Stashbot 1,243,504 bytes +12,592 thcipriani@deploy1002: Synchronized php-1.36.0-wmf.36/extensions/3D/package.json: No-op demo sync (duration: 01m 07s)
  • curprev 01:3301:33, 25 March 2021imported>Stashbot 1,230,912 bytes +22,060 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)

24 March 2021

  • curprev 00:2200:22, 24 March 2021imported>Stashbot 1,208,852 bytes +46,207 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2378.codfw.wmnet with reason: REIMAGE

23 March 2021

  • curprev 00:0800:08, 23 March 2021imported>Stashbot 1,162,645 bytes +17,284 tstarling@deploy1002: Synchronized wmf-config: use RequestTimeout library step 3: clean up (duration: 00m 58s)

21 March 2021

  • curprev 10:2510:25, 21 March 2021imported>Stashbot 1,145,361 bytes +368 _joe_: restarting gerrit on gerrit1001, using 45G of reserved memory

20 March 2021

  • curprev 00:2200:22, 20 March 2021imported>Stashbot 1,144,993 bytes +6,348 tzatziki: altering emails for STei (WMF) and SGrabarczuk (WMF)

19 March 2021

  • curprev 00:5100:51, 19 March 2021imported>Stashbot 1,138,645 bytes +9,921 dancy@deploy1002: Synchronized php-1.36.0-wmf.35/extensions/LiquidThreads/classes/Thread.php: T277772 (duration: 00m 58s)

18 March 2021

  • curprev 00:0500:05, 18 March 2021imported>Stashbot 1,128,724 bytes +22,211 eileen: tools revision changed from b7b4060c30 to ef54260b0d

16 March 2021

  • curprev 23:5623:56, 16 March 2021imported>Stashbot 1,106,513 bytes +44,018 krinkle@deploy1002: Synchronized php-1.36.0-wmf.35/includes/Revision/: I8619ab9e92b, T277362, T275531 (duration: 00m 58s)

15 March 2021

  • curprev 23:3123:31, 15 March 2021imported>Stashbot 1,062,495 bytes +21,243 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Remove back-compat from when IRC feed servers was a string (T224579) (duration: 00m 59s)

14 March 2021

  • curprev 17:5717:57, 14 March 2021imported>Stashbot 1,041,252 bytes +1,114 marostegui@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Repool db1146:3314', diff saved to https://phabricator.wikimedia.org/P14827 and previous config saved to /var/cache/conftool/dbconfig/20210314-175751-root.json

13 March 2021

  • curprev 19:0219:02, 13 March 2021imported>Stashbot 1,040,138 bytes +699 Amir1: change default charset of all core tables in labstestwiki to binary (T269348)
  • curprev 00:5500:55, 13 March 2021imported>Stashbot 1,039,439 bytes +10,754 mutante: [wdqs1009:/etc/envoy] $ sudo /usr/local/sbin/build-envoy-config -c /etc/envoy/

12 March 2021

  • curprev 01:3001:30, 12 March 2021imported>Stashbot 1,028,685 bytes +31,013 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2215.codfw.wmnet

11 March 2021

  • curprev 00:5000:50, 11 March 2021imported>Stashbot 997,672 bytes +37,248 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2219.codfw.wmnet

10 March 2021

  • curprev 01:0801:08, 10 March 2021imported>Stashbot 960,424 bytes +27,385 krinkle@deploy1002: Synchronized php-1.36.0-wmf.34/extensions/NavigationTiming/modules/ext.navigationTiming.js: T276826 Ibd9ddf14d64 (duration: 01m 14s)

9 March 2021

  • curprev 00:5800:58, 9 March 2021imported>Stashbot 933,039 bytes +16,107 Krinkle: krinkle@mwmaint1002 Ran invalidateUserSesssions.php for one user

7 March 2021

  • curprev 08:0108:01, 7 March 2021imported>Stashbot 916,932 bytes +170 elukey: "megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll" on analytics1066 - BBU looks fine, but the raid controller was using WriteThrough

5 March 2021

  • curprev 23:1623:16, 5 March 2021imported>Stashbot 916,762 bytes +8,775 legoktm: imported pygments 2.8.0+dfsg-1 to apt.wm.o buster-wikimedia component/pygments (T276298)
  • curprev 00:5900:59, 5 March 2021imported>Stashbot 907,987 bytes +16,215 legoktm: depooled registry1001/registry1002 (old stretch VMs) - T272550

4 March 2021

3 March 2021

  • curprev 00:4200:42, 3 March 2021imported>Stashbot 861,710 bytes +24,132 Urbanecm: Finished deployment in Evening B&C window; logmsgbot is currently down, and a simple restart did not bring it back up

2 March 2021

  • curprev 00:5900:59, 2 March 2021imported>Stashbot 837,578 bytes +21,612 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 0f08e8bbe1f74220a7a01a7606b67e0f75734a53: Update the Persian Wikipedia logos (T261033; 2/2) (duration: 00m 56s)

28 February 2021

27 February 2021

  • curprev 21:1921:19, 27 February 2021imported>Stashbot 815,895 bytes +260 dwisehaupt: ran the following on frdb2002 to allow replication to continue after conversion to utf8mb4 charset: set global slave_type_conversions = ALL_NON_LOSSY;
  • curprev 00:0800:08, 27 February 2021imported>Stashbot 815,635 bytes +26,358 mutante: deploy1002 - rsyncing home dirs from deploy1001

26 February 2021

  • curprev 00:1400:14, 26 February 2021imported>Stashbot 789,277 bytes +25,251 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.32/extensions/Graph/: 9d5cf348f5dda32f8889d5160bb1fe34a4e07f8c: Do not log graph errors to WMF servers (T274557) (duration: 01m 36s)

25 February 2021

  • curprev 00:2900:29, 25 February 2021imported>Stashbot 764,026 bytes +22,501 ryankemper: T274204 Restored service health on `elastic106[0,4,5]` via `sudo apt-get remove --purge wmf-elasticsearch-search-plugins --yes && sudo dpkg -i /var/cache/apt/archives/wmf-elasticsearch-search-plugins_6.5.4-4~stretch_all.deb && sudo puppet agent -tv`. There's some sort of issue with `6.5.4-5~stretch` that we will need to circle back and investigate; for now the fleet is staying on `6.5.4-4~stretch`

24 February 2021

  • curprev 00:5800:58, 24 February 2021imported>Stashbot 741,525 bytes +17,976 volker-e@deploy1001: Finished deploy [design/style-guide@a66b5b6]: Deploy design/style-guide: a66b5b6 “Components”: Add “Dialogs” (#430) (duration: 00m 06s)

23 February 2021

  • curprev 00:4600:46, 23 February 2021imported>Stashbot 723,549 bytes +23,528 eileen: civicrm revision changed from c535ac603a to 5e042e6e57, config revision is ef64f705bb

21 February 2021

  • curprev 16:0316:03, 21 February 2021imported>Stashbot 700,021 bytes +990 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1162 - crashed', diff saved to https://phabricator.wikimedia.org/P14424 and previous config saved to /var/cache/conftool/dbconfig/20210221-160258-marostegui.json

20 February 2021

  • curprev 00:1700:17, 20 February 2021imported>Stashbot 699,031 bytes +13,319 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1317.eqiad.wmnet

19 February 2021

18 February 2021

  • curprev 01:3801:38, 18 February 2021imported>Stashbot 677,805 bytes +16,614 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

17 February 2021

  • curprev 01:0001:00, 17 February 2021imported>Stashbot 661,191 bytes +18,673 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mwdebug1001.eqiad.wmnet

15 February 2021

  • curprev 21:3321:33, 15 February 2021imported>Stashbot 642,518 bytes +7,693 eileen: civicrm revision changed from dfbb8f41bc to c535ac603a, config revision is ba9b2380b1

14 February 2021

  • curprev 13:1313:13, 14 February 2021imported>Stashbot 634,825 bytes +806 akosiaris: sudo cumin -b 1 -s 120 'cp500[2,3,5,6].eqsin.wmnet' 'systemctl restart varnish-frontend.service'

13 February 2021

  • curprev 03:2403:24, 13 February 2021imported>Stashbot 634,019 bytes +232 ryankemper: Depooled `wdqs1006` to catch up on lag
  • curprev 01:0001:00, 13 February 2021imported>Stashbot 633,787 bytes +21,925 crusnov@cumin1001: START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet

12 February 2021

  • curprev 01:2101:21, 12 February 2021imported>Stashbot 611,862 bytes +30,779 legoktm@cumin1001: conftool action : set/pooled=yes; selector: name=mw1329.eqiad.wmnet

11 February 2021

  • curprev 00:2600:26, 11 February 2021imported>Stashbot 581,083 bytes +30,413 legoktm@deploy1001: Synchronized wmf-config/profiler.php: Revert "profiler: Send data to excimer-buster pipeline" (duration: 02m 00s)

10 February 2021

9 February 2021

  • curprev 00:4800:48, 9 February 2021imported>Stashbot 514,235 bytes +25,431 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw1302.eqiad.wmnet

7 February 2021

6 February 2021

  • curprev 08:5908:59, 6 February 2021imported>Stashbot 488,709 bytes +1,136 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro-from-cdh-clients (exit_code=0) for Hadoop test cluster: Change Hadoop distribution - elukey@cumin1001
  • curprev 01:0001:00, 6 February 2021imported>Stashbot 487,573 bytes +17,493 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2265.codfw.wmnet

5 February 2021

  • curprev 00:5900:59, 5 February 2021imported>Stashbot 470,080 bytes +27,954 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@85713c1]: restore data range specifier in extract job partition spec

4 February 2021

  • curprev 01:1501:15, 4 February 2021imported>Stashbot 442,126 bytes +22,570 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@4b4872d]: transfer_to_es: Increase timeout waiting for source data to three hours (duration: 01m 16s)

3 February 2021

  • curprev 00:1600:16, 3 February 2021imported>Stashbot 419,556 bytes +21,008 jhuneidi@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .

1 February 2021

  • curprev 23:5423:54, 1 February 2021imported>Stashbot 398,548 bytes +28,951 legoktm@deploy1001: Synchronized wmf-config/profiler.php: profiler: Send data to excimer-buster pipeline (T273312) (duration: 00m 57s)

29 January 2021

  • curprev 23:2623:26, 29 January 2021imported>Stashbot 369,597 bytes +9,826 razzi@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001
  • curprev 00:1900:19, 29 January 2021imported>Stashbot 359,771 bytes +37,885 legoktm@cumin1001: conftool action : set/pooled=yes; selector: name=mw2261.codfw.wmnet

28 January 2021

  • curprev 00:5600:56, 28 January 2021imported>Stashbot 321,886 bytes +25,610 legoktm@cumin1001: conftool action : set/pooled=no; selector: name=mw2294.codfw.wmnet

27 January 2021

  • curprev 00:5400:54, 27 January 2021imported>Stashbot 296,276 bytes +26,066 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2296.codfw.wmnet with reason: REIMAGE

26 January 2021

  • curprev 00:4800:48, 26 January 2021imported>Stashbot 270,210 bytes +18,727 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2331.codfw.wmnet

23 January 2021

  • curprev 22:2122:21, 23 January 2021imported>Stashbot 251,483 bytes +1,766 volker-e@deploy1001: Finished deploy [design/style-guide@63e39e7]: Deploy design/style-guide: 63e39e7 “Components”: Amend button groups states SVG font stack (#427) (duration: 00m 06s)
  • curprev 00:4600:46, 23 January 2021imported>Stashbot 249,717 bytes +27,911 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch enwiki to use enwiki20 "Option A" logo variant (T272526) (duration: 00m 57s)

22 January 2021

21 January 2021

  • curprev 01:2701:27, 21 January 2021imported>Stashbot 205,931 bytes +34,902 ryankemper: [WDQS Deploy] Rollback complete, service health of `wdqs1003` is restored. Need to investigate source of 404 (possibly related to some recent changes we made in the `gui` repo)

20 January 2021

18 January 2021

  • curprev 21:3321:33, 18 January 2021imported>Stashbot 147,203 bytes +19,595 eileen: civicrm revision changed from 4220fc8177 to a4caad22b1, config revision is f08249ecf9

16 January 2021

  • curprev 12:1812:18, 16 January 2021imported>Stashbot 127,608 bytes +369 elukey: elukey@cumin1001:~$ sudo cumin 'A:mw-app-canary and A:mw-eqiad' 'run-puppet-agent' -b 10 - T272215

15 January 2021

  • curprev 23:5923:59, 15 January 2021imported>Stashbot 127,239 bytes +6,731 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE

14 January 2021

  • curprev 23:3523:35, 14 January 2021imported>Stashbot 120,508 bytes +16,149 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2236.codfw.wmnet
  • curprev 00:2200:22, 14 January 2021imported>Stashbot 104,359 bytes +24,462 ryankemper: T266492 Restart of `relforge` successful

12 January 2021

  • curprev 22:5522:55, 12 January 2021imported>Stashbot 79,897 bytes +10,636 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2225.codfw.wmnet

11 January 2021

  • curprev 22:1622:16, 11 January 2021imported>Stashbot 69,261 bytes +16,348 eileen: process-control config revision is f08249ecf9 eoy jobs disabled

10 January 2021

(newest | oldest) View ( | ) (20 | 50 | 100 | 250 | 500)