Server Admin Log: Revision history

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

curprev 14:4514:45, 3 October 2021‎ imported>Stashbot‎ 605,567 bytes +527‎ _joe_: restarting acmechief on acmechief1001

curprev 17:2817:28, 2 October 2021‎ imported>Stashbot‎ 605,040 bytes +230‎ bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

curprev 23:1923:19, 1 October 2021‎ imported>Stashbot‎ 604,810 bytes +38,661‎ bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .

curprev 23:2023:20, 29 September 2021‎ imported>Stashbot‎ 566,149 bytes +14,880‎ bd808@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' .
curprev 00:5200:52, 29 September 2021‎ imported>Stashbot‎ 551,269 bytes +27,094‎ eileen: civicrm revision changed from a1929b3dfd to a480bf03c9, config revision is 77cb7ec866

curprev 23:4223:42, 27 September 2021‎ imported>Stashbot‎ 524,175 bytes +27,476‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

curprev 14:5114:51, 26 September 2021‎ imported>Stashbot‎ 496,699 bytes +432‎ volker-e@deploy1002: Finished deploy [design/style-guide@aac0ae9]: Deploy design/style-guide: aac0ae9 “Apps”: Fix image path (#490) (duration: 00m 06s)

curprev 02:0002:00, 25 September 2021‎ imported>Stashbot‎ 496,267 bytes +374‎ legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .

curprev 20:0020:00, 24 September 2021‎ imported>Stashbot‎ 495,893 bytes +10,338‎ volker-e@deploy1002: Finished deploy [design/style-guide@362c6b1]: Deploy design/style-guide: 362c6b1 “Components”: Fix index link (#489) (duration: 00m 06s)
curprev 00:3900:39, 24 September 2021‎ imported>Stashbot‎ 485,555 bytes +19,157‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

curprev 00:3500:35, 23 September 2021‎ imported>Stashbot‎ 466,398 bytes +12,744‎ catrope@deploy1002: Synchronized php-1.38.0-wmf.1/extensions/MediaSearch/: Use text() instead of parse() for MediaSearch UI messages (T291590) (duration: 01m 08s)

curprev 23:2023:20, 21 September 2021‎ imported>Stashbot‎ 453,654 bytes +9,016‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
curprev 00:1600:16, 21 September 2021‎ imported>Stashbot‎ 444,638 bytes +11,548‎ tgr: Evening deploys done

curprev 01:4701:47, 18 September 2021‎ imported>Stashbot‎ 433,090 bytes +322‎ ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.23/includes/libs/rdbms/database/Database.php: (no justification provided) (duration: 00m 57s)

curprev 21:2821:28, 17 September 2021‎ imported>Stashbot‎ 432,768 bytes +3,000‎ legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
curprev 00:0400:04, 17 September 2021‎ imported>Stashbot‎ 429,768 bytes +13,476‎ legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .

curprev 23:0223:02, 15 September 2021‎ imported>Stashbot‎ 416,292 bytes +9,362‎ legoktm: upgrading lists1001 to use postorius 1.3.5

curprev 23:0123:01, 14 September 2021‎ imported>Stashbot‎ 406,930 bytes +15,959‎ legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Re-enable VipsScaler (2 of 2) (duration: 01m 04s)
curprev 00:0400:04, 14 September 2021‎ imported>Stashbot‎ 390,971 bytes +8,994‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

curprev 18:3318:33, 12 September 2021‎ imported>Stashbot‎ 381,977 bytes +422‎ vgutierrez: restart varnish-fe on cp3061, cp3063 and cp3065

curprev 19:0219:02, 11 September 2021‎ imported>Stashbot‎ 381,555 bytes +650‎ urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 27814b8eaacb5ba2fee1b6167a36ea14356a1ecf: testwiki: Fully remove securepoll-related groups (T290808) (duration: 00m 57s)

curprev 21:2921:29, 10 September 2021‎ imported>Stashbot‎ 380,905 bytes +5,151‎ legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
curprev 00:3500:35, 10 September 2021‎ imported>Stashbot‎ 375,754 bytes +15,347‎ tgr: Deployed patch for T290692

curprev 00:5000:50, 9 September 2021‎ imported>Stashbot‎ 360,407 bytes +8,465‎ legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Don't set default to Score (try #2) (duration: 00m 58s)

curprev 00:0000:00, 8 September 2021‎ imported>Stashbot‎ 351,942 bytes +12,900‎ legoktm: legoktm@lists1001:~$ sudo rm -rf /etc/mailman # cleanup as part of 4869d91b0be / T282303

curprev 23:5223:52, 6 September 2021‎ imported>Stashbot‎ 339,042 bytes +8,019‎ tstarling@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/SecurePoll/includes/Talliers/STVTallier.php: T290000 (duration: 00m 58s)

curprev 18:5418:54, 5 September 2021‎ imported>Stashbot‎ 331,023 bytes +201‎ urbanecm: wikiadmin@10.192.0.119(ptwiki)> update protected_titles set pt_create_perm='editautoreviewprotected' where pt_create_perm='autoreviewer'; # T290396

curprev 13:3513:35, 4 September 2021‎ imported>Stashbot‎ 330,822 bytes +1,989‎ marostegui@cumin1001: dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: Slowly repool T290374', diff saved to https://phabricator.wikimedia.org/P17217 and previous config saved to /var/cache/conftool/dbconfig/20210904-133532-root.json

curprev 21:4921:49, 3 September 2021‎ imported>Stashbot‎ 328,833 bytes +13,135‎ bd808@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' .
curprev 00:3100:31, 3 September 2021‎ imported>Stashbot‎ 315,698 bytes +18,404‎ tgr@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php: Backport: fixLinkRecommendationData: Try harder to avoid >10K result sets (T284531) (duration: 00m 58s)

curprev 23:5023:50, 1 September 2021‎ imported>Stashbot‎ 297,294 bytes +11,478‎ Amir1: mwscript createAndPromote.php --wiki=test2wiki --sysop --force Ladsgroup
curprev 00:5300:53, 1 September 2021‎ imported>Stashbot‎ 285,816 bytes +13,363‎ eileen: civicrm revision changed from e567b4c289 to 7da3eba4f9, config revision is 5f004d94d7

curprev 23:1423:14, 30 August 2021‎ imported>Stashbot‎ 272,453 bytes +10,249‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

curprev 00:1500:15, 29 August 2021‎ imported>Stashbot‎ 262,204 bytes +714‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

curprev 16:4616:46, 27 August 2021‎ imported>Stashbot‎ 261,490 bytes +3,953‎ hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps1005.eqiad.wmnet with reason: Resyncing from master
curprev 00:4600:46, 27 August 2021‎ imported>Stashbot‎ 257,537 bytes +9,223‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

curprev 23:2323:23, 25 August 2021‎ imported>Stashbot‎ 248,314 bytes +23,982‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
curprev 00:5300:53, 25 August 2021‎ imported>Stashbot‎ 224,332 bytes +6,127‎ legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .

curprev 00:1700:17, 24 August 2021‎ imported>Stashbot‎ 218,205 bytes +10,888‎ ryankemper: [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'`

curprev 13:3513:35, 22 August 2021‎ imported>Nintendofan885‎ 207,317 bytes −804,471‎ Archiving May 2021–July 2021 (2/3)

curprev 15:1215:12, 21 August 2021‎ imported>Stashbot‎ 1,011,788 bytes +262‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

curprev 23:1823:18, 20 August 2021‎ imported>Stashbot‎ 1,011,526 bytes +4,817‎ legoktm: deployed patch for T289385
curprev 00:1800:18, 20 August 2021‎ imported>Stashbot‎ 1,006,709 bytes +10,542‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

curprev 01:1301:13, 19 August 2021‎ imported>Stashbot‎ 996,167 bytes +6,256‎ ejegg: updated fundraising CiviCRM from 73f6ec9190 to 8ed303f2d1

curprev 00:3900:39, 18 August 2021‎ imported>Stashbot‎ 989,911 bytes +13,880‎ dpifke@deploy1002: Finished deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243) (duration: 00m 05s)

curprev 00:4500:45, 17 August 2021‎ imported>Stashbot‎ 976,031 bytes +12,823‎ eileen: civicrm revision changed from ba0c7705bb to 175a3101f7, config revision is 7bdc78073d

curprev 20:0220:02, 15 August 2021‎ imported>Stashbot‎ 963,208 bytes +299‎ addshore: restarting blazegraph on wdqs2004

curprev 03:5403:54, 14 August 2021‎ imported>Stashbot‎ 962,909 bytes +121‎ legoktm[m]: restarting mailman3 on lists1001, bounce runner crashed (T288880)

curprev 18:4318:43, 13 August 2021‎ imported>Stashbot‎ 962,788 bytes +9,163‎ bblack: reprepro: uploaded gdnsd-3.8.0-1~wmf1 to buster-wikimedia - T252132

curprev 23:5023:50, 12 August 2021‎ imported>Stashbot‎ 953,625 bytes +22,057‎ derick@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Set archive namespaces on foundationwiki to 'noindex,follow' (T288763) (duration: 00m 59s)

curprev 23:3023:30, 11 August 2021‎ imported>Stashbot‎ 931,568 bytes +15,885‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
curprev 01:4701:47, 11 August 2021‎ imported>Stashbot‎ 915,683 bytes +19,823‎ dpifke@deploy1002: Finished deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 06s)

curprev 16:1216:12, 9 August 2021‎ imported>Stashbot‎ 895,860 bytes +14,288‎ legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .

curprev 19:1719:17, 6 August 2021‎ imported>Stashbot‎ 881,572 bytes +8,445‎ cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
curprev 01:0101:01, 6 August 2021‎ imported>Stashbot‎ 873,127 bytes +23,598‎ pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be2065.codfw.wmnet with reason: REIMAGE

curprev 01:2601:26, 5 August 2021‎ imported>Stashbot‎ 849,529 bytes +19,135‎ Krinkle: krinkle@mwmaint1002 Temporarily grant myself `translationadmin` on wikimania2016wiki in order to approve an edit given FlaggedRevs-like nature of Translate

curprev 23:3423:34, 3 August 2021‎ imported>Stashbot‎ 830,394 bytes +13,656‎ mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
curprev 00:4400:44, 3 August 2021‎ imported>Stashbot‎ 816,738 bytes +12,154‎ reedy@deploy1002: Finished deploy [integration/docroot@f9d225d]: with less gref (duration: 00m 05s)

curprev 12:4012:40, 31 July 2021‎ imported>Stashbot‎ 804,584 bytes +151‎ reedy@deploy1002: Synchronized php-1.37.0-wmf.16/extensions/SecurePoll/: T287780 T287782 (duration: 00m 58s)
curprev 00:0100:01, 31 July 2021‎ imported>Stashbot‎ 804,433 bytes +8,409‎ eileen: civicrm revision changed from 158ed65e00 to d6baf291f4, config revision is 6011d9c471

curprev 23:4123:41, 29 July 2021‎ imported>Stashbot‎ 796,024 bytes +6,556‎ derick@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Merge new configs with existing testwiki definition (duration: 00m 57s)

curprev 23:5723:57, 28 July 2021‎ imported>Stashbot‎ 789,468 bytes +7,751‎ thcipriani@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: wgSkipSkins: Update defaults, hide modern (T287616) (duration: 01m 06s)

curprev 23:5323:53, 27 July 2021‎ imported>Stashbot‎ 781,717 bytes +16,222‎ thcipriani@deploy1002: Synchronized php-1.37.0-wmf.16/skins/Vector: Backport: Restore print, links, table and message box styles (T278896) (duration: 01m 07s)

curprev 23:3723:37, 26 July 2021‎ imported>Stashbot‎ 765,495 bytes +4,200‎ legoktm@deploy1002: Synchronized php-1.37.0-wmf.15/extensions/Score/includes/Score.php: Increase lilypond version cache TTL to 1 hour (duration: 00m 57s)

curprev 11:0411:04, 24 July 2021‎ imported>Stashbot‎ 761,295 bytes +338‎ urbanecm: [urbanecm@mwmaint2002 ~]$ mwscript extensions/Translate/scripts/moveTranslatablePage.php --wiki=commonswiki --reason='OTRS -> VRTS renaming process; see Phab:T280392 and Phab:T280397' --move-subpages 'Commons:OTRS' 'Commons:Volunteer Response Team' 'Martin Urbanec' # T287321

curprev 19:1119:11, 23 July 2021‎ imported>Stashbot‎ 760,957 bytes +5,409‎ topranks: Successfully re-pooled eqiad - reversed change from yesterday after successful line card replacement in cr2-codfw - T287110

curprev 23:3523:35, 22 July 2021‎ imported>Stashbot‎ 755,548 bytes +14,051‎ derick@deploy1002: Synchronized php-1.37.0-wmf.15/includes/preferences/DefaultPreferencesFactory.php: Backport: Make sure enable responsive mode UI reflects actual preference value (T285402) (duration: 00m 56s)
curprev 00:5200:52, 22 July 2021‎ imported>Stashbot‎ 741,497 bytes +10,669‎ ryankemper@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2043.codfw.wmnet with reason: REIMAGE

curprev 20:5320:53, 20 July 2021‎ imported>Stashbot‎ 730,828 bytes +12,637‎ urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: caa5a076f39b051b01622aa3e4c9d716a8643eef: Set wgGEMentorDashboardBackendEnabled properly (T285811) (duration: 00m 57s)

curprev 20:4820:48, 19 July 2021‎ imported>Stashbot‎ 718,191 bytes +14,116‎ urbanecm: Deploy security patch for T286884

curprev 19:5019:50, 16 July 2021‎ imported>Stashbot‎ 704,075 bytes +7,705‎ robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on copernicium.wikimedia.org with reason: REIMAGE
curprev 00:0600:06, 16 July 2021‎ imported>Stashbot‎ 696,370 bytes +21,594‎ hoo: Updated the Wikidata property suggester with data from the 2021-07-12 JSON dump (with pre-applied T132839 workarounds)

curprev 00:0000:00, 15 July 2021‎ imported>Stashbot‎ 674,776 bytes +12,120‎ twentyafterfour: phabricator update deployed.

curprev 00:5800:58, 14 July 2021‎ imported>Stashbot‎ 662,656 bytes +13,088‎ eileen: process control updated to c291b3c6890364281d

curprev 23:5723:57, 12 July 2021‎ imported>Stashbot‎ 649,568 bytes +13,413‎ urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 1896efc27f3de39659673091bc4c43ad874da0c5: Add sayahna.org to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T286163) (duration: 00m 56s)

curprev 23:2823:28, 9 July 2021‎ imported>Stashbot‎ 636,155 bytes +1,014‎ legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .
curprev 00:4700:47, 9 July 2021‎ imported>Stashbot‎ 635,141 bytes +9,116‎ legoktm: zotero rolling restart didn't help, filed T286360 for DNS issues

curprev 20:2220:22, 7 July 2021‎ imported>Stashbot‎ 626,025 bytes +5,635‎ legoktm: repooling eqiad - https://gerrit.wikimedia.org/r/703561

curprev 18:3418:34, 6 July 2021‎ imported>Stashbot‎ 620,390 bytes +5,171‎ otto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
curprev 00:5600:56, 6 July 2021‎ imported>Stashbot‎ 615,219 bytes +1,686‎ eileen: process-control config revision is 8d46b52ed4

curprev 00:5300:53, 5 July 2021‎ imported>Stashbot‎ 613,533 bytes +528‎ eileen: process-control config revision is a1717c7fde

curprev 17:4617:46, 3 July 2021‎ imported>Stashbot‎ 613,005 bytes +322‎ elukey: depool eqsin due to loss of power redundancy (equinix maintenance) - T286113

curprev 22:0622:06, 2 July 2021‎ imported>Stashbot‎ 612,683 bytes +8,804‎ foks: removing three files for legal compliance
curprev 01:4701:47, 2 July 2021‎ imported>Stashbot‎ 603,879 bytes +9,552‎ eileen: civicrm revision changed from e07c2be1a7 to bb62188ec6, config revision is 1739c53fcb

curprev 23:2823:28, 30 June 2021‎ imported>Stashbot‎ 594,327 bytes +11,040‎ urbanecm: Evening B&C window finished
curprev 00:3600:36, 30 June 2021‎ imported>Stashbot‎ 583,287 bytes +15,990‎ tstarling@deploy1002: Synchronized wmf-config/db-labs.php: gerrit 701995 SQL query log (duration: 01m 05s)

curprev 00:2600:26, 29 June 2021‎ imported>Stashbot‎ 567,297 bytes +16,261‎ Krinkle: krinkle@mwmaint1002: purgeParserCache.php --tag pc1, ref T282761

curprev 09:1909:19, 27 June 2021‎ imported>Elukey‎ 551,036 bytes +741‎ Add missing restart entry (logged on a different chan)

curprev 21:2821:28, 26 June 2021‎ imported>Stashbot‎ 550,295 bytes +1,342‎ volans: upgraded spicerack to v0.0.56 on the cumin hosts (includes only bug fixes for the switchdc)

curprev 21:3721:37, 25 June 2021‎ imported>Stashbot‎ 548,953 bytes +6,188‎ ebernhardson@deploy1002: Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s)

curprev 23:0223:02, 24 June 2021‎ imported>Stashbot‎ 542,765 bytes +11,139‎ legoktm: reverted cumin1001 spicerack live hacks
curprev 00:2500:25, 24 June 2021‎ imported>Stashbot‎ 531,626 bytes +19,731‎ eileen: civicrm revision changed from bd906975f0 to 6d3dd6e5a5, config revision is 821e5889f7

curprev 00:5000:50, 23 June 2021‎ imported>Stashbot‎ 511,895 bytes +14,361‎ eileen: civicrm revision changed from c745d4f075 to 03bead707d, config revision is 4ab72c1033

curprev 01:0201:02, 22 June 2021‎ imported>Stashbot‎ 497,534 bytes +26,215‎ pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE

curprev 20:5520:55, 18 June 2021‎ imported>Stashbot‎ 471,319 bytes +9,077‎ Krinkle: Remove doc1001:/srv/doc/mediawiki-core/wmf-1.36.0-wmf.31-testing

curprev 21:4921:49, 17 June 2021‎ imported>Stashbot‎ 462,242 bytes +15,302‎ legoktm: regenerating pipermail redirects to skip those with duplicate message-ids (T280731)

curprev 21:3521:35, 16 June 2021‎ imported>Stashbot‎ 446,940 bytes +4,876‎ legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .

curprev 17:5417:54, 15 June 2021‎ imported>Stashbot‎ 442,064 bytes +4,022‎ dancy: testing upcoming Scap release on beta
curprev 00:3700:37, 15 June 2021‎ imported>Stashbot‎ 438,042 bytes +26,468‎ eileen: civicrm revision changed from 31d07115a0 to 28ace1b86f, config revision is 2aed6ff89b

curprev 13:4913:49, 12 June 2021‎ imported>Stashbot‎ 411,574 bytes +317‎ rzl@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: alert noise, no impact, x2 is unused

curprev 23:3723:37, 11 June 2021‎ imported>Stashbot‎ 411,257 bytes +6,591‎ mutante: removing firewall hole for mgmt networks to install* because it turned out it cant be used for firmware upgrades
curprev 01:1001:10, 11 June 2021‎ imported>Stashbot‎ 404,666 bytes +17,032‎ eileen: process-control config revision is 2aed6ff89b

curprev 22:1222:12, 9 June 2021‎ imported>Stashbot‎ 387,634 bytes +19,207‎ dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh1002.wikimedia.org
curprev 00:3700:37, 9 June 2021‎ imported>Stashbot‎ 368,427 bytes +17,582‎ catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Enable Wikisource OCR on select Wikisources (T283898) (duration: 01m 31s)

curprev 21:2621:26, 7 June 2021‎ imported>Stashbot‎ 350,845 bytes +11,147‎ otto@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0)

curprev 16:1616:16, 5 June 2021‎ imported>Stashbot‎ 339,698 bytes +307‎ Amir1: deleting all private archives of mm2. All are inaccessible now (T282303)
curprev 00:2100:21, 5 June 2021‎ imported>Stashbot‎ 339,391 bytes +11,476‎ mutante: backup1001 - systemctl baclua-dir works again (restoring backup for non-existing host)

curprev 00:1200:12, 4 June 2021‎ imported>Stashbot‎ 327,915 bytes +24,727‎ ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)

curprev 00:4000:40, 3 June 2021‎ imported>Stashbot‎ 303,188 bytes +15,354‎ ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.7/extensions/Gadgets: Backport: Reduce message parse in GadgetHooks::getPreferences (second time) (T58633 T278650), Try II (duration: 00m 57s)

curprev 21:0921:09, 1 June 2021‎ imported>Stashbot‎ 287,834 bytes +5,348‎ andrewbogott: dropping a bunch of tables from the labswiki db as per T284108
curprev 00:4600:46, 1 June 2021‎ imported>Stashbot‎ 282,486 bytes +974‎ legoktm@deploy1002: Synchronized logos/config.yaml: Revert "Use eswiki 20th anniversary logos" (T280908) (duration: 01m 07s)

curprev 14:4414:44, 29 May 2021‎ imported>Stashbot‎ 281,512 bytes +194‎ elukey: execute apt-get clean on an-airflow1001 to free space

curprev 08:0608:06, 28 May 2021‎ imported>Stashbot‎ 281,318 bytes +368‎ oblivian@cumin1001: conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet,dc=eqiad

curprev 23:5623:56, 27 May 2021‎ imported>Stashbot‎ 280,950 bytes +10,488‎ robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab1004.eqiad.wmnet with reason: REIMAGE

curprev 23:0723:07, 26 May 2021‎ imported>Stashbot‎ 270,462 bytes +14,277‎ ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.7/includes/resourceloader/dependencystore/SqlModuleDependencyStore.php: Backport: resourceloader: Avoid primary connection in SqlModuleDependencyStore (2) (duration: 01m 06s)
curprev 00:4700:47, 26 May 2021‎ imported>Stashbot‎ 256,185 bytes +10,892‎ eileen: civicrm revision changed from 584b96452a to eac772e9c9, config revision is 2ca92c3c3c

curprev 00:4800:48, 25 May 2021‎ imported>Stashbot‎ 245,293 bytes +11,934‎ legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

curprev 14:2514:25, 23 May 2021‎ imported>Stashbot‎ 233,359 bytes +262‎ urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: EMERGENCY: f752f8b80c57a5b7e41b91873b3eac388535ac46: enwiktionary: Raise AF emergency disable treshold+count (T283460) (duration: 00m 57s)

curprev 22:1322:13, 22 May 2021‎ imported>Stashbot‎ 233,097 bytes +265‎ legoktm: reset 2FA for User:Yuvipanda on wikitech

curprev 22:3222:32, 21 May 2021‎ imported>Stashbot‎ 232,832 bytes +12,510‎ bstorm: upload nfsd-ldap: 1.2+deb10u1 to buster-wikimedia T283385

curprev 21:4521:45, 20 May 2021‎ imported>Stashbot‎ 220,322 bytes +13,158‎ herron@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
curprev 01:0101:01, 20 May 2021‎ imported>Stashbot‎ 207,164 bytes +12,243‎ mutante: signing puppet certs for doh2001 and doh2002.wikimedia.org (T283192)

curprev 18:4018:40, 18 May 2021‎ imported>Stashbot‎ 194,921 bytes +14,463‎ razzi@deploy1002: Finished deploy [analytics/refinery@9392f1d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@9392f1db6e66975304c8e9b2b7031acd3ed87fa7] (duration: 05m 16s)
curprev 00:5600:56, 18 May 2021‎ imported>Stashbot‎ 180,458 bytes +19,706‎ eileen: civicrm revision changed from 38ac15233f to b3fb3c9cb0, config revision is 1f8d0a6bfa

curprev 00:1300:13, 17 May 2021‎ imported>Stashbot‎ 160,752 bytes +408‎ legoktm: installing python3-dbg on lists1001

curprev 00:5300:53, 16 May 2021‎ imported>Stashbot‎ 160,344 bytes +324‎ legoktm: restarted mailman3-web on lists1001, uwsgi looked like it got stuck, consuming all CPU/memory

curprev 20:4220:42, 14 May 2021‎ imported>Stashbot‎ 160,020 bytes +3,428‎ dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people1002.eqiad.wmnet
curprev 00:3900:39, 14 May 2021‎ imported>Stashbot‎ 156,592 bytes +8,035‎ ryankemper: T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2003.codfw.wmnet` on `ryankemper@cumin2001` tmux session `wdqs_reimage`

curprev 23:4823:48, 12 May 2021‎ imported>Stashbot‎ 148,557 bytes +16,709‎ urbanecm@deploy1002: Synchronized php-1.37.0-wmf.4/extensions/WikiEditor/includes/WikiEditorHooks.php: 2f6af514c49d47bbec5ce51f9f7263015e039003? PHP VisualEditorFeatureUse logging: properly record session id (T281409) (duration: 01m 07s)
curprev 01:4201:42, 12 May 2021‎ imported>Stashbot‎ 131,848 bytes +17,502‎ dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host

curprev 23:3823:38, 10 May 2021‎ imported>Stashbot‎ 114,346 bytes +15,358‎ urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: 779fb53bfd7a4d9b11f865df14f8a72adb97f33b: Update messages used for tech CoC (T280886) (duration: 00m 56s)

curprev 21:4421:44, 9 May 2021‎ imported>Stashbot‎ 98,988 bytes −1,654,362‎ legoktm: restarted mailman3 again (T282348) pymysql.err.InternalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')

curprev 17:1817:18, 8 May 2021‎ imported>Stashbot‎ 1,753,350 bytes +105‎ Amir1: starting upgrade of batch G of mailing lists (T280322)

curprev 21:4021:40, 7 May 2021‎ imported>Stashbot‎ 1,753,245 bytes +7,667‎ legoktm: deleted education@ from MM3, didn't import properly

curprev 23:5023:50, 6 May 2021‎ imported>Stashbot‎ 1,745,578 bytes +25,003‎ brennen@deploy1002: rebuilt and synchronized wikiversions files: Rollback group1 and group2 to 1.37.0-wmf.3 (T282193)
curprev 00:3500:35, 6 May 2021‎ imported>Stashbot‎ 1,720,575 bytes +28,723‎ Amir1: sudo service mailman3-web restart

curprev 01:4501:45, 5 May 2021‎ imported>Stashbot‎ 1,691,852 bytes +24,478‎ ryankemper: T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1011.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph` on `ryankemper@cumin1001` tmux session `reimage`

curprev 23:1823:18, 3 May 2021‎ imported>Stashbot‎ 1,667,374 bytes +11,155‎ urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 230ef5716b34ca83348667f289180313b76ce8a3: Prepare for new configuration option (T277951) (duration: 00m 57s)

curprev 13:4013:40, 2 May 2021‎ imported>Stashbot‎ 1,656,219 bytes +311‎ dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cloudmetrics1002.eqiad.wmnet with reason: Flaky host

curprev 19:1219:12, 1 May 2021‎ imported>Stashbot‎ 1,655,908 bytes +557‎ Urbanecm: Invalidate password for MaraBot@SUL (T281586)

curprev 21:5521:55, 30 April 2021‎ imported>Stashbot‎ 1,655,351 bytes +6,117‎ mutante: people1003 - rsycncing /home from peopel1002
curprev 01:0801:08, 30 April 2021‎ imported>Stashbot‎ 1,649,234 bytes +23,564‎ ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw reboot - ryankemper@cumin1001 - T280563

curprev 00:4100:41, 29 April 2021‎ imported>Stashbot‎ 1,625,670 bytes +24,040‎ tstarling@deploy1002: Synchronized php-1.37.0-wmf.3/includes/specials/pagers/ImageListPager.php: T281405 (duration: 01m 08s)

curprev 01:0601:06, 28 April 2021‎ imported>Stashbot‎ 1,601,630 bytes +35,800‎ robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

curprev 23:2823:28, 26 April 2021‎ imported>Stashbot‎ 1,565,830 bytes +16,596‎ mutante: renewing TLS cert for peopleweb.discovery.wmnet, adding *3 hosts

curprev 15:2315:23, 25 April 2021‎ imported>Stashbot‎ 1,549,234 bytes +138‎ Amir1: sudo -u list /var/lib/mailman/bin/change_pw -l wikica-l -p $(pwgen -c1 -s 12) (T281066)

curprev 22:2422:24, 24 April 2021‎ imported>Stashbot‎ 1,549,096 bytes +78‎ bstorm: Rebooting labstore1007 from ilo after crash

curprev 21:3621:36, 23 April 2021‎ imported>Stashbot‎ 1,549,018 bytes +3,966‎ foks: removing 1 file for legal compliance

curprev 17:2617:26, 22 April 2021‎ imported>Stashbot‎ 1,545,052 bytes +1,074‎ marostegui: Stop mysql on tendril/dbtree database
curprev 01:3401:34, 22 April 2021‎ imported>Stashbot‎ 1,543,978 bytes +14,300‎ eileen: civicrm revision changed from 35a8dd33ba to 42ca3cf65a, config revision is cf07e7ba0b

curprev 00:3800:38, 21 April 2021‎ imported>Stashbot‎ 1,529,678 bytes +27,920‎ pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE

curprev 22:5622:56, 19 April 2021‎ imported>Stashbot‎ 1,501,758 bytes +29,254‎ robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE

curprev 06:4006:40, 18 April 2021‎ imported>Stashbot‎ 1,472,504 bytes +106‎ Amir1: cleaning watchlist of User:Mr._Ibrahem in wikidatawiki (in main ns only)

curprev 16:1616:16, 17 April 2021‎ imported>Stashbot‎ 1,472,398 bytes +61‎ Amir1: cleaning SuccuBot's watchlist in wikidatawiki
curprev 00:5300:53, 17 April 2021‎ imported>Stashbot‎ 1,472,337 bytes +38,446‎ dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet

curprev 01:1901:19, 15 April 2021‎ imported>Stashbot‎ 1,433,891 bytes +21,435‎ Amir1: mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P8671 --new-data-type external-id (T278427)

curprev 00:1100:11, 14 April 2021‎ imported>Stashbot‎ 1,412,456 bytes +25,158‎ legoktm@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw2411.codfw.wmnet

curprev 23:2523:25, 12 April 2021‎ imported>Stashbot‎ 1,387,298 bytes +4,421‎ krinkle@deploy1002: Synchronized wmf-config/mc.php: I390b4726d01037107 (duration: 00m 58s)

curprev 14:2114:21, 10 April 2021‎ imported>Stashbot‎ 1,382,877 bytes +815‎ andrew@deploy1002: Finished deploy [horizon/deploy@ee1be56]: fix for T279699 (duration: 04m 12s)

curprev 14:0714:07, 9 April 2021‎ imported>Stashbot‎ 1,382,062 bytes +59‎ jynus: retry es4 backup dump on eqiad (backup1002)
curprev 01:2501:25, 9 April 2021‎ imported>Stashbot‎ 1,382,003 bytes +23,659‎ pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on moss-be2002.codfw.wmnet with reason: REIMAGE

curprev 23:3823:38, 7 April 2021‎ imported>Stashbot‎ 1,358,344 bytes +22,437‎ ejegg: updated payments-wiki from b06009c099 to 70f5163816,
curprev 01:4601:46, 7 April 2021‎ imported>Stashbot‎ 1,335,907 bytes +16,883‎ pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: REIMAGE

curprev 23:1723:17, 5 April 2021‎ imported>Stashbot‎ 1,319,024 bytes +12,454‎ AaronSchulz: Running importMissingLocalNames.php on mwmaint1002 in a screen

curprev 14:4714:47, 4 April 2021‎ imported>Stashbot‎ 1,306,570 bytes +265‎ andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 01m 36s)

curprev 19:2019:20, 3 April 2021‎ imported>Stashbot‎ 1,306,305 bytes +1,408‎ andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 02m 11s)

curprev 22:3122:31, 2 April 2021‎ imported>Stashbot‎ 1,304,897 bytes +10,597‎ bstorm@cumin1001: END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0)

curprev 23:3223:32, 1 April 2021‎ imported>Stashbot‎ 1,294,300 bytes +18,843‎ thcipriani@deploy1002: Synchronized php-1.36.0-wmf.37/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: Backport: Revert "Turn on glent m1 AB test" T262612 (duration: 00m 58s)
curprev 00:5600:56, 1 April 2021‎ imported>Stashbot‎ 1,275,457 bytes +7,735‎ pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2381.codfw.wmnet with reason: REIMAGE

curprev 23:5923:59, 30 March 2021‎ imported>Stashbot‎ 1,267,722 bytes +13,942‎ Trey314159: reindexing English wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T274200)

curprev 19:0619:06, 29 March 2021‎ imported>Stashbot‎ 1,253,780 bytes +3,056‎ hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=aqs1004.eqiad.wmnet

curprev 19:2619:26, 27 March 2021‎ imported>Stashbot‎ 1,250,724 bytes +1,286‎ elukey: powercycle elastic1060 - T278630

curprev 22:2722:27, 26 March 2021‎ imported>Stashbot‎ 1,249,438 bytes +5,934‎ tzatziki: reset password for Philroc

curprev 23:4723:47, 25 March 2021‎ imported>Stashbot‎ 1,243,504 bytes +12,592‎ thcipriani@deploy1002: Synchronized php-1.36.0-wmf.36/extensions/3D/package.json: No-op demo sync (duration: 01m 07s)
curprev 01:3301:33, 25 March 2021‎ imported>Stashbot‎ 1,230,912 bytes +22,060‎ ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)

curprev 00:2200:22, 24 March 2021‎ imported>Stashbot‎ 1,208,852 bytes +46,207‎ pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2378.codfw.wmnet with reason: REIMAGE

curprev 00:0800:08, 23 March 2021‎ imported>Stashbot‎ 1,162,645 bytes +17,284‎ tstarling@deploy1002: Synchronized wmf-config: use RequestTimeout library step 3: clean up (duration: 00m 58s)

curprev 10:2510:25, 21 March 2021‎ imported>Stashbot‎ 1,145,361 bytes +368‎ _joe_: restarting gerrit on gerrit1001, using 45G of reserved memory

curprev 00:2200:22, 20 March 2021‎ imported>Stashbot‎ 1,144,993 bytes +6,348‎ tzatziki: altering emails for STei (WMF) and SGrabarczuk (WMF)

curprev 00:5100:51, 19 March 2021‎ imported>Stashbot‎ 1,138,645 bytes +9,921‎ dancy@deploy1002: Synchronized php-1.36.0-wmf.35/extensions/LiquidThreads/classes/Thread.php: T277772 (duration: 00m 58s)

curprev 00:0500:05, 18 March 2021‎ imported>Stashbot‎ 1,128,724 bytes +22,211‎ eileen: tools revision changed from b7b4060c30 to ef54260b0d

curprev 23:5623:56, 16 March 2021‎ imported>Stashbot‎ 1,106,513 bytes +44,018‎ krinkle@deploy1002: Synchronized php-1.36.0-wmf.35/includes/Revision/: I8619ab9e92b, T277362, T275531 (duration: 00m 58s)

curprev 23:3123:31, 15 March 2021‎ imported>Stashbot‎ 1,062,495 bytes +21,243‎ legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Remove back-compat from when IRC feed servers was a string (T224579) (duration: 00m 59s)

curprev 17:5717:57, 14 March 2021‎ imported>Stashbot‎ 1,041,252 bytes +1,114‎ marostegui@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Repool db1146:3314', diff saved to https://phabricator.wikimedia.org/P14827 and previous config saved to /var/cache/conftool/dbconfig/20210314-175751-root.json

curprev 19:0219:02, 13 March 2021‎ imported>Stashbot‎ 1,040,138 bytes +699‎ Amir1: change default charset of all core tables in labstestwiki to binary (T269348)
curprev 00:5500:55, 13 March 2021‎ imported>Stashbot‎ 1,039,439 bytes +10,754‎ mutante: [wdqs1009:/etc/envoy] $ sudo /usr/local/sbin/build-envoy-config -c /etc/envoy/

curprev 01:3001:30, 12 March 2021‎ imported>Stashbot‎ 1,028,685 bytes +31,013‎ dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2215.codfw.wmnet

curprev 00:5000:50, 11 March 2021‎ imported>Stashbot‎ 997,672 bytes +37,248‎ dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2219.codfw.wmnet

curprev 01:0801:08, 10 March 2021‎ imported>Stashbot‎ 960,424 bytes +27,385‎ krinkle@deploy1002: Synchronized php-1.36.0-wmf.34/extensions/NavigationTiming/modules/ext.navigationTiming.js: T276826 Ibd9ddf14d64 (duration: 01m 14s)

curprev 00:5800:58, 9 March 2021‎ imported>Stashbot‎ 933,039 bytes +16,107‎ Krinkle: krinkle@mwmaint1002 Ran invalidateUserSesssions.php for one user

curprev 08:0108:01, 7 March 2021‎ imported>Stashbot‎ 916,932 bytes +170‎ elukey: "megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll" on analytics1066 - BBU looks fine, but the raid controller was using WriteThrough

curprev 23:1623:16, 5 March 2021‎ imported>Stashbot‎ 916,762 bytes +8,775‎ legoktm: imported pygments 2.8.0+dfsg-1 to apt.wm.o buster-wikimedia component/pygments (T276298)
curprev 00:5900:59, 5 March 2021‎ imported>Stashbot‎ 907,987 bytes +16,215‎ legoktm: depooled registry1001/registry1002 (old stretch VMs) - T272550

curprev 01:2401:24, 4 March 2021‎ imported>Stashbot‎ 891,772 bytes +30,062‎ twentyafterfour: phabricator upgrade complete

curprev 00:4200:42, 3 March 2021‎ imported>Stashbot‎ 861,710 bytes +24,132‎ Urbanecm: Finished deployment in Evening B&C window; logmsgbot is currently down, and a simple restart did not bring it back up

curprev 00:5900:59, 2 March 2021‎ imported>Stashbot‎ 837,578 bytes +21,612‎ urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 0f08e8bbe1f74220a7a01a7606b67e0f75734a53: Update the Persian Wikipedia logos (T261033; 2/2) (duration: 00m 56s)

curprev 14:1714:17, 28 February 2021‎ imported>Stashbot‎ 815,966 bytes +71‎ gehel: repooled wdqs1011 - catched up on lag

curprev 21:1921:19, 27 February 2021‎ imported>Stashbot‎ 815,895 bytes +260‎ dwisehaupt: ran the following on frdb2002 to allow replication to continue after conversion to utf8mb4 charset: set global slave_type_conversions = ALL_NON_LOSSY;
curprev 00:0800:08, 27 February 2021‎ imported>Stashbot‎ 815,635 bytes +26,358‎ mutante: deploy1002 - rsyncing home dirs from deploy1001

curprev 00:1400:14, 26 February 2021‎ imported>Stashbot‎ 789,277 bytes +25,251‎ urbanecm@deploy1001: Synchronized php-1.36.0-wmf.32/extensions/Graph/: 9d5cf348f5dda32f8889d5160bb1fe34a4e07f8c: Do not log graph errors to WMF servers (T274557) (duration: 01m 36s)

curprev 00:2900:29, 25 February 2021‎ imported>Stashbot‎ 764,026 bytes +22,501‎ ryankemper: T274204 Restored service health on `elastic106[0,4,5]` via `sudo apt-get remove --purge wmf-elasticsearch-search-plugins --yes && sudo dpkg -i /var/cache/apt/archives/wmf-elasticsearch-search-plugins_6.5.4-4~stretch_all.deb && sudo puppet agent -tv`. There's some sort of issue with `6.5.4-5~stretch` that we will need to circle back and investigate; for now the fleet is staying on `6.5.4-4~stretch`

curprev 00:5800:58, 24 February 2021‎ imported>Stashbot‎ 741,525 bytes +17,976‎ volker-e@deploy1001: Finished deploy [design/style-guide@a66b5b6]: Deploy design/style-guide: a66b5b6 “Components”: Add “Dialogs” (#430) (duration: 00m 06s)

curprev 00:4600:46, 23 February 2021‎ imported>Stashbot‎ 723,549 bytes +23,528‎ eileen: civicrm revision changed from c535ac603a to 5e042e6e57, config revision is ef64f705bb

curprev 16:0316:03, 21 February 2021‎ imported>Stashbot‎ 700,021 bytes +990‎ marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1162 - crashed', diff saved to https://phabricator.wikimedia.org/P14424 and previous config saved to /var/cache/conftool/dbconfig/20210221-160258-marostegui.json

curprev 00:1700:17, 20 February 2021‎ imported>Stashbot‎ 699,031 bytes +13,319‎ dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1317.eqiad.wmnet

curprev 00:5400:54, 19 February 2021‎ imported>Stashbot‎ 685,712 bytes +7,907‎ mutante: mwmaint2001 - back from reimage - scap pull

curprev 01:3801:38, 18 February 2021‎ imported>Stashbot‎ 677,805 bytes +16,614‎ robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)

curprev 01:0001:00, 17 February 2021‎ imported>Stashbot‎ 661,191 bytes +18,673‎ dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mwdebug1001.eqiad.wmnet

curprev 21:3321:33, 15 February 2021‎ imported>Stashbot‎ 642,518 bytes +7,693‎ eileen: civicrm revision changed from dfbb8f41bc to c535ac603a, config revision is ba9b2380b1

curprev 13:1313:13, 14 February 2021‎ imported>Stashbot‎ 634,825 bytes +806‎ akosiaris: sudo cumin -b 1 -s 120 'cp500[2,3,5,6].eqsin.wmnet' 'systemctl restart varnish-frontend.service'

curprev 03:2403:24, 13 February 2021‎ imported>Stashbot‎ 634,019 bytes +232‎ ryankemper: Depooled `wdqs1006` to catch up on lag
curprev 01:0001:00, 13 February 2021‎ imported>Stashbot‎ 633,787 bytes +21,925‎ crusnov@cumin1001: START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet

curprev 01:2101:21, 12 February 2021‎ imported>Stashbot‎ 611,862 bytes +30,779‎ legoktm@cumin1001: conftool action : set/pooled=yes; selector: name=mw1329.eqiad.wmnet

curprev 00:2600:26, 11 February 2021‎ imported>Stashbot‎ 581,083 bytes +30,413‎ legoktm@deploy1001: Synchronized wmf-config/profiler.php: Revert "profiler: Send data to excimer-buster pipeline" (duration: 02m 00s)

curprev 00:5800:58, 10 February 2021‎ imported>Stashbot‎ 550,670 bytes +36,435‎ mutante: doc1001 - reloaded apache2

curprev 00:4800:48, 9 February 2021‎ imported>Stashbot‎ 514,235 bytes +25,431‎ dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw1302.eqiad.wmnet

curprev 22:5822:58, 7 February 2021‎ imported>Stashbot‎ 488,804 bytes +95‎ Urbanecm: Reset password for TheresNoTime (T274087)

curprev 08:5908:59, 6 February 2021‎ imported>Stashbot‎ 488,709 bytes +1,136‎ elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro-from-cdh-clients (exit_code=0) for Hadoop test cluster: Change Hadoop distribution - elukey@cumin1001
curprev 01:0001:00, 6 February 2021‎ imported>Stashbot‎ 487,573 bytes +17,493‎ dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2265.codfw.wmnet

curprev 00:5900:59, 5 February 2021‎ imported>Stashbot‎ 470,080 bytes +27,954‎ ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@85713c1]: restore data range specifier in extract job partition spec

curprev 01:1501:15, 4 February 2021‎ imported>Stashbot‎ 442,126 bytes +22,570‎ ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@4b4872d]: transfer_to_es: Increase timeout waiting for source data to three hours (duration: 01m 16s)

curprev 00:1600:16, 3 February 2021‎ imported>Stashbot‎ 419,556 bytes +21,008‎ jhuneidi@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .

curprev 23:5423:54, 1 February 2021‎ imported>Stashbot‎ 398,548 bytes +28,951‎ legoktm@deploy1001: Synchronized wmf-config/profiler.php: profiler: Send data to excimer-buster pipeline (T273312) (duration: 00m 57s)

curprev 23:2623:26, 29 January 2021‎ imported>Stashbot‎ 369,597 bytes +9,826‎ razzi@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001
curprev 00:1900:19, 29 January 2021‎ imported>Stashbot‎ 359,771 bytes +37,885‎ legoktm@cumin1001: conftool action : set/pooled=yes; selector: name=mw2261.codfw.wmnet

curprev 00:5600:56, 28 January 2021‎ imported>Stashbot‎ 321,886 bytes +25,610‎ legoktm@cumin1001: conftool action : set/pooled=no; selector: name=mw2294.codfw.wmnet

curprev 00:5400:54, 27 January 2021‎ imported>Stashbot‎ 296,276 bytes +26,066‎ legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2296.codfw.wmnet with reason: REIMAGE

curprev 00:4800:48, 26 January 2021‎ imported>Stashbot‎ 270,210 bytes +18,727‎ dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2331.codfw.wmnet

curprev 22:2122:21, 23 January 2021‎ imported>Stashbot‎ 251,483 bytes +1,766‎ volker-e@deploy1001: Finished deploy [design/style-guide@63e39e7]: Deploy design/style-guide: 63e39e7 “Components”: Amend button groups states SVG font stack (#427) (duration: 00m 06s)
curprev 00:4600:46, 23 January 2021‎ imported>Stashbot‎ 249,717 bytes +27,911‎ legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch enwiki to use enwiki20 "Option A" logo variant (T272526) (duration: 00m 57s)

curprev 01:0001:00, 22 January 2021‎ imported>Stashbot‎ 221,806 bytes +15,875‎ Urbanecm: Evening B&C still in process, waiting on Zuul

curprev 01:2701:27, 21 January 2021‎ imported>Stashbot‎ 205,931 bytes +34,902‎ ryankemper: [WDQS Deploy] Rollback complete, service health of `wdqs1003` is restored. Need to investigate source of 404 (possibly related to some recent changes we made in the `gui` repo)

curprev 00:4300:43, 20 January 2021‎ imported>Stashbot‎ 171,029 bytes +23,826‎ tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Update /analytics/legacy/homepagemodule/ schema version to 1.1.0 (T270309) (duration: 01m 03s)

curprev 21:3321:33, 18 January 2021‎ imported>Stashbot‎ 147,203 bytes +19,595‎ eileen: civicrm revision changed from 4220fc8177 to a4caad22b1, config revision is f08249ecf9

curprev 12:1812:18, 16 January 2021‎ imported>Stashbot‎ 127,608 bytes +369‎ elukey: elukey@cumin1001:~$ sudo cumin 'A:mw-app-canary and A:mw-eqiad' 'run-puppet-agent' -b 10 - T272215

curprev 23:5923:59, 15 January 2021‎ imported>Stashbot‎ 127,239 bytes +6,731‎ robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE

curprev 23:3523:35, 14 January 2021‎ imported>Stashbot‎ 120,508 bytes +16,149‎ dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2236.codfw.wmnet
curprev 00:2200:22, 14 January 2021‎ imported>Stashbot‎ 104,359 bytes +24,462‎ ryankemper: T266492 Restart of `relforge` successful

curprev 22:5522:55, 12 January 2021‎ imported>Stashbot‎ 79,897 bytes +10,636‎ dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2225.codfw.wmnet

curprev 22:1622:16, 11 January 2021‎ imported>Stashbot‎ 69,261 bytes +16,348‎ eileen: process-control config revision is f08249ecf9 eoy jobs disabled

curprev 12:5512:55, 10 January 2021‎ imported>Nintendofan885‎ 52,913 bytes −155,895‎ Archiving

curprev 00:1100:11, 9 January 2021‎ imported>Stashbot‎ 208,808 bytes +13,848‎ mutante: puppetmaster2003 - restarted apache after spweing 500s

curprev 01:2901:29, 8 January 2021‎ imported>Stashbot‎ 194,960 bytes +13,843‎ dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1267.eqiad.wmnet

curprev 00:5000:50, 7 January 2021‎ imported>Stashbot‎ 181,117 bytes +8,175‎ eileen: civicrm revision changed from 95fefcb421 to 2df572bdcd, config revision is 603c0ff4a0

curprev 01:2201:22, 6 January 2021‎ imported>Stashbot‎ 172,942 bytes +9,241‎ mutante: testreduce1001 rm -rf /srv/deployment/parsoid/deploy

curprev 00:4000:40, 5 January 2021‎ imported>Stashbot‎ 163,701 bytes +5,675‎ jgleeson: updated civicrm revision changed from bb8baac617 to 6be8a130df

curprev 16:1716:17, 3 January 2021‎ imported>Stashbot‎ 158,026 bytes +627‎ arturo: merged change to TLS cert used by slapd/openldap servers https://gerrit.wikimedia.org/r/c/operations/puppet/+/653871

curprev 19:2719:27, 2 January 2021‎ imported>Stashbot‎ 157,399 bytes +74‎ vgutierrez: restart acme-chief on acmechief1001

curprev 14:4914:49, 1 January 2021‎ imported>Stashbot‎ 157,325 bytes +1,317‎ milimetric@deploy1001: Finished deploy [analytics/refinery@f9281dd] (thin): [SAFE, IGNORE] Simple hotfix for a python bug, analytics refinery only, not urgent (duration: 00m 07s)

curprev 15:5215:52, 29 December 2020‎ imported>Stashbot‎ 156,008 bytes +210‎ vgutierrez: reloading nginx on cloudelastic1005 and cloudelastic1006

Server Admin Log: Revision history

3 October 2021

2 October 2021

1 October 2021

29 September 2021

27 September 2021

26 September 2021

25 September 2021

24 September 2021

23 September 2021

21 September 2021

18 September 2021

17 September 2021

15 September 2021

14 September 2021

12 September 2021

11 September 2021

10 September 2021

9 September 2021

8 September 2021

6 September 2021

5 September 2021

4 September 2021

3 September 2021

1 September 2021

30 August 2021

29 August 2021

27 August 2021

25 August 2021

24 August 2021

22 August 2021

21 August 2021

20 August 2021

19 August 2021

18 August 2021

17 August 2021

15 August 2021

14 August 2021

13 August 2021

12 August 2021

11 August 2021

9 August 2021

6 August 2021

5 August 2021

3 August 2021

31 July 2021

29 July 2021

28 July 2021

27 July 2021

26 July 2021

24 July 2021

23 July 2021

22 July 2021

20 July 2021

19 July 2021

16 July 2021

15 July 2021

14 July 2021

12 July 2021

9 July 2021

7 July 2021

6 July 2021

5 July 2021

3 July 2021

2 July 2021

30 June 2021

29 June 2021

27 June 2021

26 June 2021

25 June 2021

24 June 2021

23 June 2021

22 June 2021

18 June 2021

17 June 2021

16 June 2021

15 June 2021

12 June 2021

11 June 2021

9 June 2021