You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Revision history of "Server Admin Log"

Jump to navigation Jump to search

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 500 | ) (20 | 50 | 100 | 250 | 500)
  • curprev 01:14, 4 December 2021imported>Stashbot 424,523 bytes +12,137 mutante: mx2001 - did not come back from reboot, did not get IP on interface, could not start ferm, logged in via console with root password, in /etc/network/interfaces replaced all "ens5" with "ens13", rebooted again, selected previous kernel version
  • curprev 01:00, 3 December 2021imported>Stashbot 412,386 bytes +17,768 tgr@deploy1002: Synchronized php-1.38.0-wmf.9/extensions/GrowthExperiments: Backport: Add an image: Add test version of GEInfoboxTemplates (T291232) (duration: 00m 57s)
  • curprev 01:21, 2 December 2021imported>Stashbot 394,618 bytes +24,122 ryankemper: T280001 Rolling restart of low-traffic pybal hosts complete. All of `wcqs` is pooled and the pybal / ipvs related alerts have cleared
  • curprev 00:35, 1 December 2021imported>Stashbot 370,496 bytes +9,705 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:22, 30 November 2021imported>Stashbot 360,791 bytes +17,004 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 17:14, 28 November 2021imported>Stashbot 343,787 bytes +293 elukey@deploy1002: Finished deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 (duration: 02m 11s)
  • curprev 19:55, 27 November 2021imported>Stashbot 343,494 bytes +1,043 andrew@deploy1002: Finished deploy [horizon/deploy@6115b3b]: network UI updates for T296548 (duration: 04m 14s)
  • curprev 16:11, 26 November 2021imported>Stashbot 342,451 bytes +4,657 arnoldokoth: drain kubestage1002 node in prep for decommissioning
  • curprev 20:44, 25 November 2021imported>Stashbot 337,794 bytes +25,467 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17872 and previous config saved to /var/cache/conftool/dbconfig/20211125-204357-ladsgroup.json
  • curprev 00:37, 25 November 2021imported>Stashbot 312,327 bytes +56,025 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2066.codfw.wmnet with OS buster
  • curprev 00:25, 24 November 2021imported>Stashbot 256,302 bytes +24,082 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS buster
  • curprev 00:51, 23 November 2021imported>Stashbot 232,220 bytes +27,289 urbanecm@deploy1002: Started scap: 69aa4a7: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 4/4)
  • curprev 13:17, 21 November 2021imported>Stashbot 204,931 bytes +326 dcausse: restarting blazegraph on wdqs1007 (jvm stuck for 10h)
  • curprev 01:02, 20 November 2021imported>Stashbot 204,605 bytes +91 mutante: lists1001 - restarted apache, icinga alerts for the web UI, but recovered
  • curprev 00:27, 20 November 2021imported>Stashbot 204,514 bytes +9,398 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
  • curprev 01:45, 19 November 2021imported>Stashbot 195,116 bytes +21,872 mutante: I think git-ssh6_22 is down (see alerts lvs2008/2009) due to the v6 issue from ongoing lvs maintenance. depooled in conftool
  • curprev 01:47, 18 November 2021imported>Stashbot 173,244 bytes +19,375 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2005.codfw.wmnet
  • curprev 00:19, 17 November 2021imported>Stashbot 153,869 bytes +12,582 ryankemper: T276198 `ryankemper@cumin1001:~$ sudo cumin -b 3 '*elastic*' 'sudo run-puppet-agent --force'` Change looks good (no complaints from systemd), rolling out to rest of fleet / reenabling puppet
  • curprev 00:28, 16 November 2021imported>Stashbot 141,287 bytes +15,831 urbanecm: UTC late window done
  • curprev 11:48, 14 November 2021imported>Stashbot 125,456 bytes +111 paravoid: disable cr1-eqiad:xe-3/0/6 (IXP port) to mitigate T295650
  • curprev 18:43, 13 November 2021imported>Stashbot 125,345 bytes +836 AndyRussG: Enabled debug logging for PayPal IPN listener (updated SmashPig config a9e30591 -> 9567cc4a on frpig1001)
  • curprev 21:00, 12 November 2021imported>Stashbot 124,509 bytes +5,102 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:19, 12 November 2021imported>Stashbot 119,407 bytes +11,807 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:22, 11 November 2021imported>Stashbot 107,600 bytes +12,063 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:07, 10 November 2021imported>Stashbot 95,537 bytes +10,338 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 23:39, 8 November 2021imported>Stashbot 85,199 bytes +11,520 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:43, 6 November 2021imported>Stashbot 73,679 bytes +7,012 dduvall@deploy1002: Synchronized php-1.38.0-wmf.7/includes/parser/ParserOutput.php: Backport: Regression fix: do language conversion on ToC in ParserOutput::getText() (T295187) (duration: 00m 56s)
  • curprev 00:16, 5 November 2021imported>Stashbot 66,667 bytes +20,851 mutante: phab1001 - sudo systemctl start phabricator_clean_tmp_files.service because Icinga alerted it had failed... worked fine
  • curprev 00:26, 4 November 2021imported>Stashbot 45,816 bytes +22,857 tgr: UTC late deploys done
  • curprev 23:47, 2 November 2021imported>Stashbot 22,959 bytes +11,789 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:21, 2 November 2021imported>Stashbot 11,170 bytes −844,596 ejegg: updated SmashPig standalone deploy from dd3a81c7c2 to be68299b92
  • curprev 21:49, 31 October 2021imported>Stashbot 855,766 bytes +371 urbanecm: urbanecm@mwmaint1002:~$ mwscript userOptions.php --wiki=dewiki --nowarn --touserid 3802752 --old 'linkrecommendation' --new 'control' 'growthexperiments-homepage-variant' # T294712
  • curprev 19:25, 30 October 2021imported>Stashbot 855,395 bytes +168 dcausse: restarting blazegraph on wdqs1007 (jvm stuck)
  • curprev 22:57, 29 October 2021imported>Stashbot 855,227 bytes +3,905 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=foundationwiki --userlist users.txt # T205347, users.txt is at P17641
  • curprev 23:50, 28 October 2021imported>Stashbot 851,322 bytes +21,486 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4033.ulsfo.wmnet with OS buster
  • curprev 23:55, 27 October 2021imported>Stashbot 829,836 bytes +10,201 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 22:59, 26 October 2021imported>Stashbot 819,635 bytes +12,871 legoktm: uploaded python-logstash to buster-wikimedia for T294393
  • curprev 23:12, 25 October 2021imported>Stashbot 806,764 bytes +11,982 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Create alias for Appendix and Appendix_talk namespaces on mywiktionary (T291146) (duration: 00m 55s)
  • curprev 16:40, 23 October 2021imported>Stashbot 794,782 bytes +253 dcausse: restarting blazegraph on wdqs1004 and wdqs1006 (free allocators alert)
  • curprev 23:17, 22 October 2021imported>Stashbot 794,529 bytes +3,761 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 23:40, 21 October 2021imported>Stashbot 790,768 bytes +17,384 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:06, 21 October 2021imported>Stashbot 773,384 bytes +14,742 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:31, 20 October 2021imported>Stashbot 758,642 bytes +27,590 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:38, 19 October 2021imported>Stashbot 731,052 bytes +11,211 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
  • curprev 03:56, 16 October 2021imported>Stashbot 719,841 bytes +266 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
  • curprev 23:48, 15 October 2021imported>Stashbot 719,575 bytes +5,052 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
  • curprev 00:09, 15 October 2021imported>Stashbot 714,523 bytes +19,109 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
  • curprev 01:41, 14 October 2021imported>Stashbot 695,414 bytes +17,092 ejegg: updated payments-wiki from b329d2dea2 to 19d18c1852
  • curprev 00:38, 13 October 2021imported>Stashbot 678,322 bytes +15,911 ejegg: updated payments-wiki from 030b11da1a to b329d2dea2
  • curprev 00:11, 12 October 2021imported>Stashbot 662,411 bytes +3,728 eileen: civicrm revision changed from 598b59b0ee to 96090e4bd2, config revision is 85277466ed
  • curprev 05:01, 9 October 2021imported>Stashbot 658,683 bytes +224 jiji@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:32, 9 October 2021imported>Stashbot 658,459 bytes +8,808 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814
  • curprev 00:07, 8 October 2021imported>Stashbot 649,651 bytes +11,887 tgr_: deploy window over
  • curprev 00:11, 7 October 2021imported>Stashbot 637,764 bytes +11,864 mutante: [grafana2001:~] $ sudo systemctl start rsync-var-lib-grafana because of "PROBLEM - Check systemd state on grafana2001 is CRITICAL: CRITICAL - degraded" because of some race condition where a file vanished during sync
  • curprev 01:39, 6 October 2021imported>Stashbot 625,900 bytes +10,381 legoktm: legoktm@mwmaint1002:~$ echo "https://en.wikiversity.org/static/images/mobile/copyright/wikiversity.svg" |mwscript purgeList.php
  • curprev 23:30, 4 October 2021imported>Stashbot 615,519 bytes +9,952 foks: resetting some emails used for abuse by a globally-banned user
  • curprev 14:45, 3 October 2021imported>Stashbot 605,567 bytes +527 _joe_: restarting acmechief on acmechief1001
  • curprev 17:28, 2 October 2021imported>Stashbot 605,040 bytes +230 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 23:19, 1 October 2021imported>Stashbot 604,810 bytes +38,661 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 23:20, 29 September 2021imported>Stashbot 566,149 bytes +14,880 bd808@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 00:52, 29 September 2021imported>Stashbot 551,269 bytes +27,094 eileen: civicrm revision changed from a1929b3dfd to a480bf03c9, config revision is 77cb7ec866
  • curprev 23:42, 27 September 2021imported>Stashbot 524,175 bytes +27,476 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 14:51, 26 September 2021imported>Stashbot 496,699 bytes +432 volker-e@deploy1002: Finished deploy [design/style-guide@aac0ae9]: Deploy design/style-guide: aac0ae9 “Apps”: Fix image path (#490) (duration: 00m 06s)
  • curprev 02:00, 25 September 2021imported>Stashbot 496,267 bytes +374 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .
  • curprev 20:00, 24 September 2021imported>Stashbot 495,893 bytes +10,338 volker-e@deploy1002: Finished deploy [design/style-guide@362c6b1]: Deploy design/style-guide: 362c6b1 “Components”: Fix index link (#489) (duration: 00m 06s)
  • curprev 00:39, 24 September 2021imported>Stashbot 485,555 bytes +19,157 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:35, 23 September 2021imported>Stashbot 466,398 bytes +12,744 catrope@deploy1002: Synchronized php-1.38.0-wmf.1/extensions/MediaSearch/: Use text() instead of parse() for MediaSearch UI messages (T291590) (duration: 01m 08s)
  • curprev 23:20, 21 September 2021imported>Stashbot 453,654 bytes +9,016 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:16, 21 September 2021imported>Stashbot 444,638 bytes +11,548 tgr: Evening deploys done
  • curprev 01:47, 18 September 2021imported>Stashbot 433,090 bytes +322 ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.23/includes/libs/rdbms/database/Database.php: (no justification provided) (duration: 00m 57s)
  • curprev 21:28, 17 September 2021imported>Stashbot 432,768 bytes +3,000 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:04, 17 September 2021imported>Stashbot 429,768 bytes +13,476 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .
  • curprev 23:02, 15 September 2021imported>Stashbot 416,292 bytes +9,362 legoktm: upgrading lists1001 to use postorius 1.3.5
  • curprev 23:01, 14 September 2021imported>Stashbot 406,930 bytes +15,959 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Re-enable VipsScaler (2 of 2) (duration: 01m 04s)
  • curprev 00:04, 14 September 2021imported>Stashbot 390,971 bytes +8,994 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 18:33, 12 September 2021imported>Stashbot 381,977 bytes +422 vgutierrez: restart varnish-fe on cp3061, cp3063 and cp3065
  • curprev 19:02, 11 September 2021imported>Stashbot 381,555 bytes +650 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 27814b8eaacb5ba2fee1b6167a36ea14356a1ecf: testwiki: Fully remove securepoll-related groups (T290808) (duration: 00m 57s)
  • curprev 21:29, 10 September 2021imported>Stashbot 380,905 bytes +5,151 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • curprev 00:35, 10 September 2021imported>Stashbot 375,754 bytes +15,347 tgr: Deployed patch for T290692
  • curprev 00:50, 9 September 2021imported>Stashbot 360,407 bytes +8,465 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Don't set default to Score (try #2) (duration: 00m 58s)
  • curprev 00:00, 8 September 2021imported>Stashbot 351,942 bytes +12,900 legoktm: legoktm@lists1001:~$ sudo rm -rf /etc/mailman # cleanup as part of 4869d91b0be / T282303
  • curprev 23:52, 6 September 2021imported>Stashbot 339,042 bytes +8,019 tstarling@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/SecurePoll/includes/Talliers/STVTallier.php: T290000 (duration: 00m 58s)
  • curprev 18:54, 5 September 2021imported>Stashbot 331,023 bytes +201 urbanecm: wikiadmin@10.192.0.119(ptwiki)> update protected_titles set pt_create_perm='editautoreviewprotected' where pt_create_perm='autoreviewer'; # T290396
  • curprev 13:35, 4 September 2021imported>Stashbot 330,822 bytes +1,989 marostegui@cumin1001: dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: Slowly repool T290374', diff saved to https://phabricator.wikimedia.org/P17217 and previous config saved to /var/cache/conftool/dbconfig/20210904-133532-root.json
  • curprev 21:49, 3 September 2021imported>Stashbot 328,833 bytes +13,135 bd808@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 00:31, 3 September 2021imported>Stashbot 315,698 bytes +18,404 tgr@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php: Backport: fixLinkRecommendationData: Try harder to avoid >10K result sets (T284531) (duration: 00m 58s)
  • curprev 23:50, 1 September 2021imported>Stashbot 297,294 bytes +11,478 Amir1: mwscript createAndPromote.php --wiki=test2wiki --sysop --force Ladsgroup
  • curprev 00:53, 1 September 2021imported>Stashbot 285,816 bytes +13,363 eileen: civicrm revision changed from e567b4c289 to 7da3eba4f9, config revision is 5f004d94d7
  • curprev 23:14, 30 August 2021imported>Stashbot 272,453 bytes +10,249 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:15, 29 August 2021imported>Stashbot 262,204 bytes +714 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 16:46, 27 August 2021imported>Stashbot 261,490 bytes +3,953 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps1005.eqiad.wmnet with reason: Resyncing from master
  • curprev 00:46, 27 August 2021imported>Stashbot 257,537 bytes +9,223 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 23:23, 25 August 2021imported>Stashbot 248,314 bytes +23,982 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:53, 25 August 2021imported>Stashbot 224,332 bytes +6,127 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • curprev 00:17, 24 August 2021imported>Stashbot 218,205 bytes +10,888 ryankemper: [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'`
  • curprev 13:35, 22 August 2021imported>Nintendofan885 207,317 bytes −804,471 Archiving May 2021–July 2021 (2/3)
  • curprev 15:12, 21 August 2021imported>Stashbot 1,011,788 bytes +262 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 23:18, 20 August 2021imported>Stashbot 1,011,526 bytes +4,817 legoktm: deployed patch for T289385
  • curprev 00:18, 20 August 2021imported>Stashbot 1,006,709 bytes +10,542 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:13, 19 August 2021imported>Stashbot 996,167 bytes +6,256 ejegg: updated fundraising CiviCRM from 73f6ec9190 to 8ed303f2d1
  • curprev 00:39, 18 August 2021imported>Stashbot 989,911 bytes +13,880 dpifke@deploy1002: Finished deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243) (duration: 00m 05s)
  • curprev 00:45, 17 August 2021imported>Stashbot 976,031 bytes +12,823 eileen: civicrm revision changed from ba0c7705bb to 175a3101f7, config revision is 7bdc78073d
  • curprev 20:02, 15 August 2021imported>Stashbot 963,208 bytes +299 addshore: restarting blazegraph on wdqs2004
  • curprev 03:54, 14 August 2021imported>Stashbot 962,909 bytes +121 legoktm[m]: restarting mailman3 on lists1001, bounce runner crashed (T288880)
  • curprev 18:43, 13 August 2021imported>Stashbot 962,788 bytes +9,163 bblack: reprepro: uploaded gdnsd-3.8.0-1~wmf1 to buster-wikimedia - T252132
  • curprev 23:50, 12 August 2021imported>Stashbot 953,625 bytes +22,057 derick@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Set archive namespaces on foundationwiki to 'noindex,follow' (T288763) (duration: 00m 59s)
  • curprev 23:30, 11 August 2021imported>Stashbot 931,568 bytes +15,885 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:47, 11 August 2021imported>Stashbot 915,683 bytes +19,823 dpifke@deploy1002: Finished deploy [performance/navtiming@12d8381]: Deploying https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 (duration: 00m 06s)
  • curprev 16:12, 9 August 2021imported>Stashbot 895,860 bytes +14,288 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .
  • curprev 19:17, 6 August 2021imported>Stashbot 881,572 bytes +8,445 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:01, 6 August 2021imported>Stashbot 873,127 bytes +23,598 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ms-be2065.codfw.wmnet with reason: REIMAGE
  • curprev 01:26, 5 August 2021imported>Stashbot 849,529 bytes +19,135 Krinkle: krinkle@mwmaint1002 Temporarily grant myself `translationadmin` on wikimania2016wiki in order to approve an edit given FlaggedRevs-like nature of Translate
  • curprev 23:34, 3 August 2021imported>Stashbot 830,394 bytes +13,656 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:44, 3 August 2021imported>Stashbot 816,738 bytes +12,154 reedy@deploy1002: Finished deploy [integration/docroot@f9d225d]: with less gref (duration: 00m 05s)
  • curprev 12:40, 31 July 2021imported>Stashbot 804,584 bytes +151 reedy@deploy1002: Synchronized php-1.37.0-wmf.16/extensions/SecurePoll/: T287780 T287782 (duration: 00m 58s)
  • curprev 00:01, 31 July 2021imported>Stashbot 804,433 bytes +8,409 eileen: civicrm revision changed from 158ed65e00 to d6baf291f4, config revision is 6011d9c471
  • curprev 23:41, 29 July 2021imported>Stashbot 796,024 bytes +6,556 derick@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Merge new configs with existing testwiki definition (duration: 00m 57s)
  • curprev 23:57, 28 July 2021imported>Stashbot 789,468 bytes +7,751 thcipriani@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: wgSkipSkins: Update defaults, hide modern (T287616) (duration: 01m 06s)
  • curprev 23:53, 27 July 2021imported>Stashbot 781,717 bytes +16,222 thcipriani@deploy1002: Synchronized php-1.37.0-wmf.16/skins/Vector: Backport: Restore print, links, table and message box styles (T278896) (duration: 01m 07s)
  • curprev 23:37, 26 July 2021imported>Stashbot 765,495 bytes +4,200 legoktm@deploy1002: Synchronized php-1.37.0-wmf.15/extensions/Score/includes/Score.php: Increase lilypond version cache TTL to 1 hour (duration: 00m 57s)
  • curprev 11:04, 24 July 2021imported>Stashbot 761,295 bytes +338 urbanecm: [urbanecm@mwmaint2002 ~]$ mwscript extensions/Translate/scripts/moveTranslatablePage.php --wiki=commonswiki --reason='OTRS -> VRTS renaming process; see Phab:T280392 and Phab:T280397' --move-subpages 'Commons:OTRS' 'Commons:Volunteer Response Team' 'Martin Urbanec' # T287321
  • curprev 19:11, 23 July 2021imported>Stashbot 760,957 bytes +5,409 topranks: Successfully re-pooled eqiad - reversed change from yesterday after successful line card replacement in cr2-codfw - T287110
  • curprev 23:35, 22 July 2021imported>Stashbot 755,548 bytes +14,051 derick@deploy1002: Synchronized php-1.37.0-wmf.15/includes/preferences/DefaultPreferencesFactory.php: Backport: Make sure enable responsive mode UI reflects actual preference value (T285402) (duration: 00m 56s)
  • curprev 00:52, 22 July 2021imported>Stashbot 741,497 bytes +10,669 ryankemper@cumin2001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2043.codfw.wmnet with reason: REIMAGE
  • curprev 20:53, 20 July 2021imported>Stashbot 730,828 bytes +12,637 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: caa5a076f39b051b01622aa3e4c9d716a8643eef: Set wgGEMentorDashboardBackendEnabled properly (T285811) (duration: 00m 57s)
  • curprev 20:48, 19 July 2021imported>Stashbot 718,191 bytes +14,116 urbanecm: Deploy security patch for T286884
  • curprev 19:50, 16 July 2021imported>Stashbot 704,075 bytes +7,705 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on copernicium.wikimedia.org with reason: REIMAGE
  • curprev 00:06, 16 July 2021imported>Stashbot 696,370 bytes +21,594 hoo: Updated the Wikidata property suggester with data from the 2021-07-12 JSON dump (with pre-applied T132839 workarounds)
  • curprev 00:00, 15 July 2021imported>Stashbot 674,776 bytes +12,120 twentyafterfour: phabricator update deployed.
  • curprev 00:58, 14 July 2021imported>Stashbot 662,656 bytes +13,088 eileen: process control updated to c291b3c6890364281d
  • curprev 23:57, 12 July 2021imported>Stashbot 649,568 bytes +13,413 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 1896efc27f3de39659673091bc4c43ad874da0c5: Add sayahna.org to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T286163) (duration: 00m 56s)
  • curprev 23:28, 9 July 2021imported>Stashbot 636,155 bytes +1,014 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • curprev 00:47, 9 July 2021imported>Stashbot 635,141 bytes +9,116 legoktm: zotero rolling restart didn't help, filed T286360 for DNS issues
  • curprev 20:22, 7 July 2021imported>Stashbot 626,025 bytes +5,635 legoktm: repooling eqiad - https://gerrit.wikimedia.org/r/703561
  • curprev 18:34, 6 July 2021imported>Stashbot 620,390 bytes +5,171 otto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-analytics' for release 'production' .
  • curprev 00:56, 6 July 2021imported>Stashbot 615,219 bytes +1,686 eileen: process-control config revision is 8d46b52ed4
  • curprev 00:53, 5 July 2021imported>Stashbot 613,533 bytes +528 eileen: process-control config revision is a1717c7fde
  • curprev 17:46, 3 July 2021imported>Stashbot 613,005 bytes +322 elukey: depool eqsin due to loss of power redundancy (equinix maintenance) - T286113
  • curprev 22:06, 2 July 2021imported>Stashbot 612,683 bytes +8,804 foks: removing three files for legal compliance
  • curprev 01:47, 2 July 2021imported>Stashbot 603,879 bytes +9,552 eileen: civicrm revision changed from e07c2be1a7 to bb62188ec6, config revision is 1739c53fcb
  • curprev 23:28, 30 June 2021imported>Stashbot 594,327 bytes +11,040 urbanecm: Evening B&C window finished
  • curprev 00:36, 30 June 2021imported>Stashbot 583,287 bytes +15,990 tstarling@deploy1002: Synchronized wmf-config/db-labs.php: gerrit 701995 SQL query log (duration: 01m 05s)
  • curprev 00:26, 29 June 2021imported>Stashbot 567,297 bytes +16,261 Krinkle: krinkle@mwmaint1002: purgeParserCache.php --tag pc1, ref T282761
  • curprev 09:19, 27 June 2021imported>Elukey 551,036 bytes +741 Add missing restart entry (logged on a different chan)
  • curprev 21:28, 26 June 2021imported>Stashbot 550,295 bytes +1,342 volans: upgraded spicerack to v0.0.56 on the cumin hosts (includes only bug fixes for the switchdc)
  • curprev 21:37, 25 June 2021imported>Stashbot 548,953 bytes +6,188 ebernhardson@deploy1002: Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s)
  • curprev 23:02, 24 June 2021imported>Stashbot 542,765 bytes +11,139 legoktm: reverted cumin1001 spicerack live hacks
  • curprev 00:25, 24 June 2021imported>Stashbot 531,626 bytes +19,731 eileen: civicrm revision changed from bd906975f0 to 6d3dd6e5a5, config revision is 821e5889f7
  • curprev 00:50, 23 June 2021imported>Stashbot 511,895 bytes +14,361 eileen: civicrm revision changed from c745d4f075 to 03bead707d, config revision is 4ab72c1033
  • curprev 01:02, 22 June 2021imported>Stashbot 497,534 bytes +26,215 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE
  • curprev 20:55, 18 June 2021imported>Stashbot 471,319 bytes +9,077 Krinkle: Remove doc1001:/srv/doc/mediawiki-core/wmf-1.36.0-wmf.31-testing
  • curprev 21:49, 17 June 2021imported>Stashbot 462,242 bytes +15,302 legoktm: regenerating pipermail redirects to skip those with duplicate message-ids (T280731)
  • curprev 21:35, 16 June 2021imported>Stashbot 446,940 bytes +4,876 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • curprev 17:54, 15 June 2021imported>Stashbot 442,064 bytes +4,022 dancy: testing upcoming Scap release on beta
  • curprev 00:37, 15 June 2021imported>Stashbot 438,042 bytes +26,468 eileen: civicrm revision changed from 31d07115a0 to 28ace1b86f, config revision is 2aed6ff89b
  • curprev 13:49, 12 June 2021imported>Stashbot 411,574 bytes +317 rzl@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: alert noise, no impact, x2 is unused
  • curprev 23:37, 11 June 2021imported>Stashbot 411,257 bytes +6,591 mutante: removing firewall hole for mgmt networks to install* because it turned out it cant be used for firmware upgrades
  • curprev 01:10, 11 June 2021imported>Stashbot 404,666 bytes +17,032 eileen: process-control config revision is 2aed6ff89b
  • curprev 22:12, 9 June 2021imported>Stashbot 387,634 bytes +19,207 dzahn@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host doh1002.wikimedia.org
  • curprev 00:37, 9 June 2021imported>Stashbot 368,427 bytes +17,582 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Enable Wikisource OCR on select Wikisources (T283898) (duration: 01m 31s)
  • curprev 21:26, 7 June 2021imported>Stashbot 350,845 bytes +11,147 otto@cumin1001: END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0)
  • curprev 16:16, 5 June 2021imported>Stashbot 339,698 bytes +307 Amir1: deleting all private archives of mm2. All are inaccessible now (T282303)
  • curprev 00:21, 5 June 2021imported>Stashbot 339,391 bytes +11,476 mutante: backup1001 - systemctl baclua-dir works again (restoring backup for non-existing host)
  • curprev 00:12, 4 June 2021imported>Stashbot 327,915 bytes +24,727 ryankemper@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
  • curprev 00:40, 3 June 2021imported>Stashbot 303,188 bytes +15,354 ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.7/extensions/Gadgets: Backport: Reduce message parse in GadgetHooks::getPreferences (second time) (T58633 T278650), Try II (duration: 00m 57s)
  • curprev 21:09, 1 June 2021imported>Stashbot 287,834 bytes +5,348 andrewbogott: dropping a bunch of tables from the labswiki db as per T284108
  • curprev 00:46, 1 June 2021imported>Stashbot 282,486 bytes +974 legoktm@deploy1002: Synchronized logos/config.yaml: Revert "Use eswiki 20th anniversary logos" (T280908) (duration: 01m 07s)
  • curprev 14:44, 29 May 2021imported>Stashbot 281,512 bytes +194 elukey: execute apt-get clean on an-airflow1001 to free space
  • curprev 08:06, 28 May 2021imported>Stashbot 281,318 bytes +368 oblivian@cumin1001: conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet,dc=eqiad
  • curprev 23:56, 27 May 2021imported>Stashbot 280,950 bytes +10,488 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab1004.eqiad.wmnet with reason: REIMAGE
  • curprev 23:07, 26 May 2021imported>Stashbot 270,462 bytes +14,277 ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.7/includes/resourceloader/dependencystore/SqlModuleDependencyStore.php: Backport: resourceloader: Avoid primary connection in SqlModuleDependencyStore (2) (duration: 01m 06s)
  • curprev 00:47, 26 May 2021imported>Stashbot 256,185 bytes +10,892 eileen: civicrm revision changed from 584b96452a to eac772e9c9, config revision is 2ca92c3c3c
  • curprev 00:48, 25 May 2021imported>Stashbot 245,293 bytes +11,934 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 14:25, 23 May 2021imported>Stashbot 233,359 bytes +262 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: EMERGENCY: f752f8b80c57a5b7e41b91873b3eac388535ac46: enwiktionary: Raise AF emergency disable treshold+count (T283460) (duration: 00m 57s)
  • curprev 22:13, 22 May 2021imported>Stashbot 233,097 bytes +265 legoktm: reset 2FA for User:Yuvipanda on wikitech
  • curprev 22:32, 21 May 2021imported>Stashbot 232,832 bytes +12,510 bstorm: upload nfsd-ldap: 1.2+deb10u1 to buster-wikimedia T283385
  • curprev 21:45, 20 May 2021imported>Stashbot 220,322 bytes +13,158 herron@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:01, 20 May 2021imported>Stashbot 207,164 bytes +12,243 mutante: signing puppet certs for doh2001 and doh2002.wikimedia.org (T283192)
  • curprev 18:40, 18 May 2021imported>Stashbot 194,921 bytes +14,463 razzi@deploy1002: Finished deploy [analytics/refinery@9392f1d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@9392f1db6e66975304c8e9b2b7031acd3ed87fa7] (duration: 05m 16s)
  • curprev 00:56, 18 May 2021imported>Stashbot 180,458 bytes +19,706 eileen: civicrm revision changed from 38ac15233f to b3fb3c9cb0, config revision is 1f8d0a6bfa
  • curprev 00:13, 17 May 2021imported>Stashbot 160,752 bytes +408 legoktm: installing python3-dbg on lists1001
  • curprev 00:53, 16 May 2021imported>Stashbot 160,344 bytes +324 legoktm: restarted mailman3-web on lists1001, uwsgi looked like it got stuck, consuming all CPU/memory
  • curprev 20:42, 14 May 2021imported>Stashbot 160,020 bytes +3,428 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts people1002.eqiad.wmnet
  • curprev 00:39, 14 May 2021imported>Stashbot 156,592 bytes +8,035 ryankemper: T280382 `sudo -i wmf-auto-reimage-host -p T280382 --new wdqs2003.codfw.wmnet` on `ryankemper@cumin2001` tmux session `wdqs_reimage`
  • curprev 23:48, 12 May 2021imported>Stashbot 148,557 bytes +16,709 urbanecm@deploy1002: Synchronized php-1.37.0-wmf.4/extensions/WikiEditor/includes/WikiEditorHooks.php: 2f6af514c49d47bbec5ce51f9f7263015e039003? PHP VisualEditorFeatureUse logging: properly record session id (T281409) (duration: 01m 07s)
  • curprev 01:42, 12 May 2021imported>Stashbot 131,848 bytes +17,502 dzahn@cumin1001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host
  • curprev 23:38, 10 May 2021imported>Stashbot 114,346 bytes +15,358 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: 779fb53bfd7a4d9b11f865df14f8a72adb97f33b: Update messages used for tech CoC (T280886) (duration: 00m 56s)
  • curprev 21:44, 9 May 2021imported>Stashbot 98,988 bytes −1,654,362 legoktm: restarted mailman3 again (T282348) pymysql.err.InternalError: (1205, 'Lock wait timeout exceeded; try restarting transaction')
  • curprev 17:18, 8 May 2021imported>Stashbot 1,753,350 bytes +105 Amir1: starting upgrade of batch G of mailing lists (T280322)
  • curprev 21:40, 7 May 2021imported>Stashbot 1,753,245 bytes +7,667 legoktm: deleted education@ from MM3, didn't import properly
  • curprev 23:50, 6 May 2021imported>Stashbot 1,745,578 bytes +25,003 brennen@deploy1002: rebuilt and synchronized wikiversions files: Rollback group1 and group2 to 1.37.0-wmf.3 (T282193)
  • curprev 00:35, 6 May 2021imported>Stashbot 1,720,575 bytes +28,723 Amir1: sudo service mailman3-web restart
  • curprev 01:45, 5 May 2021imported>Stashbot 1,691,852 bytes +24,478 ryankemper: T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1006.eqiad.wmnet --dest wdqs1011.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph` on `ryankemper@cumin1001` tmux session `reimage`
  • curprev 23:18, 3 May 2021imported>Stashbot 1,667,374 bytes +11,155 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 230ef5716b34ca83348667f289180313b76ce8a3: Prepare for new configuration option (T277951) (duration: 00m 57s)
  • curprev 13:40, 2 May 2021imported>Stashbot 1,656,219 bytes +311 dcaro@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cloudmetrics1002.eqiad.wmnet with reason: Flaky host
  • curprev 19:12, 1 May 2021imported>Stashbot 1,655,908 bytes +557 Urbanecm: Invalidate password for MaraBot@SUL (T281586)
  • curprev 21:55, 30 April 2021imported>Stashbot 1,655,351 bytes +6,117 mutante: people1003 - rsycncing /home from peopel1002
  • curprev 01:08, 30 April 2021imported>Stashbot 1,649,234 bytes +23,564 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw reboot - ryankemper@cumin1001 - T280563
  • curprev 00:41, 29 April 2021imported>Stashbot 1,625,670 bytes +24,040 tstarling@deploy1002: Synchronized php-1.37.0-wmf.3/includes/specials/pagers/ImageListPager.php: T281405 (duration: 01m 08s)
  • curprev 01:06, 28 April 2021imported>Stashbot 1,601,630 bytes +35,800 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 23:28, 26 April 2021imported>Stashbot 1,565,830 bytes +16,596 mutante: renewing TLS cert for peopleweb.discovery.wmnet, adding *3 hosts
  • curprev 15:23, 25 April 2021imported>Stashbot 1,549,234 bytes +138 Amir1: sudo -u list /var/lib/mailman/bin/change_pw -l wikica-l -p $(pwgen -c1 -s 12) (T281066)
  • curprev 22:24, 24 April 2021imported>Stashbot 1,549,096 bytes +78 bstorm: Rebooting labstore1007 from ilo after crash
  • curprev 21:36, 23 April 2021imported>Stashbot 1,549,018 bytes +3,966 foks: removing 1 file for legal compliance
  • curprev 17:26, 22 April 2021imported>Stashbot 1,545,052 bytes +1,074 marostegui: Stop mysql on tendril/dbtree database
  • curprev 01:34, 22 April 2021imported>Stashbot 1,543,978 bytes +14,300 eileen: civicrm revision changed from 35a8dd33ba to 42ca3cf65a, config revision is cf07e7ba0b
  • curprev 00:38, 21 April 2021imported>Stashbot 1,529,678 bytes +27,920 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE
  • curprev 22:56, 19 April 2021imported>Stashbot 1,501,758 bytes +29,254 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: REIMAGE
  • curprev 06:40, 18 April 2021imported>Stashbot 1,472,504 bytes +106 Amir1: cleaning watchlist of User:Mr._Ibrahem in wikidatawiki (in main ns only)
  • curprev 16:16, 17 April 2021imported>Stashbot 1,472,398 bytes +61 Amir1: cleaning SuccuBot's watchlist in wikidatawiki
  • curprev 00:53, 17 April 2021imported>Stashbot 1,472,337 bytes +38,446 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1307.eqiad.wmnet
  • curprev 01:19, 15 April 2021imported>Stashbot 1,433,891 bytes +21,435 Amir1: mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P8671 --new-data-type external-id (T278427)
  • curprev 00:11, 14 April 2021imported>Stashbot 1,412,456 bytes +25,158 legoktm@cumin1001: conftool action : set/pooled=yes; selector: cluster=jobrunner,name=mw2411.codfw.wmnet
  • curprev 23:25, 12 April 2021imported>Stashbot 1,387,298 bytes +4,421 krinkle@deploy1002: Synchronized wmf-config/mc.php: I390b4726d01037107 (duration: 00m 58s)
  • curprev 14:21, 10 April 2021imported>Stashbot 1,382,877 bytes +815 andrew@deploy1002: Finished deploy [horizon/deploy@ee1be56]: fix for T279699 (duration: 04m 12s)
  • curprev 14:07, 9 April 2021imported>Stashbot 1,382,062 bytes +59 jynus: retry es4 backup dump on eqiad (backup1002)
  • curprev 01:25, 9 April 2021imported>Stashbot 1,382,003 bytes +23,659 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on moss-be2002.codfw.wmnet with reason: REIMAGE
  • curprev 23:38, 7 April 2021imported>Stashbot 1,358,344 bytes +22,437 ejegg: updated payments-wiki from b06009c099 to 70f5163816,
  • curprev 01:46, 7 April 2021imported>Stashbot 1,335,907 bytes +16,883 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephmon2004-dev.codfw.wmnet with reason: REIMAGE
  • curprev 23:17, 5 April 2021imported>Stashbot 1,319,024 bytes +12,454 AaronSchulz: Running importMissingLocalNames.php on mwmaint1002 in a screen
  • curprev 14:47, 4 April 2021imported>Stashbot 1,306,570 bytes +265 andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 01m 36s)
  • curprev 19:20, 3 April 2021imported>Stashbot 1,306,305 bytes +1,408 andrew@deploy1002: Finished deploy [horizon/deploy@df2b0b4]: upgrade labtesthorizon to the Wallaby branch (duration: 02m 11s)
  • curprev 22:31, 2 April 2021imported>Stashbot 1,304,897 bytes +10,597 bstorm@cumin1001: END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0)
  • curprev 23:32, 1 April 2021imported>Stashbot 1,294,300 bytes +18,843 thcipriani@deploy1002: Synchronized php-1.36.0-wmf.37/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: Backport: Revert "Turn on glent m1 AB test" T262612 (duration: 00m 58s)
  • curprev 00:56, 1 April 2021imported>Stashbot 1,275,457 bytes +7,735 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2381.codfw.wmnet with reason: REIMAGE
  • curprev 23:59, 30 March 2021imported>Stashbot 1,267,722 bytes +13,942 Trey314159: reindexing English wikis on elastic@eqiad, elastic@codfw, and cloudelastic (T274200)
  • curprev 19:06, 29 March 2021imported>Stashbot 1,253,780 bytes +3,056 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=aqs1004.eqiad.wmnet
  • curprev 19:26, 27 March 2021imported>Stashbot 1,250,724 bytes +1,286 elukey: powercycle elastic1060 - T278630
  • curprev 22:27, 26 March 2021imported>Stashbot 1,249,438 bytes +5,934 tzatziki: reset password for Philroc
  • curprev 23:47, 25 March 2021imported>Stashbot 1,243,504 bytes +12,592 thcipriani@deploy1002: Synchronized php-1.36.0-wmf.36/extensions/3D/package.json: No-op demo sync (duration: 01m 07s)
  • curprev 01:33, 25 March 2021imported>Stashbot 1,230,912 bytes +22,060 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-upgrade (exit_code=99)
  • curprev 00:22, 24 March 2021imported>Stashbot 1,208,852 bytes +46,207 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw2378.codfw.wmnet with reason: REIMAGE
  • curprev 00:08, 23 March 2021imported>Stashbot 1,162,645 bytes +17,284 tstarling@deploy1002: Synchronized wmf-config: use RequestTimeout library step 3: clean up (duration: 00m 58s)
  • curprev 10:25, 21 March 2021imported>Stashbot 1,145,361 bytes +368 _joe_: restarting gerrit on gerrit1001, using 45G of reserved memory
  • curprev 00:22, 20 March 2021imported>Stashbot 1,144,993 bytes +6,348 tzatziki: altering emails for STei (WMF) and SGrabarczuk (WMF)
  • curprev 00:51, 19 March 2021imported>Stashbot 1,138,645 bytes +9,921 dancy@deploy1002: Synchronized php-1.36.0-wmf.35/extensions/LiquidThreads/classes/Thread.php: T277772 (duration: 00m 58s)
  • curprev 00:05, 18 March 2021imported>Stashbot 1,128,724 bytes +22,211 eileen: tools revision changed from b7b4060c30 to ef54260b0d
  • curprev 23:56, 16 March 2021imported>Stashbot 1,106,513 bytes +44,018 krinkle@deploy1002: Synchronized php-1.36.0-wmf.35/includes/Revision/: I8619ab9e92b, T277362, T275531 (duration: 00m 58s)
  • curprev 23:31, 15 March 2021imported>Stashbot 1,062,495 bytes +21,243 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Remove back-compat from when IRC feed servers was a string (T224579) (duration: 00m 59s)
  • curprev 17:57, 14 March 2021imported>Stashbot 1,041,252 bytes +1,114 marostegui@cumin1001: dbctl commit (dc=all): 'db1146:3314 (re)pooling @ 100%: Repool db1146:3314', diff saved to https://phabricator.wikimedia.org/P14827 and previous config saved to /var/cache/conftool/dbconfig/20210314-175751-root.json
  • curprev 19:02, 13 March 2021imported>Stashbot 1,040,138 bytes +699 Amir1: change default charset of all core tables in labstestwiki to binary (T269348)
  • curprev 00:55, 13 March 2021imported>Stashbot 1,039,439 bytes +10,754 mutante: [wdqs1009:/etc/envoy] $ sudo /usr/local/sbin/build-envoy-config -c /etc/envoy/
  • curprev 01:30, 12 March 2021imported>Stashbot 1,028,685 bytes +31,013 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts mw2215.codfw.wmnet
  • curprev 00:50, 11 March 2021imported>Stashbot 997,672 bytes +37,248 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw2219.codfw.wmnet
  • curprev 01:08, 10 March 2021imported>Stashbot 960,424 bytes +27,385 krinkle@deploy1002: Synchronized php-1.36.0-wmf.34/extensions/NavigationTiming/modules/ext.navigationTiming.js: T276826 Ibd9ddf14d64 (duration: 01m 14s)
  • curprev 00:58, 9 March 2021imported>Stashbot 933,039 bytes +16,107 Krinkle: krinkle@mwmaint1002 Ran invalidateUserSesssions.php for one user
  • curprev 08:01, 7 March 2021imported>Stashbot 916,932 bytes +170 elukey: "megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll" on analytics1066 - BBU looks fine, but the raid controller was using WriteThrough
  • curprev 23:16, 5 March 2021imported>Stashbot 916,762 bytes +8,775 legoktm: imported pygments 2.8.0+dfsg-1 to apt.wm.o buster-wikimedia component/pygments (T276298)
  • curprev 00:59, 5 March 2021imported>Stashbot 907,987 bytes +16,215 legoktm: depooled registry1001/registry1002 (old stretch VMs) - T272550
  • curprev 01:24, 4 March 2021imported>Stashbot 891,772 bytes +30,062 twentyafterfour: phabricator upgrade complete
  • curprev 00:42, 3 March 2021imported>Stashbot 861,710 bytes +24,132 Urbanecm: Finished deployment in Evening B&C window; logmsgbot is currently down, and a simple restart did not bring it back up
  • curprev 00:59, 2 March 2021imported>Stashbot 837,578 bytes +21,612 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 0f08e8bbe1f74220a7a01a7606b67e0f75734a53: Update the Persian Wikipedia logos (T261033; 2/2) (duration: 00m 56s)
  • curprev 14:17, 28 February 2021imported>Stashbot 815,966 bytes +71 gehel: repooled wdqs1011 - catched up on lag
  • curprev 21:19, 27 February 2021imported>Stashbot 815,895 bytes +260 dwisehaupt: ran the following on frdb2002 to allow replication to continue after conversion to utf8mb4 charset: set global slave_type_conversions = ALL_NON_LOSSY;
  • curprev 00:08, 27 February 2021imported>Stashbot 815,635 bytes +26,358 mutante: deploy1002 - rsyncing home dirs from deploy1001
  • curprev 00:14, 26 February 2021imported>Stashbot 789,277 bytes +25,251 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.32/extensions/Graph/: 9d5cf348f5dda32f8889d5160bb1fe34a4e07f8c: Do not log graph errors to WMF servers (T274557) (duration: 01m 36s)
  • curprev 00:29, 25 February 2021imported>Stashbot 764,026 bytes +22,501 ryankemper: T274204 Restored service health on `elastic106[0,4,5]` via `sudo apt-get remove --purge wmf-elasticsearch-search-plugins --yes && sudo dpkg -i /var/cache/apt/archives/wmf-elasticsearch-search-plugins_6.5.4-4~stretch_all.deb && sudo puppet agent -tv`. There's some sort of issue with `6.5.4-5~stretch` that we will need to circle back and investigate; for now the fleet is staying on `6.5.4-4~stretch`
  • curprev 00:58, 24 February 2021imported>Stashbot 741,525 bytes +17,976 volker-e@deploy1001: Finished deploy [design/style-guide@a66b5b6]: Deploy design/style-guide: a66b5b6 “Components”: Add “Dialogs” (#430) (duration: 00m 06s)
  • curprev 00:46, 23 February 2021imported>Stashbot 723,549 bytes +23,528 eileen: civicrm revision changed from c535ac603a to 5e042e6e57, config revision is ef64f705bb
  • curprev 16:03, 21 February 2021imported>Stashbot 700,021 bytes +990 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db1162 - crashed', diff saved to https://phabricator.wikimedia.org/P14424 and previous config saved to /var/cache/conftool/dbconfig/20210221-160258-marostegui.json
  • curprev 00:17, 20 February 2021imported>Stashbot 699,031 bytes +13,319 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1317.eqiad.wmnet
  • curprev 00:54, 19 February 2021imported>Stashbot 685,712 bytes +7,907 mutante: mwmaint2001 - back from reimage - scap pull
  • curprev 01:38, 18 February 2021imported>Stashbot 677,805 bytes +16,614 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:00, 17 February 2021imported>Stashbot 661,191 bytes +18,673 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mwdebug1001.eqiad.wmnet
  • curprev 21:33, 15 February 2021imported>Stashbot 642,518 bytes +7,693 eileen: civicrm revision changed from dfbb8f41bc to c535ac603a, config revision is ba9b2380b1
  • curprev 13:13, 14 February 2021imported>Stashbot 634,825 bytes +806 akosiaris: sudo cumin -b 1 -s 120 'cp500[2,3,5,6].eqsin.wmnet' 'systemctl restart varnish-frontend.service'
  • curprev 03:24, 13 February 2021imported>Stashbot 634,019 bytes +232 ryankemper: Depooled `wdqs1006` to catch up on lag
  • curprev 01:00, 13 February 2021imported>Stashbot 633,787 bytes +21,925 crusnov@cumin1001: START - Cookbook sre.ganeti.makevm for new host mwdebug1002.eqiad.wmnet
  • curprev 01:21, 12 February 2021imported>Stashbot 611,862 bytes +30,779 legoktm@cumin1001: conftool action : set/pooled=yes; selector: name=mw1329.eqiad.wmnet
  • curprev 00:26, 11 February 2021imported>Stashbot 581,083 bytes +30,413 legoktm@deploy1001: Synchronized wmf-config/profiler.php: Revert "profiler: Send data to excimer-buster pipeline" (duration: 02m 00s)
  • curprev 00:58, 10 February 2021imported>Stashbot 550,670 bytes +36,435 mutante: doc1001 - reloaded apache2
  • curprev 00:48, 9 February 2021imported>Stashbot 514,235 bytes +25,431 dzahn@cumin1001: conftool action : set/pooled=no; selector: name=mw1302.eqiad.wmnet
  • curprev 22:58, 7 February 2021imported>Stashbot 488,804 bytes +95 Urbanecm: Reset password for TheresNoTime (T274087)
  • curprev 08:59, 6 February 2021imported>Stashbot 488,709 bytes +1,136 elukey@cumin1001: END (PASS) - Cookbook sre.hadoop.change-distro-from-cdh-clients (exit_code=0) for Hadoop test cluster: Change Hadoop distribution - elukey@cumin1001
  • curprev 01:00, 6 February 2021imported>Stashbot 487,573 bytes +17,493 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2265.codfw.wmnet
  • curprev 00:59, 5 February 2021imported>Stashbot 470,080 bytes +27,954 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@85713c1]: restore data range specifier in extract job partition spec
  • curprev 01:15, 4 February 2021imported>Stashbot 442,126 bytes +22,570 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@4b4872d]: transfer_to_es: Increase timeout waiting for source data to three hours (duration: 01m 16s)
  • curprev 00:16, 3 February 2021imported>Stashbot 419,556 bytes +21,008 jhuneidi@deploy1001: helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
  • curprev 23:54, 1 February 2021imported>Stashbot 398,548 bytes +28,951 legoktm@deploy1001: Synchronized wmf-config/profiler.php: profiler: Send data to excimer-buster pipeline (T273312) (duration: 00m 57s)
  • curprev 23:26, 29 January 2021imported>Stashbot 369,597 bytes +9,826 razzi@cumin1001: END (PASS) - Cookbook sre.kafka.reboot-workers (exit_code=0) for Kafka test cluster: Reboot kafka nodes - razzi@cumin1001
  • curprev 00:19, 29 January 2021imported>Stashbot 359,771 bytes +37,885 legoktm@cumin1001: conftool action : set/pooled=yes; selector: name=mw2261.codfw.wmnet
  • curprev 00:56, 28 January 2021imported>Stashbot 321,886 bytes +25,610 legoktm@cumin1001: conftool action : set/pooled=no; selector: name=mw2294.codfw.wmnet
  • curprev 00:54, 27 January 2021imported>Stashbot 296,276 bytes +26,066 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2296.codfw.wmnet with reason: REIMAGE
  • curprev 00:48, 26 January 2021imported>Stashbot 270,210 bytes +18,727 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2331.codfw.wmnet
  • curprev 22:21, 23 January 2021imported>Stashbot 251,483 bytes +1,766 volker-e@deploy1001: Finished deploy [design/style-guide@63e39e7]: Deploy design/style-guide: 63e39e7 “Components”: Amend button groups states SVG font stack (#427) (duration: 00m 06s)
  • curprev 00:46, 23 January 2021imported>Stashbot 249,717 bytes +27,911 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch enwiki to use enwiki20 "Option A" logo variant (T272526) (duration: 00m 57s)
  • curprev 01:00, 22 January 2021imported>Stashbot 221,806 bytes +15,875 Urbanecm: Evening B&C still in process, waiting on Zuul
  • curprev 01:27, 21 January 2021imported>Stashbot 205,931 bytes +34,902 ryankemper: [WDQS Deploy] Rollback complete, service health of `wdqs1003` is restored. Need to investigate source of 404 (possibly related to some recent changes we made in the `gui` repo)
  • curprev 00:43, 20 January 2021imported>Stashbot 171,029 bytes +23,826 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Update /analytics/legacy/homepagemodule/ schema version to 1.1.0 (T270309) (duration: 01m 03s)
  • curprev 21:33, 18 January 2021imported>Stashbot 147,203 bytes +19,595 eileen: civicrm revision changed from 4220fc8177 to a4caad22b1, config revision is f08249ecf9
  • curprev 12:18, 16 January 2021imported>Stashbot 127,608 bytes +369 elukey: elukey@cumin1001:~$ sudo cumin 'A:mw-app-canary and A:mw-eqiad' 'run-puppet-agent' -b 10 - T272215
  • curprev 23:59, 15 January 2021imported>Stashbot 127,239 bytes +6,731 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1118.eqiad.wmnet with reason: REIMAGE
  • curprev 23:35, 14 January 2021imported>Stashbot 120,508 bytes +16,149 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2236.codfw.wmnet
  • curprev 00:22, 14 January 2021imported>Stashbot 104,359 bytes +24,462 ryankemper: T266492 Restart of `relforge` successful
  • curprev 22:55, 12 January 2021imported>Stashbot 79,897 bytes +10,636 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw2225.codfw.wmnet
  • curprev 22:16, 11 January 2021imported>Stashbot 69,261 bytes +16,348 eileen: process-control config revision is f08249ecf9 eoy jobs disabled
  • curprev 12:55, 10 January 2021imported>Nintendofan885 52,913 bytes −155,895 Archiving
  • curprev 00:11, 9 January 2021imported>Stashbot 208,808 bytes +13,848 mutante: puppetmaster2003 - restarted apache after spweing 500s
  • curprev 01:29, 8 January 2021imported>Stashbot 194,960 bytes +13,843 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=mw1267.eqiad.wmnet
  • curprev 00:50, 7 January 2021imported>Stashbot 181,117 bytes +8,175 eileen: civicrm revision changed from 95fefcb421 to 2df572bdcd, config revision is 603c0ff4a0
  • curprev 01:22, 6 January 2021imported>Stashbot 172,942 bytes +9,241 mutante: testreduce1001 rm -rf /srv/deployment/parsoid/deploy
  • curprev 00:40, 5 January 2021imported>Stashbot 163,701 bytes +5,675 jgleeson: updated civicrm revision changed from bb8baac617 to 6be8a130df
  • curprev 16:17, 3 January 2021imported>Stashbot 158,026 bytes +627 arturo: merged change to TLS cert used by slapd/openldap servers https://gerrit.wikimedia.org/r/c/operations/puppet/+/653871
  • curprev 19:27, 2 January 2021imported>Stashbot 157,399 bytes +74 vgutierrez: restart acme-chief on acmechief1001
  • curprev 14:49, 1 January 2021imported>Stashbot 157,325 bytes +1,317 milimetric@deploy1001: Finished deploy [analytics/refinery@f9281dd] (thin): [SAFE, IGNORE] Simple hotfix for a python bug, analytics refinery only, not urgent (duration: 00m 07s)
  • curprev 15:52, 29 December 2020imported>Stashbot 156,008 bytes +210 vgutierrez: reloading nginx on cloudelastic1005 and cloudelastic1006
  • curprev 09:54, 28 December 2020imported>Stashbot 155,798 bytes +134 elukey: reboot an-coord1002 (puppet in D state after issues with broken disk - host in standby, no traffic)
  • curprev 12:41, 24 December 2020imported>Stashbot 155,664 bytes +475 elukey@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:45, 24 December 2020imported>Stashbot 155,189 bytes +953 legoktm: reset maxmind password
  • curprev 21:57, 22 December 2020imported>Stashbot 154,236 bytes +2,451 mutante: apt1001 - sudo systemctl status rsync-aptrepo-apt2001.wikimedia.org.service - confirmed timer job is working like the cron before
  • curprev 00:20, 22 December 2020imported>Stashbot 151,785 bytes +6,451 crusnov@deploy1001: Finished deploy [netbox/deploy@b17db99]: Redeploy of 2.9.10 to netbox-dev for dep test (duration: 00m 54s)
  • curprev 21:52, 18 December 2020imported>Stashbot 145,334 bytes +3,951 thcipriani: flushing gerrit project cache
  • curprev 01:34, 18 December 2020imported>Stashbot 141,383 bytes +18,260 legoktm: restarted gerrit (T270451)
  • curprev 00:22, 17 December 2020imported>Stashbot 123,123 bytes +13,157 mutante: running puppet on mw2266, mw2370, mw2354
  • curprev 23:47, 15 December 2020imported>Stashbot 109,966 bytes +14,076 dduvall@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
  • curprev 00:13, 15 December 2020imported>Stashbot 95,890 bytes +6,424 ejegg: updated payments-wiki from 63ae7413a8 to 3d3055c478
  • curprev 22:05, 11 December 2020imported>Stashbot 89,466 bytes +7,632 dduvall@deploy1001: helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' .
  • curprev 23:35, 10 December 2020imported>Stashbot 81,834 bytes +10,617 Urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript resetAuthenticationThrottle.php --wiki=enwiki --login --ip 'REDACTED' --user 'WP 1.0 bot' # T269898
  • curprev 00:42, 10 December 2020imported>Stashbot 71,217 bytes −965,466 ejegg: updated payments-wiki from 756c2f7ce0 to df80a99b40
  • curprev 00:53, 9 December 2020imported>Stashbot 1,036,683 bytes +9,549 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.20/vendor/: 3278ffd107888757c4620383160a6d5fa67d05b5: Bump wikimedia/parsoid to v0.13.0-a19 (T269685) (duration: 01m 16s)
  • curprev 01:44, 8 December 2020imported>Stashbot 1,027,134 bytes +11,262 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
  • curprev 16:25, 5 December 2020imported>Stashbot 1,015,872 bytes +2,765 godog: swift disable sdg1 on ms-be1054
  • curprev 00:45, 5 December 2020imported>Stashbot 1,013,107 bytes +3,441 ryankemper@cumin2001: START - Cookbook sre.wdqs.data-transfer
  • curprev 01:43, 4 December 2020imported>Stashbot 1,009,666 bytes +11,577 andrew@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • curprev 23:42, 2 December 2020imported>Stashbot 998,089 bytes +10,158 reedy@deploy1001: Synchronized php-1.36.0-wmf.20/includes/debug/logger/monolog/LogstashFormatter.php: T269286 (duration: 01m 07s)
  • curprev 00:58, 2 December 2020imported>Stashbot 987,931 bytes +14,689 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-reload (exit_code=0)
  • curprev 01:18, 1 December 2020imported>Stashbot 973,242 bytes +18,284 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
  • curprev 17:30, 27 November 2020imported>Stashbot 954,958 bytes +2,219 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • curprev 17:18, 26 November 2020imported>Stashbot 952,739 bytes +11,515 jayme: downgrade helmfile to 0.125.2-1 on deploy*
  • curprev 23:28, 25 November 2020imported>Stashbot 941,224 bytes +12,272 akosiaris@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
  • curprev 01:10, 25 November 2020imported>Stashbot 928,952 bytes +16,627 tgr_: Evening deploys done
  • curprev 00:42, 24 November 2020imported>Stashbot 912,325 bytes +9,753 reedy@deploy1001: Synchronized php-1.36.0-wmf.16/extensions/SecurePoll/includes/Crypt/GpgCrypt.php: Unbreak gpg encrypted polls T268583 (duration: 01m 05s)
  • curprev 09:18, 21 November 2020imported>Stashbot 902,572 bytes +604 joal: Drop historical logs of 'Wikidata Concepts Monitor ETL' on HDFS keeping one example - freeing 60Tb
  • curprev 23:38, 20 November 2020imported>Stashbot 901,968 bytes +7,656 mutante: synced puppet-compiler facts - new hosts should be usable in compiler
  • curprev 23:59, 19 November 2020imported>Stashbot 894,312 bytes +11,241 razzi@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0)
  • curprev 23:34, 18 November 2020imported>Stashbot 883,071 bytes +19,780 mutante: disabling puppet on memcache::mediawiki - deploying gerrit:637742
  • curprev 00:53, 18 November 2020imported>Stashbot 863,291 bytes +23,755 tgr_: also deployed Suggested Edits: Guard against task type not existing (T268012)
  • curprev 23:28, 16 November 2020imported>Stashbot 839,536 bytes +8,318 mutante: cumin1001 - sudo systemctl start cumin-check-aliases (to confirm switching cron to timer worked) T265138
  • curprev 00:55, 16 November 2020imported>Stashbot 831,218 bytes +2,135 shdubsh: re-applied mask to kafka and kafka-mirror-main-eqiad_to_main-codfw@0 on kafka-main2003 and disabled puppet to prevent restart - T267865
  • curprev 22:06, 13 November 2020imported>Stashbot 829,083 bytes +1,249 Urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript emptyUserGroup.php --wiki=myvwiki autopatrolled # T105570
  • curprev 01:30, 13 November 2020imported>Stashbot 827,834 bytes +2,596 TimStarling: on mwmaint1002 running fixT260485.php unmerged fixup script from https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaMaintenance/+/640348
  • curprev 16:44, 11 November 2020imported>Stashbot 825,238 bytes +1,480 XioNoX: Revert "temporarily route Italy to codfw"
  • curprev 00:48, 11 November 2020imported>Stashbot 823,758 bytes +6,737 ryankemper: Restarting `wdqs-categories` one host at a time across all wdqs production instances: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 60 && systemctl restart wdqs-categories && sleep 30 && pool'`
  • curprev 22:26, 9 November 2020imported>Stashbot 817,021 bytes +7,948 pt1979@cumin2001: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99)
  • curprev 23:08, 8 November 2020imported>Stashbot 809,073 bytes +654 tstarling@deploy1001: Synchronized php-1.36.0-wmf.16/resources/src/mediawiki.api/upload.js: fixing UBN T266903 (duration: 01m 06s)
  • curprev 23:38, 6 November 2020imported>Stashbot 808,419 bytes +5,409 dwisehaupt: frdata1001 upgraded to buster
  • curprev 00:53, 6 November 2020imported>Stashbot 803,010 bytes +18,393 brennen@deploy1001: Finished scap: Synchronizing to pick up i18n for gerrit:639505. Will resume moving train to group1 on Monday morning (US) (T263182) (duration: 69m 02s)
  • curprev 01:47, 5 November 2020imported>Stashbot 784,617 bytes +19,362 milimetric@deploy1001: Started deploy [analytics/refinery@6913407]: Regular analytics weekly train [analytics/refinery@6913407]
  • curprev 22:56, 3 November 2020imported>Stashbot 765,255 bytes +10,780 _joe_: repooling mw1346
  • curprev 01:01, 3 November 2020imported>Stashbot 754,475 bytes +6,652 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • curprev 22:41, 1 November 2020imported>Stashbot 747,823 bytes +627 Urbanecm: mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=metawiki Turkmen # T266976
  • curprev 01:26, 1 November 2020imported>Stashbot 747,196 bytes +356 rzl@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • curprev 00:12, 31 October 2020imported>Stashbot 746,840 bytes +5,375 mutante: removed Nuria from wmf group, she is already in nda group (T266086)
  • curprev 00:00, 30 October 2020imported>Stashbot 741,465 bytes +14,485 eileen: process-control config revision is 6891d35bce
  • curprev 01:41, 29 October 2020imported>Stashbot 726,980 bytes +9,799 mutante: scandium reimaged a second time after making puppet changes to ensure nodejs/npm is NOT installed anymore (T257906)
  • curprev 00:40, 28 October 2020imported>Stashbot 717,181 bytes +10,760 eileen: civicrm revision changed from 4fdfb8408b to e1d65b0f3a, config revision is f16003ab62
  • curprev 23:12, 26 October 2020imported>Stashbot 706,421 bytes +13,445 catrope@deploy1001: Synchronized php-1.36.0-wmf.14/extensions/GrowthExperiments/: Fix JS error when no topics set (T266501) (duration: 01m 00s)
  • curprev 15:53, 25 October 2020imported>Stashbot 692,976 bytes +136 dwisehaupt: kernel upgrade and reboot for frdb1003
  • curprev 22:56, 23 October 2020imported>Stashbot 692,840 bytes +1,858 mutante: added Nuria to "nda" LDAP group - leaving her in "wmf" until the actual last day - shell account remains so no puppet change needed in ldap_only_admins (T266086)
  • curprev 22:42, 22 October 2020imported>Stashbot 690,982 bytes +6,223 mutante: ganeti1001 - adding 2 more vcpus to VM testreduce1001 - T257940
  • curprev 23:16, 21 October 2020imported>Stashbot 684,759 bytes +8,846 catrope@deploy1001: Synchronized php-1.36.0-wmf.14/extensions/GrowthExperiments/: T266033 (duration: 01m 05s)
  • curprev 22:10, 20 October 2020imported>Stashbot 675,913 bytes +8,976 dwisehaupt: frmon2001 upgraded to buster with grafana 7.2.1
  • curprev 23:57, 19 October 2020imported>Stashbot 666,937 bytes +14,227 gehel@cumin1001: END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99)
  • curprev 13:22, 17 October 2020imported>Stashbot 652,710 bytes +173 Urbanecm: [urbanecm@mwmaint2001 ~/uploads]$ mwscript importImages.php --wiki=commonswiki --comment-ext=txt --user=Fæ . # T264529
  • curprev 21:46, 16 October 2020imported>Stashbot 652,537 bytes +3,177 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 01:01, 16 October 2020imported>Stashbot 649,360 bytes +9,475 ryankemper: Cleaning up a dangling no-longer-puppet-managed udev elasticsearch-readahead rule across all cirrus instances: `sudo cumin -b 36 C:profile::elasticsearch::cirrus 'sudo rm -fv /etc/udev/rules.d/elasticsearch-readahead.rules && sudo /sbin/udevadm control --reload && sudo /sbin/udevadm trigger'`
  • curprev 00:36, 15 October 2020imported>Stashbot 639,885 bytes +8,944 ryankemper: Beginning rolling upgrade for cirrussearch `eqiad`. Cookbook will restart elasticsearch on 36 nodes total, 3 nodes at a time
  • curprev 23:22, 13 October 2020imported>Stashbot 630,941 bytes +8,939 catrope@deploy1001: Synchronized php-1.36.0-wmf.13/extensions/GrowthExperiments/: Revert removal of variant A (T265372) (duration: 01m 04s)
  • curprev 17:03, 12 October 2020imported>Stashbot 622,002 bytes +1,861 jayme: fixed /var/lock/ permission (1777) on ms-be2036 - T265208
  • curprev 01:32, 10 October 2020imported>Stashbot 620,141 bytes +173 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Enable session-ip log channel everywhere (T264799) (duration: 00m 59s)
  • curprev 00:54, 10 October 2020imported>Stashbot 619,968 bytes +8,015 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Config: Enable session-ip log channel on all but enwiki (T264799) (duration: 01m 01s)
  • curprev 00:16, 9 October 2020imported>Stashbot 611,953 bytes +10,880 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0)
  • curprev 00:31, 8 October 2020imported>Stashbot 601,073 bytes +9,125 tgr_: evening deploys done
  • curprev 01:00, 7 October 2020imported>Stashbot 591,948 bytes +13,893 cdanis: repool esams; cr2-esams router upgrade complete
  • curprev 23:11, 5 October 2020imported>Stashbot 578,055 bytes +8,516 ejegg: updated payments staging from 52704ffe24 to db03677b2d
  • curprev 15:52, 3 October 2020imported>Stashbot 569,539 bytes +246 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: emergency: 840545f1d9115ea6b672cecce1762d850d8b1f54: Restrict flow-hide right to autoconfirmed users on zhwiki (T264489) (duration: 01m 17s)
  • curprev 00:08, 3 October 2020imported>Stashbot 569,293 bytes +9,618 ejegg: updated fundraising CiviCRM from 256adda03c to a30da7f92a
  • curprev 23:38, 1 October 2020imported>Stashbot 559,675 bytes +14,499 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@6101b56]: mjolnir: increase training memory overhead by 10% (duration: 00m 34s)
  • curprev 22:52, 30 September 2020imported>Stashbot 545,176 bytes +12,360 robh@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • curprev 00:41, 30 September 2020imported>Stashbot 532,816 bytes +14,059 tgr@deploy1001: Synchronized php-1.36.0-wmf.11/extensions/GrowthExperiments/: Backport: Ensure variant A homepage sidebar is always at least 300px (T263905) (duration: 01m 01s)
  • curprev 00:32, 29 September 2020imported>Stashbot 518,757 bytes +19,144 tgr_: B&C done
  • curprev 06:36, 27 September 2020imported>Stashbot 499,613 bytes +59 elukey: powercycle analytics1048
  • curprev 19:20, 26 September 2020imported>Stashbot 499,554 bytes +672 chrisalbon: sudo service uwsgi-ores restart
  • curprev 01:17, 26 September 2020imported>Stashbot 498,882 bytes +5,669 cdanis: ❌cdanis@ores2001.codfw.wmnet ~ 🕤🍺 sudo systemctl restart uwsgi-ores.service
  • curprev 23:39, 24 September 2020imported>Stashbot 493,213 bytes +19,136 andrew@deploy1001: Finished deploy [horizon/deploy@7b61460]: (no justification provided) (duration: 01m 58s)
  • curprev 23:52, 23 September 2020imported>Stashbot 474,077 bytes +20,309 mutante: alert1001 - systemctl restar ircecho because icinga-wm left the chat
  • curprev 23:27, 22 September 2020imported>Stashbot 453,768 bytes +20,420 legoktm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: clientError: enable on ja,es,de,ru,it,zh,pt wikipedias (T255585) (duration: 01m 04s)
  • curprev 23:42, 21 September 2020imported>Stashbot 433,348 bytes +16,298 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' .
  • curprev 08:46, 20 September 2020imported>Stashbot 417,050 bytes +345 Urbanecm: [urbanecm@mwmaint2001 ~]$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Tepig10102020' 'Davidfromtheworld' # T263317
  • curprev 19:03, 19 September 2020imported>Stashbot 416,705 bytes +407 ariel@deploy1001: Finished deploy [dumps/dumps@14ba6e9]: defer getting db creds until really needed (duration: 00m 04s)
  • curprev 21:48, 18 September 2020imported>Stashbot 416,298 bytes +10,576 tzatziki: changed password for Millennium bug@ptwiki
  • curprev 23:41, 17 September 2020imported>Stashbot 405,722 bytes +16,325 ejegg: updated payments-wiki from 86c997fdb2 to 7bb99ce03a
  • curprev 23:41, 16 September 2020imported>Stashbot 389,397 bytes +13,843 catrope@deploy1001: Synchronized php-1.36.0-wmf.8/extensions/FlaggedRevs: T262970 (duration: 01m 06s)
  • curprev 23:20, 15 September 2020imported>Stashbot 375,554 bytes +10,278 urbanecm@deploy1001: Synchronized php-1.36.0-wmf.9/extensions/FlaggedRevs/backend/FlaggedRevsHooks.php: 1c0b0d161fe1024d6d08a27bbacf5b62c56c9c01: Fix APCOND_FR_NEVERBLOCKED handling (T262970) (duration: 00m 56s)
  • curprev 22:59, 14 September 2020imported>Stashbot 365,276 bytes +15,318 mholloway-shell@deploy1001: helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' .
  • curprev 23:47, 13 September 2020imported>Stashbot 349,958 bytes +218 Urbanecm: Change email address of User:Oversight@enwiki to oversight-l@lists.wikimedia.org as part of OTRS downtime preparation (T262733)
  • curprev 01:07, 12 September 2020imported>Stashbot 349,740 bytes +4,036 mutante: people2001 - rsyncing user home dirs from people1002
  • curprev 00:41, 11 September 2020imported>Stashbot 345,704 bytes +10,447 milimetric@deploy1001: Finished deploy [analytics/refinery@7f5a6ca] (thin): Regular analytics weekly train THIN [analytics/refinery@7f5a6ca] (duration: 00m 08s)
  • curprev 00:31, 10 September 2020imported>Stashbot 335,257 bytes +14,554 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@00b0e20]: Update to current master (duration: 06m 42s)
  • curprev 23:47, 8 September 2020imported>Stashbot 320,703 bytes +17,255 eileen: civicrm revision is 4e40a59d42, config revision is d26334fa36
  • curprev 23:35, 7 September 2020imported>Stashbot 303,448 bytes +9,007 Reedy: Deployed patch for T262213
  • curprev 19:45, 6 September 2020imported>Stashbot 294,441 bytes +598 marostegui@cumin1001: dbctl commit (dc=all): 'Decrease db2127's weight a bit', diff saved to https://phabricator.wikimedia.org/P12496 and previous config saved to /var/cache/conftool/dbconfig/20200906-194512-marostegui.json
  • curprev 00:23, 5 September 2020imported>Stashbot 293,843 bytes +4,779 foks: removing 2 files for legal compliance
  • curprev 01:35, 4 September 2020imported>Stashbot 289,064 bytes +20,066 pt1979@cumin2001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 22:55, 2 September 2020imported>Stashbot 268,998 bytes +11,333 shdubsh: restart rsyslog on centrallog[12]001
  • curprev 22:39, 1 September 2020imported>Stashbot 257,665 bytes +5,601 Urbanecm: [urbanecm@mwmaint2001 ~]$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=sysop_itwiki Pierpao (T261722)
  • curprev 00:02, 1 September 2020imported>Stashbot 252,064 bytes +8,872 mutante: wb2-grrrri was not running and wikibugs had no more Gerrit updates since a while
  • curprev 16:13, 30 August 2020imported>Stashbot 243,192 bytes +64 herron: restarted eqiad v5 logstashes
  • curprev 18:05, 29 August 2020imported>Stashbot 243,128 bytes +402 Amir1: end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T261451)
  • curprev 21:53, 28 August 2020imported>Stashbot 242,726 bytes +5,179 ryankemper: `sudo systemctl reload nginx.service` on `cloudelastic100[5,6].wikimedia.org` to try to resolve certificate warning issues
  • curprev 00:06, 28 August 2020imported>Stashbot 237,547 bytes +27,285 eileen: process-control config revision is dd541a25dc
  • curprev 23:35, 26 August 2020imported>Stashbot 210,262 bytes +10,539 eileen: civicrm revision changed from d2e80f7522 to 481ab742db, config revision is e2ab4d7c1f
  • curprev 21:51, 25 August 2020imported>Stashbot 199,723 bytes +10,845 mutante: xhgui1001/xhgui2001 - Unpacking xhgui (0.12.0-2-wmf1) over (0.9.0-1-wmf1) (T260397)
  • curprev 00:57, 25 August 2020imported>Stashbot 188,878 bytes +12,712 cdanis: T259621 ❌cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 homer 'cr*' commit 'drain cr2-esams transport link'
  • curprev 20:38, 23 August 2020imported>Stashbot 176,166 bytes +518 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • curprev 19:33, 22 August 2020imported>Stashbot 175,648 bytes +367 ryankemper: depooled wdqs1006 (still has 2.5 hours to catch up on)
  • curprev 17:43, 21 August 2020imported>Stashbot 175,281 bytes +1,663 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 22:31, 20 August 2020imported>Stashbot 173,618 bytes +10,495 eileen: civicrm revision changed from 27d5900f7d to ce28723709, config revision is 706cf3c898
  • curprev 00:51, 20 August 2020imported>Stashbot 163,123 bytes +12,052 mutante: ms-be1039 - started failed ferm service
  • curprev 01:05, 19 August 2020imported>Stashbot 151,071 bytes +12,992 dpifke@deploy1001: Synchronized wmf-config/profiler.php: Deploying https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/620139 (duration: 01m 18s)
  • curprev 00:53, 18 August 2020imported>Stashbot 138,079 bytes +29,239 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0)
  • curprev 11:12, 16 August 2020imported>Stashbot 108,840 bytes +72 gehel: repooling wdqs1004 - catched up on lag
  • curprev 21:18, 15 August 2020imported>Stashbot 108,768 bytes +123 gehel: depooling wdqs1004 and restarting services, will wait to catch up on lag before repooling
  • curprev 19:41, 14 August 2020imported>Stashbot 108,645 bytes +742 effie: restart mwdebug1002
  • curprev 23:39, 13 August 2020imported>Stashbot 107,903 bytes +7,860 tzatziki: removing 3 files for legal compliance
  • curprev 00:37, 13 August 2020imported>Stashbot 100,043 bytes +31,681 mutante: removing jenkins_service_running checks from secondary servers where it's stopped, manually from icinga config, running puppet on icinga
  • curprev 23:41, 11 August 2020imported>Stashbot 68,362 bytes +9,831 Urbanecm: Evening B&C window completed
  • curprev 00:33, 11 August 2020imported>Stashbot 58,531 bytes +11,201 dpifke@deploy1001: Finished deploy [performance/arc-lamp@fc5f1c6]: Deploying latest attempt to fix T259167 (duration: 01m 03s)
  • curprev 21:58, 9 August 2020imported>Stashbot 47,330 bytes +195 ejegg: updated payments-wiki from cd012f37f1 to 932aacde54
  • curprev 02:23, 8 August 2020imported>Stashbot 47,135 bytes +243 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-reload
  • curprev 16:42, 7 August 2020imported>Stashbot 46,892 bytes +2,081 jforrester@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/DiscussionTools/: T259855 Revert new reply API (duration: 01m 06s)
  • curprev 23:21, 6 August 2020imported>Stashbot 44,811 bytes +12,330 catrope@deploy1001: Synchronized php-1.36.0-wmf.3/extensions/GrowthExperiments/: Fixes for WelcomeSurvey language question (T232410) (duration: 00m 59s)
  • curprev 00:35, 6 August 2020imported>Stashbot 32,481 bytes +9,981 mutante: wtp2019 - reimaging - parsoid service does not work, unlike on all other wtp*, making sure it's clean
  • curprev 22:41, 4 August 2020imported>Stashbot 22,500 bytes +12,987 brennen: restarting php7.2-fpm on mw1404 for opcache issues
  • curprev 23:43, 3 August 2020imported>Stashbot 9,513 bytes +9,109 mutante: mwdebug1001 - temp installing apt-file for debugging an issue on mwmaint
  • curprev 16:30, 1 August 2020imported>Stashbot 404 bytes −726,353 Amir1: wikiadmin@10.64.32.197(avkwiki)> delete from site_identifiers; (T259122)
  • curprev 23:48, 31 July 2020imported>Stashbot 726,757 bytes +6,358 ejegg: updated payments-wiki from c365c136d2 to cd012f37f1
  • curprev 00:47, 31 July 2020imported>Stashbot 720,399 bytes +10,664 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: disable lilypond execution again (duration: 01m 10s)
  • curprev 00:52, 30 July 2020imported>Stashbot 709,735 bytes +18,579 dzahn@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
  • curprev 00:53, 29 July 2020imported>Stashbot 691,156 bytes +25,595 dzahn@cumin1001: conftool action : set/pooled=yes; selector: name=wtp1046.eqiad.wmnet
  • curprev 00:19, 28 July 2020imported>Stashbot 665,561 bytes +13,224 andrew@cumin1001: conftool action : set/pooled=inactive; selector: name=cloudcephmon1003.eqiad.wmnet
  • curprev 12:41, 25 July 2020imported>Stashbot 652,337 bytes +620 marostegui@cumin1001: dbctl commit (dc=all): 'Pool db1096:3315 into s5 api afte db1082 crashed T258336', diff saved to https://phabricator.wikimedia.org/P12041 and previous config saved to /var/cache/conftool/dbconfig/20200725-124104-marostegui.json
  • curprev 00:47, 25 July 2020imported>Stashbot 651,717 bytes +3,102 mutante: ganeti - removing disk 3 (20G) from webperf1002. the disks are 0-indexed, so the ones actually mounted are 0 (50G) and 1 (300G) (T257931)
  • curprev 01:19, 24 July 2020imported>Stashbot 648,615 bytes +12,821 ejegg: updated payments-wiki from 31a3de1130 to c365c136d2
  • curprev 00:24, 23 July 2020imported>Stashbot 635,794 bytes +17,409 legoktm@deploy1001: Synchronized php-1.35.0-wmf.41/includes/: T258664: Revert "Add a new type of database to the installer from extension" (2/2) (duration: 01m 08s)
  • curprev 23:37, 21 July 2020imported>Stashbot 618,385 bytes +16,371 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump cirrus MLR models to latest (duration: 01m 06s)
  • curprev 00:02, 21 July 2020imported>Stashbot 602,014 bytes +20,177 ryankemper: Began Elasticsearch reindex job on index `dewiki_content` across [`eqiad`, `codfw`, `cloudelastic`], on `rkemper@mwmaint1002` under tmux session `reindex`. Should complete in <24 hours
  • curprev 19:16, 19 July 2020imported>Stashbot 581,837 bytes +546 marostegui: Upgrade and reboot db1085 T258360
  • curprev 21:41, 18 July 2020imported>Stashbot 581,291 bytes +539 shdubsh: restart logstash on logstash200[456]
  • curprev 00:13, 18 July 2020imported>Stashbot 580,752 bytes +4,067 dpifke: Performing one-time expiration of ArcLamp files older than 40 days (normal retention is 45 days), to solve disk space issue until either Ganeti issue is solved or compressed logfile support is merged.
  • curprev 22:16, 16 July 2020imported>Stashbot 576,685 bytes +13,808 mutante: testreduce1001 manually git clone 'scandium' branch of integration/visualdiff into /srv/visualdiff (T257906)
  • curprev 23:54, 15 July 2020imported>Stashbot 562,877 bytes +11,781 eileen: tools revision changed from 7b6018a16e to 711d671600
  • curprev 01:45, 15 July 2020imported>Stashbot 551,096 bytes +15,235 eileen: tools revision changed from a9e7dc1559 to 7b6018a16e
  • curprev 01:23, 14 July 2020imported>Stashbot 535,861 bytes +9,041 ryankemper: Started long-running Elasticsearch reindex of `eqiad`, `codfw`, and `cloudelastic`. tmux session `reindex` under `ryankemper` on `mwmaint1002`
  • curprev 19:16, 11 July 2020imported>Stashbot 526,820 bytes +852 qchris: Restarting Gerrit on gerrit1001 to switch to new gerrit.war and zuul plugin
  • curprev 21:52, 10 July 2020imported>Stashbot 525,968 bytes +7,988 ryankemper: Started long-running reindex of Elasticsearch indices in `eqiad`, `codfw`, and `dewiki` on `mwmaint1002` under tmux session `reindex` for user `ryankemper`
  • curprev 01:44, 10 July 2020imported>Stashbot 517,980 bytes +10,370 mutante: LDAP - adding coka to wmde and nda (T257038)
  • curprev 00:58, 9 July 2020imported>Stashbot 507,610 bytes +15,943 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6c1b646e T256395"'
  • curprev 22:41, 7 July 2020imported>Stashbot 491,667 bytes +21,022 mutante: new Wikimedia Annual Report 2019 now available on annual.wikimedia.org
  • curprev 00:22, 7 July 2020imported>Stashbot 470,645 bytes +11,069 cstone: civicrm revision changed from a48caf0f37 to d73ee2e73f
  • curprev 21:50, 5 July 2020imported>Stashbot 459,576 bytes +1,640 qchris: Restarting gerrit on gerrit1001 to pick up new war and jars.
  • curprev 19:23, 4 July 2020imported>Stashbot 457,936 bytes +722 qchris@deploy1001: Finished deploy [gerrit/gerrit@b78914b]: Bump gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1002 (duration: 00m 08s)
  • curprev 21:49, 3 July 2020imported>Stashbot 457,214 bytes +4,336 reedy@deploy1001: Synchronized php-1.35.0-wmf.39/extensions/Score/: Sync maintenance script (duration: 00m 58s)
  • curprev 23:22, 2 July 2020imported>Stashbot 452,878 bytes +4,683 jhuneidi@deploy1001: helmfile [EQIAD] Ran 'sync' command on namespace 'blubberoid' for release 'production' .
  • curprev 00:53, 2 July 2020imported>Stashbot 448,195 bytes +11,373 eileen: tools revision changed from 806e2b4412 to e38f7a83d4
  • curprev 00:14, 1 July 2020imported>Stashbot 436,822 bytes +12,688 krinkle@deploy1001: Synchronized private/PrivateSettings.php: T254795 - Set $wmgXhguiDBuser and $wmgXhguiDBpasswor (duration: 01m 06s)
  • curprev 23:28, 29 June 2020imported>Stashbot 424,134 bytes +9,692 eileen: civicrm revision changed from 52a32f2d66 to 391d0fdf75, config revision is f1b4bdb7b7
  • curprev 21:43, 28 June 2020imported>Stashbot 414,442 bytes +271 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: no-op I56eb4a802 (duration: 00m 58s)
  • curprev 20:22, 27 June 2020imported>Stashbot 414,171 bytes +2,305 qchris: Gerrit upgrade done.
  • curprev 18:42, 26 June 2020imported>Stashbot 411,866 bytes +4,536 robh: all ulsfo onsite work completed as of 30 minutes ago
  • curprev 01:41, 26 June 2020imported>Stashbot 407,330 bytes +15,922 cdanis: ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕘🍺 sudo cumin A:cp 'enable-puppet "cdanis deploying I6cc5f3e6 T256395"'
  • curprev 00:48, 25 June 2020imported>Stashbot 391,408 bytes +12,937 twentyafterfour: restart php-fpm on phab1001 to fix T256343
  • curprev 00:35, 24 June 2020imported>Stashbot 378,471 bytes +8,585 ejegg: restarted fundraising jobs on main CiviCRM box
  • curprev 23:41, 22 June 2020imported>Stashbot 369,886 bytes +10,612 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: touch for T247330 (duration: 00m 56s)
  • curprev 22:56, 20 June 2020imported>Stashbot 359,274 bytes +414 cdanis@cumin2001: dbctl commit (dc=all): 'db1088 seems to have crashed', diff saved to https://phabricator.wikimedia.org/P11611 and previous config saved to /var/cache/conftool/dbconfig/20200620-225624-cdanis.json
  • curprev 18:10, 19 June 2020imported>Stashbot 358,860 bytes +6,598 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Bump eventlogging_Test schema version to 1.1.0 to pick up client_dt - T238230 (duration: 00m 59s)
  • curprev 22:30, 18 June 2020imported>Stashbot 352,262 bytes +9,559 otto@deploy1001: Synchronized wmf-config/InitialiseSettings.php: EventLogging to EventGate: - SearchSatisfaction on all wikis - T249261 (duration: 00m 56s)
  • curprev 23:25, 17 June 2020imported>Stashbot 342,703 bytes +17,347 urbanecm@deploy1001: Synchronized wmf-config/InitialiseSettings.php: 0e7079d: Install DiscussionTools on all wikis (attempt 2) (T252264; T253943) (duration: 00m 56s)
  • curprev 01:43, 17 June 2020imported>Stashbot 325,356 bytes +20,465 shdubsh: restart elasticsearch on logstash1011
  • curprev 00:28, 16 June 2020imported>Stashbot 304,891 bytes +12,502 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: limit HTTP client timeout T245170 (duration: 00m 56s)
  • curprev 13:51, 14 June 2020imported>Stashbot 292,389 bytes +113 qchris: Disabling puppet on gerrit1002 (test instance) to do some more upgrade testing
  • curprev 21:12, 13 June 2020imported>Stashbot 292,276 bytes +335 qchris: Enabling puppet on gerrit1002 (test instance). Done with testing for today.
  • curprev 17:44, 12 June 2020imported>Stashbot 291,941 bytes +6,714 herron: restarting logstash1011 elasticsearch instance
  • curprev 01:02, 12 June 2020imported>Stashbot 285,227 bytes +14,139 ejegg: updated payments-wiki from aceddff8b5 to 5fd4eb1519
  • curprev 23:55, 10 June 2020imported>Stashbot 271,088 bytes +14,140 catrope@deploy1001: Synchronized php-1.35.0-wmf.36/includes/skins/SkinTemplate.php: T255073 (duration: 01m 07s)
  • curprev 23:18, 9 June 2020imported>Stashbot 256,948 bytes +12,465 Reedy: run namespaceDupes.php --fix for hiwikibooks T254012
  • curprev 00:39, 9 June 2020imported>Stashbot 244,483 bytes +10,746 ejegg: updated payments-wiki from c1d14a5db7 to aceddff8b5
  • curprev 16:45, 5 June 2020imported>Stashbot 233,737 bytes +4,799 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@f7e4f78]: Upgrade to 1.24.0 (duration: 00m 11s)
  • curprev 23:45, 4 June 2020imported>Stashbot 228,938 bytes +11,913 catrope@deploy1001: Synchronized wmf-config/mc.php: Set coalesceKeys=non-global for WANCache on enwiki (duration: 00m 59s)
  • curprev 00:36, 4 June 2020imported>Stashbot 217,025 bytes +14,843 reedy@deploy1001: Synchronized php-1.35.0-wmf.35/includes/specials/SpecialUserrights.php: T254417 T251534 (duration: 01m 06s)
  • curprev 01:00, 3 June 2020imported>Stashbot 202,182 bytes +10,631 reedy@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: wgLocalVirtualHosts (duration: 01m 06s)
  • curprev 20:14, 1 June 2020imported>Stashbot 191,551 bytes +5,297 shdubsh: downgrade mtail to rc5 in ulsfo -- T254192
  • curprev 09:56, 31 May 2020imported>Stashbot 186,254 bytes +199 Urbanecm: mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki 'Vox Golf' 'Colonel Chicken' (T254068)
  • curprev 22:32, 29 May 2020imported>Stashbot 186,055 bytes +4,498 bstorm_: updated views on labsdb1010 T252219
  • curprev 23:48, 28 May 2020imported>Stashbot 181,557 bytes +7,100 jforrester@deploy1001: Synchronized php-1.35.0-wmf.34/skins/Vector/resources/skins.vector.styles/Menu.less: T253912 Hotfix: Cannot rename emptyPortlet to empty-portlet yet (duration: 00m 59s)
  • curprev 23:20, 27 May 2020imported>Stashbot 174,457 bytes +16,759 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add autoreviewrestore right to rollbacker group on hiwiki (T252986) (duration: 01m 05s)
  • curprev 21:34, 26 May 2020imported>Stashbot 157,698 bytes +8,943 krinkle@deploy1001: Synchronized wmf-config/mc.php: I0fb124b3593 (duration: 01m 05s)
  • curprev 23:34, 25 May 2020imported>Stashbot 148,755 bytes +7,068 ejegg: re-enabled fundraising queue consumers and job runners, except audits, dedupe, and recurring
  • curprev 17:36, 24 May 2020imported>Stashbot 141,687 bytes +170 gehel: restarting elasticsearch psi on elastic1052
  • curprev 19:04, 23 May 2020imported>Stashbot 141,517 bytes +906 krinkle@deploy1001: Synchronized php-1.35.0-wmf.31/includes/filerepo/file/LocalFile.php: I0f7e885997d60 (duration: 01m 06s)
(newest | oldest) View (newer 500 | ) (20 | 50 | 100 | 250 | 500)