Revision history of "Server Admin Log"

Jump to navigation Jump to search

Diff selection: Mark the radio boxes of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)
  • curprev 16:11, 26 November 2021imported>Stashbot 342,451 bytes +4,657 arnoldokoth: drain kubestage1002 node in prep for decommissioning
  • curprev 20:44, 25 November 2021imported>Stashbot 337,794 bytes +25,467 ladsgroup@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T296143)', diff saved to https://phabricator.wikimedia.org/P17872 and previous config saved to /var/cache/conftool/dbconfig/20211125-204357-ladsgroup.json
  • curprev 00:37, 25 November 2021imported>Stashbot 312,327 bytes +56,025 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2066.codfw.wmnet with OS buster
  • curprev 00:25, 24 November 2021imported>Stashbot 256,302 bytes +24,082 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wdqs2012.codfw.wmnet with OS buster
  • curprev 00:51, 23 November 2021imported>Stashbot 232,220 bytes +27,289 urbanecm@deploy1002: Started scap: 69aa4a7: 7c0e074: Revert "Create redirect Special Pages for delete and protect action" (T295611; T296203; 4/4)
  • curprev 13:17, 21 November 2021imported>Stashbot 204,931 bytes +326 dcausse: restarting blazegraph on wdqs1007 (jvm stuck for 10h)
  • curprev 01:02, 20 November 2021imported>Stashbot 204,605 bytes +91 mutante: lists1001 - restarted apache, icinga alerts for the web UI, but recovered
  • curprev 00:27, 20 November 2021imported>Stashbot 204,514 bytes +9,398 cdanis@cumin1001: END (PASS) - Cookbook sre.network.cf (exit_code=0)
  • curprev 01:45, 19 November 2021imported>Stashbot 195,116 bytes +21,872 mutante: I think git-ssh6_22 is down (see alerts lvs2008/2009) due to the v6 issue from ongoing lvs maintenance. depooled in conftool
  • curprev 01:47, 18 November 2021imported>Stashbot 173,244 bytes +19,375 legoktm@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host thumbor2005.codfw.wmnet
  • curprev 00:19, 17 November 2021imported>Stashbot 153,869 bytes +12,582 ryankemper: T276198 `ryankemper@cumin1001:~$ sudo cumin -b 3 '*elastic*' 'sudo run-puppet-agent --force'` Change looks good (no complaints from systemd), rolling out to rest of fleet / reenabling puppet
  • curprev 00:28, 16 November 2021imported>Stashbot 141,287 bytes +15,831 urbanecm: UTC late window done
  • curprev 11:48, 14 November 2021imported>Stashbot 125,456 bytes +111 paravoid: disable cr1-eqiad:xe-3/0/6 (IXP port) to mitigate T295650
  • curprev 18:43, 13 November 2021imported>Stashbot 125,345 bytes +836 AndyRussG: Enabled debug logging for PayPal IPN listener (updated SmashPig config a9e30591 -> 9567cc4a on frpig1001)
  • curprev 21:00, 12 November 2021imported>Stashbot 124,509 bytes +5,102 cmjohnson@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:19, 12 November 2021imported>Stashbot 119,407 bytes +11,807 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:22, 11 November 2021imported>Stashbot 107,600 bytes +12,063 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:07, 10 November 2021imported>Stashbot 95,537 bytes +10,338 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 23:39, 8 November 2021imported>Stashbot 85,199 bytes +11,520 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:43, 6 November 2021imported>Stashbot 73,679 bytes +7,012 dduvall@deploy1002: Synchronized php-1.38.0-wmf.7/includes/parser/ParserOutput.php: Backport: Regression fix: do language conversion on ToC in ParserOutput::getText() (T295187) (duration: 00m 56s)
  • curprev 00:16, 5 November 2021imported>Stashbot 66,667 bytes +20,851 mutante: phab1001 - sudo systemctl start phabricator_clean_tmp_files.service because Icinga alerted it had failed... worked fine
  • curprev 00:26, 4 November 2021imported>Stashbot 45,816 bytes +22,857 tgr: UTC late deploys done
  • curprev 23:47, 2 November 2021imported>Stashbot 22,959 bytes +11,789 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:21, 2 November 2021imported>Stashbot 11,170 bytes −844,596 ejegg: updated SmashPig standalone deploy from dd3a81c7c2 to be68299b92
  • curprev 21:49, 31 October 2021imported>Stashbot 855,766 bytes +371 urbanecm: urbanecm@mwmaint1002:~$ mwscript userOptions.php --wiki=dewiki --nowarn --touserid 3802752 --old 'linkrecommendation' --new 'control' 'growthexperiments-homepage-variant' # T294712
  • curprev 19:25, 30 October 2021imported>Stashbot 855,395 bytes +168 dcausse: restarting blazegraph on wdqs1007 (jvm stuck)
  • curprev 22:57, 29 October 2021imported>Stashbot 855,227 bytes +3,905 urbanecm: [urbanecm@mwmaint1002 ~]$ mwscript extensions/CentralAuth/maintenance/attachAccount.php --wiki=foundationwiki --userlist users.txt # T205347, users.txt is at P17641
  • curprev 23:50, 28 October 2021imported>Stashbot 851,322 bytes +21,486 robh@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4033.ulsfo.wmnet with OS buster
  • curprev 23:55, 27 October 2021imported>Stashbot 829,836 bytes +10,201 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 22:59, 26 October 2021imported>Stashbot 819,635 bytes +12,871 legoktm: uploaded python-logstash to buster-wikimedia for T294393
  • curprev 23:12, 25 October 2021imported>Stashbot 806,764 bytes +11,982 catrope@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Create alias for Appendix and Appendix_talk namespaces on mywiktionary (T291146) (duration: 00m 55s)
  • curprev 16:40, 23 October 2021imported>Stashbot 794,782 bytes +253 dcausse: restarting blazegraph on wdqs1004 and wdqs1006 (free allocators alert)
  • curprev 23:17, 22 October 2021imported>Stashbot 794,529 bytes +3,761 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 23:40, 21 October 2021imported>Stashbot 790,768 bytes +17,384 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:06, 21 October 2021imported>Stashbot 773,384 bytes +14,742 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:31, 20 October 2021imported>Stashbot 758,642 bytes +27,590 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:38, 19 October 2021imported>Stashbot 731,052 bytes +11,211 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
  • curprev 03:56, 16 October 2021imported>Stashbot 719,841 bytes +266 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
  • curprev 23:48, 15 October 2021imported>Stashbot 719,575 bytes +5,052 ryankemper@cumin1001: START - Cookbook sre.wdqs.data-transfer
  • curprev 00:09, 15 October 2021imported>Stashbot 714,523 bytes +19,109 ryankemper@cumin1001: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0)
  • curprev 01:41, 14 October 2021imported>Stashbot 695,414 bytes +17,092 ejegg: updated payments-wiki from b329d2dea2 to 19d18c1852
  • curprev 00:38, 13 October 2021imported>Stashbot 678,322 bytes +15,911 ejegg: updated payments-wiki from 030b11da1a to b329d2dea2
  • curprev 00:11, 12 October 2021imported>Stashbot 662,411 bytes +3,728 eileen: civicrm revision changed from 598b59b0ee to 96090e4bd2, config revision is 85277466ed
  • curprev 05:01, 9 October 2021imported>Stashbot 658,683 bytes +224 jiji@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:32, 9 October 2021imported>Stashbot 658,459 bytes +8,808 ryankemper@cumin1001: END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) restart without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic restart - ryankemper@cumin1001 - T292814
  • curprev 00:07, 8 October 2021imported>Stashbot 649,651 bytes +11,887 tgr_: deploy window over
  • curprev 00:11, 7 October 2021imported>Stashbot 637,764 bytes +11,864 mutante: [grafana2001:~] $ sudo systemctl start rsync-var-lib-grafana because of "PROBLEM - Check systemd state on grafana2001 is CRITICAL: CRITICAL - degraded" because of some race condition where a file vanished during sync
  • curprev 01:39, 6 October 2021imported>Stashbot 625,900 bytes +10,381 legoktm: legoktm@mwmaint1002:~$ echo "https://en.wikiversity.org/static/images/mobile/copyright/wikiversity.svg" |mwscript purgeList.php
  • curprev 23:30, 4 October 2021imported>Stashbot 615,519 bytes +9,952 foks: resetting some emails used for abuse by a globally-banned user
  • curprev 14:45, 3 October 2021imported>Stashbot 605,567 bytes +527 _joe_: restarting acmechief on acmechief1001
  • curprev 17:28, 2 October 2021imported>Stashbot 605,040 bytes +230 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 23:19, 1 October 2021imported>Stashbot 604,810 bytes +38,661 bd808@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 23:20, 29 September 2021imported>Stashbot 566,149 bytes +14,880 bd808@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 00:52, 29 September 2021imported>Stashbot 551,269 bytes +27,094 eileen: civicrm revision changed from a1929b3dfd to a480bf03c9, config revision is 77cb7ec866
  • curprev 23:42, 27 September 2021imported>Stashbot 524,175 bytes +27,476 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 14:51, 26 September 2021imported>Stashbot 496,699 bytes +432 volker-e@deploy1002: Finished deploy [design/style-guide@aac0ae9]: Deploy design/style-guide: aac0ae9 “Apps”: Fix image path (#490) (duration: 00m 06s)
  • curprev 02:00, 25 September 2021imported>Stashbot 496,267 bytes +374 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .
  • curprev 20:00, 24 September 2021imported>Stashbot 495,893 bytes +10,338 volker-e@deploy1002: Finished deploy [design/style-guide@362c6b1]: Deploy design/style-guide: 362c6b1 “Components”: Fix index link (#489) (duration: 00m 06s)
  • curprev 00:39, 24 September 2021imported>Stashbot 485,555 bytes +19,157 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:35, 23 September 2021imported>Stashbot 466,398 bytes +12,744 catrope@deploy1002: Synchronized php-1.38.0-wmf.1/extensions/MediaSearch/: Use text() instead of parse() for MediaSearch UI messages (T291590) (duration: 01m 08s)
  • curprev 23:20, 21 September 2021imported>Stashbot 453,654 bytes +9,016 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:16, 21 September 2021imported>Stashbot 444,638 bytes +11,548 tgr: Evening deploys done
  • curprev 01:47, 18 September 2021imported>Stashbot 433,090 bytes +322 ladsgroup@deploy1002: Synchronized php-1.37.0-wmf.23/includes/libs/rdbms/database/Database.php: (no justification provided) (duration: 00m 57s)
  • curprev 21:28, 17 September 2021imported>Stashbot 432,768 bytes +3,000 legoktm@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • curprev 00:04, 17 September 2021imported>Stashbot 429,768 bytes +13,476 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .
  • curprev 23:02, 15 September 2021imported>Stashbot 416,292 bytes +9,362 legoktm: upgrading lists1001 to use postorius 1.3.5
  • curprev 23:01, 14 September 2021imported>Stashbot 406,930 bytes +15,959 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Re-enable VipsScaler (2 of 2) (duration: 01m 04s)
  • curprev 00:04, 14 September 2021imported>Stashbot 390,971 bytes +8,994 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 18:33, 12 September 2021imported>Stashbot 381,977 bytes +422 vgutierrez: restart varnish-fe on cp3061, cp3063 and cp3065
  • curprev 19:02, 11 September 2021imported>Stashbot 381,555 bytes +650 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 27814b8eaacb5ba2fee1b6167a36ea14356a1ecf: testwiki: Fully remove securepoll-related groups (T290808) (duration: 00m 57s)
  • curprev 21:29, 10 September 2021imported>Stashbot 380,905 bytes +5,151 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • curprev 00:35, 10 September 2021imported>Stashbot 375,754 bytes +15,347 tgr: Deployed patch for T290692
  • curprev 00:50, 9 September 2021imported>Stashbot 360,407 bytes +8,465 legoktm@deploy1002: Synchronized wmf-config/CommonSettings.php: Don't set default to Score (try #2) (duration: 00m 58s)
  • curprev 00:00, 8 September 2021imported>Stashbot 351,942 bytes +12,900 legoktm: legoktm@lists1001:~$ sudo rm -rf /etc/mailman # cleanup as part of 4869d91b0be / T282303
  • curprev 23:52, 6 September 2021imported>Stashbot 339,042 bytes +8,019 tstarling@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/SecurePoll/includes/Talliers/STVTallier.php: T290000 (duration: 00m 58s)
  • curprev 18:54, 5 September 2021imported>Stashbot 331,023 bytes +201 urbanecm: wikiadmin@10.192.0.119(ptwiki)> update protected_titles set pt_create_perm='editautoreviewprotected' where pt_create_perm='autoreviewer'; # T290396
  • curprev 13:35, 4 September 2021imported>Stashbot 330,822 bytes +1,989 marostegui@cumin1001: dbctl commit (dc=all): 'db2137:3314 (re)pooling @ 100%: Slowly repool T290374', diff saved to https://phabricator.wikimedia.org/P17217 and previous config saved to /var/cache/conftool/dbconfig/20210904-133532-root.json
  • curprev 21:49, 3 September 2021imported>Stashbot 328,833 bytes +13,135 bd808@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' .
  • curprev 00:31, 3 September 2021imported>Stashbot 315,698 bytes +18,404 tgr@deploy1002: Synchronized php-1.37.0-wmf.21/extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php: Backport: fixLinkRecommendationData: Try harder to avoid >10K result sets (T284531) (duration: 00m 58s)
  • curprev 23:50, 1 September 2021imported>Stashbot 297,294 bytes +11,478 Amir1: mwscript createAndPromote.php --wiki=test2wiki --sysop --force Ladsgroup
  • curprev 00:53, 1 September 2021imported>Stashbot 285,816 bytes +13,363 eileen: civicrm revision changed from e567b4c289 to 7da3eba4f9, config revision is 5f004d94d7
  • curprev 23:14, 30 August 2021imported>Stashbot 272,453 bytes +10,249 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:15, 29 August 2021imported>Stashbot 262,204 bytes +714 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 16:46, 27 August 2021imported>Stashbot 261,490 bytes +3,953 hnowlan@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps1005.eqiad.wmnet with reason: Resyncing from master
  • curprev 00:46, 27 August 2021imported>Stashbot 257,537 bytes +9,223 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 23:23, 25 August 2021imported>Stashbot 248,314 bytes +23,982 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 00:53, 25 August 2021imported>Stashbot 224,332 bytes +6,127 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • curprev 00:17, 24 August 2021imported>Stashbot 218,205 bytes +10,888 ryankemper: [WDQS Deploy] Restarting `wdqs-categories` across lvs-managed hosts, one node at a time: `sudo -E cumin -b 1 'A:wdqs-all and not A:wdqs-test' 'depool && sleep 45 && systemctl restart wdqs-categories && sleep 45 && pool'`
  • curprev 13:35, 22 August 2021imported>Nintendofan885 207,317 bytes −804,471 Archiving May 2021–July 2021 (2/3)
  • curprev 15:12, 21 August 2021imported>Stashbot 1,011,788 bytes +262 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 23:18, 20 August 2021imported>Stashbot 1,011,526 bytes +4,817 legoktm: deployed patch for T289385
  • curprev 00:18, 20 August 2021imported>Stashbot 1,006,709 bytes +10,542 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • curprev 01:13, 19 August 2021imported>Stashbot 996,167 bytes +6,256 ejegg: updated fundraising CiviCRM from 73f6ec9190 to 8ed303f2d1
  • curprev 00:39, 18 August 2021imported>Stashbot 989,911 bytes +13,880 dpifke@deploy1002: Finished deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243) (duration: 00m 05s)
  • curprev 00:45, 17 August 2021imported>Stashbot 976,031 bytes +12,823 eileen: civicrm revision changed from ba0c7705bb to 175a3101f7, config revision is 7bdc78073d
  • curprev 20:02, 15 August 2021imported>Stashbot 963,208 bytes +299 addshore: restarting blazegraph on wdqs2004
  • curprev 03:54, 14 August 2021imported>Stashbot 962,909 bytes +121 legoktm[m]: restarting mailman3 on lists1001, bounce runner crashed (T288880)
  • curprev 18:43, 13 August 2021imported>Stashbot 962,788 bytes +9,163 bblack: reprepro: uploaded gdnsd-3.8.0-1~wmf1 to buster-wikimedia - T252132
  • curprev 23:50, 12 August 2021imported>Stashbot 953,625 bytes +22,057 derick@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Set archive namespaces on foundationwiki to 'noindex,follow' (T288763) (duration: 00m 59s)
  • curprev 23:30, 11 August 2021imported>Stashbot 931,568 bytes +15,885 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
(newest | oldest) View (newer 100 | ) (20 | 50 | 100 | 250 | 500)