You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(ejegg: updated SmashPig standalone deploy from dd3a81c7c2 to be68299b92)
imported>Stashbot
(mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .)
Line 1: Line 1:
== 2021-11-02 ==
== 2021-11-02 ==
* 23:47 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 23:46 tgr: UTC late deploys done
* 23:45 tgr@deploy1002: Synchronized wmf-config: Config: Use page id for GrowthExperiments image recommendations, except for testwiki ([[gerrit:736314{{!}}736314]] [[gerrit:736317{{!}}736317]] ([[phab:T290949|T290949]] [[phab:T292154|T292154]]) (duration: 01m 03s)
* 23:44 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 23:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 23:34 tgr@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:735094{{!}}Use url-downloader proxy for GrowthExperiments (T290949)]] (duration: 01m 14s)
* 23:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 22:14 robh@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-db1002.eqiad.wmnet with OS buster
* 21:50 robh@cumin1001: START - Cookbook sre.hosts.reimage for host an-db1002.eqiad.wmnet with OS buster
* 21:32 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-db1002.eqiad.wmnet with OS buster
* 21:03 robh@cumin1001: START - Cookbook sre.hosts.reimage for host an-db1002.eqiad.wmnet with OS buster
* 20:52 robh@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-db1001.eqiad.wmnet with OS buster
* 20:28 robh@cumin1001: START - Cookbook sre.hosts.reimage for host an-db1001.eqiad.wmnet with OS buster
* 20:01 thcipriani: 1.38.0-wmf.7 on testwikis, leaving it there for today for US holiday ([[phab:T293948|T293948]])
* 19:58 thcipriani@deploy1002: Pruned MediaWiki: 1.38.0-wmf.5 (duration: 04m 08s)
* 19:53 thcipriani@deploy1002: Finished scap: testwikis wikis to 1.38.0-wmf.7  refs [[phab:T293948|T293948]] (duration: 50m 13s)
* 19:50 moritzm: imported ganeti 2.16.0-1~bpo9+1+wmf1to component/ganeti216 for stretch-wikimedia (with additional cherrypicked patches for compat with KVM 3.1)  [[phab:T284811|T284811]]
* 19:47 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 19:39 robh@cumin1001: START - Cookbook sre.dns.netbox
* 19:35 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts an-db1002.eqiad.wmnet
* 19:08 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 19:08 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-db1001.eqiad.wmnet with OS buster
* 19:05 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 19:02 thcipriani@deploy1002: Started scap: testwikis wikis to 1.38.0-wmf.7  refs [[phab:T293948|T293948]]
* 18:46 thcipriani: starting to stage train for 1.38.0-wmf.7 ([[phab:T293948|T293948]])
* 18:33 robh@cumin1001: START - Cookbook sre.hosts.decommission for hosts an-db1002.eqiad.wmnet
* 18:32 robh@cumin1001: START - Cookbook sre.hosts.reimage for host an-db1001.eqiad.wmnet with OS buster
* 18:23 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
* 18:18 robh@cumin1001: START - Cookbook sre.dns.netbox
* 18:15 robh@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-db1001.eqiad.wmnet
* 18:14 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 18:11 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 18:01 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 17:59 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.6/extensions/DiscussionTools/modules/dt-ve/dt.ui.UsernameCompletionAction.js: {{Gerrit|494af124b95e2eabff94fde79aed6b6f6f81feab}}: UsernameCompletion: Filter out users with indefinite sitewide blocks from API results ([[phab:T294783|T294783]]) (duration: 00m 55s)
* 17:58 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 17:57 robh@cumin1001: START - Cookbook sre.hosts.decommission for hosts an-db1001.eqiad.wmnet
* 17:48 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 17:45 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 17:44 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: {{Gerrit|339be07a35de1fa3846b845376695d68a9d743fd}}: foundationwiki: Set wgCentralAuthCookies to true ([[phab:T205347|T205347]]) (duration: 00m 54s)
* 17:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 17:33 moritzm: installing opencv security updates
* 17:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 17:24 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|e3227703a662ecda744bb159f39b128ed289c76d}}: Revert "Revert "foundationwiki: Enable Translate extension"" ([[phab:T205349|T205349]]) (duration: 00m 55s)
* 17:22 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.6/includes/cache/LinkCache.php: {{Gerrit|1e78aeabd682537d8c284559e1356d15c62da810}}: LinkCache: Try invalidating cache before throwing ([[phab:T205349|T205349]]) (duration: 00m 56s)
* 17:22 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 17:18 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 16:38 mmandere: pool cp4036.ulsfo.wmnet - [[phab:T290694|T290694]]
* 16:30 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4036.ulsfo.wmnet with OS buster
* 15:41 mmandere: pool cp4034.ulsfo.wmnet - [[phab:T290694|T290694]]
* 15:38 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp4036.ulsfo.wmnet with OS buster
* 15:32 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4034.ulsfo.wmnet with OS buster
* 15:12 jgiannelos@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
* 15:11 jgiannelos@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
* 15:07 jgiannelos@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
* 14:34 mmandere: pool cp4035.ulsfo.wmnet - [[phab:T290694|T290694]]
* 14:31 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp4034.ulsfo.wmnet with OS buster
* 14:24 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4035.ulsfo.wmnet with OS buster
* 14:19 hnowlan: roll-restart restart-php7.2-fpm on A:mw-app-canary and A:mw-api-canary
* 14:15 hnowlan: debdeploying wikidiff2-1.13.0-1 to A:mw-app-canary and A:mw-api-canary for [[phab:T285857|T285857]]
* 14:05 hashar@deploy1002: Finished deploy [integration/docroot@4e4d14a]: Add landing page for code metrics (duration: 00m 09s)
* 14:05 hashar@deploy1002: Started deploy [integration/docroot@4e4d14a]: Add landing page for code metrics
* 13:45 mmandere: pool cp4033.ulsfo.wmnet  - [[phab:T290694|T290694]]
* 11:26 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw1002.eqiad.wmnet
* 11:06 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1003.eqiad.wmnet
* 11:00 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudnet1003.eqiad.wmnet
* 11:00 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1004.eqiad.wmnet
* 10:57 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1008.eqiad.wmnet
* 10:54 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudnet1004.eqiad.wmnet
* 10:53 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2002-dev.codfw.wmnet
* 10:48 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw2002-dev.codfw.wmnet
* 10:48 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2001-dev.codfw.wmnet
* 10:46 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1008.eqiad.wmnet
* 10:46 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1005.eqiad.wmnet
* 10:41 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw2001-dev.codfw.wmnet
* 10:40 aborrero@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudgw2001-dev.codfw.wmnet
* 10:40 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw2001-dev.codfw.wmnet
* 10:36 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet
* 10:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 10:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 10:30 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|dbff998f40e438556345185408e495f429440a1b}}: dewiki: Set wgGEHomepageDefaultVariant to control ([[phab:T294712|T294712]]) (duration: 00m 55s)
* 10:03 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1118 with weight 0 [[phab:T293964|T293964]]', diff saved to https://phabricator.wikimedia.org/P17652 and previous config saved to /var/cache/conftool/dbconfig/20211102-100348-root.json
* 09:46 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 09:42 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 09:40 legoktm: restarted apache2 on lists1001
* 09:39 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|b2594347041ae61ef88661bc0d5aa57fc501540d}}: QuickSurveys: Show Growth IP editors survey to 0.1% of users ([[phab:T294568|T294568]]) (duration: 00m 57s)
* 09:03 marostegui@cumin1001: dbctl commit (dc=all): 'Remove recentchanges replicas from s6 eqiad [[phab:T263127|T263127]]', diff saved to https://phabricator.wikimedia.org/P17651 and previous config saved to /var/cache/conftool/dbconfig/20211102-090306-marostegui.json
* 08:29 moritzm: installing sdl2 security updates
* 07:23 marostegui@cumin1001: dbctl commit (dc=all): 'Remove recentchangeslinked replicas from s6 eqiad [[phab:T263127|T263127]]', diff saved to https://phabricator.wikimedia.org/P17650 and previous config saved to /var/cache/conftool/dbconfig/20211102-072320-marostegui.json
* 07:13 elukey: `apt-get purge dkms` (rc state) on stat100[5,8]
* 06:45 marostegui: Rename oauth2_access_tokens oauth_accepted_consumer oauth_registered_consumer tables on db1123 [[phab:T294595|T294595]]
* 02:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 02:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 02:11 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 02:07 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
* 01:56 cstone: civicrm revision changed from {{Gerrit|403be9ce05}} to {{Gerrit|93caef68ef}}
* 01:21 ejegg: updated SmashPig standalone deploy from {{Gerrit|dd3a81c7c2}} to {{Gerrit|be68299b92}}
* 01:21 ejegg: updated SmashPig standalone deploy from {{Gerrit|dd3a81c7c2}} to {{Gerrit|be68299b92}}
* 01:18 ejegg: updated payments-wiki from {{Gerrit|5b9fdd0fe1}} to {{Gerrit|73de4731bd}}
* 01:18 ejegg: updated payments-wiki from {{Gerrit|5b9fdd0fe1}} to {{Gerrit|73de4731bd}}

Revision as of 23:47, 2 November 2021

2021-11-02

  • 23:47 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 23:46 tgr: UTC late deploys done
  • 23:45 tgr@deploy1002: Synchronized wmf-config: Config: Use page id for GrowthExperiments image recommendations, except for testwiki (736314 736317 (T290949 T292154) (duration: 01m 03s)
  • 23:44 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 23:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 23:34 tgr@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: Use url-downloader proxy for GrowthExperiments (T290949) (duration: 01m 14s)
  • 23:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 22:14 robh@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-db1002.eqiad.wmnet with OS buster
  • 21:50 robh@cumin1001: START - Cookbook sre.hosts.reimage for host an-db1002.eqiad.wmnet with OS buster
  • 21:32 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-db1002.eqiad.wmnet with OS buster
  • 21:03 robh@cumin1001: START - Cookbook sre.hosts.reimage for host an-db1002.eqiad.wmnet with OS buster
  • 20:52 robh@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-db1001.eqiad.wmnet with OS buster
  • 20:28 robh@cumin1001: START - Cookbook sre.hosts.reimage for host an-db1001.eqiad.wmnet with OS buster
  • 20:01 thcipriani: 1.38.0-wmf.7 on testwikis, leaving it there for today for US holiday (T293948)
  • 19:58 thcipriani@deploy1002: Pruned MediaWiki: 1.38.0-wmf.5 (duration: 04m 08s)
  • 19:53 thcipriani@deploy1002: Finished scap: testwikis wikis to 1.38.0-wmf.7 refs T293948 (duration: 50m 13s)
  • 19:50 moritzm: imported ganeti 2.16.0-1~bpo9+1+wmf1to component/ganeti216 for stretch-wikimedia (with additional cherrypicked patches for compat with KVM 3.1) T284811
  • 19:47 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:39 robh@cumin1001: START - Cookbook sre.dns.netbox
  • 19:35 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts an-db1002.eqiad.wmnet
  • 19:08 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 19:08 robh@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-db1001.eqiad.wmnet with OS buster
  • 19:05 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 19:02 thcipriani@deploy1002: Started scap: testwikis wikis to 1.38.0-wmf.7 refs T293948
  • 18:46 thcipriani: starting to stage train for 1.38.0-wmf.7 (T293948)
  • 18:33 robh@cumin1001: START - Cookbook sre.hosts.decommission for hosts an-db1002.eqiad.wmnet
  • 18:32 robh@cumin1001: START - Cookbook sre.hosts.reimage for host an-db1001.eqiad.wmnet with OS buster
  • 18:23 robh@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:18 robh@cumin1001: START - Cookbook sre.dns.netbox
  • 18:15 robh@cumin1001: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-db1001.eqiad.wmnet
  • 18:14 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 18:11 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 18:01 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:59 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.6/extensions/DiscussionTools/modules/dt-ve/dt.ui.UsernameCompletionAction.js: 494af12: UsernameCompletion: Filter out users with indefinite sitewide blocks from API results (T294783) (duration: 00m 55s)
  • 17:58 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:57 robh@cumin1001: START - Cookbook sre.hosts.decommission for hosts an-db1001.eqiad.wmnet
  • 17:48 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:45 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:44 urbanecm@deploy1002: Synchronized wmf-config/CommonSettings.php: 339be07: foundationwiki: Set wgCentralAuthCookies to true (T205347) (duration: 00m 54s)
  • 17:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:33 moritzm: installing opencv security updates
  • 17:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:24 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: e322770: Revert "Revert "foundationwiki: Enable Translate extension"" (T205349) (duration: 00m 55s)
  • 17:22 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.6/includes/cache/LinkCache.php: 1e78aea: LinkCache: Try invalidating cache before throwing (T205349) (duration: 00m 56s)
  • 17:22 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:18 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 16:38 mmandere: pool cp4036.ulsfo.wmnet - T290694
  • 16:30 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4036.ulsfo.wmnet with OS buster
  • 15:41 mmandere: pool cp4034.ulsfo.wmnet - T290694
  • 15:38 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp4036.ulsfo.wmnet with OS buster
  • 15:32 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4034.ulsfo.wmnet with OS buster
  • 15:12 jgiannelos@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
  • 15:11 jgiannelos@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
  • 15:07 jgiannelos@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' .
  • 14:34 mmandere: pool cp4035.ulsfo.wmnet - T290694
  • 14:31 mmandere@cumin1001: START - Cookbook sre.hosts.reimage for host cp4034.ulsfo.wmnet with OS buster
  • 14:24 mmandere@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4035.ulsfo.wmnet with OS buster
  • 14:19 hnowlan: roll-restart restart-php7.2-fpm on A:mw-app-canary and A:mw-api-canary
  • 14:15 hnowlan: debdeploying wikidiff2-1.13.0-1 to A:mw-app-canary and A:mw-api-canary for T285857
  • 14:05 hashar@deploy1002: Finished deploy [integration/docroot@4e4d14a]: Add landing page for code metrics (duration: 00m 09s)
  • 14:05 hashar@deploy1002: Started deploy [integration/docroot@4e4d14a]: Add landing page for code metrics
  • 13:45 mmandere: pool cp4033.ulsfo.wmnet - T290694
  • 11:26 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw1002.eqiad.wmnet
  • 11:06 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1003.eqiad.wmnet
  • 11:00 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudnet1003.eqiad.wmnet
  • 11:00 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet1004.eqiad.wmnet
  • 10:57 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1008.eqiad.wmnet
  • 10:54 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudnet1004.eqiad.wmnet
  • 10:53 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2002-dev.codfw.wmnet
  • 10:48 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw2002-dev.codfw.wmnet
  • 10:48 aborrero@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudgw2001-dev.codfw.wmnet
  • 10:46 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1008.eqiad.wmnet
  • 10:46 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1005.eqiad.wmnet
  • 10:41 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw2001-dev.codfw.wmnet
  • 10:40 aborrero@cumin1001: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudgw2001-dev.codfw.wmnet
  • 10:40 aborrero@cumin1001: START - Cookbook sre.hosts.reboot-single for host cloudgw2001-dev.codfw.wmnet
  • 10:36 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet
  • 10:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 10:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 10:30 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: dbff998: dewiki: Set wgGEHomepageDefaultVariant to control (T294712) (duration: 00m 55s)
  • 10:03 marostegui@cumin1001: dbctl commit (dc=all): 'Set db1118 with weight 0 T293964', diff saved to https://phabricator.wikimedia.org/P17652 and previous config saved to /var/cache/conftool/dbconfig/20211102-100348-root.json
  • 09:46 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 09:42 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 09:40 legoktm: restarted apache2 on lists1001
  • 09:39 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: b259434: QuickSurveys: Show Growth IP editors survey to 0.1% of users (T294568) (duration: 00m 57s)
  • 09:03 marostegui@cumin1001: dbctl commit (dc=all): 'Remove recentchanges replicas from s6 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17651 and previous config saved to /var/cache/conftool/dbconfig/20211102-090306-marostegui.json
  • 08:29 moritzm: installing sdl2 security updates
  • 07:23 marostegui@cumin1001: dbctl commit (dc=all): 'Remove recentchangeslinked replicas from s6 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P17650 and previous config saved to /var/cache/conftool/dbconfig/20211102-072320-marostegui.json
  • 07:13 elukey: `apt-get purge dkms` (rc state) on stat100[5,8]
  • 06:45 marostegui: Rename oauth2_access_tokens oauth_accepted_consumer oauth_registered_consumer tables on db1123 T294595
  • 02:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 02:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 02:11 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 02:07 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 01:56 cstone: civicrm revision changed from 403be9ce05 to 93caef68ef
  • 01:21 ejegg: updated SmashPig standalone deploy from dd3a81c7c2 to be68299b92
  • 01:18 ejegg: updated payments-wiki from 5b9fdd0fe1 to 73de4731bd
  • 00:45 mutante: upgraded php-fpm on cloudweb2001-dev - https://labtestwikitech.wikimedia.org/wiki/Main_Page
  • 00:24 mutante: parsoid-canary (scandium, wtp1025, wtp1026, parse2001, parse2002) - upgrading php-fpm and php-* packages
  • 00:17 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 00:13 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 00:07 mutante: scandium - installing package upgrades, incl. apache, php7.2- packages
  • 00:03 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 00:02 legoktm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Add event stream config for discussiontools (T286076) (duration: 00m 55s)
  • 00:00 legoktm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Enable ArticlePlaceholder for kswiki (T294632) (duration: 00m 55s)
  • 00:00 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .

2021-11-01

  • 21:34 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 21:30 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 21:30 urbanecm: Deploy a security patch for T290808
  • 21:28 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 8f5008d: votewiki: Grant election admins securepoll-view-voter-pii (T290808) (duration: 00m 55s)
  • 20:59 mutante: mwmaint1002:/# systemctl start mediawiki_job_growthexperiments-purgeExpiredMentorStatus (T280307)
  • 20:56 legoktm: upgrading PHP 7.2 on A:mw-canary servers
  • 20:44 legoktm: upgrading PHP 7.2 on mwdebug* servers
  • 20:34 mutante: mwmaint* - new timer/service mediawiki_job_growthexperiments-purgeExpiredMentorStatus created by puppet - T280307
  • 20:33 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • 20:32 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • 20:30 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' .
  • 20:24 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .
  • 20:22 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .
  • 20:18 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox-media' for release 'main' .
  • 20:14 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .
  • 20:12 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .
  • 20:10 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' .
  • 20:08 mutante: planet1002 - systemctl start update-en-planet after merging config change btw. legoktm: it should be included in a sec
  • 19:35 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 19:31 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 19:29 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: cba805c: Prepare a QuickSurvey for Growth IP research (T294568) (duration: 00m 55s)
  • 19:26 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • 19:23 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • 19:19 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox' for release 'main' .
  • 18:49 legoktm@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .
  • 18:37 legoktm@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .
  • 18:26 legoktm@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'shellbox-constraints' for release 'main' .
  • 18:25 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 18:19 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 18:09 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: fb433d6: Amend wordmark for the Meetei (Manipuri) Wikipedia (T294189; 2/2) (duration: 00m 55s)
  • 18:09 urbanecm: Purge https://en.wikipedia.org/static/images/mobile/copyright/wikipedia-wordmark-mni.svg (T294189)
  • 18:09 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 18:08 urbanecm@deploy1002: Synchronized static/images/mobile/copyright/wikipedia-wordmark-mni.svg: fb433d6: Amend wordmark for the Meetei (Manipuri) Wikipedia (T294189; 1/2) (duration: 00m 55s)
  • 18:06 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:52 topranks: force-resetting FPC 0 on cr2-codfw as it appears hard down.
  • 17:46 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:46 mutante: removing mediawiki font packages from the 8 canary API servers, in addition to 11 canary appservers T294378
  • 17:43 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 17:06 mutante: removing font packages from canary appservers (T294378, gerrit:735685)
  • 16:53 otto@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
  • 16:53 otto@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
  • 15:52 otto@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'canary' .
  • 15:52 otto@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
  • 15:50 otto@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'eventgate-main' for release 'production' .
  • 15:49 moritzm: installing opencv security updates on stretch
  • 15:28 moritzm: rolling restart of mw canaries to pick up tiff security updates
  • 15:12 moritzm: installing tiff security updates
  • 14:54 moritzm: uploaded PHP 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf3 to apt.wikimedia.org (buster-wikimedia/component/php72) T294317
  • 14:37 moritzm: updating PHP on mwdebug1001
  • 13:31 moritzm: installing jbig2dec security updates
  • 12:25 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1101.eqiad.wmnet
  • 12:18 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1101.eqiad.wmnet
  • 12:08 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1100.eqiad.wmnet
  • 12:08 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.6/extensions/GrowthExperiments/includes/Mentorship/QuitMentorship.php: 4671528: QuitMentorship: Pass a logger (T294665; 2/2) (duration: 00m 55s)
  • 12:07 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.6/extensions/GrowthExperiments/includes/Mentorship/QuitMentorshipFactory.php: 4671528: QuitMentorship: Pass a logger (T294665; 1/2) (duration: 00m 56s)
  • 11:59 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1100.eqiad.wmnet
  • 11:58 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1099.eqiad.wmnet
  • 11:50 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 11:49 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1099.eqiad.wmnet
  • 11:48 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1098.eqiad.wmnet
  • 11:47 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 11:41 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1098.eqiad.wmnet
  • 11:31 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1097.eqiad.wmnet
  • 11:22 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host an-worker1097.eqiad.wmnet
  • 11:20 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1096.eqiad.wmnet
  • 11:17 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 11:14 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 11:01 urbanecm: 11:01:21 Synchronized wmf-config/CommonSettings.php: b9aa3d2: Add edit-legal to editprotected grant (duration: 00m 54s)
  • 11:00 urbanecm: 10:59:03 Synchronized wmf-config/InitialiseSettings.php: c236232: foundationwiki: Disable direct account creation (T205347) (duration: 00m 56s)
  • 10:46 moritzm: installing libdatetime-timezone-perl updates (updates for latest tz changes)
  • 10:17 urbanecm: Deploy a security patch for T294686
  • 09:03 dcausse: restarting blazegraph on wdqs2003 (jvm stuck for the last 22hours)
  • 02:46 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 02:41 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 02:31 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 02:28 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
  • 02:24 reedy@deploy1002: Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 01m 49s)
  • 02:22 reedy@deploy1002: Synchronized langlist: Add ami to langlist T294717 T292414 (duration: 00m 55s)

Archives

See Server Admin Log/Archives.