You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(gehel: restarting tilerator on all maps servers)
imported>Stashbot
(jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/RepoHooks.php: T213227 Don't have onApiCheckCanExecute die for inactive entity types (duration: 00m 53s))
Line 1: Line 1:
== 2019-01-09 ==
* 01:15 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/RepoHooks.php: [[phab:T213227|T213227]] Don't have onApiCheckCanExecute die for inactive entity types (duration: 00m 53s)
* 01:04 jforrester@deploy1001: Synchronized docroot/: [[phab:T187716|T187716]] Remove mobilelanding.php, no longer pointed to by Apache (duration: 00m 52s)
* 00:58 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [Wikimania] Add 2019 content to default search (duration: 00m 53s)
* 00:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T202683|T202683]] [Wikimania] Create year namespaces for each Wikimania, 2005–2019 (duration: 00m 53s)
* 00:34 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:481115{{!}}Make password policy and logging code saner]] (duration: 00m 52s)
* 00:33 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481115{{!}}Make password policy and logging code saner]] (duration: 00m 55s)
== 2019-01-08 ==
== 2019-01-08 ==
* 23:44 SMalyshev: repooled wdqs1004
* 23:35 eileen: process-control config revision is {{Gerrit|9dc6e63fcd}}
* 23:00 XioNoX: Update pfw3-codfw/eqiad security policies - [[phab:T213100|T213100]]
* 22:39 XioNoX: deactivate policy-statement BGP_fundraising_aggregates term nat on pfw3-eqiad/codfw - [[phab:T211028|T211028]]
* 22:29 gehel: starting data copy from wdqs1007 to wdqs1008 (both will be depooled) - [[phab:T213217|T213217]]
* 22:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Add default search NSes (duration: 00m 51s)
* 22:22 James_F: Ran /docroot/noc/createTxtFileSymlinks.sh for new dblist
* 22:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new wikidatarepo dblist where appropriate (duration: 00m 52s)
* 22:20 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: dblists: Load wikibaserepo (duration: 00m 52s)
* 22:15 jforrester@deploy1001: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
* 22:14 jforrester@deploy1001: Synchronized dblists/wikidata.dblist: dblists: Remove testcommons from wikidata list (duration: 00m 52s)
* 22:13 jforrester@deploy1001: Synchronized dblists/wikidatarepo.dblist: dblists: Add wikidatarepo list (duration: 00m 53s)
* 22:12 urandom: forcing removal of restbase1016-b (host down way too long to salvage) -- [[phab:T212418|T212418]]
* 22:08 marostegui: Drop valid_tag table from db2043 with replication (s3 codfw master - lag will be generated) - [[phab:T212254|T212254]]
* 22:03 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: cleanup - {{Gerrit|Idfa129a65a41}} (duration: 00m 53s)
* 21:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 [[phab:T212254|T212254]] (duration: 00m 52s)
* 21:49 marostegui: Drop valid_tag table from db1078 (s3) - [[phab:T212254|T212254]]
* 21:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 [[phab:T212254|T212254]] (duration: 00m 53s)
* 21:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 [[phab:T212254|T212254]] (duration: 00m 53s)
* 21:38 marostegui: Drop valid_tag table from db1123 (s3) - [[phab:T212254|T212254]]
* 21:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 [[phab:T212254|T212254]] (duration: 00m 53s)
* 21:31 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.12
* 21:03 dduvall@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache (duration: 39m 22s)
* 20:42 ejegg: updated payments-wiki from {{Gerrit|b8acb95a2a}} to {{Gerrit|c455bbc6bb}}
* 20:24 dduvall@deploy1001: Started scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache
* 20:24 gehel: starting data copy from wdqs1004 to wdqs1007 (both will be depooled) - [[phab:T213217|T213217]]
* 20:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Don't enable entities, we're not Wikidata.org (duration: 01m 44s)
* 20:11 XioNoX: change BGP_fundraising_aggregates term nat from static to aggregate on pfw3-eqiad - [[phab:T211028|T211028]]
* 19:51 ejegg: updated fundraising CiviCRM from {{Gerrit|b8e3a71845}} to {{Gerrit|5580f0b11c}}
* 19:48 krinkle@deploy1001: Finished deploy [performance/navtiming@68fd54d]: (no justification provided) (duration: 00m 05s)
* 19:48 krinkle@deploy1001: Started deploy [performance/navtiming@68fd54d]: (no justification provided)
* 19:48 dduvall@deploy1001: Pruned MediaWiki: 1.33.0-wmf.12 (duration: 06m 26s)
* 19:11 arlolra: Updated Parsoid to {{Gerrit|2c5dc7b}} ([[phab:T197616|T197616]], [[phab:T205491|T205491]], [[phab:T209772|T209772]], [[phab:T199926|T199926]], [[phab:T209194|T209194]], [[phab:T204622|T204622]])
* 19:06 marostegui: Drop valid_tag table from s1 - [[phab:T212254|T212254]]
* 19:00 arlolra@deploy1001: Finished deploy [parsoid/deploy@4b82683]: Updating Parsoid to {{Gerrit|2c5dc7b}} (duration: 10m 40s)
* 18:54 XioNoX: make pfw3-codfw source NAT similar to pfw3-eqiad - [[phab:T211028|T211028]]
* 18:54 ejegg: updated SmashPig standalone install from {{Gerrit|fb3268897b}} to {{Gerrit|25713ca232}}
* 18:50 marostegui: Drop valid_tag table from s4 - [[phab:T212254|T212254]]
* 18:50 XioNoX: add NAT workaround to pfw3-eqiad - [[phab:T211028|T211028]]
* 18:49 arlolra@deploy1001: Started deploy [parsoid/deploy@4b82683]: Updating Parsoid to {{Gerrit|2c5dc7b}}
* 18:38 XioNoX: temporarily permit ssh from frpm1001 to pfw3-eqiad on pfw3-eqiad
* 18:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 18:27 jynus: restarting s5 replication on labsdb1009/10/11
* 17:41 moritzm: installing libseccomp updates from stretch point release
* 17:40 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2 (duration: 02m 29s)
* 17:38 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2
* 17:37 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - [[phab:T210752|T210752]] [[phab:T197616|T197616]] (duration: 96m 50s)
* 17:33 _joe_: applying the new apache configuration to jobrunners in eqiad
* 17:24 elukey: roll restart of aqs on aqs100* to pick up new Druid settings
* 17:20 _joe_: depooling mw1299 for testing of the apache change
* 17:16 SMalyshev: restarted Blazegraph wdqs1006 due to unresponsiveness (caused by load?)
* 16:56 urandom: forcing removal of restbase1016-a (host down way too long to salvage) -- [[phab:T212418|T212418]]
* 16:56 jynus: changing db1124:s5 replication to db2066
* 16:55 marostegui: Deploy schema change on db1105:3311 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 16:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 44s)
* 16:54 jynus: stopping s5 replication on labsdb1009/10/11 to prevent undoable mistakes
* 16:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2019 - [[phab:T212833|T212833]] (duration: 02m 51s)
* 16:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 16:12 XioNoX: add BGP sessions to AS64050 in AMS-IX
* 16:04 marostegui: Drop valid_tag table from s7 - [[phab:T212254|T212254]]
* 16:00 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - [[phab:T210752|T210752]] [[phab:T197616|T197616]]
* 15:59 marostegui: Deploy schema change on db1089 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 15:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 15:45 marostegui: Drop valid_tag table from s2 - [[phab:T212254|T212254]]
* 15:32 marostegui: Stop MySQL on es2019 for upgrade - [[phab:T212833|T212833]]
* 15:23 godog: briefly stop carbon daemons on graphite1004 to move /srv/whisper -> /srv/carbon/whisper
* 15:17 marostegui: Increase connections from 10 to 50 for recommendationapiservice on m2 - [[phab:T212154|T212154]]
* 15:10 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2019 - [[phab:T212833|T212833]] (duration: 00m 44s)
* 15:04 hashar: Restarted CI Jenkins
* 13:02 zeljkof: EU SWAT finished
* 12:59 jynus: transfering db1102:s5 mariadb datadir to db1082
* 12:57 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481107{{!}}Give all users (including IPs) the pagequality right in plwikisource (T212478)]] (duration: 00m 45s)
* 12:45 akosiaris@deploy1001: scap-helm zotero finished
* 12:45 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
* 12:45 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
* 12:44 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481662{{!}}Allow ptwikis bureaucrats to grant/revoke rollbacker user group (T212735)]] (duration: 00m 45s)
* 12:39 akosiaris@deploy1001: scap-helm zotero upgrade production2 -f zoterov2-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
* 12:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:481663{{!}}Use localized wgMetaNamespace and wgMetaNamespaceTalk in satwiki (T211294)]] (duration: 00m 45s)
* 12:23 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: [[gerrit:481240{{!}}New throttle rule for students writing Wikipedia program (T212226)]] (duration: 00m 44s)
* 12:14 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: [[gerrit:482586{{!}}New throttle rule for University of Southern California editathon (T212917)]] (duration: 00m 45s)
* 12:07 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: [[phab:T212768|T212768]] [cirrus] re-enable HHVM connection pooling (duration: 00m 45s)
* 12:01 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c] (dev-cluster): Add test-commons and nap.wikisource (duration: 12m 38s)
* 11:49 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c] (dev-cluster): Add test-commons and nap.wikisource
* 11:46 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Increase time out on the MW side to 60s - [[phab:T204183|T204183]] (duration: 00m 51s)
* 11:36 akosiaris@deploy1001: scap-helm zotero finished
* 11:36 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 11:36 akosiaris@deploy1001: scap-helm zotero upgrade production -f zoterov2-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 11:35 akosiaris@deploy1001: scap-helm zotero finished
* 11:35 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 11:35 akosiaris@deploy1001: scap-helm zotero upgrade production -f zoterov2-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 11:33 mobrovac@deploy1001: Started restart [electron-render/deploy@94d27d7]: Electron strugling, restart - [[phab:T213154|T213154]]
* 11:29 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=zotero,name=codfw
* 11:24 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=zotero,name=codfw
* 11:07 jynus: stoping and restarting db1102 (s5, s4) for upgrade
* 11:04 moritzm: rebooting mw1261
* 10:48 moritzm: installing libseccomp updates from stretch point release
* 10:34 dcausse: elastic@eqiad setting crosscluster conf on production search cluster ([[phab:T213150|T213150]])
* 10:25 banyek: executing schema change on db1062 - [[phab:T85757|T85757]]
* 09:39 foks: reset user email for Zergiorubio
* 09:26 akosiaris@deploy1001: scap-helm zotero finished
* 09:26 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
* 09:26 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
* 09:22 jynus: stop replication on db1124:s5 [[phab:T213108|T213108]]
* 09:21 akosiaris@deploy1001: scap-helm zotero finished
* 09:21 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 09:21 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 09:19 hashar: gerrit: resaved configuration for All-Projects by changing "Max Reviewers" from 3 to 4. Might enable adding reviewers automatically based on git blame. See task for config diff # [[phab:T101131|T101131]]
* 09:12 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@f91cf04]: Increase the concurrency of categoryMembershipJob - [[phab:T192691|T192691]] (duration: 00m 59s)
* 09:12 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@f91cf04]: Increase the concurrency of categoryMembershipJob - [[phab:T192691|T192691]]
* 05:39 SMalyshev: restarted some Blazegraph servers as precaution against corruption issues
* 04:26 onimisionipe: depooling wdqs1008 - [[phab:T213134|T213134]]
* 03:23 kartik@deploy1001: Finished deploy [cxserver/deploy@b669f95]: Update cxserver to {{Gerrit|d6b1d6f}} (duration: 05m 00s)
* 03:18 kartik@deploy1001: Started deploy [cxserver/deploy@b669f95]: Update cxserver to {{Gerrit|d6b1d6f}}
* 00:22 gehel: restarting tilerator on all maps servers
* 00:22 gehel: restarting tilerator on all maps servers
* 00:06 gehel: depooling wdqs1007 (something looks like DB corruption)
* 00:06 gehel: depooling wdqs1007 (something looks like DB corruption)

Revision as of 01:15, 9 January 2019

2019-01-09

  • 01:15 jforrester@deploy1001: Synchronized php-1.33.0-wmf.12/extensions/Wikibase/repo/RepoHooks.php: T213227 Don't have onApiCheckCanExecute die for inactive entity types (duration: 00m 53s)
  • 01:04 jforrester@deploy1001: Synchronized docroot/: T187716 Remove mobilelanding.php, no longer pointed to by Apache (duration: 00m 52s)
  • 00:58 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [Wikimania] Add 2019 content to default search (duration: 00m 53s)
  • 00:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202683 [Wikimania] Create year namespaces for each Wikimania, 2005–2019 (duration: 00m 53s)
  • 00:34 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Make password policy and logging code saner (duration: 00m 52s)
  • 00:33 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make password policy and logging code saner (duration: 00m 55s)

2019-01-08

  • 23:44 SMalyshev: repooled wdqs1004
  • 23:35 eileen: process-control config revision is 9dc6e63fcd
  • 23:00 XioNoX: Update pfw3-codfw/eqiad security policies - T213100
  • 22:39 XioNoX: deactivate policy-statement BGP_fundraising_aggregates term nat on pfw3-eqiad/codfw - T211028
  • 22:29 gehel: starting data copy from wdqs1007 to wdqs1008 (both will be depooled) - T213217
  • 22:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Add default search NSes (duration: 00m 51s)
  • 22:22 James_F: Ran /docroot/noc/createTxtFileSymlinks.sh for new dblist
  • 22:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new wikidatarepo dblist where appropriate (duration: 00m 52s)
  • 22:20 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: dblists: Load wikibaserepo (duration: 00m 52s)
  • 22:15 jforrester@deploy1001: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details)
  • 22:14 jforrester@deploy1001: Synchronized dblists/wikidata.dblist: dblists: Remove testcommons from wikidata list (duration: 00m 52s)
  • 22:13 jforrester@deploy1001: Synchronized dblists/wikidatarepo.dblist: dblists: Add wikidatarepo list (duration: 00m 53s)
  • 22:12 urandom: forcing removal of restbase1016-b (host down way too long to salvage) -- T212418
  • 22:08 marostegui: Drop valid_tag table from db2043 with replication (s3 codfw master - lag will be generated) - T212254
  • 22:03 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: cleanup - Idfa129a65a41 (duration: 00m 53s)
  • 21:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 T212254 (duration: 00m 52s)
  • 21:49 marostegui: Drop valid_tag table from db1078 (s3) - T212254
  • 21:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 T212254 (duration: 00m 53s)
  • 21:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 T212254 (duration: 00m 53s)
  • 21:38 marostegui: Drop valid_tag table from db1123 (s3) - T212254
  • 21:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 T212254 (duration: 00m 53s)
  • 21:31 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.12
  • 21:03 dduvall@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache (duration: 39m 22s)
  • 20:42 ejegg: updated payments-wiki from b8acb95a2a to c455bbc6bb
  • 20:24 dduvall@deploy1001: Started scap: testwiki to php-1.33.0-wmf.12 and rebuild l10n cache
  • 20:24 gehel: starting data copy from wdqs1004 to wdqs1007 (both will be depooled) - T213217
  • 20:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Don't enable entities, we're not Wikidata.org (duration: 01m 44s)
  • 20:11 XioNoX: change BGP_fundraising_aggregates term nat from static to aggregate on pfw3-eqiad - T211028
  • 19:51 ejegg: updated fundraising CiviCRM from b8e3a71845 to 5580f0b11c
  • 19:48 krinkle@deploy1001: Finished deploy [performance/navtiming@68fd54d]: (no justification provided) (duration: 00m 05s)
  • 19:48 krinkle@deploy1001: Started deploy [performance/navtiming@68fd54d]: (no justification provided)
  • 19:48 dduvall@deploy1001: Pruned MediaWiki: 1.33.0-wmf.12 (duration: 06m 26s)
  • 19:11 arlolra: Updated Parsoid to 2c5dc7b (T197616, T205491, T209772, T199926, T209194, T204622)
  • 19:06 marostegui: Drop valid_tag table from s1 - T212254
  • 19:00 arlolra@deploy1001: Finished deploy [parsoid/deploy@4b82683]: Updating Parsoid to 2c5dc7b (duration: 10m 40s)
  • 18:54 XioNoX: make pfw3-codfw source NAT similar to pfw3-eqiad - T211028
  • 18:54 ejegg: updated SmashPig standalone install from fb3268897b to 25713ca232
  • 18:50 marostegui: Drop valid_tag table from s4 - T212254
  • 18:50 XioNoX: add NAT workaround to pfw3-eqiad - T211028
  • 18:49 arlolra@deploy1001: Started deploy [parsoid/deploy@4b82683]: Updating Parsoid to 2c5dc7b
  • 18:38 XioNoX: temporarily permit ssh from frpm1001 to pfw3-eqiad on pfw3-eqiad
  • 18:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 T86338 T202167 (duration: 00m 45s)
  • 18:27 jynus: restarting s5 replication on labsdb1009/10/11
  • 17:41 moritzm: installing libseccomp updates from stretch point release
  • 17:40 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2 (duration: 02m 29s)
  • 17:38 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource, take #2
  • 17:37 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - T210752 T197616 (duration: 96m 50s)
  • 17:33 _joe_: applying the new apache configuration to jobrunners in eqiad
  • 17:24 elukey: roll restart of aqs on aqs100* to pick up new Druid settings
  • 17:20 _joe_: depooling mw1299 for testing of the apache change
  • 17:16 SMalyshev: restarted Blazegraph wdqs1006 due to unresponsiveness (caused by load?)
  • 16:56 urandom: forcing removal of restbase1016-a (host down way too long to salvage) -- T212418
  • 16:56 jynus: changing db1124:s5 replication to db2066
  • 16:55 marostegui: Deploy schema change on db1105:3311 T86338 T202167
  • 16:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 T86338 T202167 (duration: 00m 44s)
  • 16:54 jynus: stopping s5 replication on labsdb1009/10/11 to prevent undoable mistakes
  • 16:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2019 - T212833 (duration: 02m 51s)
  • 16:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 T86338 T202167 (duration: 00m 45s)
  • 16:12 XioNoX: add BGP sessions to AS64050 in AMS-IX
  • 16:04 marostegui: Drop valid_tag table from s7 - T212254
  • 16:00 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c]: Add test-commons and nap.wikisource - T210752 T197616
  • 15:59 marostegui: Deploy schema change on db1089 T86338 T202167
  • 15:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 T86338 T202167 (duration: 00m 45s)
  • 15:45 marostegui: Drop valid_tag table from s2 - T212254
  • 15:32 marostegui: Stop MySQL on es2019 for upgrade - T212833
  • 15:23 godog: briefly stop carbon daemons on graphite1004 to move /srv/whisper -> /srv/carbon/whisper
  • 15:17 marostegui: Increase connections from 10 to 50 for recommendationapiservice on m2 - T212154
  • 15:10 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2019 - T212833 (duration: 00m 44s)
  • 15:04 hashar: Restarted CI Jenkins
  • 13:02 zeljkof: EU SWAT finished
  • 12:59 jynus: transfering db1102:s5 mariadb datadir to db1082
  • 12:57 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Give all users (including IPs) the pagequality right in plwikisource (T212478) (duration: 00m 45s)
  • 12:45 akosiaris@deploy1001: scap-helm zotero finished
  • 12:45 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 12:45 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 12:44 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow ptwikis bureaucrats to grant/revoke rollbacker user group (T212735) (duration: 00m 45s)
  • 12:39 akosiaris@deploy1001: scap-helm zotero upgrade production2 -f zoterov2-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 12:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use localized wgMetaNamespace and wgMetaNamespaceTalk in satwiki (T211294) (duration: 00m 45s)
  • 12:23 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for students writing Wikipedia program (T212226) (duration: 00m 44s)
  • 12:14 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for University of Southern California editathon (T212917) (duration: 00m 45s)
  • 12:07 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T212768 [cirrus] re-enable HHVM connection pooling (duration: 00m 45s)
  • 12:01 mobrovac@deploy1001: Finished deploy [restbase/deploy@503b29c] (dev-cluster): Add test-commons and nap.wikisource (duration: 12m 38s)
  • 11:49 mobrovac@deploy1001: Started deploy [restbase/deploy@503b29c] (dev-cluster): Add test-commons and nap.wikisource
  • 11:46 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Increase time out on the MW side to 60s - T204183 (duration: 00m 51s)
  • 11:36 akosiaris@deploy1001: scap-helm zotero finished
  • 11:36 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 11:36 akosiaris@deploy1001: scap-helm zotero upgrade production -f zoterov2-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 11:35 akosiaris@deploy1001: scap-helm zotero finished
  • 11:35 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 11:35 akosiaris@deploy1001: scap-helm zotero upgrade production -f zoterov2-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 11:33 mobrovac@deploy1001: Started restart [electron-render/deploy@94d27d7]: Electron strugling, restart - T213154
  • 11:29 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=zotero,name=codfw
  • 11:24 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=zotero,name=codfw
  • 11:07 jynus: stoping and restarting db1102 (s5, s4) for upgrade
  • 11:04 moritzm: rebooting mw1261
  • 10:48 moritzm: installing libseccomp updates from stretch point release
  • 10:34 dcausse: elastic@eqiad setting crosscluster conf on production search cluster (T213150)
  • 10:25 banyek: executing schema change on db1062 - T85757
  • 09:39 foks: reset user email for Zergiorubio
  • 09:26 akosiaris@deploy1001: scap-helm zotero finished
  • 09:26 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 09:26 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 09:22 jynus: stop replication on db1124:s5 T213108
  • 09:21 akosiaris@deploy1001: scap-helm zotero finished
  • 09:21 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 09:21 akosiaris@deploy1001: scap-helm zotero install --name production2 -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 09:19 hashar: gerrit: resaved configuration for All-Projects by changing "Max Reviewers" from 3 to 4. Might enable adding reviewers automatically based on git blame. See task for config diff # T101131
  • 09:12 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@f91cf04]: Increase the concurrency of categoryMembershipJob - T192691 (duration: 00m 59s)
  • 09:12 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@f91cf04]: Increase the concurrency of categoryMembershipJob - T192691
  • 05:39 SMalyshev: restarted some Blazegraph servers as precaution against corruption issues
  • 04:26 onimisionipe: depooling wdqs1008 - T213134
  • 03:23 kartik@deploy1001: Finished deploy [cxserver/deploy@b669f95]: Update cxserver to d6b1d6f (duration: 05m 00s)
  • 03:18 kartik@deploy1001: Started deploy [cxserver/deploy@b669f95]: Update cxserver to d6b1d6f
  • 00:22 gehel: restarting tilerator on all maps servers
  • 00:06 gehel: depooling wdqs1007 (something looks like DB corruption)

2019-01-07

  • 23:56 eileen: update civicrm revision changed from bcb4b7a7d1 to b8e3a71845, config revision is 260be32d0a
  • 22:08 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Re-enable uploading of files, accidentally prevented (duration: 00m 44s)
  • 21:19 XioNoX: push NAT changes to pfw3-eqiad - T211028
  • 21:16 awight@deploy1001: Finished deploy [ores/deploy@9253beb]: T212530: new ORES models; revscoring 2.3.0 (duration: 15m 28s)
  • 21:13 mforns@deploy1001: Finished deploy [analytics/refinery@faac592]: deploying analytics/refinery to account with refinery-source v0.0.83 (duration: 06m 52s)
  • 21:06 mforns@deploy1001: Started deploy [analytics/refinery@faac592]: deploying analytics/refinery to account with refinery-source v0.0.83
  • 21:00 awight@deploy1001: Started deploy [ores/deploy@9253beb]: T212530: new ORES models; revscoring 2.3.0
  • 20:19 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: TestCommons: Final go-switch for WBMI Ie52b8af006ba (duration: 00m 45s)
  • 19:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove redundant namespace talk definitions (T206952) (duration: 00m 44s)
  • 19:46 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set $wgMetaNamespace for bewikibooks (T212665) (duration: 00m 45s)
  • 19:43 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable WikibaseRepo and WikibaseMediaInfo on testcommonswiki (duration: 00m 44s)
  • 19:42 XioNoX: push firewall change to pfw3-codfw/eqiad - T211712
  • 19:40 catrope@deploy1001: Synchronized wmf-config/Wikibase.php: Set empty clientDbList for testcommonswiki (duration: 00m 44s)
  • 19:38 catrope@deploy1001: Synchronized dblists/wikidata.dblist: Enable Wikidata on testcommonswiki (duration: 00m 44s)
  • 19:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add importupload to sysops on testcommons (duration: 00m 45s)
  • 19:14 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on viwikisource (T212929) (duration: 00m 45s)
  • 19:13 catrope@deploy1001: Synchronized dblists/flow.dblist: Enable Flow on viwikisource (T212929) (duration: 00m 45s)
  • 19:11 RoanKattouw: Ran emptyUserGroup.php for autoreview, reviewer and editor groups on srwikinews (T212058)
  • 18:51 XioNoX: re-deactivate bgp sessions to Zayo on cr1-eqiad - T212791
  • 18:20 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@d8f911c]: new GUI, Updater & Blazegraph build (duration: 10m 13s)
  • 18:18 XioNoX: activate bgp sessions to Zayo on cr1-eqiad - T212791
  • 18:10 jynus: manually creating tables on es1015, es1017 with replication for testcommonswiki
  • 18:10 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@d8f911c]: new GUI, Updater & Blazegraph build
  • 18:07 onimisionipe@deploy1001: deploy aborted: (no justification provided) (duration: 00m 04s)
  • 18:06 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@d8f911c]: (no justification provided)
  • 18:05 XioNoX: deactivate bgp sessions to Zayo on cr1-eqiad T212791
  • 17:35 akosiaris: restart pdfrender on scb1004
  • 17:35 akosiaris: restart pdfrender
  • 17:23 kartik@deploy1001: Finished deploy [cxserver/deploy@594420b]: Update cxserver to 7632c43 (duration: 04m 06s)
  • 17:19 kartik@deploy1001: Started deploy [cxserver/deploy@594420b]: Update cxserver to 7632c43
  • 16:24 jynus: shutting down mariadb again and rebooting db1107
  • 16:15 jynus: starting mariadb on db1107
  • 16:12 onimisionipe: starting inplace reindexing for enwiki - T212224
  • 16:07 volans: powercycle db1107
  • 16:03 elukey: stop eventlogging mysql consumers on eventlog1002 and eventlogging replication on db1108 due to issues with db1107
  • 16:02 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 (duration: 00m 45s)
  • 15:46 cmjohnson1: replacing bad fuse on the PDU rack A2 eqiad
  • 14:19 moritzm: added jbond to WMF-LDAP group in Phabricator (T213079)
  • 13:56 ariel@deploy1001: Finished deploy [dumps/dumps@acd9bca]: logging and quiet mode for adds-changes and other dumps (duration: 00m 05s)
  • 13:56 ariel@deploy1001: Started deploy [dumps/dumps@acd9bca]: logging and quiet mode for adds-changes and other dumps
  • 13:02 zeljkof: EU SWAT finished
  • 13:01 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: cirrus: increase number of shards (T212224) (duration: 00m 44s)
  • 12:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Restrict moving categories for users at srwiki (T213050) (duration: 00m 44s)
  • 12:40 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Cleanup old throttle rules (duration: 00m 44s)
  • 12:34 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: To lift a cap on account creation from IP for mrwiki community (T212921) (duration: 00m 43s)
  • 12:30 Zoranzoki21: tools.zoranzoki21wiki Archived https://www.mediawiki.org/w/index.php?title=Extension:Woopra (https://www.wikidata.org/wiki/Q21679347) - T212994
  • 12:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reader trust survey (T209882) (duration: 00m 45s)
  • 12:21 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Quiz extension on ru.wikibooks (T212622) (duration: 00m 45s)
  • 12:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add suppressredirect user right to editor user group at pl.wikisource (T212655) (duration: 00m 44s)
  • 12:11 gtirloni: disabled notifications for cloudvirt0124 (T212360)
  • 12:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable extendedmover user group at en.wiktionary (T212662) (duration: 00m 46s)
  • 12:07 kartik@deploy1001: Finished deploy [cxserver/deploy@2d54a64]: Deploy Google Translation (T90208) (duration: 05m 07s)
  • 12:02 kartik@deploy1001: Started deploy [cxserver/deploy@2d54a64]: Deploy Google Translation (T90208)
  • 10:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1079 after schema change - T85757 (duration: 00m 44s)
  • 10:31 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Move group1 to new logging infrastructure - T211124 (duration: 00m 45s)
  • 10:30 banyek: repooling db1079 after schema change - T85757
  • 10:27 banyek: restarting replication on db1079 - T85757
  • 09:55 banyek: executing schema change on db1079 with replication enabled - T85757
  • 09:53 banyek: stopping replication on db1079 - T85757
  • 09:47 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1079 for schema change - T85757 (duration: 01m 02s)
  • 09:36 banyek: depooling db1079 for schema change - T85757
  • 08:30 moritzm: rolling restart of swift backend servers to pick up OpenSSL security update
  • 07:24 elukey: restart pdfrender on scb1002

2019-01-06

  • 14:50 ariel@deploy1001: Finished deploy [dumps/dumps@cb30b6c]: check xml files for closing mediawiki tag (duration: 00m 06s)
  • 14:50 ariel@deploy1001: Started deploy [dumps/dumps@cb30b6c]: check xml files for closing mediawiki tag

2019-01-05

  • 20:23 elukey: manually clean up of big logs under /var/log/.. on analytics-tool1002 due to root partition almost filled up

2019-01-04

  • 23:07 mutante: scandium apt-get remove nodejs nodes-legacy ; puppet agent -tv - after merging gerrit:482150 this fixed "you have held broken packages" issue, now we are at a puppet dependecy cycle with apt::pin T201366
  • 15:42 bawolff@deploy1001: Synchronized private/PrivateSettings.php: T212667 - More aggressive anti-spam measures for account creation on kowiki (duration: 00m 48s)
  • 14:08 moritzm: rebooting etcd1001-1003 to pick up SSBD-enabled qemu
  • 13:52 moritzm: rebooting etcd1004-1006 to pick up SSBD-enabled qemu
  • 13:33 moritzm: rebooting kubernetes staging etcd hosts to pick up SSBD-enabled qemu
  • 13:11 moritzm: rebooting kubernetes staging master to pick up SSBD-enabled qemu
  • 12:57 moritzm: rebooting kubernetes staging workers for kernel security update
  • 11:58 moritzm: installing libsndfile security updates
  • 11:33 moritzm: installing jasper security updates
  • 11:31 moritzm: installing libdatetime-timezone-perl updates for recent tz changes
  • 10:47 arturo: T212898 reimaging cloudvirt1024 as stretch
  • 10:46 moritzm: rolling restart of swift proxies to pick up OpenSSL update
  • 09:57 jijiki: restarting thumbor services to pick up 481141
  • 09:50 onimisionipe: restarting nginx on all wdqs hosts
  • 09:40 banyek: executing schema change on dbstore1002 - T85757
  • 09:13 moritzm: restarting nginx on puppetdb hosts to pick up new OpenSSL
  • 09:03 banyek: executing schema change on db1116 - T85757
  • 08:44 moritzm: restarting nginx on francium to pick up new OpenSSL
  • 08:16 elukey: restart eventlogging daemons on eventlog1002 to pick up openssl updates
  • 07:56 moritzm: installing OpenSSL security updates
  • 00:07 mutante: an-coord1001 - apt-get clean to free disk space, reacting to Icinga alert for running out of disk

2019-01-03

  • 23:08 volans: restarted pdfrender on scb1004
  • 22:29 volans: restarted all slaves on dbstore1002 (relayed from banyek)
  • 22:14 banyek: stopping all slaves on dbstore1002 (NOT labsdb)
  • 22:14 banyek: stopping all slaves on labsdb1002
  • 20:50 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: Fix error for testcommons (duration: 00m 44s)
  • 20:46 reedy@deploy1001: Synchronized dblists/group0.dblist: Add testcommonswiki to group0 (duration: 00m 43s)
  • 20:43 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 05s)
  • 20:24 reedy@deploy1001: Synchronized wmf-config/db-codfw.php: T197616 (duration: 00m 44s)
  • 20:23 reedy@deploy1001: Synchronized wmf-config/db-eqiad.php: T197616 (duration: 00m 44s)
  • 20:13 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197616 (duration: 00m 44s)
  • 20:12 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: T197616 (duration: 00m 44s)
  • 20:11 reedy@deploy1001: rebuilt and synchronized wikiversions files: T197616
  • 20:09 reedy@deploy1001: Synchronized dblists/: T197616 (duration: 00m 45s)
  • 18:51 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@1182b3b]: Update mobileapps to f6ad0e5: Set timeout for backend /page/html requests, part 2 (duration: 05m 27s)
  • 18:46 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@1182b3b]: Update mobileapps to f6ad0e5: Set timeout for backend /page/html requests, part 2
  • 18:37 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@c470ed2]: Update mobileapps to f6ad0e5: Set timeout for backend /page/html requests (duration: 04m 11s)
  • 18:33 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@c470ed2]: Update mobileapps to f6ad0e5: Set timeout for backend /page/html requests
  • 18:21 volans: restart pdfrender on scb1003
  • 17:58 ariel@deploy1001: Finished deploy [dumps/dumps@10dc8ad]: return properly if commands failed (duration: 00m 08s)
  • 17:58 ariel@deploy1001: Started deploy [dumps/dumps@10dc8ad]: return properly if commands failed
  • 16:32 XioNoX: remove old 10.64.22.0/24 IPs from cloud-instance-transport1-b-eqiad - T207663
  • 16:22 moritzm: rebooting kubernetes workers in eqiad for kernel security update
  • 16:02 arturo: reimaging cloudvirt1013 cloudvirt1026-1028 to stretch
  • 15:48 moritzm: restart parsoid on wtp1025 to pick up OpenSSL update for nodejs
  • 15:43 jijiki: Enabled puppet on mw servers after merging 481796 - T197616
  • 15:31 jijiki: Disabling puppet on mw servers to test 481796 - T197616
  • 15:14 ejegg: updated Fundraising CiviCRM from b33dcd3c94 to bcb4b7a7d1
  • 14:37 moritzm: rebooting kubernetes workers in codfw for kernel security update
  • 14:37 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1101:3317 after schema change - T85757 (duration: 00m 44s)
  • 14:32 banyek: repooling db1101:3317 after schema change - T85757
  • 14:21 moritzm: rebooting kubernetes masters in eqiad to pick up SSBD-enabled qemu
  • 14:14 moritzm: rebooting kubernetes mastes in codfw to pick up SSBD-enabled qemu
  • 14:05 arturo: T209616 reimage cloudvirt1029 as debian stretch
  • 13:43 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1101:3317 for schema change - T85757 (duration: 00m 44s)
  • 13:41 banyek: depooling db1101:3317 for schema change - T85757
  • 13:38 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1098:3317 after schema change - T85757 (duration: 00m 44s)
  • 13:34 banyek: repooling db1098:3317 after schema change - T85757
  • 13:24 kartik@deploy1001: Finished deploy [cxserver/deploy@3b2ede7]: Update cxserver to 2369a18 (duration: 04m 30s)
  • 13:20 kartik@deploy1001: Started deploy [cxserver/deploy@3b2ede7]: Update cxserver to 2369a18
  • 12:58 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1098:3317 for schema change - T85757 (duration: 00m 45s)
  • 12:55 banyek: depooling db1098:3317 for schema change - T85757
  • 12:54 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1094 after schema change - T85757 (duration: 00m 45s)
  • 12:49 banyek: repooling db1094 after schema change - T85757
  • 12:41 arturo: T212302 reimaging again cloudvirt1030 to test final puppet code
  • 12:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1094 for schema change - T85757 (duration: 00m 46s)
  • 12:28 banyek: depooling db1094 for schema change - T85757
  • 12:27 moritzm: restarting tor on torrelay1001 to pick up OpenSSL security update
  • 11:02 _joe_: manually reloading icinga to pick up changes to commands.cfg
  • 10:55 moritzm: installing apache updates on puppetmasters
  • 10:22 moritzm: installing ghostscript security updates on jessie
  • 09:51 elukey: restart memcached on mc1023 to apply -R 200 - T208844
  • 09:46 moritzm: remove imagemagick remnants from ATS hosts (obsoleted by upstream packaging change which dropped the webp plugin)
  • 09:39 moritzm: installing nginx updates on puppetdb*
  • 09:26 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: repool es2019 - T212833 (duration: 01m 33s)
  • 09:18 banyek: repooling es2019 - T212833
  • 08:46 moritzm: rolling restart of proton to pick up OpenSSL update
  • 08:35 banyek: depooled es2019 as host was unsresponsive - T212833
  • 08:35 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: depool es2019, host is unsresponsible - T212833 (duration: 00m 49s)
  • 08:11 moritzm: installing OpenSSL security updates
  • 00:21 mutante: notebook1004 - started nagios-nrpe-server one more time

2019-01-02

  • 23:59 mutante: notebook1004 still keeps running out of memory from some user actions and that kills nagios-nrpe-server and that causes a bunch of Icinga alerts
  • 23:39 mutante: notebook1004 - systemctl start nagios-nrpe-server
  • 23:39 mutante: notebook1004 - systemctl status nagios-nrpe-server
  • 20:59 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=parsoid,service=parsoid,name=wtp1028.eqiad.wmnet
  • 20:59 herron: repooling wtp1028 T212624
  • 20:52 herron: rebooting wtp1028 — looking for POST errors T212624
  • 20:05 Krinkle: mwmaint1002: foreachwikiindblist s5 deleteEqualMessages.php
  • 20:04 Krinkle: mwmaint1002: foreachwikiindblist s2 deleteEqualMessages.php
  • 18:35 volans: restarting icinga on icinga1001 T212669
  • 16:50 XioNoX: create BGP sessions to AS3214 in AMS-IX
  • 16:46 XioNoX: remove BGP sessions to AS42949 in AMS-IX (leaving the IX)
  • 16:43 XioNoX: remove BGP sessions to AS6866 in AMS-IX (leaving the IX)
  • 16:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1090:3317 after schema change - T85757 (duration: 00m 46s)
  • 16:30 arturo: reimaging cloudvirt1030 with stretch, server cleanup after puppet refactoring
  • 16:29 moritzm: restarting Superset to pick up openssl security update
  • 16:25 moritzm: restarting Hue to pick up openssl security update
  • 16:23 arturo: T212302 re-enable puppet in all {cloud,lab}virt* servers, all was fine
  • 16:22 banyek: repooling db1090:3317 after schema change (T85757)
  • 16:11 arturo: T212302 disable puppet in all {cloud,lab}virt* servers to merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/481194/
  • 15:39 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1090:3317 for schema change - T85757 (duration: 00m 44s)
  • 15:34 moritzm: installing OpenSSL security updates
  • 15:31 banyek: depooling db1090:3317 for schema change (T85757)
  • 15:13 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1086 after schema change - T85757 (duration: 00m 44s)
  • 15:07 banyek: repooling db1086 after schema change (T85757)
  • 14:49 banyek: executing schema change on db1086 - T85757
  • 14:48 moritzm: installing ghostscript security update for jessie
  • 14:47 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1086 for schema change - T85757 (duration: 00m 45s)
  • 14:38 banyek: depooling db1086 for schema change (T85757)
  • 14:15 ema: cp hosts: upgrade OpenSSL from 1.1.0f to 1.1.0j
  • 13:39 moritzm: installing ghostscript update for stretch
  • 13:33 moritzm: installing libav security updates
  • 13:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 T86338 T202167 (duration: 00m 44s)
  • 13:17 moritzm: installing openjpeg2 security updates
  • 13:17 banyek: executing schema change on db2040 (s7 codfw master) replication lag could be expected on codfw - T85757
  • 13:13 banyek: stopping replication on db2077 prior to executing schema change on codfw s7 master (db2040) - T85757
  • 13:06 marostegui: Deploy schema change on db1119 - T86338 T202167
  • 13:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 T86338 T202167 (duration: 00m 45s)
  • 13:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T86338 T202167 (duration: 00m 47s)
  • 12:00 moritzm: rebooting labtestpuppetmaster2001 for kernel security update
  • 11:53 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe1006.eqiad.wmnet
  • 11:51 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1006.eqiad.wmnet
  • 11:50 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1006.codfw.wmnet
  • 11:46 ema: replace TLS certificates on ms-fe eqiad hosts T212215
  • 11:41 moritzm: rebooting labtestweb2001 for kernel security update
  • 11:24 marostegui: Deploy schema change on db1099:3311 - T86338 T202167
  • 11:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T86338 T202167 (duration: 00m 45s)
  • 11:17 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe2006.codfw.wmnet
  • 11:10 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe2006.codfw.wmnet
  • 10:59 ema: replace TLS certificates on ms-fe codfw hosts T212215
  • 10:52 moritzm: rebooting centrallog1001 for kernel security update
  • 10:48 volans: testing the new spicerack package on cumin2001, in the unlikely event you need to use spicerack cookbooks today please use cumin1001
  • 10:45 godog: ms-be2018 Flashing Smart Array P840 in Slot 3 [ 3.00 -> 6.60 ]
  • 10:43 moritzm: removed labvirt1013 from debmonitor, got renamed in T212513
  • 10:42 volans: uploaded spicerack_0.0.10-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • 10:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2096 (duration: 00m 44s)
  • 09:50 marostegui: Stop MySQL on db2096 for kernel and mysql upgrade
  • 09:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2096 (duration: 00m 45s)
  • 09:48 marostegui@deploy1001: sync-file aborted: Depool db2096 (duration: 00m 01s)
  • 09:18 moritzm: installing c3p0 security updates
  • 09:07 Zoranzoki21: Drop valid_tag from s8 by Marostegui - T212254
  • 09:06 godog: eqiad-prod: final weight for ms-be10[44-50].eqiad.wmnet - T209618
  • 08:56 moritzm: installing libarchive security updates
  • 07:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T212692 (duration: 00m 46s)
  • 07:30 marostegui: Fix login.logging table on db1078 - T212692
  • 07:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T212692 (duration: 00m 47s)
  • 07:01 marostegui: Deploy schema change on s1 codfw master (lag will be generated on s1 codfw) - T202167 T86338
  • 06:54 marostegui: Drop empty valid_tag table from labswiki labtestwiki - T212254
  • 06:49 marostegui: Drop empty valid_tag table from s5 - T212254
  • 06:25 marostegui: Drop valid_tag from s6 - T212254
  • 06:15 marostegui: Fix last chunks on db1124:338 - T212574


Archives

See Server admin log/Archives.