You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(elukey: manually clean up of old log files on an-coord1001 (disk space issues))
imported>Stashbot
(marostegui: Fix replication on db1124:3318)
Line 1: Line 1:
== 2018-12-23 ==
* 11:53 marostegui: Fix replication on db1124:3318
* 10:25 ariel@deploy1001: Finished deploy [dumps/dumps@af74350]: python3 fixup for show runtimes (duration: 00m 05s)
* 10:25 ariel@deploy1001: Started deploy [dumps/dumps@af74350]: python3 fixup for show runtimes
* 08:53 apergos: restarted pdfrender on scb1003
== 2018-12-22 ==
== 2018-12-22 ==
* 18:45 elukey: manually clean up of old log files on an-coord1001 (disk space issues)
* 18:45 elukey: manually clean up of old log files on an-coord1001 (disk space issues)

Revision as of 11:53, 23 December 2018

2018-12-23

  • 11:53 marostegui: Fix replication on db1124:3318
  • 10:25 ariel@deploy1001: Finished deploy [dumps/dumps@af74350]: python3 fixup for show runtimes (duration: 00m 05s)
  • 10:25 ariel@deploy1001: Started deploy [dumps/dumps@af74350]: python3 fixup for show runtimes
  • 08:53 apergos: restarted pdfrender on scb1003

2018-12-22

  • 18:45 elukey: manually clean up of old log files on an-coord1001 (disk space issues)
  • 15:58 godog: reboot ms-be2018, stuck on sd 0:1:0:1: rejecting I/O to offline device
  • 10:21 cwd: re-enabled process-control
  • 08:36 cwd: took down sidebar fr campaign
  • 08:36 cwd: disabled process-control
  • 01:00 AaronSchulz: Deployed b47e9fcfece99 to navtiming
  • 00:59 aaron@deploy1001: Finished deploy [performance/navtiming@b47e9fc]: (no justification provided) (duration: 00m 05s)
  • 00:59 aaron@deploy1001: Started deploy [performance/navtiming@b47e9fc]: (no justification provided)

2018-12-21

  • 21:29 mutante: phab1002 - temp hack to unbreak phd / systemd alert, real fix will be phab deployment to new server
  • 21:28 mutante: phab1002 - mkdir -p /srv/phab/libext/ava/src  ; touch __phutil_library_init__.php
  • 21:05 mutante: phab1002 - apt autoremove
  • 21:04 mutante: phab1002 - removing all php related packages and letting puppet reinstall them
  • 20:38 mutante: phab1002 - restart php-fpm, restart phd for testing. phd fails
  • 18:50 mutante: [scb1001:~] $ sudo systemctl restart pdfrender
  • 14:20 dcausse: elastic@eqiad deleting unused index enwiki_general_1537906513
  • 13:24 moritzm: installing subversion updates from stretch point release
  • 12:34 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: T211494 (duration: 00m 44s)
  • 12:31 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211494 (duration: 00m 45s)
  • 12:10 moritzm: rebooting url downloaders to pick up SSBD-enabled QEMU
  • 11:38 moritzm: rebooting debug proxies to pick up SSBD-enabled QEMU
  • 09:58 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe2006.codfw.wmnet
  • 09:57 ema: repool ms-fe2006 with old certs, test successful T212215#4839960
  • 09:23 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe2006.codfw.wmnet
  • 09:22 ema: depool ms-fe2006 to test new TLS certs T212215
  • 08:45 moritzm: upgrading nginx on sodium
  • 08:41 moritzm: upgrading nginx on debug proxies
  • 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove old comments T211973 (duration: 00m 46s)
  • 00:52 twentyafterfour: SWAT Finished. See you all next year!
  • 00:50 twentyafterfour@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/MobileFrontend/: SWAT: sync https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/481026 (duration: 00m 48s)

2018-12-20

  • 22:46 mutante: phab1001 / phabricator: upgraded nodejs package
  • 22:39 mutante: phab1001 / phabricator: installing php5 package upgrades
  • 19:29 kaldari@deploy1001: Synchronized wmf-config/Wikibase.php: syncing Wikibase for SWAT deployment (duration: 00m 45s)
  • 19:27 kaldari@deploy1001: Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for SWAT deployment (duration: 00m 46s)
  • 19:03 elukey: restart hdfs namenode on an-master1002 with new heap settings (currently standby, 8->12G)
  • 18:30 elukey: remove hdfs journalnode config+packages from analytics10(28|35) - not used anymore - T209929
  • 18:29 elukey: restart hdfs namenode on an-master1001 with new heap settings (currently standby, 8->12G)
  • 18:06 mutante: doc1001 - meged gerrit:480881 and then manually moved the entire /srv/org/wikimedia/doc/ structure into /srv/docroot/srv/org/wikimedia/ and deleted the old dirs T137890
  • 17:53 arturo: updating puppet compiler facts: `PUPPET_COMPILER=compiler1002.puppet-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts`
  • 17:49 arturo: updating puppet compiler facts: `PUPPET_COMPILER=compiler1001.puppet-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts`
  • 17:05 XioNoX: add 208.80.155.88/29 to cloud-in4 term icmp - T207663
  • 17:02 XioNoX: configure additional 208.80.155.88/29 IPs on cloud-instance-transport1-b-eqiad - T207663
  • 16:59 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.9
  • 16:51 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/Wikibase: SWAT: Revert "Fail hard if an entity namespace is not configured." (T212427) (duration: 01m 17s)
  • 16:31 elukey: remove two journal nodes from the Analytics hadoop cluster - T209929
  • 14:53 moritzm: installing nodejs security updates on maps* (was tested via T211419)
  • 14:41 moritzm: restarted etherpad for nodejs security updates
  • 14:39 elukey: add two journal nodes to the Analytics Hadoop cluster - T209929
  • 14:30 moritzm: installing libdap updates from stretch point release
  • 14:26 moritzm: rearmed keyholder on netmon1002 after reboot
  • 14:22 moritzm: rebooting netmon1002 for kernel security update
  • 13:09 moritzm: installing xapian-core updates from stretch point release
  • 12:53 arturo: T209616 installing cloudvirt1030, icinga downtime for 1 day
  • 12:48 kartik@deploy1001: Finished deploy [cxserver/deploy@16f65cb]: Update cxserver to 803baa4 (T210581, T211889, T144467, T209473) (duration: 04m 42s)
  • 12:47 moritzm: installing libxcursor security updates
  • 12:43 kartik@deploy1001: Started deploy [cxserver/deploy@16f65cb]: Update cxserver to 803baa4 (T210581, T211889, T144467, T209473)
  • 12:31 moritzm: installing fuse updates from stretch point release
  • 12:09 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546, T202497) (duration: 00m 52s)
  • 12:08 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546, T202497) (duration: 00m 53s)
  • 12:02 moritzm: draining restbase1016 for eventual reboot for kernel security update
  • 11:46 moritzm: draining restbase1015 for eventual reboot for kernel security update
  • 11:34 moritzm: powercycling restbase1014, similar EFI ASSERT error to T212305
  • 11:34 moritzm: powercycling restbase1014, similar EFI ASSEER error to T212305
  • 11:21 moritzm: draining restbase1014 for eventual reboot for kernel security update
  • 10:59 banyek: executing schema change on db1068 (s4 master) - T85757
  • 10:58 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1121 after schema change - T85757 (duration: 00m 52s)
  • 10:53 banyek: repooling db1121 after schema change T85757
  • 10:45 moritzm: draining restbase1013 for eventual reboot for kernel security update
  • 10:44 banyek: stopping replication on db1121 - T85757
  • 10:40 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1121 for schema change - T85757 (duration: 00m 52s)
  • 10:35 banyek: depooling db1121 for schema change T85757
  • 10:33 banyek: executing schema change on dbstore1002 - T85757
  • 10:22 banyek: executing schema change on db1102 - T85757
  • 10:20 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1081 after schema change - T85757 (duration: 00m 51s)
  • 10:19 moritzm: draining restbase1012 for eventual reboot for kernel security update
  • 10:15 banyek: repooling db1081 after schema change T85757
  • 10:10 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1081 for schema change - T85757 (duration: 00m 51s)
  • 10:05 banyek: depooling db1081 for schema change T85757
  • 10:00 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1103:3314 after schema change - T85757 (duration: 00m 52s)
  • 09:56 banyek: repooling db1103:3314 after schema change T85757
  • 09:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1103:3314 for schema change - T85757 (duration: 00m 51s)
  • 09:46 banyek: depooling db1103:3314 for schema change T85757
  • 09:45 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1097:3314 after schema change - T85757 (duration: 00m 51s)
  • 09:41 legoktm@deploy1001: Synchronized php-1.33.0-wmf.8/includes/: T199540 (duration: 01m 06s)
  • 09:40 legoktm@deploy1001: Synchronized php-1.33.0-wmf.9/includes/: T199540 (duration: 01m 14s)
  • 09:39 banyek: repooling db1097:3314 after schema change T85757
  • 09:35 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T187299 Increase ruwiki navtiming rate + frwiki survey rate (duration: 00m 52s)
  • 09:29 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1097:3314 for schema change - T85757 (duration: 00m 52s)
  • 09:23 banyek: depooling db1097:3314 for schema change T85757
  • 08:06 elukey: roll restart of druid middlemanagers on druid* to pick up new port settings
  • 07:17 marostegui: Re-start codfw s4 backup as the previous one failed
  • 07:11 elukey: restart pdfrender on scb1002
  • 07:10 elukey: restart rsyslog on lithium - in:imtcp stuck in recvfrom ms-be2047.codfw.wmnet - T199406
  • 06:35 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2057 T212277 (duration: 00m 57s)
  • 01:41 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@c7977a7]: Update mobileapps to 42c011e (duration: 04m 08s)
  • 01:37 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@c7977a7]: Update mobileapps to 42c011e
  • 01:09 eileen_: civicrm revision changed from 9d727e4708 to b33dcd3c94, config revision is 0f94a475b7
  • 00:09 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/479911/ (duration: 00m 53s)

2018-12-19

  • 23:24 mutante: syncing facts from puppetmaster1001 to compiler1001/compiler1002
  • 23:21 eileen_: civicrm revision changed from d5c3d5fd17 to 9d727e4708, config revision is 0f94a475b7
  • 23:05 krinkle@deploy1001: Finished deploy [performance/navtiming@64e3f63]: (no justification provided) (duration: 00m 05s)
  • 23:05 krinkle@deploy1001: Started deploy [performance/navtiming@64e3f63]: (no justification provided)
  • 22:15 mutante: scb1003: systemctl restart pdfrender
  • 20:44 hashar: 1.33.0-wmf.9 on group1 looks fine.
  • 20:03 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.9 (duration: 00m 51s)
  • 20:02 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.9
  • 19:33 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad" - T210467
  • 19:32 XioNoX: repool codfw - T210467
  • 19:28 XioNoX: reactive BGP sessions to telia on cr1-codfw - T211715
  • 19:07 chasemp: cp of files to ext drive on labstore1007
  • 18:41 marostegui: Stop MySQL on db2057
  • 18:28 XioNoX: replace `interface-range vlan-private1-b-eqiad member ge-6/0/*` with individual interfaces on asw2-b-eqiad
  • 18:09 ejegg: updated fundraising CiviCRM from 8e18485697 to d5c3d5fd17
  • 17:32 addshore: SWAT done
  • 17:22 addshore@deploy1001: Synchronized wmf-config: Wikibase: wikidatawiki upsert idGenerator, T194299 (duration: 00m 52s)
  • 17:13 addshore@deploy1001: Synchronized wmf-config: Wikibase: testwikidatawiki upsert idGenerator, T194299 (duration: 00m 52s)
  • 17:05 XioNoX: remove 2nd port to AS8220 (cf. email to peering@)
  • 17:04 addshore@deploy1001: Synchronized wmf-config: Wikibase: prepare to set $wgWBRepoSettings idGenerator, T194299 (duration: 00m 53s)
  • 16:59 XioNoX: deactive BGP sessions to telia on cr1-codfw - T211715
  • 16:58 fsero: DNS: updating wmnet to include new registries T212212
  • 16:57 hashar@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/ExtensionDistributor/includes/specials/SpecialBaseDistributor.php: Follow-up f686d348: No need for an <img> tag any more - T212217 (duration: 00m 52s)
  • 16:52 XioNoX: codfw row D maintenance finished without issues - T210467
  • 16:39 cmjohnson1: swapping disk in slot 2 on db1072
  • 16:33 XioNoX: shutdown asw-d4-codfw - T210467
  • 16:21 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting comment migration to new on group 2 (T166733) (duration: 00m 52s)
  • 16:04 XioNoX: Redirect eqsin/ulsfo caches to eqiad - T210467
  • 16:01 moritzm: installing php5 security updates on jessie
  • 15:48 XioNoX: depool codfw - T210467
  • 15:39 chasemp: various cp jobs on labstore1007 to ext media
  • 15:33 chasemp: labstore1007 mount /dev/sde /mnt/T211327
  • 15:13 moritzm: draining restbase1011 for eventual reboot for kernel security update
  • 15:07 marostegui: Drop image_comment_temp from labswiki and labtestwiki - T209591
  • 14:58 moritzm: draining restbase1010 for eventual reboot for kernel security update
  • 14:42 moritzm: draining restbase1009 for eventual reboot for kernel security update
  • 14:16 moritzm: draining restbase1008 for eventual reboot for kernel security update
  • 14:03 moritzm: draining restbase1007 for eventual reboot for kernel security update
  • 13:57 moritzm: installing nodejs updates on wtp*
  • 13:36 marostegui: Correction from the previous !log: Rename table valid_tag on db1089 (s1) - T212254
  • 13:35 marostegui: Rename table valid_tag on db1081 (s1) - T212254
  • 13:31 marostegui: Drop image_comment_temp on s4 - T209591
  • 12:46 marostegui: Drop image_comment_temp on s3 - T209591
  • 12:24 zeljkof: EU SWAT finished
  • 12:23 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/Kartographer/: SWAT: Fix using at-ease functions in namespaced class (T212218) (duration: 00m 53s)
  • 12:06 moritzm: rearmed keyholder after netmon2001 reboot
  • 11:58 moritzm: rebooting netmon2001 for kernel security update
  • 11:50 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1091 after schema change - T85757 (duration: 00m 52s)
  • 11:49 moritzm: rebooting matomo1001 to pick up SSBD-enabled qemu
  • 11:46 banyek: repooling db1091 after schema change - T85757
  • 11:41 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1091 for schema change - T85757 (duration: 00m 52s)
  • 11:37 banyek: depooling db1091 for schema change - T85757
  • 11:34 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1084 after schema change - T85757 (duration: 00m 51s)
  • 11:30 banyek: repooling db1084 after schema change - T85757
  • 11:29 moritzm: upgrading nodejs on restbase2013-2018
  • 11:26 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2050 after recloning db2057 T212275 (duration: 00m 52s)
  • 11:25 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1084 for schema change - T85757 (duration: 00m 52s)
  • 11:15 banyek: depooling db1084 for schema change - T85757
  • 11:13 marostegui: Stop MySQL and power off db2057 for firmware upgrade - T212277
  • 11:11 marostegui: Drop image_comment_temp from s5 - T209591
  • 10:37 moritzm: draining restbase2012 for eventual reboot for kernel security update
  • 10:28 banyek: executing schema change in db2051 (s4 codfw master) with replication enabled - T85757
  • 10:25 banyek: stopping replication on db2073 as executing schema change on codfw master - T85757
  • 10:14 moritzm: draining restbase2011 for eventual reboot for kernel security update
  • 09:53 banyek: dropping tables 'flagged%' on db1066 ptwiki with replication enabled - T211544
  • 09:41 moritzm: draining restbase2010 for eventual reboot for kernel security update
  • 09:37 banyek: dropping tables with 'T211544' prefix on db1122 - T211544
  • 09:24 moritzm: draining restbase2009 for eventual reboot for kernel security update
  • 09:14 moritzm: rebooting restbase2008 for kernel security update
  • 08:53 elukey: roll restart of cassandra on aqs1005-1009 for opendjdk upgrades
  • 08:50 marostegui: Drop image_comment_temp from s7 - T209591
  • 08:46 marostegui: Drop image_comment_temp from s6 - T209591
  • 08:43 akosiaris: rebalance row_A ganeti01.svc.codfw.wmnet nodegroup after recabling T210447
  • 08:40 marostegui: Drop image_comment_temp from s8 - T209591
  • 08:37 moritzm: draining restbase2007 for eventual reboot for kernel security update
  • 08:28 marostegui: Drop image_comment_temp from s2 - T209591
  • 08:27 marostegui: Drop image_comment_temp from s1 - T209591
  • 08:16 godog: swift eqiad-prod: more weight for ms-be10[44-50].eqiad.wmnet - T209618
  • 07:37 marostegui: Drop nodepooldb on m5 master - T212230
  • 07:22 marostegui: Stop MySQL on db2050 to clone db2057 - T212275
  • 07:15 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2050 to clone db2057 T212275 (duration: 00m 52s)
  • 07:00 marostegui: Enable GTID on s8 codfw master (db2045) - T211973
  • 06:58 marostegui: Enable GTID on s1 codfw master (db2048) - T211973
  • 06:44 marostegui: Remove nodepool@10.64.16.155 user from m5 master - T212230
  • 06:34 marostegui: Hard reboot db2057 - T212275
  • 06:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2057 - storage crashed T212275 (duration: 01m 08s)
  • 01:27 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: g 480681 make maintenance script dry run more useful (duration: 00m 52s)
  • 01:25 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/includes/AbuseFilter.php: g 480680 fix exception in maintenance script (duration: 00m 54s)
  • 00:11 mutante: contint1001 - rsyncing /srv/org/wikimedia/docs to rsync://docs1001.eqiad.wmnet/docs T211974

2018-12-18

  • 23:49 XioNoX: remove BGP session to AS50629 from cr2-esams (not in AMS-IX anymore)
  • 20:33 otto@deploy1001: Finished deploy [eventlogging/analytics@104adb5]: Send JSON string of event for validation errors in EventError (duration: 00m 04s)
  • 20:33 otto@deploy1001: Started deploy [eventlogging/analytics@104adb5]: Send JSON string of event for validation errors in EventError
  • 19:43 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@963d704]: Parse summaries from lead objects only (T202642) (duration: 05m 26s)
  • 19:38 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@963d704]: Parse summaries from lead objects only (T202642)
  • 19:30 robh: migration of ulsfo pdus complete
  • 19:19 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Production configuration for GrowthExperiments Help Panel (duration: 00m 52s)
  • 19:06 XioNoX: redirect ns1 back to authdns2001 - T210447
  • 18:51 moritzm: installing libx11 security updates
  • 18:48 XioNoX: depool ulsfo - T209101
  • 18:42 XioNoX: repool codfw - T210447
  • 18:41 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad" - T210447
  • 18:05 chasemp: stat1004:~# umount /mnt/T211327
  • 17:58 XioNoX: shutdown fpc4 for replacement - T210447
  • 17:51 bblack: deploying php7 cache-splitter patch to cache_text - https://gerrit.wikimedia.org/r/c/operations/puppet/+/478680 - T206339
  • 17:49 godog: bounce rsyslog on lithium, tls listener timeout
  • 17:34 volans: triggered restart of ircecho on icinga1001 while applying https://gerrit.wikimedia.org/r/480509
  • 16:37 jijiki: librsvg* 2.40.20-3+wmf1+stretch1 uploaded to components/thumor to stretch-wikimedia - T209886
  • 16:22 bblack: powercycle cp1075 from console (crashed, apparently)
  • 16:07 XioNoX: starting codfw row A recabling - T210447
  • 16:05 ejegg: updated fundraising python tools from af5dbee8eb to 5f44d9dd43
  • 15:53 XioNoX: redirect ns1 to authdns1001 for T210447
  • 15:47 XioNoX: redirect eqsin/ulsfo caches to eqiad for T210447
  • 15:44 XioNoX: depool codfw for T210447
  • 15:30 akosiaris: empty ganeti2005, ganeti2006 for T210447
  • 15:16 Amir1: mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=testwikidatawiki --config-format=wgConf | tee WikibaseQualityConstraints-config.php
  • 14:55 akosiaris: restart pybal on lvs1006, lvs2003 for blubberoid LVS deployment. T205919
  • 14:40 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.9
  • 14:26 akosiaris: restart ircecho, seems to have croaked with Dec 17 19:39:52 icinga1001 ircecho[861]: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 174: ordinal not in range(128)
  • 14:25 zfilipin@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.9 and rebuild l10n cache (duration: 34m 11s)
  • 14:15 akosiaris: restart pybal on lvs1016, lvs2006 for blubberoid LVS deployment. T205919
  • 13:51 zfilipin@deploy1001: Started scap: testwiki to php-1.33.0-wmf.9 and rebuild l10n cache
  • 13:12 addshore@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/Wikibase/lib/includes/Formatters/ControlledFallbackEntityIdFormatter.php: T201930 ControlledFallbackEntityIdFormatter, track unique value formats (duration: 00m 46s)
  • 12:52 Amir1: deployed a patch on wmf.8 for T207814
  • 12:41 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 45s)
  • 12:17 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Perform even more PHP constraint checks before falling back (T209504) (duration: 00m 46s)
  • 12:16 moritzm: installing fuse updates from stretch point release
  • 11:41 moritzm: installing remaining libgd2 security updates
  • 09:38 godog: swift eqiad-prod: initial weights for ms-be10[44-50].eqiad.wmnet - T209618
  • 08:44 marostegui: Enable GTID on s7 codfw master (db2040) - T211973
  • 08:26 marostegui: Enable GTID on es3 - T211973
  • 08:23 marostegui: Enable GTID on es2 - T211973
  • {{safesubst:SAL entry|1=07:57 elukey: restart cassandra-{a,b} on aqs1004 for openjdk upgrades}}
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 T86338 T202167 (duration: 00m 45s)
  • 07:49 marostegui: Deploy schema change on db1075 (s3 master) T86338 T202167
  • 06:07 marostegui: Deploy schema change on db1078 T86338 T202167
  • 06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 T86338 T202167 (duration: 00m 45s)
  • 06:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2038 after mysql and kernel upgrade (duration: 00m 47s)
  • 00:13 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable MediaViewer thumbnail URL guessing for private wikis (T212099) (duration: 00m 45s)

2018-12-17

  • 23:21 tgr: deployed security patch per T207750#4829275
  • 23:17 chasemp: launch gpg jobs for files on stat1004
  • 23:17 chasemp: launch gpg jobs for files on labstore1007
  • 22:00 mutante: puppetmaster - signed cert for doc1001 (ganeti VM), initial puppet run
  • 21:58 chasemp: stat1004:/var/log# mkfs.exfat /dev/sde && mkdir /mnt/T211327 && mount /dev/sde /mnt/T211327/
  • 21:55 arlolra: Updated Parsoid to 4eba44e (T204622, T211941)
  • 21:42 arlolra@deploy1001: Finished deploy [parsoid/deploy@1bf4dab]: Updating Parsoid to 4eba44e (duration: 08m 55s)
  • 21:33 arlolra@deploy1001: Started deploy [parsoid/deploy@1bf4dab]: Updating Parsoid to 4eba44e
  • 21:20 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@92fdf43]: Update mobileapps to d244439 (duration: 04m 31s)
  • 21:15 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@92fdf43]: Update mobileapps to d244439
  • 20:25 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@c1b6b32]: Rollback to c1b6b32 until the checks are fixed (duration: 01m 56s)
  • 20:23 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@c1b6b32]: Rollback to c1b6b32 until the checks are fixed
  • 20:17 mutante: creating new ganeti VM doc1001.eqiad.wmnet for doc.wikimedia.org - specs as requested by hashar on T211974
  • 20:05 XioNoX: remove sandbox-out4 from all routers - T212155
  • 20:02 XioNoX: remove sandbox-out4 from ulsfo - T212155
  • 19:47 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@f183af7]: Update to 657a515. All hosts (duration: 10m 45s)
  • 19:36 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@f183af7]: Update to 657a515. All hosts
  • 19:33 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@f183af7]: Update to 657a515. Canary on scb2001 (duration: 00m 18s)
  • 19:32 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@f183af7]: Update to 657a515. Canary on scb2001
  • 19:29 raynor: Morning SWAT finished
  • 19:27 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Page issues treatment enabled on all wikis except enwiki(T210553) (duration: 00m 45s)
  • 18:58 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@8036f9b]: Update to 2991db1. Canary on scb2001 (duration: 00m 51s)
  • 18:57 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@8036f9b]: Update to 2991db1. Canary on scb2001
  • 18:42 herron: manually restarting pybal on lvs2003 to add kibana service T205850
  • 18:34 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2006.codfw.wmnet
  • 18:34 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2005.codfw.wmnet
  • 18:34 mobrovac@deploy1001: Finished deploy [proton/deploy@ff7c8a2]: Config: Use extdomain and add printable=yes to the req query - T210793 (duration: 01m 32s)
  • 18:32 herron: performing manual puppet run and manually restarting pybal on lvs2006 to add kibana service T205850
  • 18:32 mobrovac@deploy1001: Started deploy [proton/deploy@ff7c8a2]: Config: Use extdomain and add printable=yes to the req query - T210793
  • 18:29 gtirloni: prometheus-node-exporter: Ignore Docker/Kubelet mount points T211810
  • 18:26 akosiaris@deploy1001: scap-helm blubberoid finished
  • 18:26 akosiaris@deploy1001: scap-helm blubberoid cluster codfw completed
  • 18:26 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.2.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: codfw]
  • 18:25 akosiaris@deploy1001: scap-helm blubberoid finished
  • 18:25 akosiaris@deploy1001: scap-helm blubberoid cluster eqiad completed
  • 18:25 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.1.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad]
  • 18:24 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.1.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad]
  • 18:23 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad,codfw]
  • 18:21 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@69c000a]: DelayQueue fix and mitigation of stale index for updater (duration: 11m 52s)
  • 18:21 moritzm: installing libgd2 security updates
  • 18:10 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@69c000a]: DelayQueue fix and mitigation of stale index for updater
  • 18:07 akosiaris@deploy1001: scap-helm blubberoid finished
  • 18:07 akosiaris@deploy1001: scap-helm blubberoid cluster staging completed
  • 18:07 akosiaris@deploy1001: scap-helm blubberoid install --name staging --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production stable/blubberoid [namespace: blubberoid, clusters: staging]
  • 18:04 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2004.codfw.wmnet
  • 16:41 moritzm: installing libarchive security updates on jessie
  • 16:33 moritzm: installing php5 security updates on jessie
  • 16:13 anomie: Aborted migrateActors.php run, queries were too slow.
  • 16:07 anomie@mwmaint1002: Running migrateActors.php on test wikis and mediawikiwiki for T188327
  • 16:07 mobrovac@deploy1001: Started restart [electron-render/deploy@94d27d7]: Electron stuck
  • 14:33 marostegui: Upgrade mysql and kernel on db2093 (tendril sby host on codfw)
  • 14:10 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere T201837 (duration: 00m 44s)
  • 14:09 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere T201837 (duration: 00m 44s)
  • 14:06 marostegui: Stop MySQL on db2038 for mysql and kernel upgrade
  • 14:06 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2038 for mysql and kernel upgrade (duration: 00m 44s)
  • 13:58 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 10million for wikidatawiki T201837 (duration: 00m 45s)
  • 13:49 marostegui: Enable GTID on s3 codfw master db2043 - T211973
  • 13:49 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 1million for wikidatawiki T201837 (duration: 00m 45s)
  • 13:46 fsero: installing new version of php-excimer on mwdebug* - T205059
  • 13:41 fsero: installing new version of php-excimer on mwdebug2001 - T205059
  • 13:31 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki incubatorwiki --force-protocol https
  • 13:29 Amir1: deleting everything from site_identifiers@incubatorwiki
  • 13:15 ariel@deploy1001: Finished deploy [dumps/dumps@0393ca7]: switch dumps to python3 (duration: 00m 03s)
  • 13:15 ariel@deploy1001: Started deploy [dumps/dumps@0393ca7]: switch dumps to python3
  • 13:04 Amir1: ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T209820)
  • 13:04 Amir1: ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https
  • 13:02 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=fawiki --force-protocol https
  • 12:52 ema: lvs1016: bounce pybal for new service kartotherian-ssl T211970
  • 12:50 ladsgroup@deploy1001: Synchronized wmf-config/InterwikiSortOrders.php: Update InterwikiSortOrders.php (T209820) (duration: 00m 45s)
  • 12:49 ladsgroup@deploy1001: sync-file aborted: Use the right index for change_tag (T211896) (duration: 00m 02s)
  • 12:38 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add be.wikisource new 1.5x and 2x logos to wgLogoHD (T150618) (duration: 00m 44s)
  • 12:38 ema: lvs2003: bounce pybal for new service kartotherian-ssl T211970
  • 12:29 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Update be.wikisource logo (T211795) (duration: 00m 45s)
  • 12:29 ema: lvs1006: bounce pybal to pick up new service kartotherian-ssl T211970
  • 12:24 ema: lvs2006: bounce pybal to pick up new service kartotherian-ssl
  • 12:22 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable extendedmover user group at ur.wiki (T211978) (duration: 00m 45s)
  • 12:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use new minerva logos for cswiki in IS.php (T210979) (duration: 00m 45s)
  • 12:08 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/: SWAT: Upload custom minerva logo for cswiki (T210979) (duration: 00m 44s)
  • 12:02 ladsgroup@deploy1001: Finished deploy [ores/deploy@18d3657]: T206333 T211267 (duration: 14m 14s)
  • 11:48 ladsgroup@deploy1001: Started deploy [ores/deploy@18d3657]: T206333 T211267
  • 10:52 paravoid: reprepro include python 3.4, 3.6, 3.7 to component/pyall (use with care)
  • 10:28 moritzm: installing openssl security updates on stretch
  • 10:18 marostegui: Repool labsdb1011 - T86338
  • 09:55 moritzm: remove debmonitor entries for restbase2001-restbase2006
  • 09:44 marostegui: Enable GTID on s4 codfw master - T211973
  • 09:08 marostegui: Depool labsdb1011 - T86338
  • 09:04 marostegui: Repool labsdb1010 - T86338
  • 09:01 elukey: stop kafkatee on oxygen and rsync /srv/log data to weblog1001
  • 08:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 T86338 T202167 (duration: 00m 45s)
  • 08:21 marostegui: Deploy schema change on db1077 with replication (lag will be generated on labsdb:s3) T86338 T202167
  • 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 T86338 T202167 (duration: 00m 45s)
  • 08:16 marostegui: Stop replication on labsdb1011
  • 08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 T86338 T202167 (duration: 00m 45s)
  • 07:58 marostegui: Enable GTID on db2052 (s5 master) - T211973
  • 06:48 marostegui: Enable GTID on db2035 (s2 master) - T211973
  • 06:36 marostegui: Enable GTID on db2034 (x1 master) - T211973
  • 06:36 marostegui: Enable GTID on db2034 (x1 master)
  • 06:29 marostegui: Enable replication consistency options on codfw masters - T211973
  • 06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 T86338 T202167 (duration: 00m 46s)
  • 06:28 marostegui: Depool labsdb1010 - T86338
  • 04:22 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 44s)
  • 04:20 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/includes/AbuseFilter.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 59s)

2018-12-16

  • 09:52 elukey: mask + reset-failed kafkatee default instance on sulfur (kafkatee-webrequest works fine)
  • 07:30 marostegui: Reboot db1115 after OOM
  • 07:28 marostegui: Stop MySQL on db1115 so tendril can get back to work

2018-12-15

  • off: restarted pdfrender on scb1004
  • 09:22 elukey: mask + reset-failed kafkatee default instance on weblog1001

2018-12-14

  • 23:59 mutante: LDAP: added aezell to wmf group (T211945) for grafana access
  • 22:00 XioNoX: increase accepted-prefix-limit for HE to 200000
  • 20:38 mutante: sulfur systemctl restart nagios-nrpe-server
  • 20:32 andrew@deploy1001: Finished deploy [horizon/deploy@1a830b9]: Rolling out fix for T177855 (duration: 03m 17s)
  • 20:29 andrew@deploy1001: Started deploy [horizon/deploy@1a830b9]: Rolling out fix for T177855
  • 20:13 otto@deploy1001: Finished deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for T211833 (duration: 06m 04s)
  • 20:07 otto@deploy1001: Started deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for T211833
  • 20:07 otto@deploy1001: deploy aborted: (no justification provided) (duration: 00m 00s)
  • 20:07 otto@deploy1001: Started deploy [analytics/refinery@ef1f7c6]: (no justification provided)
  • 17:22 cmjohnson1: mw1272 down for h/w troubleshooting
  • 14:38 marostegui: Enable GTID on db2039 (s6 codfw master) - T211973
  • 14:03 marostegui: Compare ruwiki.revision between db2039 (s6 master) and db1085 - T211973
  • 13:59 marostegui: Enable notifications for db1095 (s3 lag check)- T211973
  • 13:57 marostegui: Enable notifications for db2068 (s7 lag check)- T211973
  • 13:47 dcausse: elastic@codfw copying index data from the main cluster to psi & omega (test disk usage & import speed)
  • 13:42 marostegui: Enable GTID on db1124:3318 - T211973
  • 11:11 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2083 db2068 db2067 after mysql and kernel upgrade (duration: 00m 44s)
  • 10:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2083 db2068 db2067 for mysql and kernel upgrade (duration: 00m 45s)
  • 10:50 marostegui: Stop MySQL on db2083, db2068 and db2067 for mysql and kernel upgrade
  • 10:19 filippo@deploy1001: Synchronized wmf-config/logging.php: wmf-config/InitialiseSettings-labs.php (duration: 00m 44s)
  • 10:18 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 45s)
  • 09:33 marostegui: Deploy schema change on db1095:3313 T86338 T202167
  • 09:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2049, db2059, db2070 after mysql and kernel upgrade (duration: 00m 45s)
  • 09:21 banyek: global user rename is in progress - T209488
  • 08:58 marostegui: Stop MySQL on db2049, db2059 and db2070 for mysql and kernel upgrade
  • 08:56 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2049, db2059, db2070 for mysql and kernel upgrade (duration: 00m 43s)
  • 08:50 elukey: swap oxygen with weblog1001
  • 08:48 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2084 and db2088 after mysql and kernel upgrade (duration: 00m 44s)
  • 08:47 elukey: disabled kafkatee-webrequest logstash output on oxygen (prep step before weblog1001)
  • 07:59 marostegui: Deploy schema change on dbstore1002:s3 T86338 T202167
  • 07:49 marostegui: Upgrade mysql and kernel on db2084 and db2088
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2084 and db2088 for mysql and kernel upgrade (duration: 00m 45s)
  • 06:54 marostegui: Deploy schema change on db2043 (s3 codfw master) - this will generate lag on s3 codfw T86338 T202167
  • 06:53 marostegui: Deploy schema change on db2043 (s3 codfw master) - this will generate lag on s3 codfw T86338 T20216
  • 06:14 marostegui: Deploy schema change on db1062 (s7 primary master) T86338 T202167
  • 06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 T86338 T202167 (duration: 00m 44s)
  • 06:10 marostegui: Deployed schema change on db1094 T86338 T202167
  • 05:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 T86338 T202167 (duration: 00m 47s)
  • 01:39 mutante: install2002 deleted /srv/ contents,then mounted /mnt/vdb on /srv so same content but now / is used only 7% and /srv 57% (T211850)
  • 00:43 mutante: install2002 (T211850) restarted instance, created ext4 filesystem on new /dev/vdb, mounted on /mnt/vdb, rsyncing /srv/ to /mnt/vdb/
  • 00:33 mutante: rebooting install2002 via ganeti2003, to add new virtual disk
  • 00:31 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Enforce a 10-byte password for +staff users, I4ecac70e (duration: 00m 44s)
  • 00:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT I7867277d Make wmgBabelMainCategory consistent for sr* wikis (duration: 00m 45s)

2018-12-13

  • 23:57 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208246: Increase default minimum new password length to 10 for privileged groups (duration: 00m 44s)
  • 23:17 andrew@deploy1001: Finished deploy [horizon/deploy@18c4ca6]: Rolling out fix for T131367 (duration: 03m 25s)
  • 23:13 andrew@deploy1001: Started deploy [horizon/deploy@18c4ca6]: Rolling out fix for T131367
  • 22:31 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/HttpRequestFactory.php: T211886 (duration: 00m 44s)
  • 22:29 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/MobileFrontend: T211903 (duration: 00m 48s)
  • 21:55 mutante: Ganeti - creating new 120G virtual disk on install2002 (T211850)
  • 21:00 otto@deploy1001: Finished deploy [analytics/superset/deploy@UNKNOWN]: revert to version 0.26.3 (duration: 00m 32s)
  • 20:59 otto@deploy1001: Started deploy [analytics/superset/deploy@UNKNOWN]: revert to version 0.26.3
  • 20:33 dcausse: creating 300+ wikis indices in elastic-psi @eqiad and @codfw
  • 20:21 volans: imported elasticsearch-curator_5.2.0-1~deb9u1 into apt.w.o stretch-wikimedia component/spicerack - T205884
  • 20:10 mutante: LDAP - added mneisler to wmf (T211742) - existing shell user, so no gerrit change needed
  • 19:53 hoo: Ran scap pull on snapshot1005 to undo live changes done for dump performance testing
  • 19:49 dcausse: creating 300 wiki indices in elastic-omega@eqiad
  • 19:46 dcausse: SF Morning SWAT done
  • 19:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] fix cluster settings for temp clusters psi&omega (duration: 00m 44s)
  • 19:40 moritzm: removed labvirt1014 from debmonitor DB, has been renamed to cloudvirt1014
  • 19:39 volans: imported python-elasticsearch_5.4.0-1~deb9u1 into apt.w.o stretch-wikimedia component/spicerack - T205884
  • 19:35 dcausse@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/TwoColConflict/: T210501: Add missing code to not loose edits on the other side (duration: 00m 45s)
  • 19:29 bblack: authdns2001: upgrading gdnsd to 2.99.9944-beta
  • 19:27 bblack: multatuli: upgrading gdnsd to 2.99.9944-beta
  • 19:22 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211234: Enable Block notice stats on top blocking wikis (duration: 00m 45s)
  • 19:19 bblack: authdns1001: upgrading gdnsd to 2.99.9944-beta
  • 18:52 arlolra: Updated Parsoid to 4242ad0 (T204622, T211738)
  • 18:40 arlolra@deploy1001: Finished deploy [parsoid/deploy@e27574c]: Updating Parsoid to 4242ad0 (duration: 09m 17s)
  • 18:31 arlolra@deploy1001: Started deploy [parsoid/deploy@e27574c]: Updating Parsoid to 4242ad0
  • 18:30 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@93443fe]: Refine MW API queries (duration: 03m 41s)
  • 18:26 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@93443fe]: Refine MW API queries
  • 18:24 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: T204748 [Beta only] Use newly-fixed config for Wikibase->Commons federation (duration: 00m 44s)
  • 18:22 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T204748 Rename repo-only Wikibase config for clarity [no-op] (duration: 00m 45s)
  • 18:11 bblack: restart pybal on lvs1006 for config updates
  • 18:07 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: Use the right index for change_tag (T211896) (duration: 00m 46s)
  • 17:45 ladsgroup@deploy1001: Finished deploy [ores/deploy@1a3de73]: T211267 (duration: 13m 53s)
  • 17:41 akosiaris: reapply the zotero calico policy to allow LVS endpoints
  • 17:35 dcausse: elastic@eqiad created cirrus metastore on psi&omega
  • 17:32 ladsgroup@deploy1001: Started deploy [ores/deploy@1a3de73]: T211267
  • 17:31 arturo: T168967 added shiny-server .deb to stretch-wikimedia
  • 17:20 godog: run puppet and bounce pybal on lvs in eqiad to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/479184
  • 17:19 akosiaris: T205919 create namespace for blubberoid on eqiad/codfw/staging clusters
  • 17:00 anomie: Set comment migration to new on group 1 (T166733)
  • 16:33 mutante: icinga1001 - started service again, enabeld puppet
  • 16:27 mutante: icinga1001 - disable puppet, stopped icinga, for cable replacement
  • 16:18 anomie: Deployed fix for T210937: API: Use parenthesized join in ApiQueryBase::showHiddenUsersAddBlockInfo
  • 15:44 moritzm: installing openssl 1.1 security updates on Hadoop workers
  • 15:33 moritzm: rebooting pybal-test hosts to pick up SSBD-enabled qemu
  • 15:18 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on all wikis (T188327) (duration: 00m 45s)
  • 15:12 anomie@deploy1001: Synchronized php-1.33.0-wmf.8/includes/api/ApiPageSet.php: Backport fix for T211804: ApiPageSet::initFromPageIds: Default $filterIds to true (duration: 00m 46s)
  • 15:03 akosiaris: ores2* deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479206/
  • 14:55 moritzm: installing openssl 1.1 security updates on mw canaries (along with nginx restart/upgrade)
  • 14:46 cdanis: updating grafana/stretch-wikimedia to 5.4.2: reprepro --restrict grafana update stretch-wikimedia
  • 14:40 akosiaris: disable puppet on ores1* and ores2* machines to deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479206/
  • 14:36 ladsgroup@deploy1001: Finished deploy [ores/deploy@a9d5e95]: noop (duration: 14m 59s)
  • 14:23 ladsgroup@deploy1001: Started deploy [ores/deploy@a9d5e95]: noop
  • 13:22 dcausse: creating 300+ wiki indices on elastic-omega@codfw
  • 13:15 godog: stop restbase and cassandra on restbase200[1-6] - T211070
  • 12:59 elukey: superset on analytics-tool1003 upgraded to 0.28.1
  • 12:41 arturo: icinga downtime (30 mins) cloudcontrol1003, cloudnet1003 and cloudnet1004 for package upgrades
  • 12:38 zeljkof: EU SWAT finished
  • 12:35 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Update maiwiki logo (T211845) (duration: 00m 52s)
  • 12:35 oblivian@deploy1001: scap-helm zotero finished
  • 12:35 oblivian@deploy1001: scap-helm zotero cluster codfw completed
  • 12:35 oblivian@deploy1001: scap-helm zotero cluster eqiad completed
  • 12:35 oblivian@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: eqiad,codfw]
  • 12:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable flood user group at ne.wiki (T211181) (duration: 00m 51s)
  • 12:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on shnwiki (T210699) (duration: 00m 53s)
  • 12:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - T211070 T211871 T211411 (duration: 18m 59s)
  • 12:10 oblivian@deploy1001: scap-helm -h finished
  • 12:10 oblivian@deploy1001: scap-helm -h cluster codfw completed
  • 12:10 oblivian@deploy1001: scap-helm -h cluster eqiad completed
  • 12:10 oblivian@deploy1001: scap-helm -h [namespace: -h, clusters: eqiad,codfw]
  • 12:00 moritzm: rebooting ununpentium to pick up SSBD-enabled qemu
  • 11:53 mobrovac@deploy1001: Started deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - T211070 T211871 T211411
  • 11:51 elukey@deploy1001: Finished deploy [analytics/superset/deploy@35841a7]: (no justification provided) (duration: 00m 38s)
  • 11:51 elukey@deploy1001: Started deploy [analytics/superset/deploy@35841a7]: (no justification provided)
  • 11:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871 (duration: 06m 08s)
  • 11:43 moritzm: rebooting vega/bromine to pick up SSBD-enabled qemu
  • 11:39 mobrovac@deploy1001: Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871
  • 11:39 mobrovac@deploy1001: Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871 (duration: 07m 08s)
  • 11:38 fsero@deploy1001: scap-helm zotero finished
  • 11:38 fsero@deploy1001: scap-helm zotero cluster codfw completed
  • 11:38 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 11:38 moritzm: rebooting krypton to pick up SSBD-enabled qemu
  • 11:31 mobrovac@deploy1001: Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871
  • 11:31 fsero@deploy1001: scap-helm zotero finished
  • 11:31 fsero@deploy1001: scap-helm zotero cluster eqiad completed
  • 11:31 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 11:14 moritzm: rebooting webperf hosts to pick up SSBD-enabled qemu
  • 10:51 moritzm: rebooting dbmonitor hosts to pick up SSBD-enabled qemu
  • 10:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 T86338 T202167 (duration: 00m 51s)
  • 10:00 elukey: upgrade nodejs on aqs100[5-9]
  • 09:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 T86338 T202167 (duration: 00m 51s)
  • 09:55 moritzm: removed openssl 1.1.0f-3+deb9u2+wmf1 from stretch-wikimedia/component/node10 (superseded by openssl update in DSA 4348 for stretch)
  • 09:35 moritzm: rebooting etcd/kubernetes hosts in codfw to pick up SSBD-enabled qemu
  • 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 T86338 T202167 (duration: 00m 51s)
  • 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2006.codfw.wmnet
  • 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2005.codfw.wmnet
  • 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2004.codfw.wmnet
  • 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2003.codfw.wmnet
  • 09:05 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2002.codfw.wmnet
  • 09:05 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
  • 08:50 godog: stress-test ms-be10[44-50] - T209618
  • 08:45 marostegui: Deploy schema change on db1090:3317 T86338 T202167
  • 08:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 T86338 T202167 (duration: 00m 53s)
  • 08:45 moritzm: installing openssl security updates on stretch
  • 08:26 vgutierrez: Use certcentral managed TLS certificates in mx[12]001.wikimedia.org - T207050
  • 08:15 marostegui: Drop unused flaggedrevs tables from srwikinews - T209761
  • 08:11 moritzm: rolling reboot of scb in eqiad for kernel security update (combined with nodejs update)
  • 08:09 marostegui: Repool labsdb1011 T86338
  • 08:08 moritzm: installing nodejs updates on restbase1007
  • 07:28 marostegui: Depool labsdb1011 T86338
  • 07:25 marostegui: Repool labsdb1010 T86338
  • 07:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 T86338 T202167 (duration: 00m 52s)
  • 06:46 marostegui: Deploy schema change on db1079 with replication, lag will be generated on labsdb:s7 T86338 T202167
  • 06:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 T86338 T202167 (duration: 00m 53s)
  • 06:43 marostegui: Depool labsdb1010 T86338
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 T86338 T202167 (duration: 00m 51s)
  • 06:17 marostegui: Deploy schema change on db1086 T86338 T202167
  • 06:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 T86338 T202167 (duration: 00m 51s)
  • 06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1098:3316 db1098:3317 after kernel and mysql upgrade (duration: 00m 54s)
  • 00:52 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/GuzzleHttpRequest.php: T211806 (duration: 00m 51s)
  • 00:41 mutante: einsteinium - rm /lib/systemd/system/update-etcd-mw-config-lastindex.service ; systemctl reset-failed
  • 00:36 tzatziki: changing two passwords for compromised accounts
  • 00:32 dcausse: elastic@codfw created cirrus metastore on psi&omega clusters
  • 00:22 dcausse@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/MobileFrontend/extension.json: T210390: Reset default mobilefrontend provider (duration: 00m 53s)
  • 00:13 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] fix temp clusters for codfw (duration: 00m 52s)

2018-12-12

  • 22:47 tzatziki: change email for User:Denrique
  • 21:48 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/EventBus: T211805 (duration: 00m 53s)
  • 21:26 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@ced6fab]: Update mobileapps to 55981a8. Summary: Get modified date with regexes to avoid unneeded Document parse (duration: 04m 03s)
  • 21:22 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@ced6fab]: Update mobileapps to 55981a8. Summary: Get modified date with regexes to avoid unneeded Document parse
  • 19:45 hashar: contint1001: sudo chown -R zuul:zuul /etc/zuul/wikimedia/.git
  • 19:28 XioNoX: repool codfw - T210456
  • 19:28 XioNoX: revert redirecting eqsin/ulsfo caches to eqiad - T210456
  • 19:16 XioNoX: re-enable BGP to telia on cr1-codfw - T211715
  • 19:08 XioNoX: disable BGP to telia on cr1-codfw - T211715
  • 19:06 bblack: uploading gdnsd 2.99.9944-beta-1+wmf1 to stretch-wikimedia
  • 19:04 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 00m 50s)
  • 18:53 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 02m 58s)
  • 18:46 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 03m 00s)
  • 18:41 banyek: pool db1098 for recentchanges and recenlchangeslinked
  • 18:12 catrope@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/GrowthExperiments/extension.json: Temporarily disable help panel / VisualEditor integration (duration: 03m 00s)
  • 17:55 banyek: pooling db1098
  • 17:54 XioNoX: shutting down asw-b4-codfw - T210456
  • 17:02 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 50s)
  • 16:42 moritzm: installing lxml security updates
  • 16:40 moritzm: installing cups updates on trusty (only client libs used)
  • 16:34 jijiki: Merged 477472 "mcrouter: replace codfw proxy before maintenance", eqiad mcrouters are picking up the change - T210467
  • 16:19 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/specials/pagers/ImageListPager.php: T211774 (duration: 00m 52s)
  • 16:17 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/specials/pagers/ImageListPager.php: T211774 (duration: 00m 52s)
  • 16:01 XioNoX: Redirect eqsin/ulsfo caches to eqiad - T210456
  • 15:54 XioNoX: Depool codfw for row B recabling - T210456
  • 15:35 elukey: upload matomo 3.7.0 to stretch-wikimedia, removed 3.5.1 from jessie-wikimedia
  • 15:27 moritzm: installing PHP security updates on matomo1001 (piwik host)
  • 15:24 godog: poweroff ms-be2044 for hardware inspection - T209921
  • 15:22 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/api/ApiBase.php: T211769 (duration: 00m 52s)
  • 15:21 urandom: decommissioning cassandra-c, restbase2006 -- T210843
  • 15:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1098:3316 db1098:3317 after kernel and mysql upgrade (duration: 00m 53s)
  • 14:43 banyek: renaming tables on db1122 ptwiki: flagged* -> T211544_flagged* - T211544
  • 14:29 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.33.0-wmf.8"
  • 14:18 moritzm: restart uwsgi-netbox on netmon2001
  • 14:16 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 51s)
  • 14:07 mobrovac@deploy1001: Started restart [restbase/deploy@5946231]: Restart RB to pick up the new seeds in codfw - T211416
  • 14:04 marostegui: Stopy MySQL on db1098:3316 and db1098:3317 for kernel and mysql upgrade
  • 13:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 db1098:3317 for kernel and mysql upgrade (duration: 00m 52s)
  • 13:26 moritzm: installing PHP security updates
  • 12:52 zeljkof: EU SWAT finished
  • 12:49 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add NS_PROJECT localised name for tt.wiktionary (T211312) (duration: 00m 52s)
  • 12:41 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new namespace aliases for zhwikiversity (T207544) (duration: 00m 52s)
  • 12:35 arturo: T205969 icinga downtime load-avg check for labstore1007 until January (1 month)
  • 12:32 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable extension SandboxLink for nowiki (T210325) (duration: 00m 52s)
  • 12:27 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos for cawikimedia in IS.php (T198507) (duration: 00m 52s)
  • 12:17 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload new logos for cawikimedia (T198507) (duration: 00m 52s)
  • 12:03 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Display Kartographer mapframes for geocoordinate statements (T184933) (duration: 00m 52s)
  • 11:30 volans: re-enabled puppet on icinga[12]001, re-activated crontab to sync files on 2001 and manually run it + run puppet
  • 11:14 volans: restarting icinga with dropped downtimes from last night (start_date > 1544489652)
  • 11:03 volans: restarting Icinga with debug log on icinga1001
  • 10:51 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2b - T211416 (duration: 10m 11s)
  • 10:41 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2b - T211416
  • 10:40 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2 - T211416 (duration: 00m 15s)
  • 10:40 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2 - T211416
  • 10:26 volans: Icinga is having issue restarting properly, investigation ongoing
  • 10:08 banyek: executing schema change on db1070 (s5 master) - T85757
  • 09:40 banyek: repooling labsdb1010 - T210693
  • 09:34 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1082 (duration: 00m 52s)
  • 09:29 banyek: repooling db1082 - T85757
  • 09:25 banyek: restarting replication on db1082 after schema change - T85757
  • 09:25 banyek: fixing triggers on db1124:3315- T85757
  • 09:22 banyek: executing schema change with replication on db1082 - T85757
  • 09:20 banyek: stopping replication on db1082 for schema change - T85757
  • 09:15 moritzm: installing pixman security updates on trusty (Debian already fixed)
  • 08:48 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1082 (duration: 00m 51s)
  • 08:38 banyek: depooling db1082 for schema change - T85757
  • 08:37 marostegui: Remove old backup directory from db1116 - T206743
  • 08:18 godog: decommissioning cassandra-b, restbase2006 -- T210843
  • 08:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1088 (duration: 00m 51s)
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 52s)
  • 07:38 marostegui: Deploy schema change on db2040 (s7 codfw master), this will generate lag on codfw T86338 T202167
  • 07:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 51s)
  • 07:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 51s)
  • 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 52s)
  • 06:44 marostegui: Stop MySQL on db1088 for mysql and kernel upgrade
  • 06:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for mysql upgrade (duration: 01m 07s)
  • 06:37 marostegui: Deploy schema change on s4 primary master (db1068) T86338
  • 06:00 marostegui: Deploy schema change on s8 primary master (db1071) T86338 T202167
  • 00:47 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add some missing groups to the privileged list (duration: 00m 51s)
  • 00:46 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Bring up password change logging to the same standards as login logging Add some missing groups to the privileged list (duration: 00m 53s)

2018-12-11

  • 22:50 chasemp: ssh to tar archive data from logstash1006 /mnt (external) to labstore1007
  • 22:06 XioNoX: push loopback filter term return-tcp to all routers - T207962
  • 22:04 urandom: decommissioning cassandra-a, restbase2006 -- T210843
  • 19:57 XioNoX: apply BGP_IXP_RS_in and avoid HE to cr4-ulsfo - T211079
  • 19:28 jforrester@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/Flow/modules/engine/misc/flow-handlebars.js: T211707 Hot-deploy fix for StructuredDiscussions pages (duration: 00m 52s)
  • 19:22 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: T209619: Hot-deploy Ibcf3c93e (duration: 00m 52s)
  • 18:59 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@ecd5fb6]: fix site CSS URL (duration: 03m 58s)
  • 18:55 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@ecd5fb6]: fix site CSS URL
  • 18:54 XioNoX: push BGP_IXP_RS_in to all routers (but don't apply it to any peers, needs to be done manually) - T211079
  • 18:33 cmjohnson1: swapping disk slot 0 db1063
  • 18:23 XioNoX: push BGP_IXP_RS_in to cr2-eqord - T211079
  • 18:06 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621, for real this time (duration: 00m 53s)
  • 18:00 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: T211237 Hot-deploy disabling Wikibase federation, unused except in Beta Cluster (duration: 00m 52s)
  • 17:33 _joe_: restarting pybal on lvs2003
  • 17:23 XioNoX: push changes tested on cr4-ulsfo to all routers - T211079
  • 17:16 _joe_: restarting pybal on lvs2006
  • 17:15 oblivian@puppetmaster1001: conftool action : set/weight=1; selector: service=search-omega-ssl
  • 17:15 oblivian@puppetmaster1001: conftool action : set/weight=1; selector: service=search-psi-ssl
  • 17:15 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=search-psi-ssl
  • 17:14 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=search-omega-ssl
  • 17:05 XioNoX: remove redundant term classification from BGP_transit_in on cr4-ulsfo - T211079
  • 16:55 banyek: executing schema change on dbstore1002 - T85757
  • 16:50 XioNoX: replace local-preference/default-action by next policy for BGP_IXP_in and BGP_Private_Peer_in on cr4-ulsfo - T211079
  • 16:49 banyek: executing schema change on db1102 - T85757
  • 16:48 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1113:3315 (duration: 00m 51s)
  • 16:44 banyek: repooling db1113:3315 after schema change - T85757
  • 16:38 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1113:3315 (duration: 00m 51s)
  • 16:35 banyek: depooling db1113:3315 for schema change - T85757
  • 16:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1110 (duration: 00m 51s)
  • 16:32 banyek: repooling db1110 after schema change - T85757
  • 16:28 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1110 (duration: 00m 50s)
  • 16:25 banyek: depooling db1110 for schema change - T85757
  • 16:23 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1100 (duration: 00m 51s)
  • 16:22 XioNoX: Remove static routes for NS v6 IPs - T211699
  • 16:20 banyek: repooling db1100 after schema change - T85757
  • 16:15 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1100 (duration: 00m 52s)
  • 16:09 banyek: depooling db1100 for schema change - T85757
  • 15:49 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.8
  • 15:45 papaul: re-installing OS on ms-be2047.codfw.wmnet
  • 15:31 zfilipin@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache (duration: 36m 08s)
  • 15:06 vgutierrez: Use certcentral managed TLS certificate in mirrors.wikimedia.org - T207050
  • 14:55 zfilipin@deploy1001: Started scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache
  • 14:50 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.3 (duration: 02m 51s)
  • 14:49 banyek: depooling labsdb1010 - T210693
  • 14:46 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.2 (duration: 03m 12s)
  • 14:43 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.1 (duration: 11m 40s)
  • 14:42 volans: 'sudo systemctl reload icinga' on icinga1001
  • 14:32 godog: decommissioning cassandra-c, restbase2005 -- T210843
  • 14:31 bblack: removed unused public IPv6 IPs from authdnses manually with "ip -6 addr del ..." - https://gerrit.wikimedia.org/r/c/operations/puppet/+/478939
  • 14:23 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=restbase2013.codfw.wmnet
  • 14:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date (duration: 00m 38s)
  • 14:13 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date
  • 14:10 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date - T211416 (duration: 01m 53s)
  • 14:08 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date - T211416
  • 13:48 vgutierrez: Use certcentral TLS managed certificate in lists.wikimedia.org - T207050
  • 13:06 zeljkof: EU SWAT finished
  • 13:02 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: HD Logos: Add fr and fy wikibooks and fr wiiknews variants to InitaliseSettings.php (T150618) (duration: 00m 46s)
  • 12:52 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update settings to include new HD logos (T150618) (duration: 00m 47s)
  • 12:50 hoo@deploy1001: Finished deploy [wdqs/wdqs@f914415]: Fix WDQS UI embeds (T211629) (duration: 10m 31s)
  • 12:39 hoo@deploy1001: Started deploy [wdqs/wdqs@f914415]: Fix WDQS UI embeds (T211629)
  • 12:38 bblack: Authdns CI/config refactoring done, all is well, resume normal DNS ops!
  • 12:37 elukey: updated nodejs nodejs-legacy on aqs1004 (security upgrades)
  • 12:36 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: HD Logos: Add 1.5x and 2x variants of fr and fy wikibooks and fr wikinews (T150618) (duration: 00m 46s)
  • 12:16 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Add HD logos for 3 projects (T150618) (duration: 00m 47s)
  • 12:12 marostegui: Repool labsdb1011 - T86338
  • 11:59 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@dcde39f]: GUI update (duration: 01m 05s)
  • 11:58 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@dcde39f]: GUI update
  • 11:39 bblack: puppet disabled on authdnses for attempting https://gerrit.wikimedia.org/r/q/topic:%22authdns-ci%22
  • 11:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 T86338 T202167 (duration: 00m 46s)
  • 11:10 marostegui: Depool labsdb1011 - T86338
  • 11:06 marostegui: Repool labsdb1010 - T86338
  • 11:03 volans: restarted pdfrender on scb1003 [last time, we need an automatic restart]
  • 10:47 fsero: pooling mw1272
  • 10:42 fsero: scap pull mw1272
  • 09:30 ema: mw1272 down for the past 12h. Nothing in console, power-cycling
  • 09:08 marostegui: Deploy schema change on db1087 with replication (this will generate lag on labsdb:s8) T202167 T86338
  • 09:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 T86338 T202167 (duration: 00m 46s)
  • 09:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 T86338 T202167 (duration: 00m 46s)
  • 09:01 marostegui: Depool labsdb1010 - T86338
  • 08:15 marostegui: Deploy schema change on db1092 T202167 T86338
  • 08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 T86338 T202167 (duration: 00m 46s)
  • 08:10 godog: decommissioning cassandra-b, restbase2005 -- T210843
  • 07:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 T86338 T202167 (duration: 00m 46s)
  • 07:32 oblivian@deploy1001: Synchronized wmf-config/PhpAutoPrepend.php: Hotfix for logging on php7 (2/2) (duration: 02m 51s)
  • 07:29 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw1272.*
  • 07:28 oblivian@deploy1001: Synchronized wmf-config/php7.php: Hotfix for logging on php7 (1/2) (duration: 02m 50s)
  • 07:06 marostegui: Deploy schema change on db1104 T202167 T86338
  • 07:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 T86338 T202167 (duration: 02m 51s)
  • 06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 T86338 T202167 (duration: 02m 52s)
  • 06:45 marostegui: Rename flaggedrevs tables on srwikinews on db1078 - T209761
  • 06:13 marostegui: Deploy schema change on db1109 T202167 T86338
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 T86338 T202167 (duration: 02m 55s)
  • 05:57 marostegui: Deploy schema change on s4 primary master (db1068) T202167 T86338
  • 01:03 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure localized logos for nywiki (T211570) (duration: 01m 36s)
  • 01:01 catrope@deploy1001: Synchronized static/images/project-logos/: Add localised logos for nywiki (T211570) (duration: 01m 00s)
  • 00:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new HD logos for zhwiktionary, zhwikivoyage, zhwikinews (T150618) (duration: 01m 16s)
  • 00:50 RoanKattouw: mw1272 is down (does not respond to ping), but scap still tries to deploy to it
  • 00:50 catrope@deploy1001: Synchronized static/images/project-logos/: Add HD logos for zhwikinews, zhwikivoyage, zhwiktionary (T150618) (duration: 02m 30s)
  • 00:15 mutante: icinga2001 - killed all nagios processes, restarted nsca service, something is different from icinga1001, service failed when trying to restart (T211641)

2018-12-10

  • 23:51 andrewbogott: silencing the kvm process count alert on cloudvirt1023 until I can figure out why it's misfiring
  • 22:13 mutante: Welcome new Mediawiki deployer Christoph 'WMDE-Fisch' Jauera (T211014)
  • 21:29 arlolra@deploy1001: Finished deploy [parsoid/deploy@dc9b3a1]: Updating Parsoid to 19560da (duration: 11m 15s)
  • 21:20 ladsgroup@deploy1001: Finished deploy [ores/deploy@03b9c98]: Add celery4 configs back to the deploy repo (duration: 15m 25s)
  • 21:19 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@9f4b567]: More internal promisification and other performance tweaks (T202642) (duration: 04m 17s)
  • 21:17 arlolra@deploy1001: Started deploy [parsoid/deploy@dc9b3a1]: Updating Parsoid to 19560da
  • 21:14 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@9f4b567]: More internal promisification and other performance tweaks (T202642)
  • 21:05 ladsgroup@deploy1001: Started deploy [ores/deploy@03b9c98]: Add celery4 configs back to the deploy repo
  • 20:35 cdanis: T210416: grafana.wikimedia.org switched to point to grafana1001.eqiad.wmnet (running grafana 5.4.1)
  • 20:32 jforrester@deploy1001: Synchronized wmf-config/extension-list: Uninstall the ParserMigration extension, Part III I332939809 (duration: 00m 46s)
  • 20:30 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Uninstall the ParserMigration extension, Part II I1f7266f55a (duration: 00m 46s)
  • 20:29 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Uninstall the ParserMigration extension, Part I I338a3d8a87fd (duration: 00m 47s)
  • 20:26 cdanis: T210416: switching grafana.wikimedia.org to point to grafana1001.eqiad.wmnet
  • 20:25 robh: messing with ulsfo power for 103.02.23 tower b, shouldnt disrupt anything T209101
  • 20:20 cdanis: T210416: setting grafana.wikimedia.org (currently served by krypton) to read-only and copying to grafana1001 (serving grafana-beta)
  • 20:13 urandom: decommissioning cassandra-a, restbase2005 -- T210843
  • 19:58 cdanis: T210416: updating grafana to 5.4.1 in stretch-wikimedia: reprepro --restrict grafana update stretch-wikimedia
  • 18:15 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@dcde39f]: GUI Update (duration: 09m 31s)
  • 18:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@dcde39f]: GUI Update
  • 17:59 banyek: restarting mysql instance on labsdb1004 to restore replication filters to the original state - T211210
  • 17:58 banyek: restarting mysql instance on labsdb1004 to restore replication filters to the original state
  • 17:29 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T211527 Hot-deploy Disable ParserMigration now that Raggett has been dropped (duration: 00m 47s)
  • 16:03 moritzm: installing PHP updates on netmon1002
  • 15:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 T86338 T202167 (duration: 00m 46s)
  • 15:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1097:3315 (duration: 00m 45s)
  • 15:31 banyek: repooling db1097:3315 after schema change - T85757
  • 15:24 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1097:3315 (duration: 00m 46s)
  • 15:13 banyek: depooling db1097:3315 on a schema change - T85757
  • 15:05 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621 (duration: 00m 46s)
  • 14:55 marostegui: Deploy schema change db1101:3318 T86338 T202167
  • 14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 T86338 T202167 (duration: 00m 46s)
  • 14:50 _joe_: uploading php-mongodb 1.5.3 to stretch-wikimedia thirdparty/php72 T206152
  • 14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099 T86338 T202167 (duration: 00m 47s)
  • 14:00 marostegui: Deploy schema change db1099:3318 T86338 T202167
  • 14:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099 T86338 T202167 (duration: 00m 46s)
  • 13:34 ema: trafficserver 8.0.1-1wm1 uploaded to stretch-wikimedia T207048
  • 12:56 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T187299 T197607 Oversample performance survey on specific ruwiki articles (duration: 00m 46s)
  • 12:49 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 12:48 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 12:43 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add http://idb.ub.uni-tuebingen.de/digitue to the wgCopyUploadsDomains (T211466) (duration: 00m 47s)
  • 12:37 zfilipin@deploy1001: Synchronized dblists/flaggedrevs.dblist: SWAT: Remove FlaggedRevs for ptwikipedia (T211433) (duration: 00m 46s)
  • 12:25 moritzm: installing imagemagick security update for jessie
  • 12:21 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Set FileImporter config help location (T199108) (duration: 00m 47s)
  • 12:20 fsero: running puppet agent on icinga to add fsero
  • 12:13 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove the "wikibase-debug" log channel (T207850) (duration: 00m 47s)
  • 11:04 godog: decommissioning cassandra-c, restbase2004 -- T210843
  • 10:42 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 10:41 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 52s)
  • 10:39 marostegui: Deploy schema change on db1116:3318 T86338 T202167
  • 10:36 marostegui: Deploy schema change on dbstore1002:s8 T86338 T202167
  • 10:35 marostegui: Repool labsdb1011 T86338
  • 09:07 marostegui: Deploy schema change on s8 codfw master with replication (db2045) - lag will be generated on codfw - T202167 T86338
  • 09:01 marostegui: Depool labsdb1011 - T86338
  • 08:58 moritzm: installing chromium security updates on proton*
  • 08:57 marostegui: Repool labsdb1010 - T86338
  • 08:39 elukey: roll restart of aqs on aqs100* to pick up new Druid backend settings
  • 08:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 T86338 T202167 (duration: 00m 51s)
  • 08:23 godog: final round of weight addition to new ms-be codfw hosts - T209395
  • 07:18 _joe_: reenabling puppet given my changes were useless
  • 06:52 _joe_: running puppet on the puppetmasters in codfw, twice, then restarting apache to ensure cleanup of any cache
  • 06:50 _joe_: disabled puppet across the fleet for merge of hiera change
  • 06:47 marostegui: Stop slave on s4 on labsdb1011
  • 06:25 marostegui: Deploy schema change on db1121 with replication (this will generate lag on labs) - T86338 T202167
  • 06:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 T86338 T202167 (duration: 00m 49s)
  • 06:23 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T86338
  • 02:53 urandom: decommissioning cassandra-b, restbase2004 -- T210843
  • 00:55 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ic07ff9acfbe17 - T211529, T205546 (duration: 00m 47s)

2018-12-09

  • 20:16 urandom: decommissioning cassandra-a, restbase2004 -- T210843
  • 18:39 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Icb9ad2f554e1 - Fix ruwikinews logo config (duration: 00m 57s)
  • 11:55 godog: decommissioning cassandra-c, restbase2003 -- T210843
  • 03:00 urandom: decommissioning cassandra-b, restbase2003 -- T210843

2018-12-08

  • 23:07 krinkle@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/: T185446 - I6cf29d598a11 (duration: 00m 47s)
  • 22:21 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209794 (duration: 00m 56s)
  • 20:59 urandom: decommissioning cassandra-a, restbase2003 -- T210843
  • 13:54 godog: decommissioning cassandra-c, restbase2002 -- T210843

2018-12-07

  • 23:54 ejegg: updated payments-wiki from b99cd0816e to b8acb95a2a
  • 23:41 urandom: decommissioning cassandra-b, restbase2002 -- T210843
  • 22:43 ejegg: re-enabled Thank You mail sender
  • 22:31 ejegg: updated fundraising CiviCRM from 3e5d74f17e to 8e18485697
  • 22:29 ejegg: Turned off Thank You mailing job for letter update
  • 17:47 herron: rebooting logstash1006 for security updates
  • 17:17 bstorm_: T207377 rebooted labstore1007 for kernel upgrades
  • 17:15 _joe_: uploading php-tideways (rebuilt with php 7.2 support) to stretch-wikimedia thirdparty/php72 T206152
  • 15:37 moritzm: rebooting sarin/neodymium
  • 14:52 urandom: decommissioning cassandra-a, restbase2002 -- T210843
  • 14:21 godog: more weight to new ms-be codfw hosts - T209395
  • 13:37 moritzm: rolling reboot of scb in codfw (along with nodejs update)
  • 12:18 moritzm: installing nodejs security updates on stat/notebook
  • 12:05 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title (duration: 21m 11s)
  • 11:43 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title
  • 11:42 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 00m 21s)
  • 11:42 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
  • 11:41 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 18m 58s)
  • 11:22 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
  • 11:20 moritzm: rolling upgrade of nginx on swift frontends
  • 11:19 mobrovac@deploy1001: Finished deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title (duration: 03m 49s)
  • 11:15 mobrovac@deploy1001: Started deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title
  • 11:09 mobrovac@deploy1001: Finished deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero (duration: 06m 19s)
  • 11:03 mobrovac@deploy1001: Started deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero
  • 10:58 mobrovac@deploy1001: Finished deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero (duration: 02m 42s)
  • 10:55 mobrovac@deploy1001: Started deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero
  • 09:40 banyek: importing back linkwatcher_linklog into database s51230__linkwatcher on host labsdb1004.eqiad.wmnet. - T211210
  • 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
  • 09:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
  • 08:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
  • 08:21 marostegui: Stop MySQL on db1096:3315,3316 for kernel and mysql upgrade
  • 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315, db1096:3316 for kernel and mysql upgrade (duration: 00m 46s)
  • 08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1084 (duration: 00m 47s)
  • 07:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
  • 07:50 godog: decommissioning cassandra-c, restbase2001 -- T210843
  • 07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
  • 07:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1084 (duration: 00m 46s)
  • 07:11 marostegui: Stop MySQL on db1084 for mysql and kernel upgrade
  • 07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 49s)
  • 00:47 XioNoX: done troubleshoting bird bfd on dns2001/cr1-codfw
  • 00:10 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: turn off wbsearchentities ab test T209402 (duration: 00m 47s)

2018-12-06

  • 23:47 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192 (duration: 02m 06s)
  • 23:47 XioNoX: troubleshoot bird bfd on dns2001/cr1-codfw
  • 23:45 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192
  • 23:21 ppchelko@deploy1001: Finished deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192 (duration: 22m 46s)
  • 22:58 ppchelko@deploy1001: Started deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192
  • 22:12 urandom: decommissioning cassandra-b, restbase2001 -- T210843
  • 21:39 gtirloni: reimaging cloudvirt1019 with jessie T196507
  • 21:33 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event (duration: 01m 15s)
  • 21:32 ppchelko@deploy1001: Started deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event
  • 21:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA (duration: 09m 18s)
  • 21:00 XioNoX: remove 2 esams avoid path + 4 prefered/selected transits - T194542
  • 20:58 smalyshev@deploy1001: Started deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA
  • 20:51 XioNoX: remove 2 eqiad avoid path - T194542
  • 20:48 gtirloni: reimaging cloudvirt1019 with stretch T196507
  • 20:45 XioNoX: remove codfw/eqdfw avoid path - T194542
  • 19:32 gehel: shutting down elasticsearch on elastic2001-2024 (third time is a charm) - T211023
  • 18:21 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642) (duration: 03m 54s)
  • 18:17 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642)
  • 17:31 moritzm: installing nodejs updates on proton*
  • 17:11 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for jessie-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
  • 16:27 urandom: decommissioning cassandra-a, restbase2001 -- T210843
  • 16:17 gehel: shutting down elasticsearch on elastic2001-2024 (second try) - T211023
  • 15:51 moritzm: upgrading spamassassin on mx1001/fermium
  • 15:45 fsero@deploy1001: scap-helm zotero finished
  • 15:45 fsero@deploy1001: scap-helm zotero cluster codfw completed
  • 15:45 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 15:42 fsero: modifying zotero deploy CLUSTER=codfw scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero - T211322
  • 15:42 _joe_: disabling puppet fleet-wide for a change in the role() function
  • 15:38 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for stretch-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
  • 15:08 gehel: restartign new elasticsearch masters on codfw - T211023
  • 15:05 gehel: upgrade nginx on wdqs servers
  • 14:59 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10 (duration: 00m 09s)
  • 14:59 elukey@deploy1001: Started deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10
  • 14:46 moritzm: uploaded nodejs 10.4.0~dfsg-1+wmf2 to apt.wikimedia.org/component/node10 (backports of recent security fixes)
  • 14:16 moritzm: installing nginx security updates on mw in eqiad
  • 13:46 moritzm: upgrading spamassassin on mx2001
  • 12:56 gehel: depooling and shutting down elasticsearch on elastic2001-2024 - T211023
  • 12:55 moritzm: installing nginx updates on mw in codfw
  • 11:59 moritzm: installing nginx updates on mw canaries
  • 10:59 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3315 (duration: 00m 47s)
  • 10:56 volans: disable event handler on Icinga for ms-be2047 MD Raid and MegaRAID checks, it's spamming Phabricator - T209921
  • 10:56 banyek: repooling db1096 for schema change - T85757
  • 10:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3315 (duration: 00m 49s)
  • 10:29 banyek: depooling db1096 for schema change - T85757
  • 09:56 dcausse: elastic@codfw cleanup: deleting wikidatawiki_content_1537469318 index (failed reindex probably)
  • 01:03 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
  • 00:55 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 47s)
  • 00:42 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
  • 00:40 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
  • 00:38 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
  • 00:15 mutante: MPM prefork tweaks for high load systems are applied again (apparently they were not since a change in the past that resulted in 2 competing configs in mods-enabled and conf-enabled with the latter one being loaded last and containing the package defaults
  • 00:13 mutante: re-enabling puppet on phabricator, applying change that adds php-fpm support on stretch ..which doesnt affect phab1001 (prod) on jessie.. BUT re-adds tuning config from the past for mpm_prefork.conf (more SpareServers etc) that was not actually applied due to a bug
  • 00:10 urandom: bootstrapping cassandra-c, restbase2018 -- T210843

2018-12-05

  • 23:50 ejegg: updated payments-wiki from 20595cca97 to b99cd0816e
  • 23:40 ejegg: re-enabled fundraising queue consumer jobs
  • 23:33 ejegg: updated fundraising CiviCRM from e757753a46 to 3e5d74f17e
  • 23:32 ejegg: turned off fundraising queue jobs for base queue consumer logic update
  • 22:43 jijiki: restarting pdfreder on scb* hosts in eqiad
  • 21:44 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 02m 47s)
  • 21:41 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
  • 21:41 mdholloway: mobileapps deployment failed for group default03, rolling back and retrying
  • 21:39 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 18m 46s)
  • 21:39 arlolra: Updated Parsoid to a6058e3 (T210647, T208360, T205333)
  • 21:37 urandom: bootstrapping cassandra-b, restbase2018 -- T210843
  • 21:23 arlolra@deploy1001: Finished deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3 (duration: 11m 36s)
  • 21:20 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
  • 21:12 arlolra@deploy1001: Started deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3
  • 20:08 banyek: repooling labsdb1010 - T210693
  • 19:12 cmjohnson1: cloudvirt1019 for an all inclusive part swap by HPE
  • 18:53 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Fix formatting of help panel links (duration: 00m 47s)
  • 18:01 jijiki: uploaded thumbor_6.3.2+git20170607-1+deb9u1 to stretch-wikimedia
  • 17:23 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Enable Kartographer maps on testwikidatawiki (T184933) (duration: 00m 46s)
  • 17:22 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
  • 17:20 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
  • 17:19 XioNoX: remove private IPs from codfw cloud-instance-transport1-b T207663
  • 17:11 XioNoX: add public IPs to codfw cloud-instance-transport1-b T207663
  • 16:58 XioNoX: re-deactivate ams-ix prefix list entry on cr2-esams
  • 16:53 XioNoX: activate ams-ix prefix list entry on cr2-esams
  • 16:27 jijiki: uploaded python-thumbor-wikimedia_2.2-1+deb9u1 to stretch-wikimedia
  • 16:18 akosiaris@deploy1001: scap-helm zotero finished
  • 16:18 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 16:18 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 16:17 akosiaris@deploy1001: scap-helm zotero finished
  • 16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 16:17 akosiaris@deploy1001: scap-helm zotero finished
  • 16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 16:15 akosiaris@deploy1001: scap-helm zotero finished
  • 16:15 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:15 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 16:08 fsero: redeploying zotero on eqiad
  • 16:02 akosiaris@deploy1001: scap-helm zotero finished
  • 16:02 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 16:02 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 stable/zotero [namespace: zotero, clusters: eqiad,codfw]
  • 16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 [namespace: zotero, clusters: eqiad,codfw]
  • 15:34 thcipriani: restarting ci jenkins for update
  • 15:33 akosiaris: add back pods/portforward right to kubernetes deploy user. T211040
  • 15:07 anomie: Running cleanupUsersWithNoId.php on potentially missed s3 and s7 wikis for T181731
  • 15:03 fsero: repooling citoid mathoid eqiad
  • 15:03 godog: bootstrap cassandra-c, restbase2017 - T210843
  • 14:56 banyek: executing schema change on s5 codfw master replication lag could be expected - T85757
  • 14:54 fsero: upgrading k8s on eqiad to 1.10.11
  • 14:54 elukey: restart HDFS namenode and Yarn resource manager on an-master100[1,2] to update rack topology config - T209929
  • 14:51 fsero: depool mathoid/eqiad: pooled changed True => False
  • 14:51 fsero: depool citoid/eqiad: pooled changed True => False
  • 14:43 anomie: Running cleanupUsersWithNoId.php on metawiki for T181731 / T210985
  • 14:09 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary item/property access for all wiktionaries (T175273) (duration: 00m 47s)
  • 13:53 akosiaris: repool citoid/mathoid codfw
  • 12:59 onimisionipe: banning elastic2001-elastic2024 from codfw production, psi and omega clusters
  • 12:53 jijiki: uploaded python-thumbor-community-core_0.4.0-1+deb9u1 to stretch-wikimedia
  • 12:47 dcausse: EU SWAT done
  • 12:45 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=(cit|math)oid,name=codfw
  • 12:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] Add temp clusters but still write to the old ones 2/2 (duration: 00m 46s)
  • 12:42 dcausse@deploy1001: Synchronized wmf-config/CommonSettings.php: T210381: [cirrus] Add temp clusters but still write to the old ones 1/2 (duration: 00m 46s)
  • 12:31 fsero: pooling mathoid and citoid again on codfw
  • 12:30 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 46s)
  • 12:29 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 49s)
  • 12:23 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211188: Increase autoconfirmed count for Meta-Wiki to 5 (duration: 00m 47s)
  • 12:15 fsero: upgrading codfw k8s cluster to 1.10.11
  • 12:15 dcausse: running namespaceDupes & cirrus indexNamespaces on yuewiktionary
  • 12:11 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205546: Define 2 new namespaces for yuewiktionary (duration: 00m 47s)
  • 12:02 fsero: depooling mathoid and citoid servers on codfw for k8s upgrade
  • 11:28 mobrovac@deploy1001: Finished deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127 (duration: 05m 55s)
  • 11:23 mobrovac@deploy1001: Started deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127
  • 11:05 akosiaris: upgrade kubernetes-client and kubernetes-master on staging to 1.10.11
  • 11:04 godog: bootstrap cassandra-b, restbase2017 - T210843
  • 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 (duration: 00m 45s)
  • 10:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 45s)
  • 10:51 ema: cache hosts: begin nginx rolling upgrade to 1.13.6-2+wmf2
  • 10:46 marostegui: Reboot db1090 for kernel upgrade
  • 10:44 moritzm: uploaded jenkins 2.138.4 to jessie-wikimedia/thirdparty and stretch-wikimedia/thirdpary/ci
  • 10:42 marostegui: Stop MySQL on db1090:3312 and db1090:3317 for MySQL upgrade
  • 10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 46s)
  • 10:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 46s)
  • 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1080 (duration: 00m 46s)
  • 10:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
  • 10:17 arturo: T205969 icinga downtime the load avg check in labstore1007 for 1 week
  • 10:10 banyek: depooling labsdb1010 for testing materialized views - T210693
  • 10:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1080 (duration: 00m 46s)
  • 09:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly pool db1080 (duration: 00m 46s)
  • 09:32 gehel: setting up new elasticsearch servers on codfw - elastic2045-2054 - T210265
  • 09:22 marostegui: Stop MySQL on db1080 for mysql and kernel upgrade
  • 09:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for MySQL upgrade (duration: 00m 46s)
  • 09:07 elukey: matomo read only + upgrade to matomo 3.7.0 on matomo1001 - T209808
  • 09:00 _joe_: disabed puppet on mw1261, used for logging tests for T211184
  • 08:48 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207850 Define a new Wikibase log channel to use (duration: 00m 47s)
  • 08:01 moritzm: installing pdns-recursor security update in esams
  • 07:59 godog: bootstrap cassandra-a, restbase2017 - T210843
  • 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 T86338 T202167 (duration: 00m 46s)
  • 06:55 marostegui: Deploy schema change on db1091 T86338 T202167
  • 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 T86338 T202167 (duration: 00m 51s)
  • 04:03 urandom: bootstrapping cassandra-c, restbase2016 -- T210843
  • 03:11 kartik@deploy1001: Finished deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578) (duration: 04m 26s)
  • 03:06 kartik@deploy1001: Started deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578)
  • 00:32 dcausse: Evening SWAT done
  • 00:30 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
  • 00:28 dcausse@deploy1001: Synchronized wmf-config/ProductionServices.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
  • 00:16 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Block notice stats on itwiki (T210452) (duration: 00m 47s)
  • 00:10 urandom: bootstrapping cassandra-b, restbase2016 -- T210843

2018-12-04

  • 23:46 eileen: civicrm revision changed from a411d6bd64 to e757753a46, config revision is 0e6ccc37fe
  • 23:33 XioNoX: update prefix-list peering4 on cr1-eqsin to match jnt
  • 22:46 XioNoX: remove neodymium/sarin from term labs-in4 on cr1/2-eqiad - T210612
  • 22:40 ejegg: Updated payments-wiki from 7403a196b4 to 20595cca97
  • 21:58 XioNoX: clear ethernet-swtiching table for labvirt1004:eth1's switch port
  • 21:57 XioNoX: clear ethernet-swtiching table for labvirt1009:eth1's switch port
  • 21:49 XioNoX: make cr1/2-codfw conform to jnt
  • 21:44 XioNoX: make cr2-eqord/eqdfw conform to jnt
  • 21:41 XioNoX: make cr3/4-ulsfo conform to jnt
  • 20:04 urandom: bootstrapping cassandra-a, restbase2016 -- T210843
  • 19:35 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation (duration: 10m 36s)
  • 19:24 smalyshev@deploy1001: Started deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation
  • 18:04 joal@deploy1001: Finished deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint (duration: 17m 33s)
  • 18:03 akosiaris: bump zotero pod number from 4 to 16 in eqiad/codfw
  • 17:47 joal@deploy1001: Started deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint
  • 17:46 ppchelko@deploy1001: Finished deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465 (duration: 01m 13s)
  • 17:45 ppchelko@deploy1001: Started deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465
  • 17:22 Reedy: created oathauth tables on punjabiwikimedia T211110
  • 17:16 godog: bootstrap cassandra-c on restbase2015 - T210843
  • 16:48 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure 'api-warning' log channel (duration: 00m 47s)
  • 15:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 T86338 T202167 (duration: 00m 47s)
  • 14:04 elukey: upgrade turnilo on analytics-tools1002 to nodejs-10 - T210705
  • 13:56 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=bnwikisource --fix --add-prefix=T210472
  • 13:53 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=euwiki --fix
  • 13:52 marostegui: Deploy schema change on db1103:3314 T86338 T202167
  • 13:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 T86338 T202167 (duration: 00m 47s)
  • 13:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 T86338 T202167 (duration: 00m 46s)
  • 13:34 cdanis: T210416: adding grafana 5 to wikimedia-stretch: reprepro --restrict grafana update stretch-wikimedia
  • 13:33 moritzm: installing nodejs security updates on restbase in codfw
  • 13:32 godog: bootstrap cassandra-b on restbase2015 - T210843
  • 13:29 moritzm: installing nodejs security updates on proton*
  • 13:19 marostegui: Deploy schema change on db1081 T86338 T202167
  • 13:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 T86338 T202167 (duration: 00m 46s)
  • 13:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 T86338 T202167 (duration: 00m 46s)
  • 12:39 Lucas_WMDE: EU SWAT done
  • 12:38 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create namespace "Work" on bnwikisource (T210472) (duration: 00m 46s)
  • 12:33 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create List namespace on euwiki (T209834) (duration: 00m 47s)
  • 12:28 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Milestone logo for atjwiki" (T200713) (duration: 00m 47s)
  • 12:17 moritzm: installing tiff security updates
  • 11:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided) (duration: 04m 47s)
  • 11:44 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided)
  • 11:41 moritzm: rebooting puppetboard2001 to pick up SSBD-enabled qemu
  • 11:38 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242 (duration: 00m 21s)
  • 11:38 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242
  • 11:37 moritzm: rebooting puppetboard1001 to pick up SSBD-enabled qemu
  • 11:36 akosiaris: enable puppet on scb1004, run puppet T197242
  • 11:36 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242 (duration: 00m 20s)
  • 11:35 marostegui: Deploy schema change on db1084 T86338 T202167
  • 11:35 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242
  • 11:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 T86338 T202167 (duration: 00m 47s)
  • 11:34 akosiaris: enable puppet on scb1003, run puppet T197242
  • 11:33 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242 (duration: 00m 28s)
  • 11:33 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242
  • 11:31 akosiaris: enable puppet on scb1002, run puppet T197242
  • 11:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 T86338 T202167 (duration: 00m 46s)
  • 11:18 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242 (duration: 00m 30s)
  • 11:18 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242
  • 11:17 akosiaris: enable puppet on scb1001, run puppet T197242
  • 11:12 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201 (duration: 09m 06s)
  • 11:04 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242 (duration: 20m 59s)
  • 11:03 elukey@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201
  • 10:59 fdans@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques (duration: 00m 37s)
  • 10:58 fdans@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques
  • 10:57 fdans: deploying AQS to expose offset and underestimate numbers on unique devices
  • 10:51 moritzm: rebooting analytics-tool1003 to pick up SSBD-enabled qemu
  • 10:47 moritzm: rebooting analytics-tool1002 to pick up SSBD-enabled qemu
  • 10:43 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242
  • 10:41 moritzm: rebooting analytics-tool1001 to pick up SSBD-enabled qemu
  • 10:03 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242 (duration: 01m 45s)
  • 10:02 godog: bootstrap cassandra-a on restbase2015 - T210843
  • 10:01 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242
  • 09:59 gehel: upgrading nginx on elasticsearch eqiad
  • 09:54 akosiaris: enable puppet on all scb2*, run puppet T197242
  • 09:52 gehel: upgrading nginx on elasticsearch codfw
  • 09:52 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242 (duration: 00m 30s)
  • 09:51 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242
  • 09:50 akosiaris: enable puppet on scb2001, run puppet T197242
  • 09:46 akosiaris: disable puppet on scb for citoid migration to zoterov2 T197242
  • 09:46 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
  • 09:31 gehel: add elastic2039-2044 to cirrus eqiad (new server) - T210265
  • 09:11 gehel: add elastic2038 to cirrus eqiad (new server) - T210265
  • 09:00 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/MediaWiki/electronpdf/action # Ran https://phabricator.wikimedia.org/P7882 for T157012
  • 08:54 marostegui: Deploy schema change on db1097:3314 T86338 T202167
  • 08:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 T86338 T202167 (duration: 00m 47s)
  • 08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*-*.wsp" -delete # T120639
  • 08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*_*.wsp" -delete # T120639
  • 08:43 marostegui: Deploy schema change on db1102:3314 T86338 T202167
  • 08:42 marostegui: Deploy schema change on dbstore1002:s4 T86338 T202167
  • 08:41 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/datamodel$ sudo -u _graphite rm wikipedia_references.wsp # T121521
  • 08:36 gehel: restarting stuck tilerator on maps* - T204047
  • 08:35 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/wbgetclaims$ sudo -u _graphite find . -type f -name "*.wsp" -delete # T140280
  • 08:11 moritzm: installing perl security updates on jessie/trusty (stretch already updated)
  • 07:49 godog: bootstrap cassandra-c on restbase2014 - T209615
  • 07:26 marostegui: Deploy schema change on s4 codfw master (db2051) with replication T86338 T202167
  • 07:22 marostegui: Deploy schema change on wikitech primary master (db1073) for labswiki and labtestwiki T86338 T202167
  • 06:39 marostegui: Deploy schema change on s5 primary master (db1070) T86338 T202167
  • 06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 T86338 T202167 (duration: 00m 49s)
  • 06:12 marostegui: Deploy schema change on db1110 T86338 T202167
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 T86338 T202167 (duration: 00m 53s)
  • 01:22 foks: Removing 2FA per request at https://phabricator.wikimedia.org/T210703
  • 01:22 foks: Reset password for user "Orangemike"

2018-12-03

  • 23:54 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/tests/: for completeness (duration: 00m 58s)
  • 23:53 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/resources/src/mediawiki.legacy/: Restore gray coloring for autocomments (T165189 part 2) (duration: 00m 47s)
  • 23:51 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Linker.php: Restore old HTML structure for history section links (T165189 part 1) (duration: 00m 47s)
  • 22:56 sbassett@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/AbuseFilter/includes/api/ApiQueryAbuseLog.php: Deploy security fix for T210329 (duration: 00m 47s)
  • 21:24 mutante: temp. disabling puppet on logstash1007 and logstash1008 to carefully deploy gerrit:476916
  • 21:17 XioNoX: push firewall change to pfw3-eqiad - T211028
  • 21:10 catrope@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: Fix ChangesListFilters validation errors (duration: 00m 49s)
  • 21:00 urandom: bootstrapping restbase2014-a -- T210843
  • 20:20 ppchelko@deploy1001: Finished deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema (duration: 01m 13s)
  • 20:19 ppchelko@deploy1001: Started deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema
  • 20:14 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build (duration: 09m 31s)
  • 20:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build
  • 20:05 XioNoX: push firewall change to pfw3-eqiad - T211028
  • 19:59 ppchelko@deploy1001: Finished deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change (duration: 01m 13s)
  • 19:58 ppchelko@deploy1001: Started deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change
  • 19:39 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/resources/mobile.toc/TableOfContents.js: SWAT T210869 Fix Table of contents rendering (duration: 00m 47s)
  • 19:30 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT I3b906e8b1 CS part of setting enhanced RC (duration: 00m 46s)
  • 19:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Ic2787309e59e IS part of setting enhanced RC (duration: 00m 47s)
  • 19:25 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Echo/modules/styles/mw.echo.ui.PaginationWidget.less: SWAT T210487 I914b94515 (duration: 00m 47s)
  • 19:15 jforrester@deploy1001: Synchronized wmf-config/ProductionServices.php: SWAT T210381 I73c7596818b Actual config (duration: 00m 46s)
  • 19:09 jforrester@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: SWAT T210381 I2ae162f5 Part II (duration: 00m 47s)
  • 19:08 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT T210381 I2ae162f5 Part I (duration: 00m 46s)
  • 18:42 anomie@deploy1001: Synchronized wmf-config/CommonSettings.php: Updating SkinBuildSidebar hook function for T210528 (duration: 00m 47s)
  • 18:08 gehel: add elastic2037 to cirrus eqiad (new server) - T210265
  • 18:06 godog: bootstrap cassandra-c on restbase2013 - T209615
  • 17:09 bstorm_: T207377 reboot labstore1006 for upgrades
  • 16:28 godog: poweroff ms-be2021 for battery replacement - T208269
  • 15:23 gehel: start configuration of elastic2037-2044 (new servers) - T210265
  • 15:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 T86338 T202167 (duration: 00m 46s)
  • 14:54 marostegui: Deploy schema change on db1100 T86338 T202167
  • 14:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 T86338 T202167 (duration: 00m 48s)
  • 14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 T86338 T202167 (duration: 00m 47s)
  • 14:17 marostegui: Deploy schema change on db1113:3315 T86338 T202167
  • 14:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 T86338 T202167 (duration: 00m 46s)
  • 14:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 T86338 T202167 (duration: 00m 46s)
  • 13:47 marostegui: Deploy schema change on db1082 (sanitarium master) with replication, lag will be generated on labs (s5) T86338 T202167
  • 13:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 T86338 T202167 (duration: 00m 47s)
  • 13:01 Lucas_WMDE: EU SWAT done
  • 12:44 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleaning of wgLogoHD (T150618) (duration: 00m 46s)
  • 12:39 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for multiple projects (T150618) (duration: 00m 47s)
  • 12:37 moritzm: installing nodejs security updates on scb1001
  • 12:34 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Close internalwiki (T205584) (duration: 00m 46s)
  • 12:28 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change sitename of shnwiki (T206777) (duration: 00m 47s)
  • 12:23 godog: bootstrap cassandra-b on restbase2013 - T209615
  • 12:19 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Perform more PHP constraint checks before falling back (T209504) (duration: 00m 48s)
  • 12:14 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Don’t send SPARQL prefixes in WikibaseQualityConstraints (T204317) (duration: 00m 49s)
  • 11:39 godog: more weight to new ms-be codfw hosts - T209395
  • 11:02 moritzm: installing nodejs security updates on stat/notebook hosts
  • 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 T86338 T202167 (duration: 00m 47s)
  • 10:52 moritzm: rolling upgrade of scb in codfw to nodejs security update
  • 10:39 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 10:39 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 47s)
  • 10:34 moritzm: installing nodejs security updates on scb2001
  • 10:31 marostegui: Deploy schema change on db1097:3315 T86338 T202167
  • 10:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 T86338 T202167 (duration: 00m 46s)
  • 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 T86338 T202167 (duration: 00m 45s)
  • 09:48 marostegui: Deploy schema change on db1096:3315 T86338 T202167
  • 09:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 T86338 T202167 (duration: 00m 47s)
  • 09:27 banyek: executing schema change on db1066 (s2 master) - T85757
  • 09:22 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1074 (duration: 00m 47s)
  • 09:16 banyek: repooling db1074 - T85757
  • 09:16 banyek: repooling db1074
  • 08:50 banyek: stopping replication on db1074 - T85757
  • 08:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1074 (duration: 00m 48s)
  • 08:44 godog: bootstrap cassandra-a on restbase2013 - T209615
  • 08:43 banyek: depooling db1074 - T85757
  • 08:32 moritzm: restarted keyholder agents/proxies on netmon1002/netmon2001 to pick up removal of netbox key
  • 08:30 marostegui: Deploy schema change on s5 codfw master (db2052) with replication, lag will be generated on codfw T86338 T202167
  • 08:13 moritzm: rearmed keyholders on netmon1002/netmon2001
  • 08:07 marostegui: Deploy schema change on s2 master (db1066) T86338 T202167
  • 07:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 T86338 T202167 (duration: 00m 47s)
  • 07:32 marostegui: Deploy schema change db1074 with replication (lag will appear on labs) T86338 T202167
  • 07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 T86338 T202167 (duration: 00m 46s)
  • 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 T86338 T202167 (duration: 00m 47s)
  • 07:09 marostegui: Stop MySQL on pc1004, pc1005 and pc1006 as they will be decommissioned - T210969
  • 06:52 marostegui: Remove pc1004, pc1005 and pc1006 from tendril and zarcillo - T210969
  • 06:38 marostegui: Deploy schema change db1122 T86338 T202167
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 T86338 T202167 (duration: 00m 48s)
  • 06:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 T86338 T202167 (duration: 00m 47s)
  • 06:16 marostegui: Deploy schema change db1076 T86338 T202167
  • 06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 T86338 T202167 (duration: 00m 50s)
  • 00:34 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Title.php: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/477182 (duration: 00m 52s)

2018-12-02

  • 22:39 addshore: addshore@mwmaint1002:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=testwikidatawiki addless # This is my account, and apparently I no longer have the 2fa for it

2018-12-01

  • 16:48 andrewbogott: rebuilding labvirt1014 as cloudvirt1014, T210904


Archives

See Server admin log/Archives.