You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(elukey: restart pdfrender on scb1002 (alarms flapping))
imported>Nhatminh01
(archive December)
Line 1: Line 1:
== 2018-12-30 ==
* 07:17 elukey: restart pdfrender on scb1002 (alarms flapping)
* 02:07 bawolff@deploy1001: Synchronized private/PrivateSettings.php: fine-tune antispam measure [[phab:T212667|T212667]] (duration: 00m 47s)
== 2018-12-29 ==
* 14:53 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T212667|T212667]] {{Gerrit|218371fd35}} - Adjust mw.org abusefilter emergency shutoff threshold down to 0.3 (duration: 00m 46s)
* 14:30 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T212667|T212667]] {{Gerrit|fe72284c}} Adjust account throttle limits (duration: 00m 46s)
* 13:30 bawolff@deploy1001: Synchronized private/PrivateSettings.php: [[phab:T212667|T212667]] - adjust spam block (duration: 00m 44s)
* 12:34 bawolff@deploy1001: Synchronized private/PrivateSettings.php: [[phab:T212667|T212667]] - make spam mitigation global (duration: 00m 49s)
* off: manually recreated /dev/log symlink on kubernetes1001, restarting systemd-journald.socket didn't worked (this should fix cron-spam emails from the host every hour)
* off: restarted pdfrender on scb1004
* 09:21 elukey: restart pdfrender on scb1004
== 2018-12-28 ==
* 18:14 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|97446843a27}} [[phab:T212667|T212667]] - Temp increase abusefilter emergency cutoff on mw.org to deal with spam attack (duration: 00m 46s)
* 18:02 bawolff@deploy1001: Synchronized private/PrivateSettings.php: [[phab:T212667|T212667]] - adjust account creation (duration: 00m 47s)
* 15:28 bawolff@deploy1001: Synchronized private/PrivateSettings.php: Attempt to adjust captcha settings for [[phab:T212667|T212667]] (duration: 00m 46s)
== 2018-12-27 ==
* 19:31 mobrovac@deploy1001: Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3 (duration: 25m 49s)
* 19:05 mobrovac@deploy1001: Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #3
* 18:15 mobrovac@deploy1001: Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays), take #2
* 18:15 mobrovac@deploy1001: Finished deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - [[phab:T212631|T212631]] (duration: 11m 20s)
* 18:04 mobrovac@deploy1001: Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - [[phab:T212631|T212631]]
* 18:04 mobrovac@deploy1001: deploy aborted: Fix rate-limiter crash (with increased deploy delays) - [[phab:T212631|T212631]] (duration: 00m 09s)
* 18:04 mobrovac@deploy1001: Started deploy [restbase/deploy@a67f38e]: Fix rate-limiter crash (with increased deploy delays) - [[phab:T212631|T212631]]
* 18:03 mobrovac@deploy1001: deploy aborted: Fix rate-limiter crash - [[phab:T212631|T212631]] (duration: 13m 09s)
* 17:50 mobrovac@deploy1001: Started deploy [restbase/deploy@70c4752]: Fix rate-limiter crash - [[phab:T212631|T212631]]
* 17:20 mobrovac@deploy1001: Finished deploy [restbase/deploy@ae7a537]: Fix rate-limiter crash - [[phab:T212631|T212631]] - deploy only on canary restbase1007 (duration: 04m 24s)
* 17:15 mobrovac@deploy1001: Started deploy [restbase/deploy@ae7a537]: Fix rate-limiter crash - [[phab:T212631|T212631]] - deploy only on canary restbase1007
* 16:00 cmjohnson1: powercycling frdb1001 for troubleshooting
* 15:21 bawolff: Updated interwiki cache https://biblio.wiki to https://wikilivres.org -> [[phab:T212650|T212650]]
* 15:19 bawolff@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 03m 34s)
== 2018-12-26 ==
* 16:37 godog: clear logstash persistent queue /var/lib/logstash on logstash100[789]
* 15:47 godog: roll-restart logstash on logstash100[789]
* 14:26 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=wtp1028.eqiad.wmnet
* 14:18 godog: powercycle wtp1028 - nothing on console
* 09:13 godog: swift eqiad-prod: more weight for ms-be10[44-50].eqiad.wmnet - [[phab:T209618|T209618]]
* 07:20 marostegui: Fix replication db1124:3318
== 2018-12-24 ==
* 05:38 _joe_: powercycling lvs3001
== 2018-12-23 ==
* 11:53 marostegui: Fix replication on db1124:3318
* 10:25 ariel@deploy1001: Finished deploy [dumps/dumps@af74350]: python3 fixup for show runtimes (duration: 00m 05s)
* 10:25 ariel@deploy1001: Started deploy [dumps/dumps@af74350]: python3 fixup for show runtimes
* 08:53 apergos: restarted pdfrender on scb1003
== 2018-12-22 ==
* 18:45 elukey: manually clean up of old log files on an-coord1001 (disk space issues)
* 15:58 godog: reboot ms-be2018, stuck on sd 0:1:0:1: rejecting I/O to offline device
* 10:21 cwd: re-enabled process-control
* 08:36 cwd: took down sidebar fr campaign
* 08:36 cwd: disabled process-control
* 01:00 AaronSchulz: Deployed {{Gerrit|b47e9fcfece99}} to navtiming
* 00:59 aaron@deploy1001: Finished deploy [performance/navtiming@b47e9fc]: (no justification provided) (duration: 00m 05s)
* 00:59 aaron@deploy1001: Started deploy [performance/navtiming@b47e9fc]: (no justification provided)
== 2018-12-21 ==
* 21:29 mutante: phab1002 - temp hack to unbreak phd / systemd alert, real fix will be phab deployment to new server
* 21:28 mutante: phab1002 - mkdir -p /srv/phab/libext/ava/src  ; touch  __phutil_library_init__.php
* 21:05 mutante: phab1002 - apt autoremove
* 21:04 mutante: phab1002 - removing all php related packages and letting puppet reinstall them
* 20:38 mutante: phab1002 - restart php-fpm, restart phd for testing. phd fails
* 18:50 mutante: [scb1001:~] $ sudo systemctl restart pdfrender
* 14:20 dcausse: elastic@eqiad deleting unused index enwiki_general_1537906513
* 13:24 moritzm: installing subversion updates from stretch point release
* 12:34 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: [[phab:T211494|T211494]] (duration: 00m 44s)
* 12:31 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T211494|T211494]] (duration: 00m 45s)
* 12:10 moritzm: rebooting url downloaders to pick up SSBD-enabled QEMU
* 11:38 moritzm: rebooting debug proxies to pick up SSBD-enabled QEMU
* 09:58 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe2006.codfw.wmnet
* 09:57 ema: repool ms-fe2006 with old certs, test successful [[phab:T212215|T212215]]#4839960
* 09:23 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe2006.codfw.wmnet
* 09:22 ema: depool ms-fe2006 to test new TLS certs [[phab:T212215|T212215]]
* 08:45 moritzm: upgrading nginx on sodium
* 08:41 moritzm: upgrading nginx on debug proxies
* 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove old comments [[phab:T211973|T211973]] (duration: 00m 46s)
* 00:52 twentyafterfour: SWAT Finished. See you all next year!
* 00:50 twentyafterfour@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/MobileFrontend/: SWAT: sync https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/481026 (duration: 00m 48s)
== 2018-12-20 ==
* 22:46 mutante: phab1001 / phabricator: upgraded nodejs package
* 22:39 mutante: phab1001 / phabricator: installing php5 package upgrades
* 19:29 kaldari@deploy1001: Synchronized wmf-config/Wikibase.php: syncing Wikibase for SWAT deployment (duration: 00m 45s)
* 19:27 kaldari@deploy1001: Synchronized wmf-config/InitialiseSettings.php: syncing InitialiseSettings for SWAT deployment (duration: 00m 46s)
* 19:03 elukey: restart hdfs namenode on an-master1002 with new heap settings (currently standby, 8->12G)
* 18:30 elukey: remove hdfs journalnode config+packages from analytics10(28{{!}}35) - not used anymore - [[phab:T209929|T209929]]
* 18:29 elukey: restart hdfs namenode on an-master1001 with new heap settings (currently standby, 8->12G)
* 18:06 mutante: doc1001 - meged gerrit:480881 and then manually moved the entire /srv/org/wikimedia/doc/ structure into /srv/docroot/srv/org/wikimedia/ and deleted the old dirs [[phab:T137890|T137890]]
* 17:53 arturo: updating puppet compiler facts: `PUPPET_COMPILER=compiler1002.puppet-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts`
* 17:49 arturo: updating puppet compiler facts: `PUPPET_COMPILER=compiler1001.puppet-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts`
* 17:05 XioNoX: add 208.80.155.88/29 to cloud-in4 term icmp - [[phab:T207663|T207663]]
* 17:02 XioNoX: configure additional 208.80.155.88/29 IPs on cloud-instance-transport1-b-eqiad - [[phab:T207663|T207663]]
* 16:59 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.9
* 16:51 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/Wikibase: SWAT: [[gerrit:480978{{!}}Revert "Fail hard if an entity namespace is not configured." (T212427)]] (duration: 01m 17s)
* 16:31 elukey: remove two journal nodes from the Analytics hadoop cluster - [[phab:T209929|T209929]]
* 14:53 moritzm: installing nodejs security updates on maps* (was tested via [[phab:T211419|T211419]])
* 14:41 moritzm: restarted etherpad for nodejs security updates
* 14:39 elukey: add two journal nodes to the Analytics Hadoop cluster - [[phab:T209929|T209929]]
* 14:30 moritzm: installing libdap updates from stretch point release
* 14:26 moritzm: rearmed keyholder on netmon1002 after reboot
* 14:22 moritzm: rebooting netmon1002 for kernel security update
* 13:09 moritzm: installing xapian-core updates from stretch point release
* 12:53 arturo: [[phab:T209616|T209616]] installing cloudvirt1030, icinga downtime for 1 day
* 12:48 kartik@deploy1001: Finished deploy [cxserver/deploy@16f65cb]: Update cxserver to {{Gerrit|803baa4}} ([[phab:T210581|T210581]], [[phab:T211889|T211889]], [[phab:T144467|T144467]], [[phab:T209473|T209473]]) (duration: 04m 42s)
* 12:47 moritzm: installing libxcursor security updates
* 12:43 kartik@deploy1001: Started deploy [cxserver/deploy@16f65cb]: Update cxserver to {{Gerrit|803baa4}} ([[phab:T210581|T210581]], [[phab:T211889|T211889]], [[phab:T144467|T144467]], [[phab:T209473|T209473]])
* 12:31 moritzm: installing fuse updates from stretch point release
* 12:09 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: [[gerrit:478659{{!}} Bumping portals to master (T128546, T202497)]] (duration: 00m 52s)
* 12:08 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:478659{{!}} Bumping portals to master (T128546, T202497)]] (duration: 00m 53s)
* 12:02 moritzm: draining restbase1016 for eventual reboot for kernel security update
* 11:46 moritzm: draining restbase1015 for eventual reboot for kernel security update
* 11:34 moritzm: powercycling restbase1014, similar EFI ASSERT error to [[phab:T212305|T212305]]
* 11:34 moritzm: powercycling restbase1014, similar EFI ASSEER error to [[phab:T212305|T212305]]
* 11:21 moritzm: draining restbase1014 for eventual reboot for kernel security update
* 10:59 banyek: executing schema change on db1068 (s4 master) - [[phab:T85757|T85757]]
* 10:58 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1121 after schema change - [[phab:T85757|T85757]] (duration: 00m 52s)
* 10:53 banyek: repooling db1121 after schema change [[phab:T85757|T85757]]
* 10:45 moritzm: draining restbase1013 for eventual reboot for kernel security update
* 10:44 banyek: stopping replication on db1121 - [[phab:T85757|T85757]]
* 10:40 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1121 for schema change - [[phab:T85757|T85757]] (duration: 00m 52s)
* 10:35 banyek: depooling db1121 for schema change [[phab:T85757|T85757]]
* 10:33 banyek: executing schema change on dbstore1002 - [[phab:T85757|T85757]]
* 10:22 banyek: executing schema change on db1102 - [[phab:T85757|T85757]]
* 10:20 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1081 after schema change - [[phab:T85757|T85757]] (duration: 00m 51s)
* 10:19 moritzm: draining restbase1012 for eventual reboot for kernel security update
* 10:15 banyek: repooling db1081 after schema change [[phab:T85757|T85757]]
* 10:10 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1081 for schema change - [[phab:T85757|T85757]] (duration: 00m 51s)
* 10:05 banyek: depooling db1081 for schema change [[phab:T85757|T85757]]
* 10:00 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1103:3314 after schema change - [[phab:T85757|T85757]] (duration: 00m 52s)
* 09:56 banyek: repooling db1103:3314 after schema change [[phab:T85757|T85757]]
* 09:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1103:3314 for schema change - [[phab:T85757|T85757]] (duration: 00m 51s)
* 09:46 banyek: depooling db1103:3314 for schema change [[phab:T85757|T85757]]
* 09:45 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1097:3314 after schema change - [[phab:T85757|T85757]] (duration: 00m 51s)
* 09:41 legoktm@deploy1001: Synchronized php-1.33.0-wmf.8/includes/: [[phab:T199540|T199540]] (duration: 01m 06s)
* 09:40 legoktm@deploy1001: Synchronized php-1.33.0-wmf.9/includes/: [[phab:T199540|T199540]] (duration: 01m 14s)
* 09:39 banyek: repooling db1097:3314 after schema change [[phab:T85757|T85757]]
* 09:35 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T187299|T187299]] Increase ruwiki navtiming rate + frwiki survey rate (duration: 00m 52s)
* 09:29 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1097:3314 for schema change - [[phab:T85757|T85757]] (duration: 00m 52s)
* 09:23 banyek: depooling db1097:3314 for schema change [[phab:T85757|T85757]]
* 08:06 elukey: roll restart of druid middlemanagers on druid* to pick up new port settings
* 07:17 marostegui: Re-start codfw s4 backup as the previous one failed
* 07:11 elukey: restart pdfrender on scb1002
* 07:10 elukey: restart rsyslog on lithium - in:imtcp stuck in recvfrom ms-be2047.codfw.wmnet  - [[phab:T199406|T199406]]
* 06:35 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2057 [[phab:T212277|T212277]] (duration: 00m 57s)
* 01:41 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@c7977a7]: Update mobileapps to {{Gerrit|42c011e}} (duration: 04m 08s)
* 01:37 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@c7977a7]: Update mobileapps to {{Gerrit|42c011e}}
* 01:09 eileen_: civicrm revision changed from {{Gerrit|9d727e4708}} to {{Gerrit|b33dcd3c94}}, config revision is {{Gerrit|0f94a475b7}}
* 00:09 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/479911/ (duration: 00m 53s)
== 2018-12-19 ==
* 23:24 mutante: syncing facts from puppetmaster1001 to compiler1001/compiler1002
* 23:21 eileen_: civicrm revision changed from {{Gerrit|d5c3d5fd17}} to {{Gerrit|9d727e4708}}, config revision is {{Gerrit|0f94a475b7}}
* 23:05 krinkle@deploy1001: Finished deploy [performance/navtiming@64e3f63]: (no justification provided) (duration: 00m 05s)
* 23:05 krinkle@deploy1001: Started deploy [performance/navtiming@64e3f63]: (no justification provided)
* 22:15 mutante: scb1003: systemctl restart pdfrender
* 20:44 hashar: 1.33.0-wmf.9 on group1 looks fine.
* 20:03 hashar@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.9 (duration: 00m 51s)
* 20:02 hashar@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.9
* 19:33 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad"  - [[phab:T210467|T210467]]
* 19:32 XioNoX: repool codfw - [[phab:T210467|T210467]]
* 19:28 XioNoX: reactive BGP sessions to telia on cr1-codfw - [[phab:T211715|T211715]]
* 19:07 chasemp: cp of files to ext drive on labstore1007
* 18:41 marostegui: Stop MySQL on db2057
* 18:28 XioNoX: replace `interface-range vlan-private1-b-eqiad member ge-6/0/*` with individual interfaces on asw2-b-eqiad
* 18:09 ejegg: updated fundraising CiviCRM from {{Gerrit|8e18485697}} to {{Gerrit|d5c3d5fd17}}
* 17:32 addshore: SWAT done
* 17:22 addshore@deploy1001: Synchronized wmf-config: Wikibase: wikidatawiki upsert idGenerator, [[phab:T194299|T194299]] (duration: 00m 52s)
* 17:13 addshore@deploy1001: Synchronized wmf-config: Wikibase: testwikidatawiki upsert idGenerator, [[phab:T194299|T194299]] (duration: 00m 52s)
* 17:05 XioNoX: remove 2nd port to AS8220 (cf. email to peering@)
* 17:04 addshore@deploy1001: Synchronized wmf-config: Wikibase: prepare to set $wgWBRepoSettings idGenerator, [[phab:T194299|T194299]] (duration: 00m 53s)
* 16:59 XioNoX: deactive BGP sessions to telia on cr1-codfw - [[phab:T211715|T211715]]
* 16:58 fsero: DNS: updating wmnet to include new registries [[phab:T212212|T212212]]
* 16:57 hashar@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/ExtensionDistributor/includes/specials/SpecialBaseDistributor.php: Follow-up {{Gerrit|f686d348}}: No need for an <img> tag any more - [[phab:T212217|T212217]] (duration: 00m 52s)
* 16:52 XioNoX: codfw row D maintenance finished without issues - [[phab:T210467|T210467]]
* 16:39 cmjohnson1: swapping disk in slot 2 on db1072
* 16:33 XioNoX: shutdown asw-d4-codfw - [[phab:T210467|T210467]]
* 16:21 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting comment migration to new on group 2 ([[phab:T166733|T166733]]) (duration: 00m 52s)
* 16:04 XioNoX: Redirect eqsin/ulsfo caches to eqiad - [[phab:T210467|T210467]]
* 16:01 moritzm: installing php5 security updates on jessie
* 15:48 XioNoX: depool codfw - [[phab:T210467|T210467]]
* 15:39 chasemp: various cp jobs on labstore1007 to ext media
* 15:33 chasemp: labstore1007 mount /dev/sde /mnt/T211327
* 15:13 moritzm: draining restbase1011 for eventual reboot for kernel security update
* 15:07 marostegui: Drop image_comment_temp from labswiki and labtestwiki - [[phab:T209591|T209591]]
* 14:58 moritzm: draining restbase1010 for eventual reboot for kernel security update
* 14:42 moritzm: draining restbase1009 for eventual reboot for kernel security update
* 14:16 moritzm: draining restbase1008 for eventual reboot for kernel security update
* 14:03 moritzm: draining restbase1007 for eventual reboot for kernel security update
* 13:57 moritzm: installing nodejs updates on wtp*
* 13:36 marostegui: Correction from the previous !log: Rename table valid_tag on db1089 (s1) - [[phab:T212254|T212254]]
* 13:35 marostegui: Rename table valid_tag on db1081 (s1) - [[phab:T212254|T212254]]
* 13:31 marostegui: Drop image_comment_temp on s4 - [[phab:T209591|T209591]]
* 12:46 marostegui: Drop image_comment_temp on s3 - [[phab:T209591|T209591]]
* 12:24 zeljkof: EU SWAT finished
* 12:23 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.9/extensions/Kartographer/: SWAT: [[gerrit:480687{{!}}Fix using at-ease functions in namespaced class (T212218)]] (duration: 00m 53s)
* 12:06 moritzm: rearmed keyholder after netmon2001 reboot
* 11:58 moritzm: rebooting netmon2001 for kernel security update
* 11:50 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1091 after schema change - [[phab:T85757|T85757]] (duration: 00m 52s)
* 11:49 moritzm: rebooting matomo1001 to pick up SSBD-enabled qemu
* 11:46 banyek: repooling db1091 after schema change - [[phab:T85757|T85757]]
* 11:41 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1091 for schema change - [[phab:T85757|T85757]] (duration: 00m 52s)
* 11:37 banyek: depooling db1091 for schema change - [[phab:T85757|T85757]]
* 11:34 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1084 after schema change - [[phab:T85757|T85757]] (duration: 00m 51s)
* 11:30 banyek: repooling db1084 after schema change - [[phab:T85757|T85757]]
* 11:29 moritzm: upgrading nodejs on restbase2013-2018
* 11:26 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2050 after recloning db2057 [[phab:T212275|T212275]] (duration: 00m 52s)
* 11:25 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1084 for schema change - [[phab:T85757|T85757]] (duration: 00m 52s)
* 11:15 banyek: depooling db1084 for schema change - [[phab:T85757|T85757]]
* 11:13 marostegui: Stop MySQL and power off db2057 for firmware upgrade - [[phab:T212277|T212277]]
* 11:11 marostegui: Drop image_comment_temp from s5 - [[phab:T209591|T209591]]
* 10:37 moritzm: draining restbase2012 for eventual reboot for kernel security update
* 10:28 banyek: executing schema change in db2051 (s4 codfw master) with replication enabled - [[phab:T85757|T85757]]
* 10:25 banyek: stopping replication on db2073 as executing schema change on codfw master - [[phab:T85757|T85757]]
* 10:14 moritzm: draining restbase2011 for eventual reboot for kernel security update
* 09:53 banyek: dropping tables 'flagged%' on db1066 ptwiki with replication enabled - [[phab:T211544|T211544]]
* 09:41 moritzm: draining restbase2010 for eventual reboot for kernel security update
* 09:37 banyek: dropping tables with '[[phab:T211544|T211544]]' prefix on db1122 - [[phab:T211544|T211544]]
* 09:24 moritzm: draining restbase2009 for eventual reboot for kernel security update
* 09:14 moritzm: rebooting restbase2008 for kernel security update
* 08:53 elukey: roll restart of cassandra on aqs1005-1009 for opendjdk upgrades
* 08:50 marostegui: Drop image_comment_temp from s7 - [[phab:T209591|T209591]]
* 08:46 marostegui: Drop image_comment_temp from s6 - [[phab:T209591|T209591]]
* 08:43 akosiaris: rebalance row_A ganeti01.svc.codfw.wmnet nodegroup after recabling [[phab:T210447|T210447]]
* 08:40 marostegui: Drop image_comment_temp from s8 - [[phab:T209591|T209591]]
* 08:37 moritzm: draining restbase2007 for eventual reboot for kernel security update
* 08:28 marostegui: Drop image_comment_temp from s2 - [[phab:T209591|T209591]]
* 08:27 marostegui: Drop image_comment_temp from s1 - [[phab:T209591|T209591]]
* 08:16 godog: swift eqiad-prod: more weight for ms-be10[44-50].eqiad.wmnet - [[phab:T209618|T209618]]
* 07:37 marostegui: Drop nodepooldb on m5 master - [[phab:T212230|T212230]]
* 07:22 marostegui: Stop MySQL on db2050 to clone db2057 - [[phab:T212275|T212275]]
* 07:15 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2050 to clone db2057 [[phab:T212275|T212275]] (duration: 00m 52s)
* 07:00 marostegui: Enable GTID on s8 codfw master (db2045) - [[phab:T211973|T211973]]
* 06:58 marostegui: Enable GTID on s1 codfw master (db2048) - [[phab:T211973|T211973]]
* 06:44 marostegui: Remove nodepool@10.64.16.155 user from m5 master - [[phab:T212230|T212230]]
* 06:34 marostegui: Hard reboot db2057 - [[phab:T212275|T212275]]
* 06:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2057 - storage crashed [[phab:T212275|T212275]] (duration: 01m 08s)
* 01:27 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: g 480681 make maintenance script dry run more useful (duration: 00m 52s)
* 01:25 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/includes/AbuseFilter.php: g 480680 fix exception in maintenance script (duration: 00m 54s)
* 00:11 mutante: contint1001 - rsyncing /srv/org/wikimedia/docs to rsync://docs1001.eqiad.wmnet/docs [[phab:T211974|T211974]]
== 2018-12-18 ==
* 23:49 XioNoX: remove BGP session to AS50629 from cr2-esams (not in AMS-IX anymore)
* 20:33 otto@deploy1001: Finished deploy [eventlogging/analytics@104adb5]: Send JSON string of event for validation errors in EventError (duration: 00m 04s)
* 20:33 otto@deploy1001: Started deploy [eventlogging/analytics@104adb5]: Send JSON string of event for validation errors in EventError
* 19:43 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@963d704]: Parse summaries from lead objects only ([[phab:T202642|T202642]]) (duration: 05m 26s)
* 19:38 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@963d704]: Parse summaries from lead objects only ([[phab:T202642|T202642]])
* 19:30 robh: migration of ulsfo pdus complete
* 19:19 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:480106{{!}}Production configuration for GrowthExperiments Help Panel]] (duration: 00m 52s)
* 19:06 XioNoX: redirect ns1 back to authdns2001 - [[phab:T210447|T210447]]
* 18:51 moritzm: installing libx11 security updates
* 18:48 XioNoX: depool ulsfo - [[phab:T209101|T209101]]
* 18:42 XioNoX: repool codfw  - [[phab:T210447|T210447]]
* 18:41 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad"  - [[phab:T210447|T210447]]
* 18:05 chasemp: stat1004:~# umount /mnt/T211327
* 17:58 XioNoX: shutdown fpc4 for replacement - [[phab:T210447|T210447]]
* 17:51 bblack: deploying php7 cache-splitter patch to cache_text - https://gerrit.wikimedia.org/r/c/operations/puppet/+/478680 - [[phab:T206339|T206339]]
* 17:49 godog: bounce rsyslog on lithium, tls listener timeout
* 17:34 volans: triggered restart of ircecho on icinga1001 while applying https://gerrit.wikimedia.org/r/480509
* 16:37 jijiki: librsvg* 2.40.20-3+wmf1+stretch1 uploaded to components/thumor to stretch-wikimedia - [[phab:T209886|T209886]]
* 16:22 bblack: powercycle cp1075 from console (crashed, apparently)
* 16:07 XioNoX: starting codfw row A recabling - [[phab:T210447|T210447]]
* 16:05 ejegg: updated fundraising python tools from {{Gerrit|af5dbee8eb}} to {{Gerrit|5f44d9dd43}}
* 15:53 XioNoX: redirect ns1 to authdns1001 for [[phab:T210447|T210447]]
* 15:47 XioNoX: redirect eqsin/ulsfo caches to eqiad for [[phab:T210447|T210447]]
* 15:44 XioNoX: depool codfw for [[phab:T210447|T210447]]
* 15:30 akosiaris: empty ganeti2005, ganeti2006 for [[phab:T210447|T210447]]
* 15:16 Amir1: mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=testwikidatawiki --config-format=wgConf {{!}} tee WikibaseQualityConstraints-config.php
* 14:55 akosiaris: restart pybal on lvs1006, lvs2003 for blubberoid LVS deployment. [[phab:T205919|T205919]]
* 14:40 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.9
* 14:26 akosiaris: restart ircecho, seems to have croaked with Dec 17 19:39:52 icinga1001 ircecho[861]: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 174: ordinal not in range(128)
* 14:25 zfilipin@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.9 and rebuild l10n cache (duration: 34m 11s)
* 14:15 akosiaris: restart pybal on lvs1016, lvs2006 for blubberoid LVS deployment. [[phab:T205919|T205919]]
* 13:51 zfilipin@deploy1001: Started scap: testwiki to php-1.33.0-wmf.9 and rebuild l10n cache
* 13:12 addshore@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/Wikibase/lib/includes/Formatters/ControlledFallbackEntityIdFormatter.php: [[phab:T201930|T201930]] ControlledFallbackEntityIdFormatter, track unique value formats (duration: 00m 46s)
* 12:52 Amir1: deployed a patch on wmf.8 for [[phab:T207814|T207814]]
* 12:41 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 45s)
* 12:17 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:478630{{!}}Perform even more PHP constraint checks before falling back (T209504)]] (duration: 00m 46s)
* 12:16 moritzm: installing fuse updates from stretch point release
* 11:41 moritzm: installing remaining libgd2 security updates
* 09:38 godog: swift eqiad-prod: initial weights for ms-be10[44-50].eqiad.wmnet - [[phab:T209618|T209618]]
* 08:44 marostegui: Enable GTID on s7 codfw master (db2040) - [[phab:T211973|T211973]]
* 08:26 marostegui: Enable GTID on es3 - [[phab:T211973|T211973]]
* 08:23 marostegui: Enable GTID on es2 - [[phab:T211973|T211973]]
* {{safesubst:SAL entry|1=07:57 elukey: restart cassandra-{a,b} on aqs1004 for openjdk upgrades}}
* 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 07:49 marostegui: Deploy schema change on db1075 (s3 master) [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:07 marostegui: Deploy schema change on db1078 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 06:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2038 after mysql and kernel upgrade (duration: 00m 47s)
* 00:13 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:479997{{!}}Disable MediaViewer thumbnail URL guessing for private wikis (T212099)]] (duration: 00m 45s)
== 2018-12-17 ==
* 23:21 tgr: deployed security patch per [[phab:T207750|T207750]]#4829275
* 23:17 chasemp: launch gpg jobs for files on stat1004
* 23:17 chasemp: launch gpg jobs for files on labstore1007
* 22:00 mutante: puppetmaster - signed cert for doc1001 (ganeti VM), initial puppet run
* 21:58 chasemp: stat1004:/var/log# mkfs.exfat /dev/sde && mkdir /mnt/T211327 && mount /dev/sde /mnt/T211327/
* 21:55 arlolra: Updated Parsoid to {{Gerrit|4eba44e}} ([[phab:T204622|T204622]], [[phab:T211941|T211941]])
* 21:42 arlolra@deploy1001: Finished deploy [parsoid/deploy@1bf4dab]: Updating Parsoid to {{Gerrit|4eba44e}} (duration: 08m 55s)
* 21:33 arlolra@deploy1001: Started deploy [parsoid/deploy@1bf4dab]: Updating Parsoid to {{Gerrit|4eba44e}}
* 21:20 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@92fdf43]: Update mobileapps to {{Gerrit|d244439}} (duration: 04m 31s)
* 21:15 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@92fdf43]: Update mobileapps to {{Gerrit|d244439}}
* 20:25 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@c1b6b32]: Rollback to {{Gerrit|c1b6b324015bd565}} until the checks are fixed (duration: 01m 56s)
* 20:23 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@c1b6b32]: Rollback to {{Gerrit|c1b6b324015bd565}} until the checks are fixed
* 20:17 mutante: creating new ganeti VM doc1001.eqiad.wmnet for doc.wikimedia.org - specs as requested by hashar on [[phab:T211974|T211974]]
* 20:05 XioNoX: remove sandbox-out4 from all routers - [[phab:T212155|T212155]]
* 20:02 XioNoX: remove sandbox-out4 from ulsfo - [[phab:T212155|T212155]]
* 19:47 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@f183af7]: Update to 657a515. All hosts (duration: 10m 45s)
* 19:36 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@f183af7]: Update to 657a515. All hosts
* 19:33 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@f183af7]: Update to 657a515. Canary on scb2001 (duration: 00m 18s)
* 19:32 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@f183af7]: Update to 657a515. Canary on scb2001
* 19:29 raynor: Morning SWAT finished
* 19:27 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:480127 {{!}} Page issues treatment enabled on all wikis except enwiki(T210553)]] (duration: 00m 45s)
* 18:58 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@8036f9b]: Update to 2991db1. Canary on scb2001 (duration: 00m 51s)
* 18:57 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@8036f9b]: Update to 2991db1. Canary on scb2001
* 18:42 herron: manually restarting pybal on lvs2003 to add kibana service [[phab:T205850|T205850]]
* 18:34 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2006.codfw.wmnet
* 18:34 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2005.codfw.wmnet
* 18:34 mobrovac@deploy1001: Finished deploy [proton/deploy@ff7c8a2]: Config: Use extdomain and add printable=yes to the req query - [[phab:T210793|T210793]] (duration: 01m 32s)
* 18:32 herron: performing manual puppet run and manually restarting pybal on lvs2006 to add kibana service [[phab:T205850|T205850]]
* 18:32 mobrovac@deploy1001: Started deploy [proton/deploy@ff7c8a2]: Config: Use extdomain and add printable=yes to the req query - [[phab:T210793|T210793]]
* 18:29 gtirloni: prometheus-node-exporter: Ignore Docker/Kubelet mount points [[phab:T211810|T211810]]
* 18:26 akosiaris@deploy1001: scap-helm blubberoid finished
* 18:26 akosiaris@deploy1001: scap-helm blubberoid cluster codfw completed
* 18:26 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.2.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: codfw]
* 18:25 akosiaris@deploy1001: scap-helm blubberoid finished
* 18:25 akosiaris@deploy1001: scap-helm blubberoid cluster eqiad completed
* 18:25 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.1.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad]
* 18:24 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.externalIP=10.2.1.31 --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad]
* 18:23 akosiaris@deploy1001: scap-helm blubberoid install --name production --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production --set service.deployment=production --set service.port=8748 stable/blubberoid [namespace: blubberoid, clusters: eqiad,codfw]
* 18:21 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@69c000a]: DelayQueue fix and mitigation of stale index for updater (duration: 11m 52s)
* 18:21 moritzm: installing libgd2 security updates
* 18:10 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@69c000a]: DelayQueue fix and mitigation of stale index for updater
* 18:07 akosiaris@deploy1001: scap-helm blubberoid finished
* 18:07 akosiaris@deploy1001: scap-helm blubberoid cluster staging completed
* 18:07 akosiaris@deploy1001: scap-helm blubberoid install --name staging --set docker.registry=docker-registry.discovery.wmnet --set main_app.version=2018-12-13-183249-production stable/blubberoid [namespace: blubberoid, clusters: staging]
* 18:04 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2004.codfw.wmnet
* 16:41 moritzm: installing libarchive security updates on jessie
* 16:33 moritzm: installing php5 security updates on jessie
* 16:13 anomie: Aborted migrateActors.php run, queries were too slow.
* 16:07 anomie@mwmaint1002: Running migrateActors.php on test wikis and mediawikiwiki for [[phab:T188327|T188327]]
* 16:07 mobrovac@deploy1001: Started restart [electron-render/deploy@94d27d7]: Electron stuck
* 14:33 marostegui: Upgrade mysql and kernel on db2093 (tendril sby host on codfw)
* 14:10 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere [[phab:T201837|T201837]] (duration: 00m 44s)
* 14:09 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 200million everywhere [[phab:T201837|T201837]] (duration: 00m 44s)
* 14:06 marostegui: Stop MySQL on db2038 for mysql and kernel upgrade
* 14:06 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2038 for mysql and kernel upgrade (duration: 00m 44s)
* 13:58 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 10million for wikidatawiki [[phab:T201837|T201837]] (duration: 00m 45s)
* 13:49 marostegui: Enable GTID on s3 codfw master db2043 - [[phab:T211973|T211973]]
* 13:49 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmgWikibaseMaxItemIdForNewItemIdHtmlFormatter 1million for wikidatawiki [[phab:T201837|T201837]] (duration: 00m 45s)
* 13:46 fsero: installing new version of php-excimer on mwdebug* - [[phab:T205059|T205059]]
* 13:41 fsero: installing new version of php-excimer on mwdebug2001 - [[phab:T205059|T205059]]
* 13:31 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki incubatorwiki --force-protocol https
* 13:29 Amir1: deleting everything from site_identifiers@incubatorwiki
* 13:15 ariel@deploy1001: Finished deploy [dumps/dumps@0393ca7]: switch dumps to python3 (duration: 00m 03s)
* 13:15 ariel@deploy1001: Started deploy [dumps/dumps@0393ca7]: switch dumps to python3
* 13:04 Amir1: ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https ([[phab:T209820|T209820]])
* 13:04 Amir1: ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https
* 13:02 Amir1: ladsgroup@mwmaint1002:~$ mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php --wiki=fawiki --force-protocol https
* 12:52 ema: lvs1016: bounce pybal for new service kartotherian-ssl [[phab:T211970|T211970]]
* 12:50 ladsgroup@deploy1001: Synchronized wmf-config/InterwikiSortOrders.php: Update InterwikiSortOrders.php ([[phab:T209820|T209820]]) (duration: 00m 45s)
* 12:49 ladsgroup@deploy1001: sync-file aborted: Use the right index for change_tag ([[phab:T211896|T211896]]) (duration: 00m 02s)
* 12:38 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:480004{{!}}Add be.wikisource new 1.5x and 2x logos to wgLogoHD (T150618)]] (duration: 00m 44s)
* 12:38 ema: lvs2003: bounce pybal for new service kartotherian-ssl [[phab:T211970|T211970]]
* 12:29 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: [[gerrit:479713{{!}}Update be.wikisource logo (T211795)]] (duration: 00m 45s)
* 12:29 ema: lvs1006: bounce pybal to pick up new service kartotherian-ssl [[phab:T211970|T211970]]
* 12:24 ema: lvs2006: bounce pybal to pick up new service kartotherian-ssl
* 12:22 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:479723{{!}}Enable extendedmover user group at ur.wiki (T211978)]] (duration: 00m 45s)
* 12:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:479963{{!}}Use new minerva logos for cswiki in IS.php (T210979)]] (duration: 00m 45s)
* 12:08 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/: SWAT: [[gerrit:479379{{!}}Upload custom minerva logo for cswiki (T210979)]] (duration: 00m 44s)
* 12:02 ladsgroup@deploy1001: Finished deploy [ores/deploy@18d3657]: [[phab:T206333|T206333]] [[phab:T211267|T211267]] (duration: 14m 14s)
* 11:48 ladsgroup@deploy1001: Started deploy [ores/deploy@18d3657]: [[phab:T206333|T206333]] [[phab:T211267|T211267]]
* 10:52 paravoid: reprepro include python 3.4, 3.6, 3.7 to component/pyall (use with care)
* 10:28 moritzm: installing openssl security updates on stretch
* 10:18 marostegui: Repool labsdb1011 - [[phab:T86338|T86338]]
* 09:55 moritzm: remove debmonitor entries for restbase2001-restbase2006
* 09:44 marostegui: Enable GTID on s4 codfw master - [[phab:T211973|T211973]]
* 09:08 marostegui: Depool labsdb1011 - [[phab:T86338|T86338]]
* 09:04 marostegui: Repool labsdb1010 - [[phab:T86338|T86338]]
* 09:01 elukey: stop kafkatee on oxygen and rsync /srv/log data to weblog1001
* 08:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 08:21 marostegui: Deploy schema change on db1077 with replication (lag will be generated on labsdb:s3) [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 08:16 marostegui: Stop replication on labsdb1011
* 08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 07:58 marostegui: Enable GTID on db2052 (s5 master) - [[phab:T211973|T211973]]
* 06:48 marostegui: Enable GTID on db2035 (s2 master) - [[phab:T211973|T211973]]
* 06:36 marostegui: Enable GTID on db2034 (x1 master) - [[phab:T211973|T211973]]
* 06:36 marostegui: Enable GTID on db2034 (x1 master)
* 06:29 marostegui: Enable replication consistency options on codfw masters - [[phab:T211973|T211973]]
* 06:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 06:28 marostegui: Depool labsdb1010 - [[phab:T86338|T86338]]
* 04:22 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/maintenance/normalizeThrottleParameters.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 44s)
* 04:20 tstarling@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/AbuseFilter/includes/AbuseFilter.php: gerrit 479998 for testing of normalizeThrottleParameters.php (duration: 00m 59s)
== 2018-12-16 ==
* 09:52 elukey: mask + reset-failed kafkatee default instance on sulfur (kafkatee-webrequest works fine)
* 07:30 marostegui: Reboot db1115 after OOM
* 07:28 marostegui: Stop MySQL on db1115 so tendril can get back to work
== 2018-12-15 ==
* off: restarted pdfrender on scb1004
* 09:22 elukey: mask + reset-failed kafkatee default instance on weblog1001
== 2018-12-14 ==
* 23:59 mutante: LDAP: added aezell to wmf group ([[phab:T211945|T211945]]) for grafana access
* 22:00 XioNoX: increase accepted-prefix-limit for HE to 200000
* 20:38 mutante: sulfur systemctl restart nagios-nrpe-server
* 20:32 andrew@deploy1001: Finished deploy [horizon/deploy@1a830b9]: Rolling out fix for [[phab:T177855|T177855]] (duration: 03m 17s)
* 20:29 andrew@deploy1001: Started deploy [horizon/deploy@1a830b9]: Rolling out fix for [[phab:T177855|T177855]]
* 20:13 otto@deploy1001: Finished deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for [[phab:T211833|T211833]] (duration: 06m 04s)
* 20:07 otto@deploy1001: Started deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for [[phab:T211833|T211833]]
* 20:07 otto@deploy1001: deploy aborted: (no justification provided) (duration: 00m 00s)
* 20:07 otto@deploy1001: Started deploy [analytics/refinery@ef1f7c6]: (no justification provided)
* 17:22 cmjohnson1: mw1272 down for h/w troubleshooting
* 14:38 marostegui: Enable GTID on db2039 (s6 codfw master) - [[phab:T211973|T211973]]
* 14:03 marostegui: Compare ruwiki.revision between db2039 (s6 master) and db1085 - [[phab:T211973|T211973]]
* 13:59 marostegui: Enable notifications for db1095 (s3 lag check)- [[phab:T211973|T211973]]
* 13:57 marostegui: Enable notifications for db2068 (s7 lag check)- [[phab:T211973|T211973]]
* 13:47 dcausse: elastic@codfw copying index data from the main cluster to psi & omega (test disk usage & import speed)
* 13:42 marostegui: Enable GTID on db1124:3318 - [[phab:T211973|T211973]]
* 11:11 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2083 db2068 db2067 after mysql and kernel upgrade (duration: 00m 44s)
* 10:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2083 db2068 db2067 for mysql and kernel upgrade (duration: 00m 45s)
* 10:50 marostegui: Stop MySQL on db2083, db2068 and db2067 for mysql and kernel upgrade
* 10:19 filippo@deploy1001: Synchronized wmf-config/logging.php: wmf-config/InitialiseSettings-labs.php (duration: 00m 44s)
* 10:18 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 45s)
* 09:33 marostegui: Deploy schema change on db1095:3313 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 09:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2049, db2059, db2070 after mysql and kernel upgrade (duration: 00m 45s)
* 09:21 banyek: global user rename is in progress - [[phab:T209488|T209488]]
* 08:58 marostegui: Stop MySQL on db2049, db2059 and db2070 for mysql and kernel upgrade
* 08:56 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2049, db2059, db2070 for mysql and kernel upgrade (duration: 00m 43s)
* 08:50 elukey: swap oxygen with weblog1001
* 08:48 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2084 and db2088 after mysql and kernel upgrade (duration: 00m 44s)
* 08:47 elukey: disabled kafkatee-webrequest logstash output on oxygen (prep step before weblog1001)
* 07:59 marostegui: Deploy schema change on dbstore1002:s3 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 07:49 marostegui: Upgrade mysql and kernel on db2084 and db2088
* 07:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2084 and db2088 for mysql and kernel upgrade (duration: 00m 45s)
* 06:54 marostegui: Deploy schema change on db2043 (s3 codfw master) - this will generate lag on s3 codfw [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:53 marostegui: Deploy schema change on db2043 (s3 codfw master) - this will generate lag on s3 codfw [[phab:T86338|T86338]] [[phab:T20216|T20216]]
* 06:14 marostegui: Deploy schema change on db1062 (s7 primary master) [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 44s)
* 06:10 marostegui: Deployed schema change on db1094 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 05:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 01:39 mutante: install2002 deleted /srv/ contents,then mounted /mnt/vdb on /srv  so same content but now / is used only 7% and /srv 57% ([[phab:T211850|T211850]])
* 00:43 mutante: install2002 ([[phab:T211850|T211850]]) restarted instance, created ext4 filesystem on new /dev/vdb, mounted on /mnt/vdb, rsyncing /srv/ to /mnt/vdb/
* 00:33 mutante: rebooting install2002 via ganeti2003, to add new virtual disk
* 00:31 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Enforce a 10-byte password for +staff users, {{Gerrit|I4ecac70e}} (duration: 00m 44s)
* 00:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT {{Gerrit|I7867277d}} Make wmgBabelMainCategory consistent for sr* wikis (duration: 00m 45s)
== 2018-12-13 ==
* 23:57 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: [[phab:T208246|T208246]]: Increase default minimum new password length to 10 for privileged groups (duration: 00m 44s)
* 23:17 andrew@deploy1001: Finished deploy [horizon/deploy@18c4ca6]: Rolling out fix for [[phab:T131367|T131367]] (duration: 03m 25s)
* 23:13 andrew@deploy1001: Started deploy [horizon/deploy@18c4ca6]: Rolling out fix for [[phab:T131367|T131367]]
* 22:31 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/HttpRequestFactory.php: [[phab:T211886|T211886]] (duration: 00m 44s)
* 22:29 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/MobileFrontend: [[phab:T211903|T211903]] (duration: 00m 48s)
* 21:55 mutante: Ganeti - creating new 120G virtual disk on install2002 ([[phab:T211850|T211850]])
* 21:00 otto@deploy1001: Finished deploy [analytics/superset/deploy@UNKNOWN]: revert to version 0.26.3 (duration: 00m 32s)
* 20:59 otto@deploy1001: Started deploy [analytics/superset/deploy@UNKNOWN]: revert to version 0.26.3
* 20:33 dcausse: creating 300+ wikis indices in elastic-psi @eqiad and @codfw
* 20:21 volans: imported elasticsearch-curator_5.2.0-1~deb9u1 into apt.w.o stretch-wikimedia component/spicerack - [[phab:T205884|T205884]]
* 20:10 mutante: LDAP - added mneisler to wmf ([[phab:T211742|T211742]]) - existing shell user, so no gerrit change needed
* 19:53 hoo: Ran scap pull on snapshot1005 to undo live changes done for dump performance testing
* 19:49 dcausse: creating 300 wiki indices in elastic-omega@eqiad
* 19:46 dcausse: SF Morning SWAT done
* 19:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: [[phab:T210381|T210381]]: [cirrus] fix cluster settings for temp clusters psi&omega (duration: 00m 44s)
* 19:40 moritzm: removed labvirt1014 from debmonitor DB, has been renamed to cloudvirt1014
* 19:39 volans: imported python-elasticsearch_5.4.0-1~deb9u1 into apt.w.o stretch-wikimedia component/spicerack - [[phab:T205884|T205884]]
* 19:35 dcausse@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/TwoColConflict/: [[phab:T210501|T210501]]: Add missing code to not loose edits on the other side (duration: 00m 45s)
* 19:29 bblack: authdns2001: upgrading gdnsd to 2.99.9944-beta
* 19:27 bblack: multatuli: upgrading gdnsd to 2.99.9944-beta
* 19:22 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T211234|T211234]]: Enable Block notice stats on top blocking wikis (duration: 00m 45s)
* 19:19 bblack: authdns1001: upgrading gdnsd to 2.99.9944-beta
* 18:52 arlolra: Updated Parsoid to {{Gerrit|4242ad0}} ([[phab:T204622|T204622]], [[phab:T211738|T211738]])
* 18:40 arlolra@deploy1001: Finished deploy [parsoid/deploy@e27574c]: Updating Parsoid to {{Gerrit|4242ad0}} (duration: 09m 17s)
* 18:31 arlolra@deploy1001: Started deploy [parsoid/deploy@e27574c]: Updating Parsoid to {{Gerrit|4242ad0}}
* 18:30 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@93443fe]: Refine MW API queries (duration: 03m 41s)
* 18:26 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@93443fe]: Refine MW API queries
* 18:24 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: [[phab:T204748|T204748]] [Beta only] Use newly-fixed config for Wikibase->Commons federation (duration: 00m 44s)
* 18:22 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T204748|T204748]] Rename repo-only Wikibase config for clarity [no-op] (duration: 00m 45s)
* 18:11 bblack: restart pybal on lvs1006 for config updates
* 18:07 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: Use the right index for change_tag ([[phab:T211896|T211896]]) (duration: 00m 46s)
* 17:45 ladsgroup@deploy1001: Finished deploy [ores/deploy@1a3de73]: [[phab:T211267|T211267]] (duration: 13m 53s)
* 17:41 akosiaris: reapply the zotero calico policy to allow LVS endpoints
* 17:35 dcausse: elastic@eqiad created cirrus metastore on psi&omega
* 17:32 ladsgroup@deploy1001: Started deploy [ores/deploy@1a3de73]: [[phab:T211267|T211267]]
* 17:31 arturo: [[phab:T168967|T168967]] added shiny-server .deb to stretch-wikimedia
* 17:20 godog: run puppet and bounce pybal on lvs in eqiad to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/479184
* 17:19 akosiaris: [[phab:T205919|T205919]] create namespace for blubberoid on eqiad/codfw/staging clusters
* 17:00 anomie: Set comment migration to new on group 1 ([[phab:T166733|T166733]])
* 16:33 mutante: icinga1001 - started service again, enabeld puppet
* 16:27 mutante: icinga1001 - disable puppet, stopped icinga, for cable replacement
* 16:18 anomie: Deployed fix for [[phab:T210937|T210937]]: API: Use parenthesized join in ApiQueryBase::showHiddenUsersAddBlockInfo
* 15:44 moritzm: installing openssl 1.1 security updates on Hadoop workers
* 15:33 moritzm: rebooting pybal-test hosts to pick up SSBD-enabled qemu
* 15:18 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on all wikis ([[phab:T188327|T188327]]) (duration: 00m 45s)
* 15:12 anomie@deploy1001: Synchronized php-1.33.0-wmf.8/includes/api/ApiPageSet.php: Backport fix for [[phab:T211804|T211804]]: ApiPageSet::initFromPageIds: Default $filterIds to true (duration: 00m 46s)
* 15:03 akosiaris: ores2* deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479206/
* 14:55 moritzm: installing openssl 1.1 security updates on mw canaries (along with nginx restart/upgrade)
* 14:46 cdanis: updating grafana/stretch-wikimedia to 5.4.2: reprepro --restrict grafana update stretch-wikimedia
* 14:40 akosiaris: disable puppet on ores1* and ores2* machines to deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479206/
* 14:36 ladsgroup@deploy1001: Finished deploy [ores/deploy@a9d5e95]: noop (duration: 14m 59s)
* 14:23 ladsgroup@deploy1001: Started deploy [ores/deploy@a9d5e95]: noop
* 13:22 dcausse: creating 300+ wiki indices on elastic-omega@codfw
* 13:15 godog: stop restbase and cassandra on restbase200[1-6] - [[phab:T211070|T211070]]
* 12:59 elukey: superset on analytics-tool1003 upgraded to 0.28.1
* 12:41 arturo: icinga downtime (30 mins) cloudcontrol1003, cloudnet1003 and cloudnet1004 for package upgrades
* 12:38 zeljkof: EU SWAT finished
* 12:35 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: [[gerrit:479411{{!}}Update maiwiki logo (T211845)]] (duration: 00m 52s)
* 12:35 oblivian@deploy1001: scap-helm zotero finished
* 12:35 oblivian@deploy1001: scap-helm zotero cluster codfw completed
* 12:35 oblivian@deploy1001: scap-helm zotero cluster eqiad completed
* 12:35 oblivian@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: eqiad,codfw]
* 12:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:478138{{!}}Enable flood user group at ne.wiki (T211181)]] (duration: 00m 51s)
* 12:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:478143{{!}}Namespace configuration on shnwiki (T210699)]] (duration: 00m 53s)
* 12:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - [[phab:T211070|T211070]] [[phab:T211871|T211871]] [[phab:T211411|T211411]] (duration: 18m 59s)
* 12:10 oblivian@deploy1001: scap-helm -h finished
* 12:10 oblivian@deploy1001: scap-helm -h cluster codfw completed
* 12:10 oblivian@deploy1001: scap-helm -h cluster eqiad completed
* 12:10 oblivian@deploy1001: scap-helm -h  [namespace: -h, clusters: eqiad,codfw]
* 12:00 moritzm: rebooting ununpentium to pick up SSBD-enabled qemu
* 11:53 mobrovac@deploy1001: Started deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - [[phab:T211070|T211070]] [[phab:T211871|T211871]] [[phab:T211411|T211411]]
* 11:51 elukey@deploy1001: Finished deploy [analytics/superset/deploy@35841a7]: (no justification provided) (duration: 00m 38s)
* 11:51 elukey@deploy1001: Started deploy [analytics/superset/deploy@35841a7]: (no justification provided)
* 11:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - [[phab:T211070|T211070]] [[phab:T211871|T211871]] (duration: 06m 08s)
* 11:43 moritzm: rebooting vega/bromine to pick up SSBD-enabled qemu
* 11:39 mobrovac@deploy1001: Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - [[phab:T211070|T211070]] [[phab:T211871|T211871]]
* 11:39 mobrovac@deploy1001: Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - [[phab:T211070|T211070]] [[phab:T211871|T211871]] (duration: 07m 08s)
* 11:38 fsero@deploy1001: scap-helm zotero finished
* 11:38 fsero@deploy1001: scap-helm zotero cluster codfw completed
* 11:38 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
* 11:38 moritzm: rebooting krypton to pick up SSBD-enabled qemu
* 11:31 mobrovac@deploy1001: Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - [[phab:T211070|T211070]] [[phab:T211871|T211871]]
* 11:31 fsero@deploy1001: scap-helm zotero finished
* 11:31 fsero@deploy1001: scap-helm zotero cluster eqiad completed
* 11:31 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 11:14 moritzm: rebooting webperf hosts to pick up SSBD-enabled qemu
* 10:51 moritzm: rebooting dbmonitor hosts to pick up SSBD-enabled qemu
* 10:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 51s)
* 10:00 elukey: upgrade nodejs on aqs100[5-9]
* 09:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 51s)
* 09:55 moritzm: removed openssl 1.1.0f-3+deb9u2+wmf1 from stretch-wikimedia/component/node10 (superseded by openssl update in DSA 4348 for stretch)
* 09:35 moritzm: rebooting etcd/kubernetes hosts in codfw to pick up SSBD-enabled qemu
* 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 51s)
* 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2006.codfw.wmnet
* 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2005.codfw.wmnet
* 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2004.codfw.wmnet
* 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2003.codfw.wmnet
* 09:05 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2002.codfw.wmnet
* 09:05 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
* 08:50 godog: stress-test ms-be10[44-50] - [[phab:T209618|T209618]]
* 08:45 marostegui: Deploy schema change on db1090:3317 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 08:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 53s)
* 08:45 moritzm: installing openssl security updates on stretch
* 08:26 vgutierrez: Use certcentral managed TLS certificates in mx[12]001.wikimedia.org - [[phab:T207050|T207050]]
* 08:15 marostegui: Drop unused flaggedrevs tables from srwikinews - [[phab:T209761|T209761]]
* 08:11 moritzm: rolling reboot of scb in eqiad for kernel security update (combined with nodejs update)
* 08:09 marostegui: Repool labsdb1011 [[phab:T86338|T86338]]
* 08:08 moritzm: installing nodejs updates on restbase1007
* 07:28 marostegui: Depool labsdb1011 [[phab:T86338|T86338]]
* 07:25 marostegui: Repool labsdb1010 [[phab:T86338|T86338]]
* 07:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 52s)
* 06:46 marostegui: Deploy schema change on db1079 with replication, lag will be generated on labsdb:s7 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 53s)
* 06:43 marostegui: Depool labsdb1010 [[phab:T86338|T86338]]
* 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 51s)
* 06:17 marostegui: Deploy schema change on db1086 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 51s)
* 06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1098:3316 db1098:3317 after kernel and mysql upgrade (duration: 00m 54s)
* 00:52 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/GuzzleHttpRequest.php: [[phab:T211806|T211806]] (duration: 00m 51s)
* 00:41 mutante: einsteinium -  rm /lib/systemd/system/update-etcd-mw-config-lastindex.service ; systemctl reset-failed
* 00:36 tzatziki: changing two passwords for compromised accounts
* 00:32 dcausse: elastic@codfw created cirrus metastore on psi&omega clusters
* 00:22 dcausse@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/MobileFrontend/extension.json: [[phab:T210390|T210390]]: Reset default mobilefrontend provider (duration: 00m 53s)
* 00:13 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: [[phab:T210381|T210381]]: [cirrus] fix temp clusters for codfw (duration: 00m 52s)
== 2018-12-12 ==
* 22:47 tzatziki: change email for User:Denrique
* 21:48 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/EventBus: [[phab:T211805|T211805]] (duration: 00m 53s)
* 21:26 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@ced6fab]: Update mobileapps to 55981a8. Summary: Get modified date with regexes to avoid unneeded Document parse (duration: 04m 03s)
* 21:22 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@ced6fab]: Update mobileapps to 55981a8. Summary: Get modified date with regexes to avoid unneeded Document parse
* 19:45 hashar: contint1001: sudo chown -R zuul:zuul /etc/zuul/wikimedia/.git
* 19:28 XioNoX: repool codfw - [[phab:T210456|T210456]]
* 19:28 XioNoX: revert redirecting eqsin/ulsfo caches to eqiad - [[phab:T210456|T210456]]
* 19:16 XioNoX: re-enable BGP to telia on cr1-codfw - [[phab:T211715|T211715]]
* 19:08 XioNoX: disable BGP to telia on cr1-codfw - [[phab:T211715|T211715]]
* 19:06 bblack: uploading gdnsd 2.99.9944-beta-1+wmf1 to stretch-wikimedia
* 19:04 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 00m 50s)
* 18:53 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 02m 58s)
* 18:46 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 03m 00s)
* 18:41 banyek: pool db1098 for recentchanges and recenlchangeslinked
* 18:12 catrope@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/GrowthExperiments/extension.json: Temporarily disable help panel / VisualEditor integration (duration: 03m 00s)
* 17:55 banyek: pooling db1098
* 17:54 XioNoX: shutting down asw-b4-codfw - [[phab:T210456|T210456]]
* 17:02 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 50s)
* 16:42 moritzm: installing lxml security updates
* 16:40 moritzm: installing cups updates on trusty (only client libs used)
* 16:34 jijiki: Merged 477472 "mcrouter: replace codfw proxy before maintenance", eqiad mcrouters are picking up the change - [[phab:T210467|T210467]]
* 16:19 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/specials/pagers/ImageListPager.php: [[phab:T211774|T211774]] (duration: 00m 52s)
* 16:17 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/specials/pagers/ImageListPager.php: [[phab:T211774|T211774]] (duration: 00m 52s)
* 16:01 XioNoX: Redirect eqsin/ulsfo caches to eqiad - [[phab:T210456|T210456]]
* 15:54 XioNoX: Depool codfw for row B recabling - [[phab:T210456|T210456]]
* 15:35 elukey: upload matomo 3.7.0 to stretch-wikimedia, removed 3.5.1 from jessie-wikimedia
* 15:27 moritzm: installing PHP security updates on matomo1001 (piwik host)
* 15:24 godog: poweroff ms-be2044 for hardware inspection - [[phab:T209921|T209921]]
* 15:22 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/api/ApiBase.php: [[phab:T211769|T211769]] (duration: 00m 52s)
* 15:21 urandom: decommissioning cassandra-c, restbase2006 -- [[phab:T210843|T210843]]
* 15:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1098:3316 db1098:3317 after kernel and mysql upgrade (duration: 00m 53s)
* 14:43 banyek: renaming tables on db1122 ptwiki: flagged* -> [[phab:T211544|T211544]]_flagged* - [[phab:T211544|T211544]]
* 14:29 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.33.0-wmf.8"
* 14:18 moritzm: restart uwsgi-netbox on netmon2001
* 14:16 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 51s)
* 14:07 mobrovac@deploy1001: Started restart [restbase/deploy@5946231]: Restart RB to pick up the new seeds in codfw - [[phab:T211416|T211416]]
* 14:04 marostegui: Stopy MySQL on db1098:3316 and db1098:3317 for kernel and mysql upgrade
* 13:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 db1098:3317 for kernel and mysql upgrade (duration: 00m 52s)
* 13:26 moritzm: installing PHP security updates
* 12:52 zeljkof: EU SWAT finished
* 12:49 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:477979{{!}}Add NS_PROJECT localised name for tt.wiktionary (T211312)]] (duration: 00m 52s)
* 12:41 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:471345{{!}}Add new namespace aliases for zhwikiversity (T207544)]] (duration: 00m 52s)
* 12:35 arturo: [[phab:T205969|T205969]] icinga downtime load-avg check for labstore1007 until January (1 month)
* 12:32 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:478896{{!}}Enable extension SandboxLink for nowiki (T210325)]] (duration: 00m 52s)
* 12:27 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:478891{{!}}Use HD logos for cawikimedia in IS.php (T198507)]] (duration: 00m 52s)
* 12:17 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: [[gerrit:478890{{!}}Upload new logos for cawikimedia (T198507)]] (duration: 00m 52s)
* 12:03 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Display Kartographer mapframes for geocoordinate statements ([[phab:T184933|T184933]]) (duration: 00m 52s)
* 11:30 volans: re-enabled puppet on icinga[12]001, re-activated crontab to sync files on 2001 and manually run it + run puppet
* 11:14 volans: restarting icinga with dropped downtimes from last night (start_date > {{Gerrit|1544489652}})
* 11:03 volans: restarting Icinga with debug log on icinga1001
* 10:51 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2b - [[phab:T211416|T211416]] (duration: 10m 11s)
* 10:41 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2b - [[phab:T211416|T211416]]
* 10:40 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2 - [[phab:T211416|T211416]] (duration: 00m 15s)
* 10:40 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2 - [[phab:T211416|T211416]]
* 10:26 volans: Icinga is having issue restarting properly, investigation ongoing
* 10:08 banyek: executing schema change on db1070 (s5 master) - [[phab:T85757|T85757]]
* 09:40 banyek: repooling labsdb1010 - [[phab:T210693|T210693]]
* 09:34 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: repool db1082 (duration: 00m 52s)
* 09:29 banyek: repooling db1082 - [[phab:T85757|T85757]]
* 09:25 banyek: restarting replication on db1082 after schema change - [[phab:T85757|T85757]]
* 09:25 banyek: fixing triggers on db1124:3315- [[phab:T85757|T85757]]
* 09:22 banyek: executing schema change with replication on db1082 - [[phab:T85757|T85757]]
* 09:20 banyek: stopping replication on db1082 for schema change - [[phab:T85757|T85757]]
* 09:15 moritzm: installing pixman security updates on trusty (Debian already fixed)
* 08:48 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: depool db1082 (duration: 00m 51s)
* 08:38 banyek: depooling db1082 for schema change - [[phab:T85757|T85757]]
* 08:37 marostegui: Remove old backup directory from db1116 - [[phab:T206743|T206743]]
* 08:18 godog: decommissioning cassandra-b, restbase2006 -- [[phab:T210843|T210843]]
* 08:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1088 (duration: 00m 51s)
* 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 52s)
* 07:38 marostegui: Deploy schema change on db2040 (s7 codfw master), this will generate lag on codfw [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 07:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 51s)
* 07:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 51s)
* 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 52s)
* 06:44 marostegui: Stop MySQL on db1088 for mysql and kernel upgrade
* 06:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for mysql upgrade (duration: 01m 07s)
* 06:37 marostegui: Deploy schema change on s4 primary master (db1068) [[phab:T86338|T86338]]
* 06:00 marostegui: Deploy schema change on s8 primary master (db1071) [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 00:47 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:478467{{!}}Add some missing groups to the privileged list]] (duration: 00m 51s)
* 00:46 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:467110{{!}}Bring up password change logging to the same standards as login logging]] [[gerrit:478467{{!}}Add some missing groups to the privileged list]] (duration: 00m 53s)
== 2018-12-11 ==
* 22:50 chasemp: ssh to tar archive data from logstash1006 /mnt (external) to labstore1007
* 22:06 XioNoX: push loopback filter term return-tcp to all routers - [[phab:T207962|T207962]]
* 22:04 urandom: decommissioning cassandra-a, restbase2006 -- [[phab:T210843|T210843]]
* 19:57 XioNoX: apply BGP_IXP_RS_in and avoid HE to cr4-ulsfo - [[phab:T211079|T211079]]
* 19:28 jforrester@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/Flow/modules/engine/misc/flow-handlebars.js: [[phab:T211707|T211707]] Hot-deploy fix for StructuredDiscussions pages (duration: 00m 52s)
* 19:22 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: [[phab:T209619|T209619]]: Hot-deploy {{Gerrit|Ibcf3c93e}} (duration: 00m 52s)
* 18:59 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@ecd5fb6]: fix site CSS URL (duration: 03m 58s)
* 18:55 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@ecd5fb6]: fix site CSS URL
* 18:54 XioNoX: push BGP_IXP_RS_in to all routers (but don't apply it to any peers, needs to be done manually) - [[phab:T211079|T211079]]
* 18:33 cmjohnson1: swapping disk slot 0 db1063
* 18:23 XioNoX: push BGP_IXP_RS_in to cr2-eqord - [[phab:T211079|T211079]]
* 18:06 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for [[phab:T210621|T210621]], for real this time (duration: 00m 53s)
* 18:00 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: [[phab:T211237|T211237]] Hot-deploy disabling Wikibase federation, unused except in Beta Cluster (duration: 00m 52s)
* 17:33 _joe_: restarting pybal on lvs2003
* 17:23 XioNoX: push changes tested on cr4-ulsfo to all routers - [[phab:T211079|T211079]]
* 17:16 _joe_: restarting pybal on lvs2006
* 17:15 oblivian@puppetmaster1001: conftool action : set/weight=1; selector: service=search-omega-ssl
* 17:15 oblivian@puppetmaster1001: conftool action : set/weight=1; selector: service=search-psi-ssl
* 17:15 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=search-psi-ssl
* 17:14 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=search-omega-ssl
* 17:05 XioNoX: remove redundant term classification from BGP_transit_in on cr4-ulsfo - [[phab:T211079|T211079]]
* 16:55 banyek: executing schema change on dbstore1002 - [[phab:T85757|T85757]]
* 16:50 XioNoX: replace local-preference/default-action by next policy for BGP_IXP_in and BGP_Private_Peer_in on cr4-ulsfo - [[phab:T211079|T211079]]
* 16:49 banyek: executing schema change on db1102 - [[phab:T85757|T85757]]
* 16:48 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: repool db1113:3315 (duration: 00m 51s)
* 16:44 banyek: repooling db1113:3315 after schema change - [[phab:T85757|T85757]]
* 16:38 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: depool db1113:3315 (duration: 00m 51s)
* 16:35 banyek: depooling db1113:3315 for schema change - [[phab:T85757|T85757]]
* 16:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: repool db1110 (duration: 00m 51s)
* 16:32 banyek: repooling db1110 after schema change - [[phab:T85757|T85757]]
* 16:28 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: depool db1110 (duration: 00m 50s)
* 16:25 banyek: depooling db1110 for schema change - [[phab:T85757|T85757]]
* 16:23 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: repool db1100 (duration: 00m 51s)
* 16:22 XioNoX: Remove static routes for NS v6 IPs - [[phab:T211699|T211699]]
* 16:20 banyek: repooling db1100 after schema change - [[phab:T85757|T85757]]
* 16:15 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: depool db1100 (duration: 00m 52s)
* 16:09 banyek: depooling db1100 for schema change - [[phab:T85757|T85757]]
* 15:49 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.8
* 15:45 papaul: re-installing OS on ms-be2047.codfw.wmnet
* 15:31 zfilipin@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache (duration: 36m 08s)
* 15:06 vgutierrez: Use certcentral managed TLS certificate in mirrors.wikimedia.org - [[phab:T207050|T207050]]
* 14:55 zfilipin@deploy1001: Started scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache
* 14:50 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.3 (duration: 02m 51s)
* 14:49 banyek: depooling labsdb1010 - [[phab:T210693|T210693]]
* 14:46 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.2 (duration: 03m 12s)
* 14:43 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.1 (duration: 11m 40s)
* 14:42 volans: 'sudo systemctl reload icinga' on icinga1001
* 14:32 godog: decommissioning cassandra-c, restbase2005 -- [[phab:T210843|T210843]]
* 14:31 bblack: removed unused public IPv6 IPs from authdnses manually with "ip -6 addr del ..." - https://gerrit.wikimedia.org/r/c/operations/puppet/+/478939
* 14:23 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=restbase2013.codfw.wmnet
* 14:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date (duration: 00m 38s)
* 14:13 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date
* 14:10 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date - [[phab:T211416|T211416]] (duration: 01m 53s)
* 14:08 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date - [[phab:T211416|T211416]]
* 13:48 vgutierrez: Use certcentral TLS managed certificate in lists.wikimedia.org - [[phab:T207050|T207050]]
* 13:06 zeljkof: EU SWAT finished
* 13:02 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:478712{{!}}HD Logos: Add fr and fy wikibooks and fr wiiknews variants to InitaliseSettings.php (T150618)]] (duration: 00m 46s)
* 12:52 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:478036{{!}}Update settings to include new HD logos (T150618)]] (duration: 00m 47s)
* 12:50 hoo@deploy1001: Finished deploy [wdqs/wdqs@f914415]: Fix WDQS UI embeds ([[phab:T211629|T211629]]) (duration: 10m 31s)
* 12:39 hoo@deploy1001: Started deploy [wdqs/wdqs@f914415]: Fix WDQS UI embeds ([[phab:T211629|T211629]])
* 12:38 bblack: Authdns CI/config refactoring done, all is well, resume normal DNS ops!
* 12:37 elukey: updated nodejs nodejs-legacy on aqs1004 (security upgrades)
* 12:36 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: [[gerrit:478709{{!}}HD Logos: Add 1.5x and 2x variants of fr and fy wikibooks and fr wikinews (T150618)]] (duration: 00m 46s)
* 12:16 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: [[gerrit:478034{{!}}Add HD logos for 3 projects (T150618)]] (duration: 00m 47s)
* 12:12 marostegui: Repool labsdb1011 - [[phab:T86338|T86338]]
* 11:59 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@dcde39f]: GUI update (duration: 01m 05s)
* 11:58 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@dcde39f]: GUI update
* 11:39 bblack: puppet disabled on authdnses for attempting https://gerrit.wikimedia.org/r/q/topic:%22authdns-ci%22
* 11:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 11:10 marostegui: Depool labsdb1011 - [[phab:T86338|T86338]]
* 11:06 marostegui: Repool labsdb1010 - [[phab:T86338|T86338]]
* 11:03 volans: restarted pdfrender on scb1003 [last time, we need an automatic restart]
* 10:47 fsero: pooling mw1272
* 10:42 fsero: scap pull mw1272
* 09:30 ema: mw1272 down for the past 12h. Nothing in console, power-cycling
* 09:08 marostegui: Deploy schema change on db1087 with replication (this will generate lag on labsdb:s8) [[phab:T202167|T202167]] [[phab:T86338|T86338]]
* 09:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 09:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 09:01 marostegui: Depool labsdb1010 - [[phab:T86338|T86338]]
* 08:15 marostegui: Deploy schema change on db1092 [[phab:T202167|T202167]] [[phab:T86338|T86338]]
* 08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 08:10 godog: decommissioning cassandra-b, restbase2005 -- [[phab:T210843|T210843]]
* 07:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 07:32 oblivian@deploy1001: Synchronized wmf-config/PhpAutoPrepend.php: Hotfix for logging on php7 (2/2) (duration: 02m 51s)
* 07:29 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw1272.*
* 07:28 oblivian@deploy1001: Synchronized wmf-config/php7.php: Hotfix for logging on php7 (1/2) (duration: 02m 50s)
* 07:06 marostegui: Deploy schema change on db1104 [[phab:T202167|T202167]] [[phab:T86338|T86338]]
* 07:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 02m 51s)
* 06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 02m 52s)
* 06:45 marostegui: Rename flaggedrevs tables on srwikinews on db1078 - [[phab:T209761|T209761]]
* 06:13 marostegui: Deploy schema change on db1109 [[phab:T202167|T202167]] [[phab:T86338|T86338]]
* 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 02m 55s)
* 05:57 marostegui: Deploy schema change on s4 primary master (db1068) [[phab:T202167|T202167]] [[phab:T86338|T86338]]
* 01:03 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure localized logos for nywiki ([[phab:T211570|T211570]]) (duration: 01m 36s)
* 01:01 catrope@deploy1001: Synchronized static/images/project-logos/: Add localised logos for nywiki ([[phab:T211570|T211570]]) (duration: 01m 00s)
* 00:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new HD logos for zhwiktionary, zhwikivoyage, zhwikinews ([[phab:T150618|T150618]]) (duration: 01m 16s)
* 00:50 RoanKattouw: mw1272 is down (does not respond to ping), but scap still tries to deploy to it
* 00:50 catrope@deploy1001: Synchronized static/images/project-logos/: Add HD logos for zhwikinews, zhwikivoyage, zhwiktionary ([[phab:T150618|T150618]]) (duration: 02m 30s)
* 00:15 mutante: icinga2001 - killed all nagios processes, restarted nsca service, something is different from icinga1001, service failed when trying to restart ([[phab:T211641|T211641]])
== 2018-12-10 ==
* 23:51 andrewbogott: silencing the kvm process count alert on cloudvirt1023 until I can figure out why it's misfiring
* 22:13 mutante: Welcome new Mediawiki deployer Christoph 'WMDE-Fisch' Jauera ([[phab:T211014|T211014]])
* 21:29 arlolra@deploy1001: Finished deploy [parsoid/deploy@dc9b3a1]: Updating Parsoid to {{Gerrit|19560da}} (duration: 11m 15s)
* 21:20 ladsgroup@deploy1001: Finished deploy [ores/deploy@03b9c98]: Add celery4 configs back to the deploy repo (duration: 15m 25s)
* 21:19 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@9f4b567]: More internal promisification and other performance tweaks ([[phab:T202642|T202642]]) (duration: 04m 17s)
* 21:17 arlolra@deploy1001: Started deploy [parsoid/deploy@dc9b3a1]: Updating Parsoid to {{Gerrit|19560da}}
* 21:14 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@9f4b567]: More internal promisification and other performance tweaks ([[phab:T202642|T202642]])
* 21:05 ladsgroup@deploy1001: Started deploy [ores/deploy@03b9c98]: Add celery4 configs back to the deploy repo
* 20:35 cdanis: [[phab:T210416|T210416]]: grafana.wikimedia.org switched to point to grafana1001.eqiad.wmnet (running grafana 5.4.1)
* 20:32 jforrester@deploy1001: Synchronized wmf-config/extension-list: Uninstall the ParserMigration extension, Part III {{Gerrit|I332939809}} (duration: 00m 46s)
* 20:30 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Uninstall the ParserMigration extension, Part II {{Gerrit|I1f7266f55a}} (duration: 00m 46s)
* 20:29 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Uninstall the ParserMigration extension, Part I {{Gerrit|I338a3d8a87fd}} (duration: 00m 47s)
* 20:26 cdanis: [[phab:T210416|T210416]]: switching grafana.wikimedia.org to point to grafana1001.eqiad.wmnet
* 20:25 robh: messing with ulsfo power for 103.02.23 tower b, shouldnt disrupt anything [[phab:T209101|T209101]]
* 20:20 cdanis: [[phab:T210416|T210416]]: setting grafana.wikimedia.org (currently served by krypton) to read-only and copying to grafana1001 (serving grafana-beta)
* 20:13 urandom: decommissioning cassandra-a, restbase2005 -- [[phab:T210843|T210843]]
* 19:58 cdanis: [[phab:T210416|T210416]]: updating grafana to 5.4.1 in stretch-wikimedia: reprepro --restrict grafana update stretch-wikimedia
* 18:15 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@dcde39f]: GUI Update (duration: 09m 31s)
* 18:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@dcde39f]: GUI Update
* 17:59 banyek: restarting mysql instance on labsdb1004 to restore replication filters to the original state - [[phab:T211210|T211210]]
* 17:58 banyek: restarting mysql instance on labsdb1004 to restore replication filters to the original state
* 17:29 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: [[phab:T211527|T211527]] Hot-deploy Disable ParserMigration now that Raggett has been dropped (duration: 00m 47s)
* 16:03 moritzm: installing PHP updates on netmon1002
* 15:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 15:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: repool db1097:3315 (duration: 00m 45s)
* 15:31 banyek: repooling db1097:3315 after schema change - [[phab:T85757|T85757]]
* 15:24 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: depool db1097:3315 (duration: 00m 46s)
* 15:13 banyek: depooling db1097:3315 on a schema change - [[phab:T85757|T85757]]
* 15:05 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for [[phab:T210621|T210621]] (duration: 00m 46s)
* 14:55 marostegui: Deploy schema change db1101:3318 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 14:50 _joe_: uploading php-mongodb 1.5.3 to stretch-wikimedia thirdparty/php72 [[phab:T206152|T206152]]
* 14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 14:00 marostegui: Deploy schema change db1099:3318 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 14:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 13:34 ema: trafficserver 8.0.1-1wm1 uploaded to stretch-wikimedia [[phab:T207048|T207048]]
* 12:56 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T187299|T187299]] [[phab:T197607|T197607]] Oversample performance survey on specific ruwiki articles (duration: 00m 46s)
* 12:49 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: [[gerrit:478659{{!}} Bumping portals to master (T128546)]] (duration: 00m 46s)
* 12:48 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:478659{{!}} Bumping portals to master (T128546)]] (duration: 00m 46s)
* 12:43 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:478466{{!}}Add http://idb.ub.uni-tuebingen.de/digitue to the wgCopyUploadsDomains (T211466)]] (duration: 00m 47s)
* 12:37 zfilipin@deploy1001: Synchronized dblists/flaggedrevs.dblist: SWAT: [[gerrit:478465{{!}}Remove FlaggedRevs for ptwikipedia (T211433)]] (duration: 00m 46s)
* 12:25 moritzm: installing imagemagick security update for jessie
* 12:21 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: [[gerrit:477798{{!}}Set FileImporter config help location (T199108)]] (duration: 00m 47s)
* 12:20 fsero: running puppet agent on icinga to add fsero
* 12:13 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove the "wikibase-debug" log channel ([[phab:T207850|T207850]]) (duration: 00m 47s)
* 11:04 godog: decommissioning cassandra-c, restbase2004 -- [[phab:T210843|T210843]]
* 10:42 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: [[gerrit:478633{{!}} Bumping portals to master (T128546)]] (duration: 00m 46s)
* 10:41 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:478633{{!}} Bumping portals to master (T128546)]] (duration: 00m 52s)
* 10:39 marostegui: Deploy schema change on db1116:3318 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 10:36 marostegui: Deploy schema change on dbstore1002:s8 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 10:35 marostegui: Repool labsdb1011 [[phab:T86338|T86338]]
* 09:07 marostegui: Deploy schema change on s8 codfw master with replication (db2045) - lag will be generated on codfw - [[phab:T202167|T202167]] [[phab:T86338|T86338]]
* 09:01 marostegui: Depool labsdb1011 - [[phab:T86338|T86338]]
* 08:58 moritzm: installing chromium security updates on proton*
* 08:57 marostegui: Repool labsdb1010 - [[phab:T86338|T86338]]
* 08:39 elukey: roll restart of aqs on aqs100* to pick up new Druid backend settings
* 08:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 51s)
* 08:23 godog: final round of weight addition to new ms-be codfw hosts - [[phab:T209395|T209395]]
* 07:18 _joe_: reenabling puppet given my changes were useless
* 06:52 _joe_: running puppet on the puppetmasters in codfw, twice, then restarting apache to ensure cleanup of any cache
* 06:50 _joe_: disabled puppet across the fleet for merge of hiera change
* 06:47 marostegui: Stop slave on s4 on labsdb1011
* 06:25 marostegui: Deploy schema change on db1121 with replication (this will generate lag on labs) - [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 49s)
* 06:23 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - [[phab:T86338|T86338]]
* 02:53 urandom: decommissioning cassandra-b, restbase2004 -- [[phab:T210843|T210843]]
* 00:55 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|Ic07ff9acfbe17}} - [[phab:T211529|T211529]], [[phab:T205546|T205546]] (duration: 00m 47s)
== 2018-12-09 ==
* 20:16 urandom: decommissioning cassandra-a, restbase2004 -- [[phab:T210843|T210843]]
* 18:39 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: {{Gerrit|Icb9ad2f554e1}} - Fix ruwikinews logo config (duration: 00m 57s)
* 11:55 godog: decommissioning cassandra-c, restbase2003 -- [[phab:T210843|T210843]]
* 03:00 urandom: decommissioning cassandra-b, restbase2003 -- [[phab:T210843|T210843]]
== 2018-12-08 ==
* 23:07 krinkle@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/: [[phab:T185446|T185446]] - {{Gerrit|I6cf29d598a11}} (duration: 00m 47s)
* 22:21 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T209794|T209794]] (duration: 00m 56s)
* 20:59 urandom: decommissioning cassandra-a, restbase2003 -- [[phab:T210843|T210843]]
* 13:54 godog: decommissioning cassandra-c, restbase2002 -- [[phab:T210843|T210843]]
== 2018-12-07 ==
* 23:54 ejegg: updated payments-wiki from {{Gerrit|b99cd0816e}} to {{Gerrit|b8acb95a2a}}
* 23:41 urandom: decommissioning cassandra-b, restbase2002 -- [[phab:T210843|T210843]]
* 22:43 ejegg: re-enabled Thank You mail sender
* 22:31 ejegg: updated fundraising CiviCRM from {{Gerrit|3e5d74f17e}} to {{Gerrit|8e18485697}}
* 22:29 ejegg: Turned off Thank You mailing job for letter update
* 17:47 herron: rebooting logstash1006 for security updates
* 17:17 bstorm_: [[phab:T207377|T207377]] rebooted labstore1007 for kernel upgrades
* 17:15 _joe_: uploading php-tideways (rebuilt with php 7.2 support) to stretch-wikimedia thirdparty/php72 [[phab:T206152|T206152]]
* 15:37 moritzm: rebooting sarin/neodymium
* 14:52 urandom: decommissioning cassandra-a, restbase2002 -- [[phab:T210843|T210843]]
* 14:21 godog: more weight to new ms-be codfw hosts - [[phab:T209395|T209395]]
* 13:37 moritzm: rolling reboot of scb in codfw (along with nodejs update)
* 12:18 moritzm: installing nodejs security updates on stat/notebook
* 12:05 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title (duration: 21m 11s)
* 11:43 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title
* 11:42 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 00m 21s)
* 11:42 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
* 11:41 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 18m 58s)
* 11:22 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
* 11:20 moritzm: rolling upgrade of nginx on swift frontends
* 11:19 mobrovac@deploy1001: Finished deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title (duration: 03m 49s)
* 11:15 mobrovac@deploy1001: Started deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title
* 11:09 mobrovac@deploy1001: Finished deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero (duration: 06m 19s)
* 11:03 mobrovac@deploy1001: Started deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero
* 10:58 mobrovac@deploy1001: Finished deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero (duration: 02m 42s)
* 10:55 mobrovac@deploy1001: Started deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero
* 09:40 banyek: importing back linkwatcher_linklog into database s51230__linkwatcher on host labsdb1004.eqiad.wmnet. - [[phab:T211210|T211210]]
* 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
* 09:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
* 08:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
* 08:21 marostegui: Stop MySQL on db1096:3315,3316 for kernel and mysql upgrade
* 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315, db1096:3316 for kernel and mysql upgrade (duration: 00m 46s)
* 08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1084 (duration: 00m 47s)
* 07:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
* 07:50 godog: decommissioning cassandra-c, restbase2001 -- [[phab:T210843|T210843]]
* 07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
* 07:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1084 (duration: 00m 46s)
* 07:11 marostegui: Stop MySQL on db1084 for mysql and kernel upgrade
* 07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 49s)
* 00:47 XioNoX: done troubleshoting bird bfd on dns2001/cr1-codfw
* 00:10 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[gerrit:478103{{!}}turn off wbsearchentities ab test]] [[phab:T209402|T209402]] (duration: 00m 47s)
== 2018-12-06 ==
* 23:47 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API [[phab:T201192|T201192]] (duration: 02m 06s)
* 23:47 XioNoX: troubleshoot bird bfd on dns2001/cr1-codfw
* 23:45 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API [[phab:T201192|T201192]]
* 23:21 ppchelko@deploy1001: Finished deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification [[phab:T201192|T201192]] (duration: 22m 46s)
* 22:58 ppchelko@deploy1001: Started deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification [[phab:T201192|T201192]]
* 22:12 urandom: decommissioning cassandra-b, restbase2001 -- [[phab:T210843|T210843]]
* 21:39 gtirloni: reimaging cloudvirt1019 with jessie [[phab:T196507|T196507]]
* 21:33 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event (duration: 01m 15s)
* 21:32 ppchelko@deploy1001: Started deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event
* 21:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA (duration: 09m 18s)
* 21:00 XioNoX: remove 2 esams avoid path + 4 prefered/selected transits - [[phab:T194542|T194542]]
* 20:58 smalyshev@deploy1001: Started deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA
* 20:51 XioNoX: remove 2 eqiad avoid path - [[phab:T194542|T194542]]
* 20:48 gtirloni: reimaging cloudvirt1019 with stretch [[phab:T196507|T196507]]
* 20:45 XioNoX: remove codfw/eqdfw avoid path - [[phab:T194542|T194542]]
* 19:32 gehel: shutting down elasticsearch on elastic2001-2024 (third time is a charm) - [[phab:T211023|T211023]]
* 18:21 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps ([[phab:T202642|T202642]]) (duration: 03m 54s)
* 18:17 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps ([[phab:T202642|T202642]])
* 17:31 moritzm: installing nodejs updates on proton*
* 17:11 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for jessie-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
* 16:27 urandom: decommissioning cassandra-a, restbase2001 -- [[phab:T210843|T210843]]
* 16:17 gehel: shutting down elasticsearch on elastic2001-2024 (second try) - [[phab:T211023|T211023]]
* 15:51 moritzm: upgrading spamassassin on mx1001/fermium
* 15:45 fsero@deploy1001: scap-helm zotero finished
* 15:45 fsero@deploy1001: scap-helm zotero cluster codfw completed
* 15:45 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
* 15:42 fsero: modifying zotero deploy CLUSTER=codfw scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero - [[phab:T211322|T211322]]
* 15:42 _joe_: disabling puppet fleet-wide for a change in the role() function
* 15:38 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for stretch-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
* 15:08 gehel: restartign new elasticsearch masters on codfw - [[phab:T211023|T211023]]
* 15:05 gehel: upgrade nginx on wdqs servers
* 14:59 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10 (duration: 00m 09s)
* 14:59 elukey@deploy1001: Started deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10
* 14:46 moritzm: uploaded nodejs 10.4.0~dfsg-1+wmf2 to apt.wikimedia.org/component/node10 (backports of recent security fixes)
* 14:16 moritzm: installing nginx security updates on mw in eqiad
* 13:46 moritzm: upgrading spamassassin on mx2001
* 12:56 gehel: depooling and shutting down elasticsearch on elastic2001-2024 - [[phab:T211023|T211023]]
* 12:55 moritzm: installing nginx updates on mw in codfw
* 11:59 moritzm: installing nginx updates on mw canaries
* 10:59 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: repool db1096:3315 (duration: 00m 47s)
* 10:56 volans: disable event handler on Icinga for ms-be2047 MD Raid and MegaRAID checks, it's spamming Phabricator - [[phab:T209921|T209921]]
* 10:56 banyek: repooling db1096 for schema change - [[phab:T85757|T85757]]
* 10:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: depool db1096:3315 (duration: 00m 49s)
* 10:29 banyek: depooling db1096 for schema change - [[phab:T85757|T85757]]
* 09:56 dcausse: elastic@codfw cleanup: deleting wikidatawiki_content_1537469318 index (failed reindex probably)
* 01:03 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
* 00:55 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 47s)
* 00:42 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
* 00:40 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
* 00:38 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
* 00:15 mutante: MPM prefork tweaks for high load systems are applied again (apparently they were not since a change in the past that resulted in 2 competing configs in mods-enabled and conf-enabled with the latter one being loaded last and containing the package defaults
* 00:13 mutante: re-enabling puppet on phabricator, applying change that adds php-fpm support on stretch ..which doesnt affect phab1001 (prod) on jessie.. BUT re-adds tuning config from the past for mpm_prefork.conf (more SpareServers etc) that was not actually applied due to a bug
* 00:10 urandom: bootstrapping cassandra-c, restbase2018 -- [[phab:T210843|T210843]]
== 2018-12-05 ==
* 23:50 ejegg: updated payments-wiki from {{Gerrit|20595cca97}} to {{Gerrit|b99cd0816e}}
* 23:40 ejegg: re-enabled fundraising queue consumer jobs
* 23:33 ejegg: updated fundraising CiviCRM from {{Gerrit|e757753a46}} to {{Gerrit|3e5d74f17e}}
* 23:32 ejegg: turned off fundraising queue jobs for base queue consumer logic update
* 22:43 jijiki: restarting pdfreder on scb* hosts in eqiad
* 21:44 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to {{Gerrit|2f44362}} (duration: 02m 47s)
* 21:41 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to {{Gerrit|2f44362}}
* 21:41 mdholloway: mobileapps deployment failed for group default03, rolling back and retrying
* 21:39 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to {{Gerrit|2f44362}} (duration: 18m 46s)
* 21:39 arlolra: Updated Parsoid to {{Gerrit|a6058e3}} ([[phab:T210647|T210647]], [[phab:T208360|T208360]], [[phab:T205333|T205333]])
* 21:37 urandom: bootstrapping cassandra-b, restbase2018 -- [[phab:T210843|T210843]]
* 21:23 arlolra@deploy1001: Finished deploy [parsoid/deploy@5e9a496]: Updating Parsoid to {{Gerrit|a6058e3}} (duration: 11m 36s)
* 21:20 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to {{Gerrit|2f44362}}
* 21:12 arlolra@deploy1001: Started deploy [parsoid/deploy@5e9a496]: Updating Parsoid to {{Gerrit|a6058e3}}
* 20:08 banyek: repooling labsdb1010 - [[phab:T210693|T210693]]
* 19:12 cmjohnson1: cloudvirt1019 for an all inclusive part swap by HPE
* 18:53 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Fix formatting of help panel links (duration: 00m 47s)
* 18:01 jijiki: uploaded thumbor_6.3.2+git20170607-1+deb9u1 to stretch-wikimedia
* 17:23 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Enable Kartographer maps on testwikidatawiki ([[phab:T184933|T184933]]) (duration: 00m 46s)
* 17:22 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
* 17:20 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
* 17:19 XioNoX: remove private IPs from codfw cloud-instance-transport1-b [[phab:T207663|T207663]]
* 17:11 XioNoX: add public IPs to codfw cloud-instance-transport1-b [[phab:T207663|T207663]]
* 16:58 XioNoX: re-deactivate ams-ix prefix list entry on cr2-esams
* 16:53 XioNoX: activate ams-ix prefix list entry on cr2-esams
* 16:27 jijiki: uploaded python-thumbor-wikimedia_2.2-1+deb9u1 to stretch-wikimedia
* 16:18 akosiaris@deploy1001: scap-helm zotero finished
* 16:18 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
* 16:18 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
* 16:17 akosiaris@deploy1001: scap-helm zotero finished
* 16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 16:17 akosiaris@deploy1001: scap-helm zotero finished
* 16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 16:15 akosiaris@deploy1001: scap-helm zotero finished
* 16:15 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 16:15 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
* 16:08 fsero: redeploying zotero on eqiad
* 16:02 akosiaris@deploy1001: scap-helm zotero finished
* 16:02 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
* 16:02 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
* 16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 stable/zotero [namespace: zotero, clusters: eqiad,codfw]
* 16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 [namespace: zotero, clusters: eqiad,codfw]
* 15:34 thcipriani: restarting ci jenkins for update
* 15:33 akosiaris: add back pods/portforward right to kubernetes deploy user. [[phab:T211040|T211040]]
* 15:07 anomie: Running cleanupUsersWithNoId.php on potentially missed s3 and s7 wikis for [[phab:T181731|T181731]]
* 15:03 fsero: repooling citoid mathoid eqiad
* 15:03 godog: bootstrap cassandra-c, restbase2017 - [[phab:T210843|T210843]]
* 14:56 banyek: executing schema change on s5 codfw master replication lag could be expected - [[phab:T85757|T85757]]
* 14:54 fsero: upgrading k8s on eqiad to 1.10.11
* 14:54 elukey: restart HDFS namenode and Yarn resource manager on an-master100[1,2] to update rack topology config - [[phab:T209929|T209929]]
* 14:51 fsero: depool mathoid/eqiad: pooled changed True => False
* 14:51 fsero: depool citoid/eqiad: pooled changed True => False
* 14:43 anomie: Running cleanupUsersWithNoId.php on metawiki for [[phab:T181731|T181731]] / [[phab:T210985|T210985]]
* 14:09 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary item/property access for all wiktionaries ([[phab:T175273|T175273]]) (duration: 00m 47s)
* 13:53 akosiaris: repool citoid/mathoid codfw
* 12:59 onimisionipe: banning elastic2001-elastic2024 from codfw production, psi and omega clusters
* 12:53 jijiki: uploaded python-thumbor-community-core_0.4.0-1+deb9u1 to stretch-wikimedia
* 12:47 dcausse: EU SWAT done
* 12:45 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=(cit{{!}}math)oid,name=codfw
* 12:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: [[phab:T210381|T210381]]: [cirrus] Add temp clusters but still write to the old ones 2/2 (duration: 00m 46s)
* 12:42 dcausse@deploy1001: Synchronized wmf-config/CommonSettings.php: [[phab:T210381|T210381]]: [cirrus] Add temp clusters but still write to the old ones 1/2 (duration: 00m 46s)
* 12:31 fsero: pooling mathoid and citoid again on  codfw
* 12:30 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: [[gerrit:477768{{!}} Bumping portals to master (T202497)]] (duration: 00m 46s)
* 12:29 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:477768{{!}} Bumping portals to master (T202497)]] (duration: 00m 49s)
* 12:23 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T211188|T211188]]: Increase autoconfirmed count for Meta-Wiki to 5 (duration: 00m 47s)
* 12:15 fsero: upgrading codfw k8s cluster to 1.10.11
* 12:15 dcausse: running namespaceDupes & cirrus indexNamespaces on yuewiktionary
* 12:11 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T205546|T205546]]: Define 2 new namespaces for yuewiktionary (duration: 00m 47s)
* 12:02 fsero: depooling mathoid and citoid servers on codfw for k8s upgrade
* 11:28 mobrovac@deploy1001: Finished deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - [[phab:T211127|T211127]] (duration: 05m 55s)
* 11:23 mobrovac@deploy1001: Started deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - [[phab:T211127|T211127]]
* 11:05 akosiaris: upgrade kubernetes-client and kubernetes-master on staging to 1.10.11
* 11:04 godog: bootstrap cassandra-b, restbase2017 - [[phab:T210843|T210843]]
* 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 (duration: 00m 45s)
* 10:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 45s)
* 10:51 ema: cache hosts: begin nginx rolling upgrade to 1.13.6-2+wmf2
* 10:46 marostegui: Reboot db1090 for kernel upgrade
* 10:44 moritzm: uploaded jenkins 2.138.4 to jessie-wikimedia/thirdparty and stretch-wikimedia/thirdpary/ci
* 10:42 marostegui: Stop MySQL on db1090:3312 and db1090:3317 for MySQL upgrade
* 10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 46s)
* 10:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 46s)
* 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1080 (duration: 00m 46s)
* 10:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
* 10:17 arturo: [[phab:T205969|T205969]] icinga downtime the load avg check in labstore1007 for 1 week
* 10:10 banyek: depooling labsdb1010 for testing materialized views - [[phab:T210693|T210693]]
* 10:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1080 (duration: 00m 46s)
* 09:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly pool db1080 (duration: 00m 46s)
* 09:32 gehel: setting up new elasticsearch servers on codfw - elastic2045-2054 - [[phab:T210265|T210265]]
* 09:22 marostegui: Stop MySQL on db1080 for mysql and kernel upgrade
* 09:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for MySQL upgrade (duration: 00m 46s)
* 09:07 elukey: matomo read only + upgrade to matomo 3.7.0 on matomo1001 - [[phab:T209808|T209808]]
* 09:00 _joe_: disabed puppet on mw1261, used for logging tests for [[phab:T211184|T211184]]
* 08:48 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [[phab:T207850|T207850]] Define a new Wikibase log channel to use (duration: 00m 47s)
* 08:01 moritzm: installing pdns-recursor security update in esams
* 07:59 godog: bootstrap cassandra-a, restbase2017 - [[phab:T210843|T210843]]
* 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 06:55 marostegui: Deploy schema change on db1091 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 51s)
* 04:03 urandom: bootstrapping cassandra-c, restbase2016 -- [[phab:T210843|T210843]]
* 03:11 kartik@deploy1001: Finished deploy [cxserver/deploy@a3dd2ca]: Update cxserver to {{Gerrit|c4240e6}} and enable Youdao MT ([[phab:T208985|T208985]], [[phab:T210578|T210578]]) (duration: 04m 26s)
* 03:06 kartik@deploy1001: Started deploy [cxserver/deploy@a3dd2ca]: Update cxserver to {{Gerrit|c4240e6}} and enable Youdao MT ([[phab:T208985|T208985]], [[phab:T210578|T210578]])
* 00:32 dcausse: Evening SWAT done
* 00:30 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [cirrus] prepare multi-instance services ([[phab:T210381|T210381]]) (duration: 00m 46s)
* 00:28 dcausse@deploy1001: Synchronized wmf-config/ProductionServices.php: [cirrus] prepare multi-instance services ([[phab:T210381|T210381]]) (duration: 00m 46s)
* 00:16 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Block notice stats on itwiki ([[phab:T210452|T210452]]) (duration: 00m 47s)
* 00:10 urandom: bootstrapping cassandra-b, restbase2016 -- [[phab:T210843|T210843]]
== 2018-12-04 ==
* 23:46 eileen: civicrm revision changed from {{Gerrit|a411d6bd64}} to {{Gerrit|e757753a46}}, config revision is {{Gerrit|0e6ccc37fe}}
* 23:33 XioNoX: update prefix-list peering4 on cr1-eqsin to match jnt
* 22:46 XioNoX: remove neodymium/sarin from term labs-in4 on cr1/2-eqiad - [[phab:T210612|T210612]]
* 22:40 ejegg: Updated payments-wiki from {{Gerrit|7403a196b4}} to {{Gerrit|20595cca97}}
* 21:58 XioNoX: clear ethernet-swtiching table for labvirt1004:eth1's switch port
* 21:57 XioNoX: clear ethernet-swtiching table for labvirt1009:eth1's switch port
* 21:49 XioNoX: make cr1/2-codfw conform to jnt
* 21:44 XioNoX: make cr2-eqord/eqdfw conform to jnt
* 21:41 XioNoX: make cr3/4-ulsfo conform to jnt
* 20:04 urandom: bootstrapping cassandra-a, restbase2016 -- [[phab:T210843|T210843]]
* 19:35 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@81dac18]: Install new Updater for [[phab:T210044|T210044]] investigation (duration: 10m 36s)
* 19:24 smalyshev@deploy1001: Started deploy [wdqs/wdqs@81dac18]: Install new Updater for [[phab:T210044|T210044]] investigation
* 18:04 joal@deploy1001: Finished deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint (duration: 17m 33s)
* 18:03 akosiaris: bump zotero pod number from 4 to 16 in eqiad/codfw
* 17:47 joal@deploy1001: Started deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint
* 17:46 ppchelko@deploy1001: Finished deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance [[phab:T210465|T210465]] (duration: 01m 13s)
* 17:45 ppchelko@deploy1001: Started deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance [[phab:T210465|T210465]]
* 17:22 Reedy: created oathauth tables on punjabiwikimedia [[phab:T211110|T211110]]
* 17:16 godog: bootstrap cassandra-c on restbase2015 - [[phab:T210843|T210843]]
* 16:48 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure 'api-warning' log channel (duration: 00m 47s)
* 15:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 14:04 elukey: upgrade turnilo on analytics-tools1002 to nodejs-10 - [[phab:T210705|T210705]]
* 13:56 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=bnwikisource --fix --add-prefix=[[phab:T210472|T210472]]
* 13:53 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=euwiki --fix
* 13:52 marostegui: Deploy schema change on db1103:3314 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 13:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 13:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 13:34 cdanis: [[phab:T210416|T210416]]: adding grafana 5 to wikimedia-stretch: reprepro --restrict grafana update stretch-wikimedia
* 13:33 moritzm: installing nodejs security updates on restbase in codfw
* 13:32 godog: bootstrap cassandra-b on restbase2015 - [[phab:T210843|T210843]]
* 13:29 moritzm: installing nodejs security updates on proton*
* 13:19 marostegui: Deploy schema change on db1081 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 13:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 13:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 12:39 Lucas_WMDE: EU SWAT done
* 12:38 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:477174{{!}}Create namespace "Work" on bnwikisource (T210472)]] (duration: 00m 46s)
* 12:33 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:477175{{!}}Create List namespace on euwiki (T209834)]] (duration: 00m 47s)
* 12:28 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: [[gerrit:449445{{!}}Revert "Milestone logo for atjwiki" (T200713)]] (duration: 00m 47s)
* 12:17 moritzm: installing tiff security updates
* 11:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided) (duration: 04m 47s)
* 11:44 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided)
* 11:41 moritzm: rebooting puppetboard2001 to pick up SSBD-enabled qemu
* 11:38 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - [[phab:T197242|T197242]] (duration: 00m 21s)
* 11:38 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - [[phab:T197242|T197242]]
* 11:37 moritzm: rebooting puppetboard1001 to pick up SSBD-enabled qemu
* 11:36 akosiaris: enable puppet on scb1004, run puppet [[phab:T197242|T197242]]
* 11:36 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - [[phab:T197242|T197242]] (duration: 00m 20s)
* 11:35 marostegui: Deploy schema change on db1084 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 11:35 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - [[phab:T197242|T197242]]
* 11:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 11:34 akosiaris: enable puppet on scb1003, run puppet [[phab:T197242|T197242]]
* 11:33 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - [[phab:T197242|T197242]] (duration: 00m 28s)
* 11:33 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - [[phab:T197242|T197242]]
* 11:31 akosiaris: enable puppet on scb1002, run puppet [[phab:T197242|T197242]]
* 11:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 11:18 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - [[phab:T197242|T197242]] (duration: 00m 30s)
* 11:18 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - [[phab:T197242|T197242]]
* 11:17 akosiaris: enable puppet on scb1001, run puppet [[phab:T197242|T197242]]
* 11:12 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - [[phab:T164201|T164201]] (duration: 09m 06s)
* 11:04 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - [[phab:T211088|T211088]] [[phab:T197242|T197242]] (duration: 20m 59s)
* 11:03 elukey@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - [[phab:T164201|T164201]]
* 10:59 fdans@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques (duration: 00m 37s)
* 10:58 fdans@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques
* 10:57 fdans: deploying AQS to expose offset and underestimate numbers on unique devices
* 10:51 moritzm: rebooting analytics-tool1003 to pick up SSBD-enabled qemu
* 10:47 moritzm: rebooting analytics-tool1002 to pick up SSBD-enabled qemu
* 10:43 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - [[phab:T211088|T211088]] [[phab:T197242|T197242]]
* 10:41 moritzm: rebooting analytics-tool1001 to pick up SSBD-enabled qemu
* 10:03 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - [[phab:T197242|T197242]] (duration: 01m 45s)
* 10:02 godog: bootstrap cassandra-a on restbase2015 - [[phab:T210843|T210843]]
* 10:01 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - [[phab:T197242|T197242]]
* 09:59 gehel: upgrading nginx on elasticsearch eqiad
* 09:54 akosiaris: enable puppet on all scb2*, run puppet [[phab:T197242|T197242]]
* 09:52 gehel: upgrading nginx on elasticsearch codfw
* 09:52 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - [[phab:T197242|T197242]] (duration: 00m 30s)
* 09:51 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - [[phab:T197242|T197242]]
* 09:50 akosiaris: enable puppet on scb2001, run puppet [[phab:T197242|T197242]]
* 09:46 akosiaris: disable puppet on scb for citoid migration to zoterov2 [[phab:T197242|T197242]]
* 09:46 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
* 09:31 gehel: add elastic2039-2044 to cirrus eqiad (new server) - [[phab:T210265|T210265]]
* 09:11 gehel: add elastic2038 to cirrus eqiad (new server) - [[phab:T210265|T210265]]
* 09:00 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/MediaWiki/electronpdf/action # Ran https://phabricator.wikimedia.org/P7882 for [[phab:T157012|T157012]]
* 08:54 marostegui: Deploy schema change on db1097:3314 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 08:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*-*.wsp" -delete # [[phab:T120639|T120639]]
* 08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*_*.wsp" -delete # [[phab:T120639|T120639]]
* 08:43 marostegui: Deploy schema change on db1102:3314 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 08:42 marostegui: Deploy schema change on dbstore1002:s4 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 08:41 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/datamodel$ sudo -u _graphite rm wikipedia_references.wsp # [[phab:T121521|T121521]]
* 08:36 gehel: restarting stuck tilerator on maps* - [[phab:T204047|T204047]]
* 08:35 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/wbgetclaims$ sudo -u _graphite find . -type f -name "*.wsp" -delete # [[phab:T140280|T140280]]
* 08:11 moritzm: installing perl security updates on jessie/trusty (stretch already updated)
* 07:49 godog: bootstrap cassandra-c on restbase2014 - [[phab:T209615|T209615]]
* 07:26 marostegui: Deploy schema change on s4 codfw master (db2051) with replication [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 07:22 marostegui: Deploy schema change on wikitech primary master (db1073) for labswiki and labtestwiki [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:39 marostegui: Deploy schema change on s5 primary master (db1070) [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 49s)
* 06:12 marostegui: Deploy schema change on db1110 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 53s)
* 01:22 foks: Removing 2FA per request at https://phabricator.wikimedia.org/T210703
* 01:22 foks: Reset password for user "Orangemike"
== 2018-12-03 ==
* 23:54 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/tests/: for completeness (duration: 00m 58s)
* 23:53 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/resources/src/mediawiki.legacy/: Restore gray coloring for autocomments ([[phab:T165189|T165189]] part 2) (duration: 00m 47s)
* 23:51 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Linker.php: Restore old HTML structure for history section links ([[phab:T165189|T165189]] part 1) (duration: 00m 47s)
* 22:56 sbassett@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/AbuseFilter/includes/api/ApiQueryAbuseLog.php: Deploy security fix for [[phab:T210329|T210329]] (duration: 00m 47s)
* 21:24 mutante: temp. disabling puppet on logstash1007 and logstash1008 to carefully deploy gerrit:476916
* 21:17 XioNoX: push firewall change to pfw3-eqiad - [[phab:T211028|T211028]]
* 21:10 catrope@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: Fix ChangesListFilters validation errors (duration: 00m 49s)
* 21:00 urandom: bootstrapping restbase2014-a -- [[phab:T210843|T210843]]
* 20:20 ppchelko@deploy1001: Finished deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema (duration: 01m 13s)
* 20:19 ppchelko@deploy1001: Started deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema
* 20:14 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build (duration: 09m 31s)
* 20:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build
* 20:05 XioNoX: push firewall change to pfw3-eqiad - [[phab:T211028|T211028]]
* 19:59 ppchelko@deploy1001: Finished deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change (duration: 01m 13s)
* 19:58 ppchelko@deploy1001: Started deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change
* 19:39 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/resources/mobile.toc/TableOfContents.js: SWAT [[phab:T210869|T210869]] Fix Table of contents rendering (duration: 00m 47s)
* 19:30 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT {{Gerrit|I3b906e8b1}} CS part of setting enhanced RC (duration: 00m 46s)
* 19:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT {{Gerrit|Ic2787309e59e}} IS part of setting enhanced RC (duration: 00m 47s)
* 19:25 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Echo/modules/styles/mw.echo.ui.PaginationWidget.less: SWAT [[phab:T210487|T210487]] {{Gerrit|I914b94515}} (duration: 00m 47s)
* 19:15 jforrester@deploy1001: Synchronized wmf-config/ProductionServices.php: SWAT [[phab:T210381|T210381]] {{Gerrit|I73c7596818b}} Actual config (duration: 00m 46s)
* 19:09 jforrester@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: SWAT [[phab:T210381|T210381]] {{Gerrit|I2ae162f5}} Part II (duration: 00m 47s)
* 19:08 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT [[phab:T210381|T210381]] {{Gerrit|I2ae162f5}} Part I (duration: 00m 46s)
* 18:42 anomie@deploy1001: Synchronized wmf-config/CommonSettings.php: Updating SkinBuildSidebar hook function for [[phab:T210528|T210528]] (duration: 00m 47s)
* 18:08 gehel: add elastic2037 to cirrus eqiad (new server) - [[phab:T210265|T210265]]
* 18:06 godog: bootstrap cassandra-c on restbase2013 - [[phab:T209615|T209615]]
* 17:09 bstorm_: [[phab:T207377|T207377]] reboot labstore1006 for upgrades
* 16:28 godog: poweroff ms-be2021 for battery replacement - [[phab:T208269|T208269]]
* 15:23 gehel: start configuration of elastic2037-2044 (new servers) - [[phab:T210265|T210265]]
* 15:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 14:54 marostegui: Deploy schema change on db1100 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 14:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 48s)
* 14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 14:17 marostegui: Deploy schema change on db1113:3315 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 14:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 14:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 13:47 marostegui: Deploy schema change on db1082 (sanitarium master) with replication, lag will be generated on labs (s5) [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 13:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 13:01 Lucas_WMDE: EU SWAT done
* 12:44 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:477138{{!}}Cleaning of wgLogoHD (T150618)]] (duration: 00m 46s)
* 12:39 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: [[gerrit:477136{{!}}Upload HD logos for multiple projects (T150618)]] (duration: 00m 47s)
* 12:37 moritzm: installing nodejs security updates on scb1001
* 12:34 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:468823{{!}}Close internalwiki (T205584)]] (duration: 00m 46s)
* 12:28 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:473670{{!}}Change sitename of shnwiki (T206777)]] (duration: 00m 47s)
* 12:23 godog: bootstrap cassandra-b on restbase2013 - [[phab:T209615|T209615]]
* 12:19 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476822{{!}}Perform more PHP constraint checks before falling back (T209504)]] (duration: 00m 48s)
* 12:14 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:476267{{!}} Don’t send SPARQL prefixes in WikibaseQualityConstraints (T204317)]] (duration: 00m 49s)
* 11:39 godog: more weight to new ms-be codfw hosts - [[phab:T209395|T209395]]
* 11:02 moritzm: installing nodejs security updates on stat/notebook hosts
* 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 10:52 moritzm: rolling upgrade of scb in codfw to nodejs security update
* 10:39 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: [[gerrit:477237{{!}} Bumping portals to master (T128546)]] (duration: 00m 46s)
* 10:39 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:477237{{!}} Bumping portals to master (T128546)]] (duration: 00m 47s)
* 10:34 moritzm: installing nodejs security updates on scb2001
* 10:31 marostegui: Deploy schema change on db1097:3315 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 10:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 45s)
* 09:48 marostegui: Deploy schema change on db1096:3315 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 09:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 09:27 banyek: executing schema change on db1066 (s2 master) - [[phab:T85757|T85757]]
* 09:22 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: repool db1074 (duration: 00m 47s)
* 09:16 banyek: repooling db1074 - [[phab:T85757|T85757]]
* 09:16 banyek: repooling db1074
* 08:50 banyek: stopping replication on db1074 - [[phab:T85757|T85757]]
* 08:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: [[phab:T85757|T85757]]: depool db1074 (duration: 00m 48s)
* 08:44 godog: bootstrap cassandra-a on restbase2013 - [[phab:T209615|T209615]]
* 08:43 banyek: depooling db1074 - [[phab:T85757|T85757]]
* 08:32 moritzm: restarted keyholder agents/proxies on netmon1002/netmon2001 to pick up removal of netbox key
* 08:30 marostegui: Deploy schema change on s5 codfw master (db2052) with replication, lag will be generated on codfw [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 08:13 moritzm: rearmed keyholders on netmon1002/netmon2001
* 08:07 marostegui: Deploy schema change on s2 master (db1066) [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 07:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 07:32 marostegui: Deploy schema change db1074 with replication (lag will appear on labs) [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 46s)
* 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 07:09 marostegui: Stop MySQL on pc1004, pc1005 and pc1006 as they will be decommissioned - [[phab:T210969|T210969]]
* 06:52 marostegui: Remove pc1004, pc1005 and pc1006 from tendril and zarcillo - [[phab:T210969|T210969]]
* 06:38 marostegui: Deploy schema change db1122 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 48s)
* 06:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 47s)
* 06:16 marostegui: Deploy schema change db1076 [[phab:T86338|T86338]] [[phab:T202167|T202167]]
* 06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 [[phab:T86338|T86338]] [[phab:T202167|T202167]] (duration: 00m 50s)
* 00:34 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Title.php: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/477182 (duration: 00m 52s)
== 2018-12-02 ==
* 22:39 addshore: addshore@mwmaint1002:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=testwikidatawiki addless # This is my account, and apparently I no longer have the 2fa for it
== 2018-12-01 ==
* 16:48 andrewbogott: rebuilding labvirt1014 as cloudvirt1014, [[phab:T210904|T210904]]
__NOTOC__
__NOTOC__
==Archives==
==Archives==
See [[Server admin log/Archives]].
See [[Server admin log/Archives]].
<noinclude>[[Category:SAL]][[Category:Operations]]</noinclude>
<noinclude>[[Category:SAL]][[Category:Operations]]</noinclude>

Revision as of 00:56, 1 January 2019