You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Nhatminh01
mNo edit summary
imported>Stashbot
(elukey: mask + reset-failed kafkatee default instance on sulfur (kafkatee-webrequest works fine))
Line 1: Line 1:
== 2018-12-16 ==
* 09:52 elukey: mask + reset-failed kafkatee default instance on sulfur (kafkatee-webrequest works fine)
* 07:30 marostegui: Reboot db1115 after OOM
* 07:28 marostegui: Stop MySQL on db1115 so tendril can get back to work
== 2018-12-15 ==
== 2018-12-15 ==
* off: restarted pdfrender on scb1004
* off: restarted pdfrender on scb1004

Revision as of 09:52, 16 December 2018

2018-12-16

  • 09:52 elukey: mask + reset-failed kafkatee default instance on sulfur (kafkatee-webrequest works fine)
  • 07:30 marostegui: Reboot db1115 after OOM
  • 07:28 marostegui: Stop MySQL on db1115 so tendril can get back to work

2018-12-15

  • off: restarted pdfrender on scb1004
  • 09:22 elukey: mask + reset-failed kafkatee default instance on weblog1001

2018-12-14

  • 23:59 mutante: LDAP: added aezell to wmf group (T211945) for grafana access
  • 22:00 XioNoX: increase accepted-prefix-limit for HE to 200000
  • 20:38 mutante: sulfur systemctl restart nagios-nrpe-server
  • 20:32 andrew@deploy1001: Finished deploy [horizon/deploy@1a830b9]: Rolling out fix for T177855 (duration: 03m 17s)
  • 20:29 andrew@deploy1001: Started deploy [horizon/deploy@1a830b9]: Rolling out fix for T177855
  • 20:13 otto@deploy1001: Finished deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for T211833 (duration: 06m 04s)
  • 20:07 otto@deploy1001: Started deploy [analytics/refinery@ef1f7c6]: deploying refinery-source 0.0.82 with fix for T211833
  • 20:07 otto@deploy1001: deploy aborted: (no justification provided) (duration: 00m 00s)
  • 20:07 otto@deploy1001: Started deploy [analytics/refinery@ef1f7c6]: (no justification provided)
  • 17:22 cmjohnson1: mw1272 down for h/w troubleshooting
  • 14:38 marostegui: Enable GTID on db2039 (s6 codfw master) - T211973
  • 14:03 marostegui: Compare ruwiki.revision between db2039 (s6 master) and db1085 - T211973
  • 13:59 marostegui: Enable notifications for db1095 (s3 lag check)- T211973
  • 13:57 marostegui: Enable notifications for db2068 (s7 lag check)- T211973
  • 13:47 dcausse: elastic@codfw copying index data from the main cluster to psi & omega (test disk usage & import speed)
  • 13:42 marostegui: Enable GTID on db1124:3318 - T211973
  • 11:11 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2083 db2068 db2067 after mysql and kernel upgrade (duration: 00m 44s)
  • 10:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2083 db2068 db2067 for mysql and kernel upgrade (duration: 00m 45s)
  • 10:50 marostegui: Stop MySQL on db2083, db2068 and db2067 for mysql and kernel upgrade
  • 10:19 filippo@deploy1001: Synchronized wmf-config/logging.php: wmf-config/InitialiseSettings-labs.php (duration: 00m 44s)
  • 10:18 filippo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: (no justification provided) (duration: 00m 45s)
  • 09:33 marostegui: Deploy schema change on db1095:3313 T86338 T202167
  • 09:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2049, db2059, db2070 after mysql and kernel upgrade (duration: 00m 45s)
  • 09:21 banyek: global user rename is in progress - T209488
  • 08:58 marostegui: Stop MySQL on db2049, db2059 and db2070 for mysql and kernel upgrade
  • 08:56 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2049, db2059, db2070 for mysql and kernel upgrade (duration: 00m 43s)
  • 08:50 elukey: swap oxygen with weblog1001
  • 08:48 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2084 and db2088 after mysql and kernel upgrade (duration: 00m 44s)
  • 08:47 elukey: disabled kafkatee-webrequest logstash output on oxygen (prep step before weblog1001)
  • 07:59 marostegui: Deploy schema change on dbstore1002:s3 T86338 T202167
  • 07:49 marostegui: Upgrade mysql and kernel on db2084 and db2088
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2084 and db2088 for mysql and kernel upgrade (duration: 00m 45s)
  • 06:54 marostegui: Deploy schema change on db2043 (s3 codfw master) - this will generate lag on s3 codfw T86338 T202167
  • 06:53 marostegui: Deploy schema change on db2043 (s3 codfw master) - this will generate lag on s3 codfw T86338 T20216
  • 06:14 marostegui: Deploy schema change on db1062 (s7 primary master) T86338 T202167
  • 06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 T86338 T202167 (duration: 00m 44s)
  • 06:10 marostegui: Deployed schema change on db1094 T86338 T202167
  • 05:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 T86338 T202167 (duration: 00m 47s)
  • 01:39 mutante: install2002 deleted /srv/ contents,then mounted /mnt/vdb on /srv so same content but now / is used only 7% and /srv 57% (T211850)
  • 00:43 mutante: install2002 (T211850) restarted instance, created ext4 filesystem on new /dev/vdb, mounted on /mnt/vdb, rsyncing /srv/ to /mnt/vdb/
  • 00:33 mutante: rebooting install2002 via ganeti2003, to add new virtual disk
  • 00:31 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Enforce a 10-byte password for +staff users, I4ecac70e (duration: 00m 44s)
  • 00:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT I7867277d Make wmgBabelMainCategory consistent for sr* wikis (duration: 00m 45s)

2018-12-13

  • 23:57 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208246: Increase default minimum new password length to 10 for privileged groups (duration: 00m 44s)
  • 23:17 andrew@deploy1001: Finished deploy [horizon/deploy@18c4ca6]: Rolling out fix for T131367 (duration: 03m 25s)
  • 23:13 andrew@deploy1001: Started deploy [horizon/deploy@18c4ca6]: Rolling out fix for T131367
  • 22:31 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/HttpRequestFactory.php: T211886 (duration: 00m 44s)
  • 22:29 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/MobileFrontend: T211903 (duration: 00m 48s)
  • 21:55 mutante: Ganeti - creating new 120G virtual disk on install2002 (T211850)
  • 21:00 otto@deploy1001: Finished deploy [analytics/superset/deploy@UNKNOWN]: revert to version 0.26.3 (duration: 00m 32s)
  • 20:59 otto@deploy1001: Started deploy [analytics/superset/deploy@UNKNOWN]: revert to version 0.26.3
  • 20:33 dcausse: creating 300+ wikis indices in elastic-psi @eqiad and @codfw
  • 20:21 volans: imported elasticsearch-curator_5.2.0-1~deb9u1 into apt.w.o stretch-wikimedia component/spicerack - T205884
  • 20:10 mutante: LDAP - added mneisler to wmf (T211742) - existing shell user, so no gerrit change needed
  • 19:53 hoo: Ran scap pull on snapshot1005 to undo live changes done for dump performance testing
  • 19:49 dcausse: creating 300 wiki indices in elastic-omega@eqiad
  • 19:46 dcausse: SF Morning SWAT done
  • 19:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] fix cluster settings for temp clusters psi&omega (duration: 00m 44s)
  • 19:40 moritzm: removed labvirt1014 from debmonitor DB, has been renamed to cloudvirt1014
  • 19:39 volans: imported python-elasticsearch_5.4.0-1~deb9u1 into apt.w.o stretch-wikimedia component/spicerack - T205884
  • 19:35 dcausse@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/TwoColConflict/: T210501: Add missing code to not loose edits on the other side (duration: 00m 45s)
  • 19:29 bblack: authdns2001: upgrading gdnsd to 2.99.9944-beta
  • 19:27 bblack: multatuli: upgrading gdnsd to 2.99.9944-beta
  • 19:22 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211234: Enable Block notice stats on top blocking wikis (duration: 00m 45s)
  • 19:19 bblack: authdns1001: upgrading gdnsd to 2.99.9944-beta
  • 18:52 arlolra: Updated Parsoid to 4242ad0 (T204622, T211738)
  • 18:40 arlolra@deploy1001: Finished deploy [parsoid/deploy@e27574c]: Updating Parsoid to 4242ad0 (duration: 09m 17s)
  • 18:31 arlolra@deploy1001: Started deploy [parsoid/deploy@e27574c]: Updating Parsoid to 4242ad0
  • 18:30 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@93443fe]: Refine MW API queries (duration: 03m 41s)
  • 18:26 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@93443fe]: Refine MW API queries
  • 18:24 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: T204748 [Beta only] Use newly-fixed config for Wikibase->Commons federation (duration: 00m 44s)
  • 18:22 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T204748 Rename repo-only Wikibase config for clarity [no-op] (duration: 00m 45s)
  • 18:11 bblack: restart pybal on lvs1006 for config updates
  • 18:07 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: Use the right index for change_tag (T211896) (duration: 00m 46s)
  • 17:45 ladsgroup@deploy1001: Finished deploy [ores/deploy@1a3de73]: T211267 (duration: 13m 53s)
  • 17:41 akosiaris: reapply the zotero calico policy to allow LVS endpoints
  • 17:35 dcausse: elastic@eqiad created cirrus metastore on psi&omega
  • 17:32 ladsgroup@deploy1001: Started deploy [ores/deploy@1a3de73]: T211267
  • 17:31 arturo: T168967 added shiny-server .deb to stretch-wikimedia
  • 17:20 godog: run puppet and bounce pybal on lvs in eqiad to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/479184
  • 17:19 akosiaris: T205919 create namespace for blubberoid on eqiad/codfw/staging clusters
  • 17:00 anomie: Set comment migration to new on group 1 (T166733)
  • 16:33 mutante: icinga1001 - started service again, enabeld puppet
  • 16:27 mutante: icinga1001 - disable puppet, stopped icinga, for cable replacement
  • 16:18 anomie: Deployed fix for T210937: API: Use parenthesized join in ApiQueryBase::showHiddenUsersAddBlockInfo
  • 15:44 moritzm: installing openssl 1.1 security updates on Hadoop workers
  • 15:33 moritzm: rebooting pybal-test hosts to pick up SSBD-enabled qemu
  • 15:18 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on all wikis (T188327) (duration: 00m 45s)
  • 15:12 anomie@deploy1001: Synchronized php-1.33.0-wmf.8/includes/api/ApiPageSet.php: Backport fix for T211804: ApiPageSet::initFromPageIds: Default $filterIds to true (duration: 00m 46s)
  • 15:03 akosiaris: ores2* deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479206/
  • 14:55 moritzm: installing openssl 1.1 security updates on mw canaries (along with nginx restart/upgrade)
  • 14:46 cdanis: updating grafana/stretch-wikimedia to 5.4.2: reprepro --restrict grafana update stretch-wikimedia
  • 14:40 akosiaris: disable puppet on ores1* and ores2* machines to deploy https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479206/
  • 14:36 ladsgroup@deploy1001: Finished deploy [ores/deploy@a9d5e95]: noop (duration: 14m 59s)
  • 14:23 ladsgroup@deploy1001: Started deploy [ores/deploy@a9d5e95]: noop
  • 13:22 dcausse: creating 300+ wiki indices on elastic-omega@codfw
  • 13:15 godog: stop restbase and cassandra on restbase200[1-6] - T211070
  • 12:59 elukey: superset on analytics-tool1003 upgraded to 0.28.1
  • 12:41 arturo: icinga downtime (30 mins) cloudcontrol1003, cloudnet1003 and cloudnet1004 for package upgrades
  • 12:38 zeljkof: EU SWAT finished
  • 12:35 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Update maiwiki logo (T211845) (duration: 00m 52s)
  • 12:35 oblivian@deploy1001: scap-helm zotero finished
  • 12:35 oblivian@deploy1001: scap-helm zotero cluster codfw completed
  • 12:35 oblivian@deploy1001: scap-helm zotero cluster eqiad completed
  • 12:35 oblivian@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: eqiad,codfw]
  • 12:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable flood user group at ne.wiki (T211181) (duration: 00m 51s)
  • 12:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Namespace configuration on shnwiki (T210699) (duration: 00m 53s)
  • 12:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - T211070 T211871 T211411 (duration: 18m 59s)
  • 12:10 oblivian@deploy1001: scap-helm -h finished
  • 12:10 oblivian@deploy1001: scap-helm -h cluster codfw completed
  • 12:10 oblivian@deploy1001: scap-helm -h cluster eqiad completed
  • 12:10 oblivian@deploy1001: scap-helm -h [namespace: -h, clusters: eqiad,codfw]
  • 12:00 moritzm: rebooting ununpentium to pick up SSBD-enabled qemu
  • 11:53 mobrovac@deploy1001: Started deploy [restbase/deploy@55fcd4b]: Remove restbase200[1-6], ensure body.tfa exists for feed responses and disable Citoid check - T211070 T211871 T211411
  • 11:51 elukey@deploy1001: Finished deploy [analytics/superset/deploy@35841a7]: (no justification provided) (duration: 00m 38s)
  • 11:51 elukey@deploy1001: Started deploy [analytics/superset/deploy@35841a7]: (no justification provided)
  • 11:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871 (duration: 06m 08s)
  • 11:43 moritzm: rebooting vega/bromine to pick up SSBD-enabled qemu
  • 11:39 mobrovac@deploy1001: Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871
  • 11:39 mobrovac@deploy1001: Finished deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871 (duration: 07m 08s)
  • 11:38 fsero@deploy1001: scap-helm zotero finished
  • 11:38 fsero@deploy1001: scap-helm zotero cluster codfw completed
  • 11:38 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 11:38 moritzm: rebooting krypton to pick up SSBD-enabled qemu
  • 11:31 mobrovac@deploy1001: Started deploy [restbase/deploy@29a0902]: Remove restbase200[1-6] and ensure body.tfa exists for feed responses - T211070 T211871
  • 11:31 fsero@deploy1001: scap-helm zotero finished
  • 11:31 fsero@deploy1001: scap-helm zotero cluster eqiad completed
  • 11:31 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 11:14 moritzm: rebooting webperf hosts to pick up SSBD-enabled qemu
  • 10:51 moritzm: rebooting dbmonitor hosts to pick up SSBD-enabled qemu
  • 10:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 T86338 T202167 (duration: 00m 51s)
  • 10:00 elukey: upgrade nodejs on aqs100[5-9]
  • 09:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 T86338 T202167 (duration: 00m 51s)
  • 09:55 moritzm: removed openssl 1.1.0f-3+deb9u2+wmf1 from stretch-wikimedia/component/node10 (superseded by openssl update in DSA 4348 for stretch)
  • 09:35 moritzm: rebooting etcd/kubernetes hosts in codfw to pick up SSBD-enabled qemu
  • 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 T86338 T202167 (duration: 00m 51s)
  • 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2006.codfw.wmnet
  • 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2005.codfw.wmnet
  • 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2004.codfw.wmnet
  • 09:06 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2003.codfw.wmnet
  • 09:05 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2002.codfw.wmnet
  • 09:05 filippo@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase2001.codfw.wmnet
  • 08:50 godog: stress-test ms-be10[44-50] - T209618
  • 08:45 marostegui: Deploy schema change on db1090:3317 T86338 T202167
  • 08:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 T86338 T202167 (duration: 00m 53s)
  • 08:45 moritzm: installing openssl security updates on stretch
  • 08:26 vgutierrez: Use certcentral managed TLS certificates in mx[12]001.wikimedia.org - T207050
  • 08:15 marostegui: Drop unused flaggedrevs tables from srwikinews - T209761
  • 08:11 moritzm: rolling reboot of scb in eqiad for kernel security update (combined with nodejs update)
  • 08:09 marostegui: Repool labsdb1011 T86338
  • 08:08 moritzm: installing nodejs updates on restbase1007
  • 07:28 marostegui: Depool labsdb1011 T86338
  • 07:25 marostegui: Repool labsdb1010 T86338
  • 07:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 T86338 T202167 (duration: 00m 52s)
  • 06:46 marostegui: Deploy schema change on db1079 with replication, lag will be generated on labsdb:s7 T86338 T202167
  • 06:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 T86338 T202167 (duration: 00m 53s)
  • 06:43 marostegui: Depool labsdb1010 T86338
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 T86338 T202167 (duration: 00m 51s)
  • 06:17 marostegui: Deploy schema change on db1086 T86338 T202167
  • 06:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 T86338 T202167 (duration: 00m 51s)
  • 06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1098:3316 db1098:3317 after kernel and mysql upgrade (duration: 00m 54s)
  • 00:52 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/includes/http/GuzzleHttpRequest.php: T211806 (duration: 00m 51s)
  • 00:41 mutante: einsteinium - rm /lib/systemd/system/update-etcd-mw-config-lastindex.service ; systemctl reset-failed
  • 00:36 tzatziki: changing two passwords for compromised accounts
  • 00:32 dcausse: elastic@codfw created cirrus metastore on psi&omega clusters
  • 00:22 dcausse@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/MobileFrontend/extension.json: T210390: Reset default mobilefrontend provider (duration: 00m 53s)
  • 00:13 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] fix temp clusters for codfw (duration: 00m 52s)

2018-12-12

  • 22:47 tzatziki: change email for User:Denrique
  • 21:48 reedy@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/EventBus: T211805 (duration: 00m 53s)
  • 21:26 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@ced6fab]: Update mobileapps to 55981a8. Summary: Get modified date with regexes to avoid unneeded Document parse (duration: 04m 03s)
  • 21:22 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@ced6fab]: Update mobileapps to 55981a8. Summary: Get modified date with regexes to avoid unneeded Document parse
  • 19:45 hashar: contint1001: sudo chown -R zuul:zuul /etc/zuul/wikimedia/.git
  • 19:28 XioNoX: repool codfw - T210456
  • 19:28 XioNoX: revert redirecting eqsin/ulsfo caches to eqiad - T210456
  • 19:16 XioNoX: re-enable BGP to telia on cr1-codfw - T211715
  • 19:08 XioNoX: disable BGP to telia on cr1-codfw - T211715
  • 19:06 bblack: uploading gdnsd 2.99.9944-beta-1+wmf1 to stretch-wikimedia
  • 19:04 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 00m 50s)
  • 18:53 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 02m 58s)
  • 18:46 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: pool db1098 for recentchanges and recentchangeslinked (duration: 03m 00s)
  • 18:41 banyek: pool db1098 for recentchanges and recenlchangeslinked
  • 18:12 catrope@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/GrowthExperiments/extension.json: Temporarily disable help panel / VisualEditor integration (duration: 03m 00s)
  • 17:55 banyek: pooling db1098
  • 17:54 XioNoX: shutting down asw-b4-codfw - T210456
  • 17:02 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 50s)
  • 16:42 moritzm: installing lxml security updates
  • 16:40 moritzm: installing cups updates on trusty (only client libs used)
  • 16:34 jijiki: Merged 477472 "mcrouter: replace codfw proxy before maintenance", eqiad mcrouters are picking up the change - T210467
  • 16:19 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/specials/pagers/ImageListPager.php: T211774 (duration: 00m 52s)
  • 16:17 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/specials/pagers/ImageListPager.php: T211774 (duration: 00m 52s)
  • 16:01 XioNoX: Redirect eqsin/ulsfo caches to eqiad - T210456
  • 15:54 XioNoX: Depool codfw for row B recabling - T210456
  • 15:35 elukey: upload matomo 3.7.0 to stretch-wikimedia, removed 3.5.1 from jessie-wikimedia
  • 15:27 moritzm: installing PHP security updates on matomo1001 (piwik host)
  • 15:24 godog: poweroff ms-be2044 for hardware inspection - T209921
  • 15:22 ladsgroup@deploy1001: Synchronized php-1.33.0-wmf.8/includes/api/ApiBase.php: T211769 (duration: 00m 52s)
  • 15:21 urandom: decommissioning cassandra-c, restbase2006 -- T210843
  • 15:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1098:3316 db1098:3317 after kernel and mysql upgrade (duration: 00m 53s)
  • 14:43 banyek: renaming tables on db1122 ptwiki: flagged* -> T211544_flagged* - T211544
  • 14:29 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.33.0-wmf.8"
  • 14:18 moritzm: restart uwsgi-netbox on netmon2001
  • 14:16 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.8 (duration: 00m 51s)
  • 14:07 mobrovac@deploy1001: Started restart [restbase/deploy@5946231]: Restart RB to pick up the new seeds in codfw - T211416
  • 14:04 marostegui: Stopy MySQL on db1098:3316 and db1098:3317 for kernel and mysql upgrade
  • 13:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 db1098:3317 for kernel and mysql upgrade (duration: 00m 52s)
  • 13:26 moritzm: installing PHP security updates
  • 12:52 zeljkof: EU SWAT finished
  • 12:49 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add NS_PROJECT localised name for tt.wiktionary (T211312) (duration: 00m 52s)
  • 12:41 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new namespace aliases for zhwikiversity (T207544) (duration: 00m 52s)
  • 12:35 arturo: T205969 icinga downtime load-avg check for labstore1007 until January (1 month)
  • 12:32 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable extension SandboxLink for nowiki (T210325) (duration: 00m 52s)
  • 12:27 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos for cawikimedia in IS.php (T198507) (duration: 00m 52s)
  • 12:17 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload new logos for cawikimedia (T198507) (duration: 00m 52s)
  • 12:03 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Display Kartographer mapframes for geocoordinate statements (T184933) (duration: 00m 52s)
  • 11:30 volans: re-enabled puppet on icinga[12]001, re-activated crontab to sync files on 2001 and manually run it + run puppet
  • 11:14 volans: restarting icinga with dropped downtimes from last night (start_date > 1544489652)
  • 11:03 volans: restarting Icinga with debug log on icinga1001
  • 10:51 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2b - T211416 (duration: 10m 11s)
  • 10:41 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2b - T211416
  • 10:40 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2 - T211416 (duration: 00m 15s)
  • 10:40 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date, try #2 - T211416
  • 10:26 volans: Icinga is having issue restarting properly, investigation ongoing
  • 10:08 banyek: executing schema change on db1070 (s5 master) - T85757
  • 09:40 banyek: repooling labsdb1010 - T210693
  • 09:34 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1082 (duration: 00m 52s)
  • 09:29 banyek: repooling db1082 - T85757
  • 09:25 banyek: restarting replication on db1082 after schema change - T85757
  • 09:25 banyek: fixing triggers on db1124:3315- T85757
  • 09:22 banyek: executing schema change with replication on db1082 - T85757
  • 09:20 banyek: stopping replication on db1082 for schema change - T85757
  • 09:15 moritzm: installing pixman security updates on trusty (Debian already fixed)
  • 08:48 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1082 (duration: 00m 51s)
  • 08:38 banyek: depooling db1082 for schema change - T85757
  • 08:37 marostegui: Remove old backup directory from db1116 - T206743
  • 08:18 godog: decommissioning cassandra-b, restbase2006 -- T210843
  • 08:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1088 (duration: 00m 51s)
  • 07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 52s)
  • 07:38 marostegui: Deploy schema change on db2040 (s7 codfw master), this will generate lag on codfw T86338 T202167
  • 07:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 51s)
  • 07:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 51s)
  • 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1088 (duration: 00m 52s)
  • 06:44 marostegui: Stop MySQL on db1088 for mysql and kernel upgrade
  • 06:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for mysql upgrade (duration: 01m 07s)
  • 06:37 marostegui: Deploy schema change on s4 primary master (db1068) T86338
  • 06:00 marostegui: Deploy schema change on s8 primary master (db1071) T86338 T202167
  • 00:47 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add some missing groups to the privileged list (duration: 00m 51s)
  • 00:46 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Bring up password change logging to the same standards as login logging Add some missing groups to the privileged list (duration: 00m 53s)

2018-12-11

  • 22:50 chasemp: ssh to tar archive data from logstash1006 /mnt (external) to labstore1007
  • 22:06 XioNoX: push loopback filter term return-tcp to all routers - T207962
  • 22:04 urandom: decommissioning cassandra-a, restbase2006 -- T210843
  • 19:57 XioNoX: apply BGP_IXP_RS_in and avoid HE to cr4-ulsfo - T211079
  • 19:28 jforrester@deploy1001: Synchronized php-1.33.0-wmf.8/extensions/Flow/modules/engine/misc/flow-handlebars.js: T211707 Hot-deploy fix for StructuredDiscussions pages (duration: 00m 52s)
  • 19:22 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: T209619: Hot-deploy Ibcf3c93e (duration: 00m 52s)
  • 18:59 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@ecd5fb6]: fix site CSS URL (duration: 03m 58s)
  • 18:55 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@ecd5fb6]: fix site CSS URL
  • 18:54 XioNoX: push BGP_IXP_RS_in to all routers (but don't apply it to any peers, needs to be done manually) - T211079
  • 18:33 cmjohnson1: swapping disk slot 0 db1063
  • 18:23 XioNoX: push BGP_IXP_RS_in to cr2-eqord - T211079
  • 18:06 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621, for real this time (duration: 00m 53s)
  • 18:00 jforrester@deploy1001: Synchronized wmf-config/Wikibase.php: T211237 Hot-deploy disabling Wikibase federation, unused except in Beta Cluster (duration: 00m 52s)
  • 17:33 _joe_: restarting pybal on lvs2003
  • 17:23 XioNoX: push changes tested on cr4-ulsfo to all routers - T211079
  • 17:16 _joe_: restarting pybal on lvs2006
  • 17:15 oblivian@puppetmaster1001: conftool action : set/weight=1; selector: service=search-omega-ssl
  • 17:15 oblivian@puppetmaster1001: conftool action : set/weight=1; selector: service=search-psi-ssl
  • 17:15 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=search-psi-ssl
  • 17:14 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: service=search-omega-ssl
  • 17:05 XioNoX: remove redundant term classification from BGP_transit_in on cr4-ulsfo - T211079
  • 16:55 banyek: executing schema change on dbstore1002 - T85757
  • 16:50 XioNoX: replace local-preference/default-action by next policy for BGP_IXP_in and BGP_Private_Peer_in on cr4-ulsfo - T211079
  • 16:49 banyek: executing schema change on db1102 - T85757
  • 16:48 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1113:3315 (duration: 00m 51s)
  • 16:44 banyek: repooling db1113:3315 after schema change - T85757
  • 16:38 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1113:3315 (duration: 00m 51s)
  • 16:35 banyek: depooling db1113:3315 for schema change - T85757
  • 16:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1110 (duration: 00m 51s)
  • 16:32 banyek: repooling db1110 after schema change - T85757
  • 16:28 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1110 (duration: 00m 50s)
  • 16:25 banyek: depooling db1110 for schema change - T85757
  • 16:23 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1100 (duration: 00m 51s)
  • 16:22 XioNoX: Remove static routes for NS v6 IPs - T211699
  • 16:20 banyek: repooling db1100 after schema change - T85757
  • 16:15 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1100 (duration: 00m 52s)
  • 16:09 banyek: depooling db1100 for schema change - T85757
  • 15:49 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.8
  • 15:45 papaul: re-installing OS on ms-be2047.codfw.wmnet
  • 15:31 zfilipin@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache (duration: 36m 08s)
  • 15:06 vgutierrez: Use certcentral managed TLS certificate in mirrors.wikimedia.org - T207050
  • 14:55 zfilipin@deploy1001: Started scap: testwiki to php-1.33.0-wmf.8 and rebuild l10n cache
  • 14:50 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.3 (duration: 02m 51s)
  • 14:49 banyek: depooling labsdb1010 - T210693
  • 14:46 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.2 (duration: 03m 12s)
  • 14:43 zfilipin@deploy1001: Pruned MediaWiki: 1.33.0-wmf.1 (duration: 11m 40s)
  • 14:42 volans: 'sudo systemctl reload icinga' on icinga1001
  • 14:32 godog: decommissioning cassandra-c, restbase2005 -- T210843
  • 14:31 bblack: removed unused public IPv6 IPs from authdnses manually with "ip -6 addr del ..." - https://gerrit.wikimedia.org/r/c/operations/puppet/+/478939
  • 14:23 filippo@puppetmaster1001: conftool action : set/pooled=yes; selector: name=restbase2013.codfw.wmnet
  • 14:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date (duration: 00m 38s)
  • 14:13 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date
  • 14:10 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date - T211416 (duration: 01m 53s)
  • 14:08 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Bring restbase201[3-8] up to date - T211416
  • 13:48 vgutierrez: Use certcentral TLS managed certificate in lists.wikimedia.org - T207050
  • 13:06 zeljkof: EU SWAT finished
  • 13:02 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: HD Logos: Add fr and fy wikibooks and fr wiiknews variants to InitaliseSettings.php (T150618) (duration: 00m 46s)
  • 12:52 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update settings to include new HD logos (T150618) (duration: 00m 47s)
  • 12:50 hoo@deploy1001: Finished deploy [wdqs/wdqs@f914415]: Fix WDQS UI embeds (T211629) (duration: 10m 31s)
  • 12:39 hoo@deploy1001: Started deploy [wdqs/wdqs@f914415]: Fix WDQS UI embeds (T211629)
  • 12:38 bblack: Authdns CI/config refactoring done, all is well, resume normal DNS ops!
  • 12:37 elukey: updated nodejs nodejs-legacy on aqs1004 (security upgrades)
  • 12:36 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: HD Logos: Add 1.5x and 2x variants of fr and fy wikibooks and fr wikinews (T150618) (duration: 00m 46s)
  • 12:16 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Add HD logos for 3 projects (T150618) (duration: 00m 47s)
  • 12:12 marostegui: Repool labsdb1011 - T86338
  • 11:59 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@dcde39f]: GUI update (duration: 01m 05s)
  • 11:58 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@dcde39f]: GUI update
  • 11:39 bblack: puppet disabled on authdnses for attempting https://gerrit.wikimedia.org/r/q/topic:%22authdns-ci%22
  • 11:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 T86338 T202167 (duration: 00m 46s)
  • 11:10 marostegui: Depool labsdb1011 - T86338
  • 11:06 marostegui: Repool labsdb1010 - T86338
  • 11:03 volans: restarted pdfrender on scb1003 [last time, we need an automatic restart]
  • 10:47 fsero: pooling mw1272
  • 10:42 fsero: scap pull mw1272
  • 09:30 ema: mw1272 down for the past 12h. Nothing in console, power-cycling
  • 09:08 marostegui: Deploy schema change on db1087 with replication (this will generate lag on labsdb:s8) T202167 T86338
  • 09:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 T86338 T202167 (duration: 00m 46s)
  • 09:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 T86338 T202167 (duration: 00m 46s)
  • 09:01 marostegui: Depool labsdb1010 - T86338
  • 08:15 marostegui: Deploy schema change on db1092 T202167 T86338
  • 08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 T86338 T202167 (duration: 00m 46s)
  • 08:10 godog: decommissioning cassandra-b, restbase2005 -- T210843
  • 07:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 T86338 T202167 (duration: 00m 46s)
  • 07:32 oblivian@deploy1001: Synchronized wmf-config/PhpAutoPrepend.php: Hotfix for logging on php7 (2/2) (duration: 02m 51s)
  • 07:29 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw1272.*
  • 07:28 oblivian@deploy1001: Synchronized wmf-config/php7.php: Hotfix for logging on php7 (1/2) (duration: 02m 50s)
  • 07:06 marostegui: Deploy schema change on db1104 T202167 T86338
  • 07:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 T86338 T202167 (duration: 02m 51s)
  • 06:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 T86338 T202167 (duration: 02m 52s)
  • 06:45 marostegui: Rename flaggedrevs tables on srwikinews on db1078 - T209761
  • 06:13 marostegui: Deploy schema change on db1109 T202167 T86338
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 T86338 T202167 (duration: 02m 55s)
  • 05:57 marostegui: Deploy schema change on s4 primary master (db1068) T202167 T86338
  • 01:03 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure localized logos for nywiki (T211570) (duration: 01m 36s)
  • 01:01 catrope@deploy1001: Synchronized static/images/project-logos/: Add localised logos for nywiki (T211570) (duration: 01m 00s)
  • 00:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new HD logos for zhwiktionary, zhwikivoyage, zhwikinews (T150618) (duration: 01m 16s)
  • 00:50 RoanKattouw: mw1272 is down (does not respond to ping), but scap still tries to deploy to it
  • 00:50 catrope@deploy1001: Synchronized static/images/project-logos/: Add HD logos for zhwikinews, zhwikivoyage, zhwiktionary (T150618) (duration: 02m 30s)
  • 00:15 mutante: icinga2001 - killed all nagios processes, restarted nsca service, something is different from icinga1001, service failed when trying to restart (T211641)

2018-12-10

  • 23:51 andrewbogott: silencing the kvm process count alert on cloudvirt1023 until I can figure out why it's misfiring
  • 22:13 mutante: Welcome new Mediawiki deployer Christoph 'WMDE-Fisch' Jauera (T211014)
  • 21:29 arlolra@deploy1001: Finished deploy [parsoid/deploy@dc9b3a1]: Updating Parsoid to 19560da (duration: 11m 15s)
  • 21:20 ladsgroup@deploy1001: Finished deploy [ores/deploy@03b9c98]: Add celery4 configs back to the deploy repo (duration: 15m 25s)
  • 21:19 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@9f4b567]: More internal promisification and other performance tweaks (T202642) (duration: 04m 17s)
  • 21:17 arlolra@deploy1001: Started deploy [parsoid/deploy@dc9b3a1]: Updating Parsoid to 19560da
  • 21:14 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@9f4b567]: More internal promisification and other performance tweaks (T202642)
  • 21:05 ladsgroup@deploy1001: Started deploy [ores/deploy@03b9c98]: Add celery4 configs back to the deploy repo
  • 20:35 cdanis: T210416: grafana.wikimedia.org switched to point to grafana1001.eqiad.wmnet (running grafana 5.4.1)
  • 20:32 jforrester@deploy1001: Synchronized wmf-config/extension-list: Uninstall the ParserMigration extension, Part III I332939809 (duration: 00m 46s)
  • 20:30 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Uninstall the ParserMigration extension, Part II I1f7266f55a (duration: 00m 46s)
  • 20:29 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Uninstall the ParserMigration extension, Part I I338a3d8a87fd (duration: 00m 47s)
  • 20:26 cdanis: T210416: switching grafana.wikimedia.org to point to grafana1001.eqiad.wmnet
  • 20:25 robh: messing with ulsfo power for 103.02.23 tower b, shouldnt disrupt anything T209101
  • 20:20 cdanis: T210416: setting grafana.wikimedia.org (currently served by krypton) to read-only and copying to grafana1001 (serving grafana-beta)
  • 20:13 urandom: decommissioning cassandra-a, restbase2005 -- T210843
  • 19:58 cdanis: T210416: updating grafana to 5.4.1 in stretch-wikimedia: reprepro --restrict grafana update stretch-wikimedia
  • 18:15 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@dcde39f]: GUI Update (duration: 09m 31s)
  • 18:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@dcde39f]: GUI Update
  • 17:59 banyek: restarting mysql instance on labsdb1004 to restore replication filters to the original state - T211210
  • 17:58 banyek: restarting mysql instance on labsdb1004 to restore replication filters to the original state
  • 17:29 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T211527 Hot-deploy Disable ParserMigration now that Raggett has been dropped (duration: 00m 47s)
  • 16:03 moritzm: installing PHP updates on netmon1002
  • 15:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 T86338 T202167 (duration: 00m 46s)
  • 15:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1097:3315 (duration: 00m 45s)
  • 15:31 banyek: repooling db1097:3315 after schema change - T85757
  • 15:24 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1097:3315 (duration: 00m 46s)
  • 15:13 banyek: depooling db1097:3315 on a schema change - T85757
  • 15:05 anomie@deploy1001: Synchronized php-1.33.0-wmf.6/includes/user/User.php: Backport fix for T210621 (duration: 00m 46s)
  • 14:55 marostegui: Deploy schema change db1101:3318 T86338 T202167
  • 14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 T86338 T202167 (duration: 00m 46s)
  • 14:50 _joe_: uploading php-mongodb 1.5.3 to stretch-wikimedia thirdparty/php72 T206152
  • 14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099 T86338 T202167 (duration: 00m 47s)
  • 14:00 marostegui: Deploy schema change db1099:3318 T86338 T202167
  • 14:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099 T86338 T202167 (duration: 00m 46s)
  • 13:34 ema: trafficserver 8.0.1-1wm1 uploaded to stretch-wikimedia T207048
  • 12:56 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T187299 T197607 Oversample performance survey on specific ruwiki articles (duration: 00m 46s)
  • 12:49 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 12:48 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 12:43 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add http://idb.ub.uni-tuebingen.de/digitue to the wgCopyUploadsDomains (T211466) (duration: 00m 47s)
  • 12:37 zfilipin@deploy1001: Synchronized dblists/flaggedrevs.dblist: SWAT: Remove FlaggedRevs for ptwikipedia (T211433) (duration: 00m 46s)
  • 12:25 moritzm: installing imagemagick security update for jessie
  • 12:21 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Set FileImporter config help location (T199108) (duration: 00m 47s)
  • 12:20 fsero: running puppet agent on icinga to add fsero
  • 12:13 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove the "wikibase-debug" log channel (T207850) (duration: 00m 47s)
  • 11:04 godog: decommissioning cassandra-c, restbase2004 -- T210843
  • 10:42 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 10:41 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 52s)
  • 10:39 marostegui: Deploy schema change on db1116:3318 T86338 T202167
  • 10:36 marostegui: Deploy schema change on dbstore1002:s8 T86338 T202167
  • 10:35 marostegui: Repool labsdb1011 T86338
  • 09:07 marostegui: Deploy schema change on s8 codfw master with replication (db2045) - lag will be generated on codfw - T202167 T86338
  • 09:01 marostegui: Depool labsdb1011 - T86338
  • 08:58 moritzm: installing chromium security updates on proton*
  • 08:57 marostegui: Repool labsdb1010 - T86338
  • 08:39 elukey: roll restart of aqs on aqs100* to pick up new Druid backend settings
  • 08:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 T86338 T202167 (duration: 00m 51s)
  • 08:23 godog: final round of weight addition to new ms-be codfw hosts - T209395
  • 07:18 _joe_: reenabling puppet given my changes were useless
  • 06:52 _joe_: running puppet on the puppetmasters in codfw, twice, then restarting apache to ensure cleanup of any cache
  • 06:50 _joe_: disabled puppet across the fleet for merge of hiera change
  • 06:47 marostegui: Stop slave on s4 on labsdb1011
  • 06:25 marostegui: Deploy schema change on db1121 with replication (this will generate lag on labs) - T86338 T202167
  • 06:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 T86338 T202167 (duration: 00m 49s)
  • 06:23 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T86338
  • 02:53 urandom: decommissioning cassandra-b, restbase2004 -- T210843
  • 00:55 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ic07ff9acfbe17 - T211529, T205546 (duration: 00m 47s)

2018-12-09

  • 20:16 urandom: decommissioning cassandra-a, restbase2004 -- T210843
  • 18:39 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Icb9ad2f554e1 - Fix ruwikinews logo config (duration: 00m 57s)
  • 11:55 godog: decommissioning cassandra-c, restbase2003 -- T210843
  • 03:00 urandom: decommissioning cassandra-b, restbase2003 -- T210843

2018-12-08

  • 23:07 krinkle@deploy1001: Synchronized docroot/wikipedia.org/speed-tests/: T185446 - I6cf29d598a11 (duration: 00m 47s)
  • 22:21 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T209794 (duration: 00m 56s)
  • 20:59 urandom: decommissioning cassandra-a, restbase2003 -- T210843
  • 13:54 godog: decommissioning cassandra-c, restbase2002 -- T210843

2018-12-07

  • 23:54 ejegg: updated payments-wiki from b99cd0816e to b8acb95a2a
  • 23:41 urandom: decommissioning cassandra-b, restbase2002 -- T210843
  • 22:43 ejegg: re-enabled Thank You mail sender
  • 22:31 ejegg: updated fundraising CiviCRM from 3e5d74f17e to 8e18485697
  • 22:29 ejegg: Turned off Thank You mailing job for letter update
  • 17:47 herron: rebooting logstash1006 for security updates
  • 17:17 bstorm_: T207377 rebooted labstore1007 for kernel upgrades
  • 17:15 _joe_: uploading php-tideways (rebuilt with php 7.2 support) to stretch-wikimedia thirdparty/php72 T206152
  • 15:37 moritzm: rebooting sarin/neodymium
  • 14:52 urandom: decommissioning cassandra-a, restbase2002 -- T210843
  • 14:21 godog: more weight to new ms-be codfw hosts - T209395
  • 13:37 moritzm: rolling reboot of scb in codfw (along with nodejs update)
  • 12:18 moritzm: installing nodejs security updates on stat/notebook
  • 12:05 mobrovac@deploy1001: Finished deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title (duration: 21m 11s)
  • 11:43 mobrovac@deploy1001: Started deploy [restbase/deploy@44e0955]: Fix: Encode recommendation api title
  • 11:42 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 00m 21s)
  • 11:42 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
  • 11:41 mobrovac@deploy1001: Finished deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title (duration: 18m 58s)
  • 11:22 mobrovac@deploy1001: Started deploy [restbase/deploy@9e4af13]: Fix: Encode recommendation api title
  • 11:20 moritzm: rolling upgrade of nginx on swift frontends
  • 11:19 mobrovac@deploy1001: Finished deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title (duration: 03m 49s)
  • 11:15 mobrovac@deploy1001: Started deploy [restbase/deploy@31c44e8]: Fix: Encode recommendation api title
  • 11:09 mobrovac@deploy1001: Finished deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero (duration: 06m 19s)
  • 11:03 mobrovac@deploy1001: Started deploy [citoid/deploy@269c9c7]: Add an explicit check for Zotero
  • 10:58 mobrovac@deploy1001: Finished deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero (duration: 02m 42s)
  • 10:55 mobrovac@deploy1001: Started deploy [citoid/deploy@6b36331]: Add an explicit check for Zotero
  • 09:40 banyek: importing back linkwatcher_linklog into database s51230__linkwatcher on host labsdb1004.eqiad.wmnet. - T211210
  • 09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
  • 09:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
  • 08:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1096:3315, db1096:3316 after kernel and mysql upgrade (duration: 00m 46s)
  • 08:21 marostegui: Stop MySQL on db1096:3315,3316 for kernel and mysql upgrade
  • 08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315, db1096:3316 for kernel and mysql upgrade (duration: 00m 46s)
  • 08:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1084 (duration: 00m 47s)
  • 07:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
  • 07:50 godog: decommissioning cassandra-c, restbase2001 -- T210843
  • 07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1084 (duration: 00m 46s)
  • 07:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1084 (duration: 00m 46s)
  • 07:11 marostegui: Stop MySQL on db1084 for mysql and kernel upgrade
  • 07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 49s)
  • 00:47 XioNoX: done troubleshoting bird bfd on dns2001/cr1-codfw
  • 00:10 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: turn off wbsearchentities ab test T209402 (duration: 00m 47s)

2018-12-06

  • 23:47 ppchelko@deploy1001: Finished deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192 (duration: 02m 06s)
  • 23:47 XioNoX: troubleshoot bird bfd on dns2001/cr1-codfw
  • 23:45 ppchelko@deploy1001: Started deploy [recommendation-api/deploy@299b268]: Add 'morelike' article recommendations API T201192
  • 23:21 ppchelko@deploy1001: Finished deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192 (duration: 22m 46s)
  • 22:58 ppchelko@deploy1001: Started deploy [restbase/deploy@be8f0c0]: Add 'morelike' recommendation public API specification T201192
  • 22:12 urandom: decommissioning cassandra-b, restbase2001 -- T210843
  • 21:39 gtirloni: reimaging cloudvirt1019 with jessie T196507
  • 21:33 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event (duration: 01m 15s)
  • 21:32 ppchelko@deploy1001: Started deploy [changeprop/deploy@f675fcc]: Added performer to the revision-scores event
  • 21:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA (duration: 09m 18s)
  • 21:00 XioNoX: remove 2 esams avoid path + 4 prefered/selected transits - T194542
  • 20:58 smalyshev@deploy1001: Started deploy [wdqs/wdqs@cbe4551]: Install new Updater with INSERT DATA
  • 20:51 XioNoX: remove 2 eqiad avoid path - T194542
  • 20:48 gtirloni: reimaging cloudvirt1019 with stretch T196507
  • 20:45 XioNoX: remove codfw/eqdfw avoid path - T194542
  • 19:32 gehel: shutting down elasticsearch on elastic2001-2024 (third time is a charm) - T211023
  • 18:21 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642) (duration: 03m 54s)
  • 18:17 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@1dba3cd]: Internally promisify page processing steps (T202642)
  • 17:31 moritzm: installing nodejs updates on proton*
  • 17:11 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for jessie-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
  • 16:27 urandom: decommissioning cassandra-a, restbase2001 -- T210843
  • 16:17 gehel: shutting down elasticsearch on elastic2001-2024 (second try) - T211023
  • 15:51 moritzm: upgrading spamassassin on mx1001/fermium
  • 15:45 fsero@deploy1001: scap-helm zotero finished
  • 15:45 fsero@deploy1001: scap-helm zotero cluster codfw completed
  • 15:45 fsero@deploy1001: scap-helm zotero upgrade production -f ../zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 15:42 fsero: modifying zotero deploy CLUSTER=codfw scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero - T211322
  • 15:42 _joe_: disabling puppet fleet-wide for a change in the role() function
  • 15:38 moritzm: uploaded nodejs 6.11~dfsg-1+wmf5 for stretch-wikimedia (the upstream patch for CVE-2018-12122 had a regression, this update fixes it)
  • 15:08 gehel: restartign new elasticsearch masters on codfw - T211023
  • 15:05 gehel: upgrade nginx on wdqs servers
  • 14:59 elukey@deploy1001: Finished deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10 (duration: 00m 09s)
  • 14:59 elukey@deploy1001: Started deploy [analytics/turnilo/deploy@6bd6e2f]: upgrade deps to nodejs 10
  • 14:46 moritzm: uploaded nodejs 10.4.0~dfsg-1+wmf2 to apt.wikimedia.org/component/node10 (backports of recent security fixes)
  • 14:16 moritzm: installing nginx security updates on mw in eqiad
  • 13:46 moritzm: upgrading spamassassin on mx2001
  • 12:56 gehel: depooling and shutting down elasticsearch on elastic2001-2024 - T211023
  • 12:55 moritzm: installing nginx updates on mw in codfw
  • 11:59 moritzm: installing nginx updates on mw canaries
  • 10:59 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1096:3315 (duration: 00m 47s)
  • 10:56 volans: disable event handler on Icinga for ms-be2047 MD Raid and MegaRAID checks, it's spamming Phabricator - T209921
  • 10:56 banyek: repooling db1096 for schema change - T85757
  • 10:36 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1096:3315 (duration: 00m 49s)
  • 10:29 banyek: depooling db1096 for schema change - T85757
  • 09:56 dcausse: elastic@codfw cleanup: deleting wikidatawiki_content_1537469318 index (failed reindex probably)
  • 01:03 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
  • 00:55 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 47s)
  • 00:42 tstarling@deploy1001: Synchronized w/fatal-error.php: (no justification provided) (duration: 00m 46s)
  • 00:40 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
  • 00:38 tstarling@deploy1001: Synchronized private/FatalErrorSettings.php: (no justification provided) (duration: 00m 46s)
  • 00:15 mutante: MPM prefork tweaks for high load systems are applied again (apparently they were not since a change in the past that resulted in 2 competing configs in mods-enabled and conf-enabled with the latter one being loaded last and containing the package defaults
  • 00:13 mutante: re-enabling puppet on phabricator, applying change that adds php-fpm support on stretch ..which doesnt affect phab1001 (prod) on jessie.. BUT re-adds tuning config from the past for mpm_prefork.conf (more SpareServers etc) that was not actually applied due to a bug
  • 00:10 urandom: bootstrapping cassandra-c, restbase2018 -- T210843

2018-12-05

  • 23:50 ejegg: updated payments-wiki from 20595cca97 to b99cd0816e
  • 23:40 ejegg: re-enabled fundraising queue consumer jobs
  • 23:33 ejegg: updated fundraising CiviCRM from e757753a46 to 3e5d74f17e
  • 23:32 ejegg: turned off fundraising queue jobs for base queue consumer logic update
  • 22:43 jijiki: restarting pdfreder on scb* hosts in eqiad
  • 21:44 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 02m 47s)
  • 21:41 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
  • 21:41 mdholloway: mobileapps deployment failed for group default03, rolling back and retrying
  • 21:39 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362 (duration: 18m 46s)
  • 21:39 arlolra: Updated Parsoid to a6058e3 (T210647, T208360, T205333)
  • 21:37 urandom: bootstrapping cassandra-b, restbase2018 -- T210843
  • 21:23 arlolra@deploy1001: Finished deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3 (duration: 11m 36s)
  • 21:20 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@243a503]: Update mobileapps to 2f44362
  • 21:12 arlolra@deploy1001: Started deploy [parsoid/deploy@5e9a496]: Updating Parsoid to a6058e3
  • 20:08 banyek: repooling labsdb1010 - T210693
  • 19:12 cmjohnson1: cloudvirt1019 for an all inclusive part swap by HPE
  • 18:53 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Fix formatting of help panel links (duration: 00m 47s)
  • 18:01 jijiki: uploaded thumbor_6.3.2+git20170607-1+deb9u1 to stretch-wikimedia
  • 17:23 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Enable Kartographer maps on testwikidatawiki (T184933) (duration: 00m 46s)
  • 17:22 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
  • 17:20 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (no justification provided) (duration: 00m 46s)
  • 17:19 XioNoX: remove private IPs from codfw cloud-instance-transport1-b T207663
  • 17:11 XioNoX: add public IPs to codfw cloud-instance-transport1-b T207663
  • 16:58 XioNoX: re-deactivate ams-ix prefix list entry on cr2-esams
  • 16:53 XioNoX: activate ams-ix prefix list entry on cr2-esams
  • 16:27 jijiki: uploaded python-thumbor-wikimedia_2.2-1+deb9u1 to stretch-wikimedia
  • 16:18 akosiaris@deploy1001: scap-helm zotero finished
  • 16:18 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 16:18 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-codfw.yaml stable/zotero [namespace: zotero, clusters: codfw]
  • 16:17 akosiaris@deploy1001: scap-helm zotero finished
  • 16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 16:17 akosiaris@deploy1001: scap-helm zotero finished
  • 16:17 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:17 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 16:15 akosiaris@deploy1001: scap-helm zotero finished
  • 16:15 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:15 akosiaris@deploy1001: scap-helm zotero upgrade production -f zotero-values-eqiad.yaml stable/zotero [namespace: zotero, clusters: eqiad]
  • 16:08 fsero: redeploying zotero on eqiad
  • 16:02 akosiaris@deploy1001: scap-helm zotero finished
  • 16:02 akosiaris@deploy1001: scap-helm zotero cluster codfw completed
  • 16:02 akosiaris@deploy1001: scap-helm zotero cluster eqiad completed
  • 16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 stable/zotero [namespace: zotero, clusters: eqiad,codfw]
  • 16:02 akosiaris@deploy1001: scap-helm zotero upgrade production --set resources.replicas=16 [namespace: zotero, clusters: eqiad,codfw]
  • 15:34 thcipriani: restarting ci jenkins for update
  • 15:33 akosiaris: add back pods/portforward right to kubernetes deploy user. T211040
  • 15:07 anomie: Running cleanupUsersWithNoId.php on potentially missed s3 and s7 wikis for T181731
  • 15:03 fsero: repooling citoid mathoid eqiad
  • 15:03 godog: bootstrap cassandra-c, restbase2017 - T210843
  • 14:56 banyek: executing schema change on s5 codfw master replication lag could be expected - T85757
  • 14:54 fsero: upgrading k8s on eqiad to 1.10.11
  • 14:54 elukey: restart HDFS namenode and Yarn resource manager on an-master100[1,2] to update rack topology config - T209929
  • 14:51 fsero: depool mathoid/eqiad: pooled changed True => False
  • 14:51 fsero: depool citoid/eqiad: pooled changed True => False
  • 14:43 anomie: Running cleanupUsersWithNoId.php on metawiki for T181731 / T210985
  • 14:09 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable arbitrary item/property access for all wiktionaries (T175273) (duration: 00m 47s)
  • 13:53 akosiaris: repool citoid/mathoid codfw
  • 12:59 onimisionipe: banning elastic2001-elastic2024 from codfw production, psi and omega clusters
  • 12:53 jijiki: uploaded python-thumbor-community-core_0.4.0-1+deb9u1 to stretch-wikimedia
  • 12:47 dcausse: EU SWAT done
  • 12:45 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=(cit|math)oid,name=codfw
  • 12:44 dcausse@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: T210381: [cirrus] Add temp clusters but still write to the old ones 2/2 (duration: 00m 46s)
  • 12:42 dcausse@deploy1001: Synchronized wmf-config/CommonSettings.php: T210381: [cirrus] Add temp clusters but still write to the old ones 1/2 (duration: 00m 46s)
  • 12:31 fsero: pooling mathoid and citoid again on codfw
  • 12:30 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 46s)
  • 12:29 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T202497) (duration: 00m 49s)
  • 12:23 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T211188: Increase autoconfirmed count for Meta-Wiki to 5 (duration: 00m 47s)
  • 12:15 fsero: upgrading codfw k8s cluster to 1.10.11
  • 12:15 dcausse: running namespaceDupes & cirrus indexNamespaces on yuewiktionary
  • 12:11 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205546: Define 2 new namespaces for yuewiktionary (duration: 00m 47s)
  • 12:02 fsero: depooling mathoid and citoid servers on codfw for k8s upgrade
  • 11:28 mobrovac@deploy1001: Finished deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127 (duration: 05m 55s)
  • 11:23 mobrovac@deploy1001: Started deploy [citoid/deploy@b10e034]: Truncate Zotero-reported time stamp to date - T211127
  • 11:05 akosiaris: upgrade kubernetes-client and kubernetes-master on staging to 1.10.11
  • 11:04 godog: bootstrap cassandra-b, restbase2017 - T210843
  • 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 (duration: 00m 45s)
  • 10:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 45s)
  • 10:51 ema: cache hosts: begin nginx rolling upgrade to 1.13.6-2+wmf2
  • 10:46 marostegui: Reboot db1090 for kernel upgrade
  • 10:44 moritzm: uploaded jenkins 2.138.4 to jessie-wikimedia/thirdparty and stretch-wikimedia/thirdpary/ci
  • 10:42 marostegui: Stop MySQL on db1090:3312 and db1090:3317 for MySQL upgrade
  • 10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 46s)
  • 10:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 46s)
  • 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1080 (duration: 00m 46s)
  • 10:21 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
  • 10:17 arturo: T205969 icinga downtime the load avg check in labstore1007 for 1 week
  • 10:10 banyek: depooling labsdb1010 for testing materialized views - T210693
  • 10:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1080 (duration: 00m 46s)
  • 09:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly pool db1080 (duration: 00m 46s)
  • 09:32 gehel: setting up new elasticsearch servers on codfw - elastic2045-2054 - T210265
  • 09:22 marostegui: Stop MySQL on db1080 for mysql and kernel upgrade
  • 09:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for MySQL upgrade (duration: 00m 46s)
  • 09:07 elukey: matomo read only + upgrade to matomo 3.7.0 on matomo1001 - T209808
  • 09:00 _joe_: disabed puppet on mw1261, used for logging tests for T211184
  • 08:48 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T207850 Define a new Wikibase log channel to use (duration: 00m 47s)
  • 08:01 moritzm: installing pdns-recursor security update in esams
  • 07:59 godog: bootstrap cassandra-a, restbase2017 - T210843
  • 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 T86338 T202167 (duration: 00m 46s)
  • 06:55 marostegui: Deploy schema change on db1091 T86338 T202167
  • 06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 T86338 T202167 (duration: 00m 51s)
  • 04:03 urandom: bootstrapping cassandra-c, restbase2016 -- T210843
  • 03:11 kartik@deploy1001: Finished deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578) (duration: 04m 26s)
  • 03:06 kartik@deploy1001: Started deploy [cxserver/deploy@a3dd2ca]: Update cxserver to c4240e6 and enable Youdao MT (T208985, T210578)
  • 00:32 dcausse: Evening SWAT done
  • 00:30 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
  • 00:28 dcausse@deploy1001: Synchronized wmf-config/ProductionServices.php: [cirrus] prepare multi-instance services (T210381) (duration: 00m 46s)
  • 00:16 dcausse@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Block notice stats on itwiki (T210452) (duration: 00m 47s)
  • 00:10 urandom: bootstrapping cassandra-b, restbase2016 -- T210843

2018-12-04

  • 23:46 eileen: civicrm revision changed from a411d6bd64 to e757753a46, config revision is 0e6ccc37fe
  • 23:33 XioNoX: update prefix-list peering4 on cr1-eqsin to match jnt
  • 22:46 XioNoX: remove neodymium/sarin from term labs-in4 on cr1/2-eqiad - T210612
  • 22:40 ejegg: Updated payments-wiki from 7403a196b4 to 20595cca97
  • 21:58 XioNoX: clear ethernet-swtiching table for labvirt1004:eth1's switch port
  • 21:57 XioNoX: clear ethernet-swtiching table for labvirt1009:eth1's switch port
  • 21:49 XioNoX: make cr1/2-codfw conform to jnt
  • 21:44 XioNoX: make cr2-eqord/eqdfw conform to jnt
  • 21:41 XioNoX: make cr3/4-ulsfo conform to jnt
  • 20:04 urandom: bootstrapping cassandra-a, restbase2016 -- T210843
  • 19:35 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation (duration: 10m 36s)
  • 19:24 smalyshev@deploy1001: Started deploy [wdqs/wdqs@81dac18]: Install new Updater for T210044 investigation
  • 18:04 joal@deploy1001: Finished deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint (duration: 17m 33s)
  • 18:03 akosiaris: bump zotero pod number from 4 to 16 in eqiad/codfw
  • 17:47 joal@deploy1001: Started deploy [analytics/aqs/deploy@e7d48e9]: Add underestimate and offset to uniques-devices endpoint
  • 17:46 ppchelko@deploy1001: Finished deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465 (duration: 01m 13s)
  • 17:45 ppchelko@deploy1001: Started deploy [changeprop/deploy@e1aeb27]: Do not initialize scores and errors arrays in advance T210465
  • 17:22 Reedy: created oathauth tables on punjabiwikimedia T211110
  • 17:16 godog: bootstrap cassandra-c on restbase2015 - T210843
  • 16:48 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Configure 'api-warning' log channel (duration: 00m 47s)
  • 15:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 T86338 T202167 (duration: 00m 47s)
  • 14:04 elukey: upgrade turnilo on analytics-tools1002 to nodejs-10 - T210705
  • 13:56 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=bnwikisource --fix --add-prefix=T210472
  • 13:53 addshore: addshore@mwmaint1002:~$ mwscript namespaceDupes.php --wiki=euwiki --fix
  • 13:52 marostegui: Deploy schema change on db1103:3314 T86338 T202167
  • 13:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 T86338 T202167 (duration: 00m 47s)
  • 13:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 T86338 T202167 (duration: 00m 46s)
  • 13:34 cdanis: T210416: adding grafana 5 to wikimedia-stretch: reprepro --restrict grafana update stretch-wikimedia
  • 13:33 moritzm: installing nodejs security updates on restbase in codfw
  • 13:32 godog: bootstrap cassandra-b on restbase2015 - T210843
  • 13:29 moritzm: installing nodejs security updates on proton*
  • 13:19 marostegui: Deploy schema change on db1081 T86338 T202167
  • 13:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 T86338 T202167 (duration: 00m 46s)
  • 13:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 T86338 T202167 (duration: 00m 46s)
  • 12:39 Lucas_WMDE: EU SWAT done
  • 12:38 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create namespace "Work" on bnwikisource (T210472) (duration: 00m 46s)
  • 12:33 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create List namespace on euwiki (T209834) (duration: 00m 47s)
  • 12:28 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Milestone logo for atjwiki" (T200713) (duration: 00m 47s)
  • 12:17 moritzm: installing tiff security updates
  • 11:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided) (duration: 04m 47s)
  • 11:44 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda] (dev-cluster): (no justification provided)
  • 11:41 moritzm: rebooting puppetboard2001 to pick up SSBD-enabled qemu
  • 11:38 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242 (duration: 00m 21s)
  • 11:38 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1004 - T197242
  • 11:37 moritzm: rebooting puppetboard1001 to pick up SSBD-enabled qemu
  • 11:36 akosiaris: enable puppet on scb1004, run puppet T197242
  • 11:36 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242 (duration: 00m 20s)
  • 11:35 marostegui: Deploy schema change on db1084 T86338 T202167
  • 11:35 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1003 - T197242
  • 11:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 T86338 T202167 (duration: 00m 47s)
  • 11:34 akosiaris: enable puppet on scb1003, run puppet T197242
  • 11:33 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242 (duration: 00m 28s)
  • 11:33 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1002 - T197242
  • 11:31 akosiaris: enable puppet on scb1002, run puppet T197242
  • 11:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 T86338 T202167 (duration: 00m 46s)
  • 11:18 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242 (duration: 00m 30s)
  • 11:18 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb1001 - T197242
  • 11:17 akosiaris: enable puppet on scb1001, run puppet T197242
  • 11:12 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201 (duration: 09m 06s)
  • 11:04 mobrovac@deploy1001: Finished deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242 (duration: 20m 59s)
  • 11:03 elukey@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Expose offset and underestimate numbers on unique devices - T164201
  • 10:59 fdans@deploy1001: Finished deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques (duration: 00m 37s)
  • 10:58 fdans@deploy1001: Started deploy [analytics/aqs/deploy@e9a63cc]: Deploying offset and underestimate numbers for uniques
  • 10:57 fdans: deploying AQS to expose offset and underestimate numbers on unique devices
  • 10:51 moritzm: rebooting analytics-tool1003 to pick up SSBD-enabled qemu
  • 10:47 moritzm: rebooting analytics-tool1002 to pick up SSBD-enabled qemu
  • 10:43 mobrovac@deploy1001: Started deploy [restbase/deploy@8abcbda]: Disable Citoid test for switching it to Zotero v2 - T211088 T197242
  • 10:41 moritzm: rebooting analytics-tool1001 to pick up SSBD-enabled qemu
  • 10:03 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242 (duration: 01m 45s)
  • 10:02 godog: bootstrap cassandra-a on restbase2015 - T210843
  • 10:01 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 in codfw - T197242
  • 09:59 gehel: upgrading nginx on elasticsearch eqiad
  • 09:54 akosiaris: enable puppet on all scb2*, run puppet T197242
  • 09:52 gehel: upgrading nginx on elasticsearch codfw
  • 09:52 mobrovac@deploy1001: Finished deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242 (duration: 00m 30s)
  • 09:51 mobrovac@deploy1001: Started deploy [citoid/deploy@b902865]: Switch Citoid to Zotero v2 on scb2001 - T197242
  • 09:50 akosiaris: enable puppet on scb2001, run puppet T197242
  • 09:46 akosiaris: disable puppet on scb for citoid migration to zoterov2 T197242
  • 09:46 gehel@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=codfw,cluster=elasticsearch
  • 09:31 gehel: add elastic2039-2044 to cirrus eqiad (new server) - T210265
  • 09:11 gehel: add elastic2038 to cirrus eqiad (new server) - T210265
  • 09:00 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/MediaWiki/electronpdf/action # Ran https://phabricator.wikimedia.org/P7882 for T157012
  • 08:54 marostegui: Deploy schema change on db1097:3314 T86338 T202167
  • 08:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 T86338 T202167 (duration: 00m 47s)
  • 08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*-*.wsp" -delete # T120639
  • 08:46 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/actions$ sudo -u _graphite find . -type f -name "*_*.wsp" -delete # T120639
  • 08:43 marostegui: Deploy schema change on db1102:3314 T86338 T202167
  • 08:42 marostegui: Deploy schema change on dbstore1002:s4 T86338 T202167
  • 08:41 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/datamodel$ sudo -u _graphite rm wikipedia_references.wsp # T121521
  • 08:36 gehel: restarting stuck tilerator on maps* - T204047
  • 08:35 addshore: graphite1004 & graphite2003, /var/lib/carbon/whisper/daily/wikidata/api/wbgetclaims$ sudo -u _graphite find . -type f -name "*.wsp" -delete # T140280
  • 08:11 moritzm: installing perl security updates on jessie/trusty (stretch already updated)
  • 07:49 godog: bootstrap cassandra-c on restbase2014 - T209615
  • 07:26 marostegui: Deploy schema change on s4 codfw master (db2051) with replication T86338 T202167
  • 07:22 marostegui: Deploy schema change on wikitech primary master (db1073) for labswiki and labtestwiki T86338 T202167
  • 06:39 marostegui: Deploy schema change on s5 primary master (db1070) T86338 T202167
  • 06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 T86338 T202167 (duration: 00m 49s)
  • 06:12 marostegui: Deploy schema change on db1110 T86338 T202167
  • 06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 T86338 T202167 (duration: 00m 53s)
  • 01:22 foks: Removing 2FA per request at https://phabricator.wikimedia.org/T210703
  • 01:22 foks: Reset password for user "Orangemike"

2018-12-03

  • 23:54 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/tests/: for completeness (duration: 00m 58s)
  • 23:53 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/resources/src/mediawiki.legacy/: Restore gray coloring for autocomments (T165189 part 2) (duration: 00m 47s)
  • 23:51 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Linker.php: Restore old HTML structure for history section links (T165189 part 1) (duration: 00m 47s)
  • 22:56 sbassett@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/AbuseFilter/includes/api/ApiQueryAbuseLog.php: Deploy security fix for T210329 (duration: 00m 47s)
  • 21:24 mutante: temp. disabling puppet on logstash1007 and logstash1008 to carefully deploy gerrit:476916
  • 21:17 XioNoX: push firewall change to pfw3-eqiad - T211028
  • 21:10 catrope@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/WikimediaEvents/includes/WikimediaEventsHooks.php: Fix ChangesListFilters validation errors (duration: 00m 49s)
  • 21:00 urandom: bootstrapping restbase2014-a -- T210843
  • 20:20 ppchelko@deploy1001: Finished deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema (duration: 01m 13s)
  • 20:19 ppchelko@deploy1001: Started deploy [changeprop/deploy@7470c85]: Start emitting revision-score events with new schema
  • 20:14 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build (duration: 09m 31s)
  • 20:05 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@0c94e5f]: New GUI and updater build
  • 20:05 XioNoX: push firewall change to pfw3-eqiad - T211028
  • 19:59 ppchelko@deploy1001: Finished deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change (duration: 01m 13s)
  • 19:58 ppchelko@deploy1001: Started deploy [changeprop/deploy@867c571]: TEMP: stop production of revision-scor events for schema change
  • 19:39 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/MobileFrontend/resources/mobile.toc/TableOfContents.js: SWAT T210869 Fix Table of contents rendering (duration: 00m 47s)
  • 19:30 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT I3b906e8b1 CS part of setting enhanced RC (duration: 00m 46s)
  • 19:27 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Ic2787309e59e IS part of setting enhanced RC (duration: 00m 47s)
  • 19:25 jforrester@deploy1001: Synchronized php-1.33.0-wmf.6/extensions/Echo/modules/styles/mw.echo.ui.PaginationWidget.less: SWAT T210487 I914b94515 (duration: 00m 47s)
  • 19:15 jforrester@deploy1001: Synchronized wmf-config/ProductionServices.php: SWAT T210381 I73c7596818b Actual config (duration: 00m 46s)
  • 19:09 jforrester@deploy1001: Synchronized wmf-config/CirrusSearch-production.php: SWAT T210381 I2ae162f5 Part II (duration: 00m 47s)
  • 19:08 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT T210381 I2ae162f5 Part I (duration: 00m 46s)
  • 18:42 anomie@deploy1001: Synchronized wmf-config/CommonSettings.php: Updating SkinBuildSidebar hook function for T210528 (duration: 00m 47s)
  • 18:08 gehel: add elastic2037 to cirrus eqiad (new server) - T210265
  • 18:06 godog: bootstrap cassandra-c on restbase2013 - T209615
  • 17:09 bstorm_: T207377 reboot labstore1006 for upgrades
  • 16:28 godog: poweroff ms-be2021 for battery replacement - T208269
  • 15:23 gehel: start configuration of elastic2037-2044 (new servers) - T210265
  • 15:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 T86338 T202167 (duration: 00m 46s)
  • 14:54 marostegui: Deploy schema change on db1100 T86338 T202167
  • 14:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 T86338 T202167 (duration: 00m 48s)
  • 14:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 T86338 T202167 (duration: 00m 47s)
  • 14:17 marostegui: Deploy schema change on db1113:3315 T86338 T202167
  • 14:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 T86338 T202167 (duration: 00m 46s)
  • 14:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 T86338 T202167 (duration: 00m 46s)
  • 13:47 marostegui: Deploy schema change on db1082 (sanitarium master) with replication, lag will be generated on labs (s5) T86338 T202167
  • 13:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 T86338 T202167 (duration: 00m 47s)
  • 13:01 Lucas_WMDE: EU SWAT done
  • 12:44 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleaning of wgLogoHD (T150618) (duration: 00m 46s)
  • 12:39 lucaswerkmeister-wmde@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for multiple projects (T150618) (duration: 00m 47s)
  • 12:37 moritzm: installing nodejs security updates on scb1001
  • 12:34 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Close internalwiki (T205584) (duration: 00m 46s)
  • 12:28 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Change sitename of shnwiki (T206777) (duration: 00m 47s)
  • 12:23 godog: bootstrap cassandra-b on restbase2013 - T209615
  • 12:19 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Perform more PHP constraint checks before falling back (T209504) (duration: 00m 48s)
  • 12:14 lucaswerkmeister-wmde@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Don’t send SPARQL prefixes in WikibaseQualityConstraints (T204317) (duration: 00m 49s)
  • 11:39 godog: more weight to new ms-be codfw hosts - T209395
  • 11:02 moritzm: installing nodejs security updates on stat/notebook hosts
  • 10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 T86338 T202167 (duration: 00m 47s)
  • 10:52 moritzm: rolling upgrade of scb in codfw to nodejs security update
  • 10:39 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 46s)
  • 10:39 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 47s)
  • 10:34 moritzm: installing nodejs security updates on scb2001
  • 10:31 marostegui: Deploy schema change on db1097:3315 T86338 T202167
  • 10:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 T86338 T202167 (duration: 00m 46s)
  • 10:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 T86338 T202167 (duration: 00m 45s)
  • 09:48 marostegui: Deploy schema change on db1096:3315 T86338 T202167
  • 09:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 T86338 T202167 (duration: 00m 47s)
  • 09:27 banyek: executing schema change on db1066 (s2 master) - T85757
  • 09:22 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: repool db1074 (duration: 00m 47s)
  • 09:16 banyek: repooling db1074 - T85757
  • 09:16 banyek: repooling db1074
  • 08:50 banyek: stopping replication on db1074 - T85757
  • 08:49 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T85757: depool db1074 (duration: 00m 48s)
  • 08:44 godog: bootstrap cassandra-a on restbase2013 - T209615
  • 08:43 banyek: depooling db1074 - T85757
  • 08:32 moritzm: restarted keyholder agents/proxies on netmon1002/netmon2001 to pick up removal of netbox key
  • 08:30 marostegui: Deploy schema change on s5 codfw master (db2052) with replication, lag will be generated on codfw T86338 T202167
  • 08:13 moritzm: rearmed keyholders on netmon1002/netmon2001
  • 08:07 marostegui: Deploy schema change on s2 master (db1066) T86338 T202167
  • 07:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 T86338 T202167 (duration: 00m 47s)
  • 07:32 marostegui: Deploy schema change db1074 with replication (lag will appear on labs) T86338 T202167
  • 07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 T86338 T202167 (duration: 00m 46s)
  • 07:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 T86338 T202167 (duration: 00m 47s)
  • 07:09 marostegui: Stop MySQL on pc1004, pc1005 and pc1006 as they will be decommissioned - T210969
  • 06:52 marostegui: Remove pc1004, pc1005 and pc1006 from tendril and zarcillo - T210969
  • 06:38 marostegui: Deploy schema change db1122 T86338 T202167
  • 06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 T86338 T202167 (duration: 00m 48s)
  • 06:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 T86338 T202167 (duration: 00m 47s)
  • 06:16 marostegui: Deploy schema change db1076 T86338 T202167
  • 06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 T86338 T202167 (duration: 00m 50s)
  • 00:34 legoktm@deploy1001: Synchronized php-1.33.0-wmf.6/includes/Title.php: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/477182 (duration: 00m 52s)

2018-12-02

  • 22:39 addshore: addshore@mwmaint1002:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=testwikidatawiki addless # This is my account, and apparently I no longer have the 2fa for it

2018-12-01

  • 16:48 andrewbogott: rebuilding labvirt1014 as cloudvirt1014, T210904


Archives

See Server admin log/Archives.