You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

Server Admin Log

From Wikitech
Jump to: navigation, search

2018-11-14

2018-11-13

  • 22:42 XioNoX: restart librenms irc bot
  • 22:24 XioNoX: add term labnet-nova-api to cloud-in4 on cr1/2-eqiad - T209424
  • 20:22 herron: updated labs realm smarthosts (via hiera) to mx-out0[12].wmflabs.org T41785
  • 19:49 otto@deploy1001: Finished deploy [analytics/refinery@62d6f4b]: Deploy hive jars from CDH 5.10.0 to workaround Refine bug: T209407 (duration: 05m 57s)
  • 19:43 otto@deploy1001: Started deploy [analytics/refinery@62d6f4b]: Deploy hive jars from CDH 5.10.0 to workaround Refine bug: T209407
  • 19:31 herron: uploaded librdkafka_0.11.6-1~bpo9+1+wikimedia1 packages to stretch-wikimedia T209300
  • 18:11 mutante: the CUSTOM message from ores.svc.codfw was the (one-time) test of the new Icinga server
  • 18:03 mutante: icinga migration has concluded, we are now on stretch and icinga1001, einsteinium is passive (T202782)
  • 17:27 mutante: re-enabled puppet on icinga1001, einsteinium becoming passive
  • 17:21 mutante: ran puppet on einsteniumr; e-enabling puppet on tegmen and icinga1001
  • 17:13 bstorm_: Added 172.16.0.0/21 to the allowed connections for wikilabels postgresql on labsdb1004
  • 17:04 mutante: disabled puppet on all 3 icinga servers, re-enabling on einsteinium , going through https://wikitech.wikimedia.org/wiki/Icinga#Failover_Icinga_between_the_active_and_passive_servers
  • 17:02 ejegg: updated payments-wiki from 20542c9184 to 5751286f1c
  • 17:01 mutante: starting migration of icinga server - maintenance windows
  • 16:33 thcipriani: restarting gerrit service for upgrade to 2.15.6
  • 16:32 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@d2763c6]: v2.15.6 to cobalt (duration: 00m 10s)
  • 16:32 thcipriani@deploy1001: Started deploy [gerrit/gerrit@d2763c6]: v2.15.6 to cobalt
  • 16:29 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@d2763c6]: v2.15.6 to gerrit2001 (duration: 00m 11s)
  • 16:29 thcipriani@deploy1001: Started deploy [gerrit/gerrit@d2763c6]: v2.15.6 to gerrit2001
  • 16:22 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-both/read-old on test wikis and mediawikiwiki (T188327) (duration: 00m 54s)
  • 16:07 anomie@mwmaint1002: Running refreshExternallinksIndex.php on labtestwiki for T209373
  • 16:07 anomie@mwmaint1002: Running refreshExternallinksIndex.php on section 3 wikis in group 0 for T209373
  • 15:48 _joe_: upgrading extensions on all appservers / jobrunners while upgrading to php 7.2
  • 15:45 gehel: restart tilerator on maps1004
  • 15:21 moritzm: draining ganeti1006 for reboot/kernel security update
  • 15:18 marostegui: Restore replication consistency options on dbstore2002:3313 as it has caught up - T208320
  • 14:59 akosiaris: increase the migration downtime for kafkamon1001. It should make live migration of these VMs easier and without the need for manual fiddling
  • 14:54 hashar@deploy1001: rebuilt and synchronized wikiversions files: group to 1.33.0-wmf.4 | T206658
  • 14:40 hashar@deploy1001: Finished scap: testwiki to php-1.33.0-wmf.4 | T206658 (duration: 19m 34s)
  • 14:27 moritzm: draining ganeti1007 for reboot/kernel security update
  • 14:20 hashar@deploy1001: Started scap: testwiki to php-1.33.0-wmf.4 | T206658
  • 14:20 akosiaris: reboot logstash1007, logstash1008, logstash1009 with 500 secs of sleep between them for the migration_downtime ganeti setting to be applied
  • 14:18 akosiaris: increase the migration downtime for logstash1007, logstash1008, logstash1009. It should make live migration of these VMs easier and without the need for manual fiddling
  • 14:15 hashar@deploy1001: Pruned MediaWiki: 1.32.0-wmf.24 (duration: 08m 55s)
  • 14:03 hashar: Applied security patches to 1.33.0-wmf.4 | T206658
  • 14:03 gehel: start plugin and JVM upgrade on elasticsearch / cirrus / codfw - T209293
  • 14:00 hashar: scap prep 1.33.0-wmf.4 # T206658
  • 13:58 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Pool pc2007 to replace pc2004 (duration: 00m 48s)
  • 13:41 marostegui: Deploy schema change on s8 codfw master (db2045) this will generate lag on s8 codfw - T203709
  • 13:40 hashar: Cutting wmf/1.33.0-wmf.4 branch | T206658
  • 13:30 moritzm: draining ganeti1008 for reboot/kernel security update
  • 12:51 phuedx: European Mid-day SWAT finished
  • 12:50 phuedx@deploy1001: Finished scap: SWAT: Define WikimediaMessages for Wikibase SEO change l18n refresh (duration: 21m 43s)
  • 12:28 phuedx@deploy1001: Started scap: SWAT: Define WikimediaMessages for Wikibase SEO change l18n refresh
  • 12:22 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/WikimediaMessages/: SWAT: Define WikimediaMessages for Wikibase SEO change (T208755) (duration: 00m 56s)
  • 10:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 52s)
  • 10:47 marostegui: Deploy schema change on db1116:3318 T203709
  • 10:40 godog: stop sending metrics to old graphite hardware
  • 10:15 gehel: restart elasticsearch on relforge for plugin upgrade - T209293
  • 09:54 moritzm: restarting jenkins on releases1001 to pick up Java security update
  • 09:25 _joe_: uploading new versions of php-msgpack, php-geoip compatible with both php 7.0 and php 7.2 to thirdparty/php72 T208433
  • 09:23 marostegui: Deploy schema change on db1092 T203709
  • 09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 52s)
  • 09:20 elukey: rollout new prometheus-mcrouter-exporter to mw* - previous rollout didn't work as expected
  • 09:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 (duration: 00m 55s)
  • 08:37 moritzm: updating remaining rsyslog on stretch to 8.38.0-1~bpo9+1wmf1
  • 07:21 marostegui: Deploy schema change on db1104 T203709
  • 07:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 00m 53s)
  • 07:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 54s)
  • 07:05 elukey: powercycle lvs2006 - mgmt/serial console blank, not responsive since hours ago
  • 06:02 marostegui: Add ipb_sitewide column to db1073:labtestwiki
  • 05:43 marostegui: Stop MySQL on pc2004 to transfer its data to pc2007 - T208383
  • 05:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2004 - T208383 (duration: 00m 53s)
  • 05:39 marostegui: Deploy schema change on db2048 (s1 codfw master), this will create lag on s1 codfw - T114117
  • 05:34 marostegui: Deploy schema change on db1109 T203709
  • 05:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 55s)

2018-11-12

  • 19:22 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T208663 4ff32d1df - Enable moving files for users with patrol and rollbacker rights on srwiki (duration: 00m 54s)
  • 18:29 onimisionipe@deploy1001: Finished deploy [wdqs/wdqs@ee91c41]: GUI update, New Thesaurus endpoint, New updater build and blazegraph update (duration: 11m 28s)
  • 18:17 onimisionipe@deploy1001: Started deploy [wdqs/wdqs@ee91c41]: GUI update, New Thesaurus endpoint, New updater build and blazegraph update
  • 18:03 elukey: rolling restart of aqs on aqs* to pick up new druid datasource settings
  • 17:44 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Optimize s2 for throughput (duration: 00m 53s)
  • 17:19 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Pool more resources into s2 api (duration: 00m 54s)
  • 17:15 _joe_: restarting HHVM on the high-cpu api hosts in eqiad, to ease the pressure and latencies
  • 17:10 _joe_: depooling mw1222 for debug
  • 16:41 banyek: disabling puppet on parsercache hosts (T208383)
  • 16:14 elukey: upgrade prometheus-mcrouter-exporter on all the mw* hosts to the new version
  • 16:09 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for large wikis
  • 16:02 volans: restarted proton on proton1002
  • 15:45 jynus: stop and upgrade db2094
  • 15:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 53s)
  • 14:49 banyek: disabling puppet on parsercache hosts - pc[12]00[456] (T208383)
  • 14:17 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for medium wikis
  • 14:08 moritzm: updating rsyslog on stretch to 8.38.0-1~bpo9+1wmf1
  • 13:59 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for small wikis (small.dblist)
  • 13:59 marostegui: Deploy schema change on db1101:3318 - T203709
  • 13:55 hashar: Upgrading Jenkins on contint1001 , contint2001, releases1001 and releases2002 | T209264
  • 13:46 moritzm: updating libfastjson on stretch to 0.99.8-1~bpo9+1wmf1
  • 13:41 gehel: starting rolling restart of elasticsearch codfw for JVM upgrade
  • 13:32 phuedx: phuedx@mwmaint1002 running restPageRandom.php maintenance script for mediawikiwiki
  • 13:23 phuedx: phuedx@mwmaint1002 running resetPageRandom.php maintenance script for testwiki
  • 13:17 zeljkof: EU SWAT finished
  • 13:16 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/maintenance/resetPageRandom.php: SWAT: Provide a script to reset the page_random column (T208909) (duration: 00m 53s)
  • 13:16 moritzm: updating liblognorm on stretch to 2.0.3-1~bpo9+1wmf1
  • 13:14 phuedx@deploy1001: Synchronized php-1.33.0-wmf.3/autoload.php: SWAT: Provide a script to reset the page_random column (T208909) (duration: 00m 55s)
  • 13:12 elukey: upgrade the Hadoop Analytics cluster to CDH 5.15 (downtime required)
  • 12:54 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add new throttle rule for Wikipedia event in Ireland on 2018-11-13 (T209037) (duration: 00m 53s)
  • 12:15 jiji: Restarting nutcracker on scb200[1-6] - T206450
  • 12:00 moritzm: uploaded jenkins 2.138.3 to apt.wikimedia.org (jessie and stretch)
  • 11:49 hashar: updating puppet CI job for mtail upgrade https://gerrit.wikimedia.org/r/#/c/integration/config/+/472962/
  • 11:37 hashar: contint1001 : cleaning disk | T209123 ?
  • 11:26 moritzm: installing Java security updates on elastic*
  • 10:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 55s)
  • 10:48 godog: upload mtail 3.0.0~rc5-1~bpo9+1wmf1 to stretch-wikimedia
  • 10:45 marostegui: Deploy schema change on db2048 (s1 codfw master), this will generate lag on s1 codfw - T51191
  • 10:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 53s)
  • 10:32 elukey: upload mcrouter exporter 0.0.0+git20181106 to stretch-wikimedia
  • 09:57 elukey: upgraded cdh packages (cdh 5.10 -> 5.15) for thirdparty/cloudera in jessie/stretch-wikimedia
  • 09:12 marostegui: Deploy schema change on db2048 (s1 codfw master) (replication will be stopped) - T67448
  • 08:53 marostegui: Deploy schema change on db1099:3318 - T203709
  • 08:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 (duration: 01m 01s)
  • 08:41 marostegui: Change sync_binlog to 0 and trx_commit to 2 on dbstore2002:3313 to let it catch up
  • {{safesubst:SAL entry|1=08:26 _joe_: uploading new php-{luasandbox,wikidiff2} to stretch main component, rebuild php-{luasandbox,wikidiff2,geoip,msgpack} for php 7.2, upload to stretch component php72, T208433}}
  • 08:23 godog: temporarily disable puppet in codfw before enabling rsyslog_exporter

2018-11-10

  • 01:10 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 39s)
  • 01:10 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting
  • 01:07 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting (duration: 00m 04s)
  • 01:07 smalyshev@deploy1001: Started deploy [wdqs/wdqs@7eeede7]: redeploy runUpdate.sh for better reporting

2018-11-09

  • 21:46 SMalyshev: repooled wdqs1004 - looks like other servers feel worse so probably makes sense to share the load equally
  • 21:16 jiji: Reimaging rdb2003, rdb2004 - T206450
  • 20:46 SMalyshev: depooled wdqs1004 to let it catch up
  • 20:40 legoktm@deploy1001: Synchronized docroot/mediawiki/keys/: Add my (20after4) PGP key to mediawiki.org/keys/keys.(txt|html) (duration: 00m 55s)
  • 20:08 andrewbogott: restarted neutron-linuxbridge-agent on cloudvirt1018 and cloudvirt1023
  • away: repooling labsdb1011 (T189158)
  • 15:24 banyek: depooling labsdb1011 (T189158)
  • 15:23 banyek: depooling labsdb1011
  • 15:08 banyek: repooling labsdb1009 (T189158)
  • 15:06 bblack: cp1008/pinkunicorn: puppet disabled, public-facing testing of new globalsign 2018 certs
  • 15:04 ladsgroup@deploy1001: Finished deploy [ores/deploy@bb39f4b]: T191842 T209060, try II (duration: 14m 43s)
  • 14:50 andrewbogott: rebooting cloudvirt1024 to (I hope) cause a page
  • 14:49 ladsgroup@deploy1001: Started deploy [ores/deploy@bb39f4b]: T191842 T209060, try II
  • 14:48 ladsgroup@deploy1001: deploy aborted: T191842 T209060 (duration: 09m 32s)
  • 14:39 ladsgroup@deploy1001: Started deploy [ores/deploy@0728805]: T191842 T209060
  • 14:18 addshore@deploy1001: Synchronized wmf-config: BETA ONLY: Enable SSR termbox for wikibase on beta - T209143 (duration: 00m 56s)
  • 13:32 moritzm: rebooting acrab for some qemu tests
  • 13:21 godog: upload graphite-web_1.0.2+debian-2.1wmf1 to stretch-wikimedia - T208782
  • 13:10 moritzm: upgrading qemu on ganeti2001 (packages supporting SSBD passthrough)
  • 12:40 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T189158: repool db1106 (duration: 00m 53s)
  • 12:34 banyek: repooling db1106 (T208954)
  • 12:17 kartik@deploy1001: Finished deploy [cxserver/deploy@fc21164]: Update cxserver to 01686f6 (T208831) (duration: 01m 09s)
  • 12:16 kartik@deploy1001: Started deploy [cxserver/deploy@fc21164]: Update cxserver to 01686f6 (T208831)
  • 11:45 banyek: data load finished restarting replication on db1106 (T208954)
  • 11:43 akosiaris: set previous normal wait for scb1001 for apertium service T206439
  • 11:39 akosiaris@puppetmaster1001: conftool action : set/weight=8; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 11:30 akosiaris: upgrade apertium apertium-cat apertium-fra apertium-fra-cat apertium-lex-tools apertium-separable cg3 libapertium3-3.5-1 libcg3-1 lttoolbox on all scb boxes and restart apertium-apy
  • 11:26 akosiaris: upgrade apertium apertium-cat apertium-fra apertium-fra-cat apertium-lex-tools apertium-separable cg3 libapertium3-3.5-1 libcg3-1 lttoolbox on scb1002
  • 11:22 jiji: switch scb*.eqiad.wmnet nutcracker rdb1003:6382 with rdb1005:6379
  • 10:51 vgutierrez: uploaded certcentral 0.6 to apt.wikimedia.org (stretch) - T208859 T208948 T208967 T208970
  • 09:48 ema: repool cp2018, cp2025 (cache_upload) T208588
  • 09:45 banyek: truncating enwiki.archive on db1124 and labsdb hosts too (T208954)
  • 09:21 banyek: stopping replication on db1106 (T208954)
  • 09:21 banyek: stopping replication on db1106 (T208672)
  • 09:08 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: T189158: depool db1106 (duration: 00m 55s)
  • 09:02 banyek: depooling db1106 (T208954)
  • 08:28 moritzm: installing nginx security updates
  • 08:05 ema: repool cp2006, cp2012 (cache_text) T208588
  • 04:33 ejegg: restarted recurring donation charge jobs
  • 04:24 ejegg: updated fundraising CiviCRM from 1154cca3f2 to 71755d021b
  • 03:25 ejegg: updated fundraising CiviCRM from 02cc1f80d4 to 1154cca3f2
  • 00:07 ejegg: updated fundraising CiviCRM from 07183ed7cc to 02cc1f80d4
  • 00:03 ejegg: updated payments-wiki from 983ce3af0f to 20542c9184

2018-11-08

  • 22:48 mutante: gerrit - adding Thomas Arrow to 'wmf-deployment' group for +2 on mw-config for T208491 access request
  • 22:37 mutante: gerrit - adding Lucas Werkmeister (WMDE) to 'wmf-deployment' group for +2 on mw-config for T208518 access request
  • 20:28 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.3
  • 19:35 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Disable wmgUseTwoColConflict everywhere" T205942 T208840 T209012 T209036 (duration: 00m 54s)
  • 18:19 shdubsh: update statsd-proxy to 0.0.9-2 on graphite1004
  • 17:29 banyek: depooling labsdb1009 (T189158)
  • 17:24 banyek: repooling labsdb1010 (T189158)
  • 17:07 godog: upload libfastjson 0.99.8-1~bpo9+1wmf1 version bump only
  • 16:59 akosiaris@deploy1001: scap-helm zotero finished
  • 16:59 akosiaris@deploy1001: scap-helm zotero cluster staging completed
  • 16:59 akosiaris@deploy1001: scap-helm zotero [namespace: zotero, clusters: staging]
  • 16:50 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
  • 16:50 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
  • 16:29 XioNoX: enable Zayo transit on cr3-ulsfo
  • 15:42 chasemp: disable /etc/logrotate.d/udp2log-mw for a bit on mwlog1001
  • 15:25 Amir1: rolling restart of celery on ores nodes (T209060)
  • 15:20 akosiaris: 'cd /srv/deployment/ores/deploy/submodules/wheels && sudo -u deploy-service git lfs pull' on all ores1* and ores2* hosts T209060
  • 15:07 XioNoX: zeroize asw-c8-codfw (decom)
  • 14:12 moritzm: rebooting releases2001 for some tests with ssbd for KVM
  • 13:52 moritzm: installing postgres updates on labsdb1006/1007
  • 13:38 jiji: Done reimaging rdb1006 - T206450
  • 13:37 moritzm: draining ganeti2001 for reboot/kernel security update
  • 13:36 moritzm: failing over ganeti master in codfw from ganeti2001 to ganeti2003
  • 13:13 godog: upload rsyslog 8.38.0-1~bpo9+1wmf1 to stretch-wikimedia, version bump only
  • 13:07 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-enable wmgUseTwoColConflict on dewiki - T205942 T208840 T209012 T209036 (duration: 00m 53s)
  • 12:56 akosiaris: increase weight of scb1001 for apertium to 99+%
  • 12:56 akosiaris@puppetmaster1001: conftool action : set/weight=3800; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 12:53 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-enable wmgUseTwoColConflict on group0 only T205942 T208840 T209012 T209036 (duration: 00m 54s)
  • 12:52 moritzm: draining ganeti2002 for reboot/kernel security update
  • 12:41 jiji: Shutdown and reimage rdb200[56] - T206450
  • 12:31 moritzm: draining ganeti2003 for reboot/kernel security update
  • 12:30 zeljkof: EU SWAT finished
  • 12:29 zfilipin@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/TwoColConflict: SWAT: Fix harmless edits turning into conflicts (T205942 T208840 T209012 T209036) (duration: 00m 55s)
  • 12:19 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set AdvancedSearch to default on group0 wikis (T207641) (duration: 00m 55s)
  • 12:18 moritzm: draining ganeti2004 for reboot/kernel security update
  • 11:57 moritzm: draining ganeti2005 for reboot/kernel security update
  • 11:51 akosiaris: increase weight of scb1001 for apertium to 50%
  • 11:50 akosiaris@puppetmaster1001: conftool action : set/weight=38; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 11:41 moritzm: draining ganeti2006 for reboot/kernel security update
  • 11:18 moritzm: draining ganeti2007 for reboot/kernel security update
  • 11:05 moritzm: draining ganeti2008 for reboot/kernel security update
  • 10:52 jiji: Reimaging rdb1006 to stretch - T206450
  • 10:52 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable wmgUseTwoColConflict everywhere T209012 T208840 T195724 (duration: 00m 58s)
  • 10:26 elukey: restart memcached on mc2029 (was depooled yesterday for network maintenance)
  • 10:23 jiji: restarting pdfrender on scb1003
  • 10:19 volans: restarting pdfrender on scb1004
  • 10:18 volans: restarting pdfrender on scb1002
  • 10:18 _joe_: restarting pdfrender on scb1001
  • 10:02 moritzm: installing ppp security updates on trusty
  • 09:37 godog: keep 2x not 3x copies of older (>15d) logstash elasticsearch indices
  • 09:29 moritzm: installing curl security updates
  • 09:29 godog: temporarily set elasticsearch logstash watermark to low:0.85 and high:0.9
  • 06:34 bawolff@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/OpenStackManager/special/SpecialNovaSudoer.php: T203885 (duration: 00m 54s)
  • 05:30 bawolff: deployed patch T208881
  • 01:18 mutante: scb1004 - systemctl restart pdfrender (T174916)
  • 00:43 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/includes/resourceloader/ResourceLoader.php: ResourceLoader: Fail less hard when JSON serialization of config fails I673f59d93 (duration: 00m 53s)
  • 00:33 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T205368 Enable BotPasswords on Governance wiki (duration: 00m 55s)
  • 00:32 ejegg: updated fundraising CiviCRM from 769dcf6456 to 07183ed7cc
  • 00:26 James_F: Created the bot_passwords table for Governance wiki T205368
  • 00:21 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208449 Disable wgWelcomeSurveyEnabled everywhere in production (duration: 00m 54s)
  • 00:18 jforrester@deploy1001: Synchronized wmf-config/extension-list: T208081 Drop the Petition extension from extension-list (duration: 00m 53s)
  • 00:16 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208081 Drop the Petition extension from InitialiseSettings (duration: 00m 52s)
  • 00:14 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: T208081 Drop the Petition extension from CommonSettings (duration: 00m 53s)
  • 00:12 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208899 Enabling wgMediaInTargetLanguage for testwiki (duration: 00m 54s)
  • 00:00 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T208081 Disable the Petition extension in production (duration: 00m 52s)

2018-11-07

  • 23:48 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Make GrowthExperiments flag operative in CommonSettings (duration: 00m 53s)
  • 23:44 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add flag for GrowthExperiments to InitialiseSettings (duration: 00m 53s)
  • 23:37 catrope@deploy1001: Finished scap: Full scap to rebuild i18n for the addition of the GrowthExperiments extension (duration: 39m 40s)
  • 23:21 jiji: Disabled nagios checks on rdb1006 and rdb2005 due to rdb1005 reimaging - T206450
  • 22:57 catrope@deploy1001: Started scap: Full scap to rebuild i18n for the addition of the GrowthExperiments extension
  • 22:13 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Revert "labswiki rollback to 1.33.0-wmf.2"
  • 22:07 thcipriani@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/LdapAuthentication/LdapAuthenticationPlugin.php: Expose methods used by OpenStackManager T208995 (duration: 00m 54s)
  • 22:06 XenoRyet: updated payments-wiki from 34506ce636 to 983ce3af0f
  • 22:02 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Allow Cloud VPS 172.16.0.0/16 for $wmgAllowLabsAnonEdits wikis T208986 (duration: 00m 54s)
  • 22:02 arlolra: Updated Parsoid to 970751a (T206940)
  • 21:54 arlolra@deploy1001: Finished deploy [parsoid/deploy@4edc771]: Updating Parsoid to 970751a (duration: 09m 34s)
  • 21:45 arlolra@deploy1001: Started deploy [parsoid/deploy@4edc771]: Updating Parsoid to 970751a
  • 21:21 ladsgroup@deploy1001: Finished deploy [ores/deploy@25dfa4f]: T191842 T197096 (duration: 17m 24s)
  • 21:18 krinkle@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/AbuseFilter/includes/AbuseFilter.php: T208144 - I0fdda5 (duration: 00m 53s)
  • 21:16 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/VipsScaler: Id9f82afd (duration: 00m 55s)
  • 21:06 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/AbuseFilter/includes/AbuseFilter.php: T208144 - I0fdda51010243 (duration: 00m 53s)
  • 21:06 banyek: stopping replication on db2072 (T208954)
  • 21:04 krinkle@deploy1001: Synchronized php-1.33.0-wmf.3/includes/jobqueue/jobs/RefreshLinksJob.php: T208147 -I7f5fafe9439d8a7b4 (duration: 00m 54s)
  • 21:03 ladsgroup@deploy1001: Started deploy [ores/deploy@25dfa4f]: T191842 T197096
  • 20:55 banyek: depool labsdb1010 (T189158)
  • 20:24 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: rollback labswiki to 1.33.0-wmf.2
  • 20:12 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.33.0-wmf.3 (duration: 00m 53s)
  • 20:11 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.3
  • 20:00 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T206173 Adding namespaces to Governance wiki (duration: 00m 55s)
  • 19:50 chasemp: labstore1007:~# mkdir /srv/security/
  • 19:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Re-sync for skipped apaches due to maintenance (whoops) (duration: 00m 55s)
  • 19:48 XioNoX: Revert "Redirect eqsin/ulsfo caches to eqiad" - T208272
  • 19:47 XioNoX: repool codfw - T208272
  • 19:45 XioNoX: asw-c-codfw maintenance finished successfuly - T208272
  • 18:51 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T201285: Disable wgRawHTML on Governance wiki (duration: 05m 12s)
  • 18:31 onimisionipe: restarting relforge-eqiad and relforge-eqiad-small-alpha clusters on relforge100[1-2]
  • 18:21 XioNoX: power down asw-c4-codfw - T208272
  • 17:31 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add sourceswiki to wikidata clients (duration: 00m 53s)
  • 17:25 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Inject wikidata rc records on wikidata itself (duration: 00m 53s)
  • 17:21 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WMEUnderstandingFirstDay on testwiki (duration: 00m 53s)
  • 16:37 XioNoX: remove asw-c-codfw FPC8 from config - T208272
  • 16:35 XioNoX: shutdown asw-c-codfw FPC8 - T208272
  • 16:33 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.ui.less: T208898 Hot-deploy to wmf.3 (duration: 00m 53s)
  • 16:22 jforrester@deploy1001: Synchronized php-1.33.0-wmf.3/extensions/Echo/modules/styles: Hot-deploy T208930 to wmf.3 (duration: 00m 54s)
  • 16:20 XioNoX: Enable all VC ports (except uplinks) on spines - T208272
  • 15:58 XioNoX: Redirect eqsin/ulsfo caches to eqiad - T208272
  • 15:57 XioNoX: depool codfw for row C maintenance - T208272
  • 15:36 moritzm: installing Java security updates on relforge*
  • 15:29 jiji@deploy1001: Synchronized wmf-config/ProductionServices.php: Remove jobqueue_redis references, T198220 (duration: 00m 54s)
  • 15:21 akosiaris: T206439 direct 30% of the apertium.svc.eqiad.wmnet traffic to scb1001. Will increase tomorrow to 50%
  • 15:20 akosiaris@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 15:16 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 15:15 akosiaris: T206439 pool upgraded scb1001 to apertium.svc.eqiad.wmnet as a form of canary
  • 15:13 moritzm: uploaded nginx 1.13.6-2+wmf2 to apt.wikimedia.org/stretch-wikimedia
  • 14:55 akosiaris@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,service=apertium,cluster=scb,name=scb1001.*
  • 14:37 oblivian@deploy1001: Synchronized docroot/wwwportal/w/search-redirect.php: Fixing redirects if no language is specified (duration: 00m 54s)
  • 14:33 moritzm: uploaded nginx 1.13.6-2+wmf2~jessie1 to apt.wikimedia.org/jessie-wikimedia
  • 14:32 akosiaris: T206439 upload apertium-cat_2.6.0-1+wmf1 apertium-fra-cat_1.5.0-1+wmf1 apertium-fra_1.5.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 14:26 bblack: rebooting graphite1004
  • 14:16 akosiaris: T206439 upload apertium-separable_0.3.2-1+wmf1 apertium-lex-tools_0.2.1-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 14:07 akosiaris: T206439 upload apertium_3.5.2-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 13:43 bblack: hi
  • 13:21 Amir1: ladsgroup@mwmaint1002:/srv/mediawiki/php-1.33.0-wmf.1$ mwscript sql.php --wiki=sourceswiki extensions/Wikibase/client/sql/entity_usage.sql (T208858)
  • 12:38 zeljkof: EU SWAT finished
  • 12:36 zfilipin@deploy1001: Synchronized wmf-config: SWAT: BC: Enable Schema.org page split test (T208763) (duration: 00m 54s)
  • 12:35 akosiaris: T206439 upload hfst-ospell_0.5.0-1+wmf1to apt.wikimedia.org/jessie-wikimedia/main
  • 12:27 akosiaris: T206439 upload cg3_1.1.7-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 12:16 zfilipin@deploy1001: Synchronized wmf-config/InterwikiSortOrders.php: SWAT: Add dty, gor, inh, kbp and lfn to InterwikiSortOrders (T208217) (duration: 00m 53s)
  • 12:12 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Art&Feminism event in Chile (T208866) (duration: 00m 54s)
  • 12:12 akosiaris: T206439 upload hfst_3.15.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 12:08 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Remove expired throttle rules (duration: 01m 05s)
  • 11:49 akosiaris: T206439 upload lttoolbox_3.5.0-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
  • 11:49 akosiaris: T206439 upload lttoolbox_3.5.0-1+wmf1
  • 10:06 hashar: CI: switched operations/puppet job to be based on Stretch ( T208422 ) and to add python3 ( T208873 )
  • 09:25 _joe_: run systemctl reset-failed on ms-be1029, had a failed debmonitor session
  • 08:16 kartik@deploy1001: Finished deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24 (duration: 04m 59s)
  • 08:11 kartik@deploy1001: Started deploy [cxserver/deploy@6f97d25]: Update cxserver to f9ffd24
  • 02:55 ejegg: disabled recurring charge jobs
  • 00:46 mutante: tegmen - shutting down for renaming and reinstall (T208824)
  • 00:11 dereckson@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 44s)

2018-11-06

  • 23:55 mutante: cp1084 - network went down, powercycled, probably T203194
  • 22:49 ejegg: updated fundraising CiviCRM from e0742d2210 to 769dcf6456
  • 21:50 mutante: icinga1001-"MediaWiki EtcdConfig up-to-date" checks were all UNKNOWN because systemd unit update-etcd-mw-config-lastindex was present but service not running. it was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" and the checks all work (T202782)
  • 21:49 mutante: icinga1001 - the "MediaWiki EtcdConfig up-to-date" checks were all unknown on the new icinga server, this was because systemd unit update-etcd-mw-config-lastindex was present but service not running. that was turned off in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/427328/ on purpose. manually ran "systemctl start update-etcd-mw-config-lastindex" to start it and the checks
  • 21:40 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Group0 wikis to 1.33.0-wmf.3
  • 20:19 thcipriani@deploy1001: Finished scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache (duration: 34m 06s)
  • 19:45 thcipriani@deploy1001: Started scap: testwiki to 1.33.0-wmf.3 and rebuild l10n cache
  • 19:44 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.23 (duration: 07m 19s)
  • 18:36 thcipriani: cutting branch for MediaWiki and extensions version 1.33.0-wmf.3
  • 17:37 XioNoX: add vlan-analytics1-a-eqiad interface-range on asw2-a-eqiad
  • 16:47 XioNoX: enable cr4-ulsfo zayo transport to cr1-codfw
  • 16:23 akosiaris@deploy1001: scap-helm zotero install --name alextest --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
  • 16:23 akosiaris@deploy1001: scap-helm zotero install --set main_app.version=20181019165254-production --set monitoring.enable=true charts/zotero [namespace: zotero, clusters: staging]
  • 16:12 mutante: einsteinium - temp disabling icinga notifications and puppet, reloading icinga (for extra caution while deploying global NRPE change)
  • 16:07 mutante: planet1001 - disabling puppet, editing NRPE config, testing allowed_hosts change
  • 16:07 akosiaris@deploy1001: scap-helm -h finished
  • 16:07 akosiaris@deploy1001: scap-helm -h cluster staging completed
  • 16:07 akosiaris@deploy1001: scap-helm -h [namespace: -h, clusters: staging]
  • 15:59 banyek: updating facts for the puppet compilers
  • 15:37 akosiaris: create zotero namespace in eqiad, codfw, staging cluster T201611
  • 15:20 godog: switch all graphite read traffic to graphite1004
  • 15:16 XioNoX: push `lldp port-id-subtype interface-name` to all compatible switches - T208630
  • 15:14 jiji: scb1001/scb1002 switched nutcracker redis from rdb1001:6382 to rdb1009:6379
  • 15:08 XioNoX: push `lldp port-id-subtype interface-name` to all routers - T208630
  • 14:40 jynus_: reducing consistenct temp. on db2048 to avoid lagging
  • 14:30 moritzm: installing Ruby 2.1 security updates
  • 14:22 godog: add graphite1004 to graphite cluster for reads
  • 14:21 moritzm: installing clamav security updates on mendelevium/ticket.wikimedia.org
  • 13:25 moritzm: restart HHVM on canaries to pick up new curl
  • 13:10 XioNoX: zeroize asw-b-eqiad (decom) - T208788
  • 12:33 moritzm: installing curl security updates
  • 12:17 moritzm: installing chromium security updates on proton* (new upstream release tested in deployment-prep)
  • 12:12 zeljkof: EU SWAT finished
  • 12:10 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Update several Wikidata-related configs (duration: 00m 55s)
  • 11:37 moritzm: installing Ruby 2.3 security updates
  • 11:37 moritzm: installing Ruby 2.3 security updates on trusty
  • 11:30 moritzm: installing Mono security updates
  • 11:23 moritzm: installing Ruby 1.9 security updates on trusty
  • 11:07 banyek: stopping replication on db2077 (T208672)
  • 07:25 _joe_: also restarting on the other eqiad nodes
  • 07:25 _joe_: restarting tilerator on maps1002
  • 04:47 kartik@deploy1001: Finished deploy [cxserver/deploy@ddb0031]: Update cxserver to 17f9a10 (T144467, T198699, T208386) (duration: 05m 26s)
  • 04:42 kartik@deploy1001: Started deploy [cxserver/deploy@ddb0031]: Update cxserver to 17f9a10 (T144467, T198699, T208386)
  • 03:40 eileen: civicrm revision changed from 99895316de to e0742d2210, config revision is e832b5a04a
  • 00:26 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471875/ (duration: 00m 51s)
  • 00:07 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/471874/ (duration: 00m 54s)
  • 00:03 eileen: civicrm revision changed from 042eeaeca9 to 99895316de, config revision is e832b5a04a

2018-11-05

  • 22:41 mutante: sodium - reboot after disk replacement (T202705)
  • 21:51 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100 (duration: 01m 02s)
  • 21:50 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@a92fce5]: Increase cirrusSearchLinksUpdate concurrency to 100
  • 21:49 arlolra: Updated Parsoid to 8ed698b (T205334, T208360)
  • 21:42 mobrovac@deploy1001: Started restart [zotero/translation-server@50f216a]: Free up some memory
  • 21:41 arlolra@deploy1001: Finished deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b (duration: 10m 59s)
  • 21:38 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: Removed decomissioned citoid url T133001 (duration: 00m 53s)
  • 21:30 arlolra@deploy1001: Started deploy [parsoid/deploy@96d739b]: Updating Parsoid to 8ed698b
  • 21:23 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2 (duration: 09m 18s)
  • 21:14 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 take 2
  • 21:10 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324 (duration: 12m 15s)
  • 21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Ensure all wikis on 1.33.0-wmf.2
  • 20:58 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c]: Update deps, removed sections table, T207904 T206048 T207324
  • 20:38 ppchelko@deploy1001: Finished deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table (duration: 03m 40s)
  • 20:35 ppchelko@deploy1001: Started deploy [restbase/deploy@5b8ad3c] (dev-cluster): Update deps, removed sections table
  • 19:53 akosiaris: do a depool, scap deploy, scap wikiversions-compile, hhvm restart and then a pool in eqiad mediawiki servers
  • {{safesubst:SAL entry|1=19:50 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:470868|Disable page issues A/B test (duration: 00m 53s)}}
  • 19:44 sbisson@deploy1001: Synchronized php-1.33.0-wmf.2/includes/block/BlockRestriction.php: SWAT: BlockRestriction::update() unnecessarily does a SELECT on the page table. (duration: 01m 00s)
  • 19:19 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable error logging for WikimediaEvents (duration: 00m 52s)
  • 19:12 sbisson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable GrowthExperiments logging channel (duration: 00m 53s)
  • 18:57 _joe_: restarting hhvm on mwdebug1002
  • 18:44 godog: pool graphite1004 for reads - T196484
  • 18:43 XioNoX: delete asw2-b - asw-b interface - T183585
  • 18:41 XioNoX: remove asw-b-eqiad from LibreNMS - T183585
  • 18:37 XioNoX: remove vrrp priority 70 on cr2-eqiad:ae2 to failback VIPs to cr2 - T183585
  • 18:26 XioNoX: re-enable ae2 on cr2-eqiad - T183585
  • 18:21 thcipriani: rollback mwdebug1001 group2 wikis
  • 18:13 thcipriani: testing php-1.33.0-wmf.2 on group2 wikis on mwdebug1001
  • 18:05 XioNoX: disable ae2 on cr2-eqiad - T183585
  • 18:02 XioNoX: set vrrp priority 70 on cr2-eqiad:ae2 to failover VIP to cr1 - T183585
  • 16:49 XioNoX: Update LLDP config on cr3-ulsfo - T208630
  • 16:48 vgutierrez: uploaded certcentral 0.5 to apt.wikimedia.org (stretch) - T208572 T208378
  • 16:06 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Setting MCR to read-new on all wikis (T198308) (duration: 00m 55s)
  • 13:57 jynus_: increase consistency of db2050, dbstore2002 s3 after them catching up replication T208462
  • 12:33 ladsgroup@deploy1001: Finished deploy [ores/deploy@096ffb3]: T208577 T181632 T208608 (duration: 22m 58s)
  • 12:23 zeljkof: EU SWAT finished
  • 12:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Increase wikidata dispatchers to 3 (duration: 00m 54s)
  • 12:16 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgForeignUploadTargets to [] for zhwiki (T208397) (duration: 00m 54s)
  • 12:10 ladsgroup@deploy1001: Started deploy [ores/deploy@096ffb3]: T208577 T181632 T208608
  • 12:05 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Revert "Anniversary logo for cswiki" (T207589) (duration: 00m 58s)
  • 10:02 godog: reformat xfs filesystems on ms-be1040 - T199198
  • 09:17 elukey@deploy1001: Finished deploy [analytics/refinery@9d39efa]: fixing stat1004 (duration: 00m 04s)
  • 09:17 elukey@deploy1001: Started deploy [analytics/refinery@9d39efa]: fixing stat1004
  • 09:08 joal@deploy1001: Finished deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy (duration: 05m 21s)
  • 09:02 joal@deploy1001: Started deploy [analytics/refinery@9d39efa]: regular analytics weekly deploy

2018-11-04

  • 23:42 jynus_: deleting the same row on all s8 broken servers
  • 23:39 jynus_: deleting one row on db1104
  • 20:38 krinkle@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/FlaggedRevs/frontend/specialpages/reports/ProblemChanges_body.php: T176232 - Ia43626584e (duration: 01m 17s)
  • 18:32 jynus_: reduce temp. consistency level of s4, s5, and s6 codfw masters to prevent excessive lagging due to ongoing mediawiki core maintenance
  • 08:42 eileen: process-control config revision is e832b5a04a renable running job list (all jobs on again now0
  • 08:38 eileen: process-control config revision is e16b2c1c61 renable jobs
  • 02:00 eileen: I think I got the rest of the jobs off process-control config revision is 4422254128
  • 01:52 eileen: process-control config revision is 6ec67b3d01 - also turn off omnirecipient repair job
  • 01:40 eileen: process-control config revision is 5b72cfe874 - reapply turn off q jobs

2018-11-03

  • 09:35 elukey: run tcpdump on mc1035 to grab memcache traffic (rotating pcaps, ~30G maximum)

2018-11-02

  • 17:04 thcipriani: rollback group2 wikis to 1.33.0-wmf.1 on mwdebug100{1,2}
  • 16:54 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1002
  • 16:43 _joe_: live-hacking removal of time limit on mwdebug1001
  • 16:32 thcipriani: deploying 1.33.0-wmf.2 to group2 wikis on mwdebug1001
  • 15:12 jynus: restarting replication @ db2074 after db2094:s3 table fix T208565
  • 15:00 jynus: stopping replication on db2074 to fix db2094:s3 T208565
  • 14:01 vgutierrez: reimaging eeden.wikimedia.org as jessie test system - T208583
  • 11:43 jynus: ignoring cawikimedia.archive replication on db2094:s3 until a reimport happens T208565
  • 11:29 jijiki: Rebooting mw2244 (spare system) for maintenance
  • 10:52 ema: restart varnish-be on cp3032 T208574
  • 08:19 jynus: performing alter table on dbstore2002 s3 and reducing consistency to improve recovery time T208462 T204006
  • 08:01 jynus: reducing consitency on db2050 to improve recovery time T208462
  • 07:59 jynus: performing alter table on db2050 T208462 T204006
  • 07:38 godog: reformat ms-be1043 xfs filesystems - T199198
  • 07:38 jynus: reducing consistency temporarily (flush, binlog sync) at db2040 to prevent lagging
  • 07:26 jynus: reducing consistency temporarily (flush, binlog sync) at db2035 to prevent lagging

2018-11-01

  • 23:01 shdubsh: restart hhvm on mw1261
  • 22:29 ejegg: restarted fundraising queue consumer jobs
  • 22:21 ejegg: updated fundraising CiviCRM from 65130ef3dd to 042eeaeca9
  • 22:18 ejegg: turned off fundraising queue jobs for civi update
  • 22:12 _joe_: rolling restart of hhvm on appservers and api in eqiad
  • 22:09 shdubsh: cumin -b 2 -s 30 "O:mediawiki::appserver and *.eqiad.wmnet" "restart-hhvm"
  • 22:05 _joe_: restarting hhvm on mw1238,1240
  • 22:02 _joe_: restart hhvm on mw1244
  • 21:52 shdubsh: restart hhvm on mw1247
  • 21:49 _joe_: depooling mw1238 for debugging
  • 21:09 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group2 back to 1.33.0-wmf.1
  • 20:55 hoo: Restarted hhvm on mwdebug2002
  • 19:40 hoo: Ran "UPDATE wb_changes_dispatch SET chd_seen = '775203911' WHERE chd_site LIKE '%wikt%' AND chd_seen < '775180000';" on wikidata master (dispatching for wiktionaries)
  • 19:00 hoo@deploy1001: Synchronized php-1.33.0-wmf.1/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 01m 01s)
  • 18:38 hoo@deploy1001: Synchronized php-1.33.0-wmf.2/includes/export/WikiExporter.php: Fix for missing end tag </page> on some exports (T207974) (duration: 00m 55s)
  • 18:25 jijiki: Enabling puppet on mw servers (T206923)
  • 18:19 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove now redundant Wikidata config for wiktionary (T208317) (duration: 00m 54s)
  • 18:12 hoo@deploy1001: Synchronized dblists/wikidataclient.dblist: Add all wiktionaries to wikidataclient.dblist, sort list (T208317) (duration: 00m 57s)
  • 18:02 gehel: restart nginx on relforge100*
  • 17:57 jijiki: Disabling puppet on mw servers (T206923)
  • 16:07 anomie@mwmaint1002: Running migrateComments.php on section 4 wikis for T166733
  • 13:46 anomie@mwmaint1002: Running migrateComments.php on remaining section 3 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on section 7 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on wikitech for T166733
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on wikitech for T188132
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on section 6 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateComments.php on section 8 wikis for T166733
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 8 wikis for T188132
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 7 wikis for T188132
  • 13:37 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 6 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateComments.php on section 5 wikis for T166733
  • 13:36 anomie@mwmaint1002: Running migrateComments.php on section 1 wikis for T166733
  • 13:36 anomie@mwmaint1002: Running migrateComments.php on section 2 wikis for T166733
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 5 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 4 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on remaining section 3 wikis for T188132
  • 13:36 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 2 wikis for T188132
  • 13:35 anomie@mwmaint1002: Running migrateImageCommentTemp.php on section 1 wikis for T188132
  • 12:50 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: List wikidataclient-test in CS.php dblists T208488 (duration: 00m 57s)
  • 09:10 elukey: added a tmux session on mw1314m mw1344, mw1316 that checks mcrouter stats every 10s
  • 00:58 onimisionipe: repooling wdqs1004. It has caught up on lag with others
  • 00:22 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/extension.json: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (invalidate RL cache) (duration: 00m 53s)
  • 00:20 tgr@deploy1001: Synchronized php-1.33.0-wmf.2/extensions/SyntaxHighlight_GeSHi/modules/ve-syntaxhighlight/ve.ui.MWSyntaxHighlightWindow.js: SWAT: Follow-up I3daca6fb: Fix exception thrown when inserting new code block (duration: 00m 54s)
  • 00:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates (duration: 03m 00s)
  • 00:10 mobrovac@deploy1001: Started deploy [restbase/deploy@e8f3a85] (dev-cluster): Add title normalisation and remove Accept-Language header duplicates
  • 00:07 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)
  • 00:05 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Move auth logging to different channels for easier counting (T150300, T123243) (duration: 00m 53s)


Archives

See Server admin log/Archives.