You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Stashbot
(dancy@deploy1002: Synchronized wmf-config/logos.php: Config: Change the Traditional Chinese and Simplified Chinese logo for zhwikinews (T298550) (duration: 01m 07s))
imported>Stashbot
(eileen: config revision changed from 3df415c1 to ecf09aa0 - disable eoy email jobs)
(One intermediate revision by the same user not shown)
Line 1: Line 1:
== 2022-01-07 ==
* 22:07 eileen: config revision changed from {{Gerrit|3df415c1}} to {{Gerrit|ecf09aa0}} - disable eoy email jobs
* 20:08 urbanecm: Purge https://en.wikipedia.org/static/images/project-logos/<nowiki>{</nowiki>zhwikinews,zhwikinews-1.5x,zhwikinews-2x,zhwikinews-hans,zhwikinews-hans-1.5x,zhwikinews-hans-2x<nowiki>}</nowiki>.png via purgeList.php
* 19:49 herron@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host apifeatureusage2001.codfw.wmnet
* 19:41 herron@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host apifeatureusage1001.eqiad.wmnet
* 19:36 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2051.codfw.wmnet with OS bullseye
* 19:21 herron@cumin1001: START - Cookbook sre.ganeti.makevm for new host apifeatureusage2001.codfw.wmnet
* 19:18 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2051.codfw.wmnet with OS bullseye
* 19:16 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2051.codfw.wmnet with OS bullseye
* 19:11 herron@cumin1001: START - Cookbook sre.ganeti.makevm for new host apifeatureusage1001.eqiad.wmnet
* 18:29 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2051.codfw.wmnet with OS bullseye
* 15:18 taavi: reset email address for Ollie Shotton developer account per [[phab:T298779|T298779]]
* 15:08 ottomata: creeating mediainfo-streaming-updater.mutation topics on kafka main-eqiad and main-codfw and setting retention to 30 days - [[phab:T296470|T296470]]
* 14:05 ema: upgrade varnish on deployment-cache-text06 to 6.0.9 [[phab:T298758|T298758]]
* 12:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 12:18 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 12:18 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 12:17 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 12:14 taavi@deploy1002: Synchronized php-1.38.0-wmf.16/extensions/ProofreadPage/modules/page: Backport: [[gerrit:751843{{!}}Makes sure $imgContHorizontal is always initialized (T298694)]] (duration: 00m 59s)
* 11:57 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 11:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 11:56 taavi@deploy1002: Synchronized php-1.38.0-wmf.16/extensions/Flow: Backport: [[gerrit:752014{{!}}Revert "Use strict equality when safe to do so" (T298760)]] (duration: 01m 00s)
* 11:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 11:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 10:40 btullis@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 10:33 btullis@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
* 07:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T297191|T297191]])', diff saved to https://phabricator.wikimedia.org/P18413 and previous config saved to /var/cache/conftool/dbconfig/20220107-072742-marostegui.json
* 07:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18412 and previous config saved to /var/cache/conftool/dbconfig/20220107-071237-marostegui.json
* 06:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18411 and previous config saved to /var/cache/conftool/dbconfig/20220107-065733-marostegui.json
* 06:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 ([[phab:T297191|T297191]])', diff saved to https://phabricator.wikimedia.org/P18410 and previous config saved to /var/cache/conftool/dbconfig/20220107-064228-marostegui.json
* 06:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 ([[phab:T297191|T297191]])', diff saved to https://phabricator.wikimedia.org/P18409 and previous config saved to /var/cache/conftool/dbconfig/20220107-064119-marostegui.json
* 06:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 06:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2114.codfw.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2114.codfw.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2089.codfw.wmnet with reason: Maintenance
* 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2089.codfw.wmnet with reason: Maintenance
* 06:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db[2076,2095].codfw.wmnet with reason: Maintenance
* 06:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db[2076,2095].codfw.wmnet with reason: Maintenance
* 05:47 marostegui: rename wikishared.wikimedia_editor_tasks_targets_passed on db1120 [[phab:T264225|T264225]]
* 00:23 thcipriani@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:752036{{!}}viwiktionary: add namespaces "Appendix" and "Appendix talk" (T298289)]] (duration: 00m 59s)
* 00:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 00:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 00:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 00:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
== 2022-01-06 ==
== 2022-01-06 ==
* 23:52 jhathaway: bouncing blazegraph on wdqs1004
* 23:23 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-test-coord1002.eqiad.wmnet with OS buster
* 22:55 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-test-coord1002.eqiad.wmnet with OS buster
* 22:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 22:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 22:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 22:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 22:25 twentyafterfour@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.16  refs [[phab:T293958|T293958]]
* 22:14 twentyafterfour@deploy1002: Synchronized php-1.38.0-wmf.16/extensions/Scribunto/: sync Scribunto to deploy https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Scribunto/+/752006/ (duration: 01m 08s)
* 22:01 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 22:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 22:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 21:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 20:23 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@3297991]: update rdf-spark-tools jar to 0.3.98 (duration: 02m 15s)
* 20:21 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@3297991]: update rdf-spark-tools jar to 0.3.98
* 20:19 inflatador: banned elastic2051 from both chi and omega search clusters - [[phab:T298674|T298674]]
* 20:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 20:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 20:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 20:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 20:01 twentyafterfour@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Sync https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/751841 (duration: 01m 08s)
* 20:01 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 19:59 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@63c162d]: generate entity revision maps for commons / wcqs (duration: 02m 07s)
* 19:57 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@63c162d]: generate entity revision maps for commons / wcqs
* 19:57 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 19:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 19:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 19:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 19:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 19:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 19:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 19:07 taavi: UTC evening deploys done
* 19:05 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:751538{{!}}Add data.nhm.ac.uk to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T298451)]] (duration: 01m 09s)
* 19:02 razzi: systemctl restart haproxy on dbproxy1018 to repool clouddb1018 for [[phab:T298505|T298505]]
* 18:59 mutante: puppetmaster1001 - creating missing Icinga contact for jgleeson in private puppet repo [[phab:T298649|T298649]]
* 18:51 mutante: contint1001 - after contint2001 also re-enabled puppet and deployed 751816 zuul-merger refactor - service git-daemon refreshed and runnning
* 18:50 razzi: run sudo maintain-views --databases centralauth --replace-all on clouddb1018 for [[phab:T298505|T298505]]
* 18:47 mutante: contint* - deploying zuul-merger puppet refactor change, first codfw-only
* 18:00 btullis@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 00m 09s)
* 18:00 btullis@deploy1002: Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production
* 17:45 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@6f5caf9]: allow for null columns in export to relforge (duration: 02m 11s)
* 17:42 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@6f5caf9]: allow for null columns in export to relforge
* 16:42 otto@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 00m 34s)
* 16:41 otto@deploy1002: Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production
* 16:37 inflatador: restarting elastic2052 for configuration change - [[phab:T298674|T298674]]
* 16:33 taavi: reset wikitech email for User:Iniquity per [[phab:T298683|T298683]]
* 16:30 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 16:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 16:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 16:25 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 16:21 taavi@deploy1002: Synchronized wmf-config/wikitech.php: wikitech: Re-enable Phabricator and Gerrit users after unblock (duration: 01m 09s)
* 16:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 16:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 16:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 16:19 btullis@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 00m 41s)
* 16:18 btullis@deploy1002: Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production
* 16:18 btullis@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 07m 16s)
* 16:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 16:10 btullis@deploy1002: Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production
* 16:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 16:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 16:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 15:57 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 15:52 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 15:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 15:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
* 15:50 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
* 15:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1004.eqiad.wmnet
* 15:00 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1004.eqiad.wmnet
* 13:51 jbond: deploy cfssl_1.6.1-0+deb9u1_amd64 to stretch systems
* 09:57 hashar: Restarting zuul-merger on contint2001 and contint1001 {{!}} https://gerrit.wikimedia.org/r/c/operations/puppet/+/738370/ {{!}} [[phab:T187897|T187897]]
* 07:06 Amir1: revoke DROP from wikiadmin globally
* 02:34 eileen: civicrm revision changed from {{Gerrit|67264062}} to {{Gerrit|3d334f30}}
* 00:32 dancy@deploy1002: Synchronized wmf-config/logos.php: Config: [[gerrit:751530{{!}}Change the Traditional Chinese and Simplified Chinese logo for zhwikinews (T298550)]] (duration: 01m 07s)
* 00:32 dancy@deploy1002: Synchronized wmf-config/logos.php: Config: [[gerrit:751530{{!}}Change the Traditional Chinese and Simplified Chinese logo for zhwikinews (T298550)]] (duration: 01m 07s)
* 00:30 dancy@deploy1002: Synchronized logos/config.yaml: Config: [[gerrit:751530{{!}}Change the Traditional Chinese and Simplified Chinese logo for zhwikinews (T298550)]] (duration: 01m 07s)
* 00:30 dancy@deploy1002: Synchronized logos/config.yaml: Config: [[gerrit:751530{{!}}Change the Traditional Chinese and Simplified Chinese logo for zhwikinews (T298550)]] (duration: 01m 07s)

Revision as of 22:07, 7 January 2022

2022-01-07

  • 22:07 eileen: config revision changed from 3df415c1 to ecf09aa0 - disable eoy email jobs
  • 20:08 urbanecm: Purge https://en.wikipedia.org/static/images/project-logos/{zhwikinews,zhwikinews-1.5x,zhwikinews-2x,zhwikinews-hans,zhwikinews-hans-1.5x,zhwikinews-hans-2x}.png via purgeList.php
  • 19:49 herron@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host apifeatureusage2001.codfw.wmnet
  • 19:41 herron@cumin1001: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host apifeatureusage1001.eqiad.wmnet
  • 19:36 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2051.codfw.wmnet with OS bullseye
  • 19:21 herron@cumin1001: START - Cookbook sre.ganeti.makevm for new host apifeatureusage2001.codfw.wmnet
  • 19:18 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2051.codfw.wmnet with OS bullseye
  • 19:16 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2051.codfw.wmnet with OS bullseye
  • 19:11 herron@cumin1001: START - Cookbook sre.ganeti.makevm for new host apifeatureusage1001.eqiad.wmnet
  • 18:29 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2051.codfw.wmnet with OS bullseye
  • 15:18 taavi: reset email address for Ollie Shotton developer account per T298779
  • 15:08 ottomata: creeating mediainfo-streaming-updater.mutation topics on kafka main-eqiad and main-codfw and setting retention to 30 days - T296470
  • 14:05 ema: upgrade varnish on deployment-cache-text06 to 6.0.9 T298758
  • 12:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:18 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:18 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:17 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:14 taavi@deploy1002: Synchronized php-1.38.0-wmf.16/extensions/ProofreadPage/modules/page: Backport: Makes sure $imgContHorizontal is always initialized (T298694) (duration: 00m 59s)
  • 11:57 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 11:56 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 11:56 taavi@deploy1002: Synchronized php-1.38.0-wmf.16/extensions/Flow: Backport: Revert "Use strict equality when safe to do so" (T298760) (duration: 01m 00s)
  • 11:56 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 11:54 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 10:40 btullis@cumin1001: END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 10:33 btullis@cumin1001: START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons.
  • 07:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T297191)', diff saved to https://phabricator.wikimedia.org/P18413 and previous config saved to /var/cache/conftool/dbconfig/20220107-072742-marostegui.json
  • 07:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18412 and previous config saved to /var/cache/conftool/dbconfig/20220107-071237-marostegui.json
  • 06:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P18411 and previous config saved to /var/cache/conftool/dbconfig/20220107-065733-marostegui.json
  • 06:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T297191)', diff saved to https://phabricator.wikimedia.org/P18410 and previous config saved to /var/cache/conftool/dbconfig/20220107-064228-marostegui.json
  • 06:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3316 (T297191)', diff saved to https://phabricator.wikimedia.org/P18409 and previous config saved to /var/cache/conftool/dbconfig/20220107-064119-marostegui.json
  • 06:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 06:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2117.codfw.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2089.codfw.wmnet with reason: Maintenance
  • 06:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2089.codfw.wmnet with reason: Maintenance
  • 06:14 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db[2076,2095].codfw.wmnet with reason: Maintenance
  • 06:14 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db[2076,2095].codfw.wmnet with reason: Maintenance
  • 05:47 marostegui: rename wikishared.wikimedia_editor_tasks_targets_passed on db1120 T264225
  • 00:23 thcipriani@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: viwiktionary: add namespaces "Appendix" and "Appendix talk" (T298289) (duration: 00m 59s)
  • 00:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 00:16 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 00:16 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 00:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn

2022-01-06

  • 23:52 jhathaway: bouncing blazegraph on wdqs1004
  • 23:23 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-test-coord1002.eqiad.wmnet with OS buster
  • 22:55 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host an-test-coord1002.eqiad.wmnet with OS buster
  • 22:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 22:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 22:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 22:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 22:25 twentyafterfour@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.16 refs T293958
  • 22:14 twentyafterfour@deploy1002: Synchronized php-1.38.0-wmf.16/extensions/Scribunto/: sync Scribunto to deploy https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Scribunto/+/752006/ (duration: 01m 08s)
  • 22:01 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 22:00 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 22:00 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 21:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:23 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@3297991]: update rdf-spark-tools jar to 0.3.98 (duration: 02m 15s)
  • 20:21 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@3297991]: update rdf-spark-tools jar to 0.3.98
  • 20:19 inflatador: banned elastic2051 from both chi and omega search clusters - T298674
  • 20:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:01 twentyafterfour@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Sync https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/751841 (duration: 01m 08s)
  • 20:01 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:59 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@63c162d]: generate entity revision maps for commons / wcqs (duration: 02m 07s)
  • 19:57 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@63c162d]: generate entity revision maps for commons / wcqs
  • 19:57 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:57 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:56 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:09 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:08 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:08 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:07 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:07 taavi: UTC evening deploys done
  • 19:05 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Add data.nhm.ac.uk to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T298451) (duration: 01m 09s)
  • 19:02 razzi: systemctl restart haproxy on dbproxy1018 to repool clouddb1018 for T298505
  • 18:59 mutante: puppetmaster1001 - creating missing Icinga contact for jgleeson in private puppet repo T298649
  • 18:51 mutante: contint1001 - after contint2001 also re-enabled puppet and deployed 751816 zuul-merger refactor - service git-daemon refreshed and runnning
  • 18:50 razzi: run sudo maintain-views --databases centralauth --replace-all on clouddb1018 for T298505
  • 18:47 mutante: contint* - deploying zuul-merger puppet refactor change, first codfw-only
  • 18:00 btullis@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 00m 09s)
  • 18:00 btullis@deploy1002: Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production
  • 17:45 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@6f5caf9]: allow for null columns in export to relforge (duration: 02m 11s)
  • 17:42 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@6f5caf9]: allow for null columns in export to relforge
  • 16:42 otto@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 00m 34s)
  • 16:41 otto@deploy1002: Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production
  • 16:37 inflatador: restarting elastic2052 for configuration change - T298674
  • 16:33 taavi: reset wikitech email for User:Iniquity per T298683
  • 16:30 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 16:26 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 16:26 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 16:25 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 16:21 taavi@deploy1002: Synchronized wmf-config/wikitech.php: wikitech: Re-enable Phabricator and Gerrit users after unblock (duration: 01m 09s)
  • 16:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 16:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 16:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 16:19 btullis@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 00m 41s)
  • 16:18 btullis@deploy1002: Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production
  • 16:18 btullis@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 07m 16s)
  • 16:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 16:10 btullis@deploy1002: Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production
  • 16:03 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 16:01 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 16:01 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 15:57 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 15:52 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 15:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 15:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 15:50 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 15:09 btullis@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1004.eqiad.wmnet
  • 15:00 btullis@cumin1001: START - Cookbook sre.hosts.reboot-single for host stat1004.eqiad.wmnet
  • 13:51 jbond: deploy cfssl_1.6.1-0+deb9u1_amd64 to stretch systems
  • 09:57 hashar: Restarting zuul-merger on contint2001 and contint1001 | https://gerrit.wikimedia.org/r/c/operations/puppet/+/738370/ | T187897
  • 07:06 Amir1: revoke DROP from wikiadmin globally
  • 02:34 eileen: civicrm revision changed from 67264062 to 3d334f30
  • 00:32 dancy@deploy1002: Synchronized wmf-config/logos.php: Config: Change the Traditional Chinese and Simplified Chinese logo for zhwikinews (T298550) (duration: 01m 07s)
  • 00:30 dancy@deploy1002: Synchronized logos/config.yaml: Config: Change the Traditional Chinese and Simplified Chinese logo for zhwikinews (T298550) (duration: 01m 07s)
  • 00:11 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 00:09 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 00:09 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 00:08 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn

2022-01-05

  • 23:50 razzi: sudo systemctl reload haproxy on dbproxy1019 to repool clouddb1014 for T298505
  • 23:26 razzi: run sudo maintain-views --databases centralauth --debug --replace-all on clouddb1014 for T298505
  • 22:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 22:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 22:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 22:02 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 21:57 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 21:51 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 21:51 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 21:45 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 21:25 eileen: civicrm revision 32d7370a -> 67264062
  • 20:50 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:44 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:44 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:39 razzi: reload haproxy on dbproxy1019 to repool clouddb1014 for T298505
  • 20:38 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:33 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:32 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:32 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:25 twentyafterfour@deploy1002: Synchronized php: group1 wikis to 1.38.0-wmf.16 refs T293957 (duration: 01m 07s)
  • 20:23 twentyafterfour@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.16 refs T293957
  • 20:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:14 twentyafterfour@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.38.0-wmf.16 refs T293957
  • 20:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:11 twentyafterfour@deploy1002: Synchronized php-1.38.0-wmf.16/includes/changetags/ChangeTags.php: unblock the train, refs T293957 (duration: 01m 09s)
  • 20:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:59 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:54 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:48 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:48 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:47 urbanecm@deploy1002: Finished scap: 485e72b: Add it namespace aliases in scn (T297844) (duration: 11m 40s)
  • 19:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:41 razzi: reload haproxy on dbproxy1019 (previously incorrectly reloaded dbproxy1018) for T298505
  • 19:37 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:35 urbanecm@deploy1002: Started scap: 485e72b: Add it namespace aliases in scn (T297844)
  • 19:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:34 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: f2da5be: Deploy sticky header (T295976) (duration: 01m 42s)
  • 19:31 razzi: reload haproxy on dbproxy1018 for T298505
  • 19:27 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.13/skins/Vector/resources/skins.vector.es6/stickyHeader.js: f6424f3: Dont use ts-ignore. It is hiding real errors (T297119) (duration: 01m 08s)
  • 19:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:12 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:08 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: aff4ac3: Add www.artsobservasjoner.no to the wgCopyUploadsDomains allowlist of Commons (T298449) (duration: 01m 08s)
  • 19:07 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:06 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 19:06 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 19:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 18:47 jgleeson: localsettings changed from 2d371ed1 to 3df415c1
  • 18:22 bd808: Toolhub: ran `poetry run ./manage.py migrate` against m5-master
  • 18:18 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/toolhub: sync on main
  • 18:16 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/toolhub: apply on main
  • 18:07 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: sync on main
  • 18:06 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply on main
  • 18:04 jgiannelos@deploy1002: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: sync on main
  • 18:03 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply on main
  • 18:03 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: sync on main
  • 18:02 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply on main
  • 18:02 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply on main
  • 17:57 btullis@deploy1002: Finished deploy [analytics/superset/deploy@09094de]: Deployment of Superset 1.3.2 to production (duration: 00m 29s)
  • 17:57 btullis@deploy1002: Started deploy [analytics/superset/deploy@09094de]: Deployment of Superset 1.3.2 to production
  • 17:55 andrew@deploy1002: Finished deploy [horizon/deploy@b300fa6]: minor code format update (duration: 04m 09s)
  • 17:53 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/toolhub: sync on main
  • 17:51 andrew@deploy1002: Started deploy [horizon/deploy@b300fa6]: minor code format update
  • 17:50 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/toolhub: apply on main
  • 17:48 btullis@deploy1002: Finished deploy [analytics/superset/deploy@09094de]: Deployment of Superset 1.3.2 to staging (duration: 00m 39s)
  • 17:47 btullis@deploy1002: Started deploy [analytics/superset/deploy@09094de]: Deployment of Superset 1.3.2 to staging
  • 17:46 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/toolhub: sync on main
  • 17:46 btullis@deploy1002: Finished deploy [analytics/superset/deploy@09094de]: Deployment of Superset 1.3.2 to staging (duration: 03m 11s)
  • 17:42 btullis@deploy1002: Started deploy [analytics/superset/deploy@09094de]: Deployment of Superset 1.3.2 to staging
  • 17:42 btullis@deploy1002: Started deploy [analytics/superset/deploy@09094de]: Deployment for something important
  • 17:36 bd808@deploy1002: helmfile [staging] START helmfile.d/services/toolhub: apply on main
  • 17:26 andrew@deploy1002: Finished deploy [horizon/deploy@15efe04]: sudo panel update (duration: 04m 00s)
  • 17:21 andrew@deploy1002: Started deploy [horizon/deploy@15efe04]: sudo panel update
  • 17:21 andrew@deploy1002: Finished deploy [horizon/deploy@15efe04]: sudo panel update (codfw1dev) (duration: 01m 54s)
  • 17:19 andrew@deploy1002: Started deploy [horizon/deploy@15efe04]: sudo panel update (codfw1dev)
  • 17:18 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 17:17 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 17:17 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 17:16 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 17:11 sbassett: Deployed security fix for T298581 to wmf.16
  • 17:04 sbassett@deploy1002: Synchronized php-1.38.0-wmf.13/extensions/MobileFrontend/includes/specials/SpecialMobileContributions.php: Deploy security fix for T298581 (duration: 01m 08s)
  • 16:51 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 16:51 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 16:38 andrew@deploy1002: Finished deploy [horizon/deploy@5e57e78]: sudo panel update (codfw1dev) (duration: 02m 08s)
  • 16:36 andrew@deploy1002: Started deploy [horizon/deploy@5e57e78]: sudo panel update (codfw1dev)
  • 16:27 andrew@deploy1002: Finished deploy [horizon/deploy@5e57e78]: sudo panel update (duration: 03m 53s)
  • 16:26 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 16:26 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 16:23 andrew@deploy1002: Started deploy [horizon/deploy@5e57e78]: sudo panel update
  • 14:54 aokoth@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 14:50 aokoth@cumin1001: START - Cookbook sre.dns.netbox
  • 13:49 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 13:48 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 13:48 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 13:48 marostegui@cumin1001: dbctl commit (dc=all): 'Repool db2087:3316, db2087:3317 after reimage T295965', diff saved to https://phabricator.wikimedia.org/P18402 and previous config saved to /var/cache/conftool/dbconfig/20220105-134827-marostegui.json
  • 13:47 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 13:42 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 13:41 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 13:41 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 13:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy2001.codfw.wmnet with OS bullseye
  • 13:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 13:38 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.16/extensions/TrustedXFF/: ce7113b: Add more Zscaler ranges (T298241) (duration: 01m 09s)
  • 13:37 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.13/extensions/TrustedXFF/: d35e36f: Add more Zscaler ranges (T298241) (duration: 01m 09s)
  • 13:33 Amir1: delete echo keys from objectchange in frwiki (T272512)
  • 13:23 jelto@deploy1002: helmfile [staging] DONE helmfile.d/services/shellbox: sync on main
  • 13:22 jelto@deploy1002: helmfile [staging] START helmfile.d/services/shellbox: apply on main
  • 13:11 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host dbproxy2001.codfw.wmnet with OS bullseye
  • 13:10 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy2002.codfw.wmnet with OS bullseye
  • 12:38 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host dbproxy2002.codfw.wmnet with OS bullseye
  • 12:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:20 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:20 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:19 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:18 taavi: UTC morning deploys done
  • 12:16 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Add akwiki as an import source for twwiki (T298296) (duration: 01m 09s)
  • 12:14 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:10 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:10 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:07 vgutierrez: pool cp5005 running envoyproxy as TLS terminator - T271421
  • 12:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 11:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy2003.codfw.wmnet with OS bullseye
  • 11:56 jbond: rollout cfssl 1.6.1
  • 11:55 jelto@deploy1002: helmfile [staging] DONE helmfile.d/services/blubberoid: sync on staging
  • 11:55 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp5005.eqsin.wmnet with OS buster
  • 11:55 jelto@deploy1002: helmfile [staging] DONE helmfile.d/services/blubberoid: apply on production
  • 11:55 jelto@deploy1002: helmfile [staging] START helmfile.d/services/blubberoid: apply on staging
  • 11:34 aokoth@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts kubestage1002.eqiad.wmnet
  • 11:24 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host dbproxy2003.codfw.wmnet with OS bullseye
  • 11:24 btullis: updating hive packages in reprepro for log4j update
  • 11:24 aokoth@cumin1001: START - Cookbook sre.hosts.decommission for hosts kubestage1002.eqiad.wmnet
  • 11:20 marostegui@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbproxy2003.codfw.wmnet with OS bullseye
  • 10:54 jbond: upload cfssl 1.6.1
  • 10:52 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host dbproxy2003.codfw.wmnet with OS bullseye
  • 10:48 hashar: CI: switching MediaWiki selenium from php built-in server to Apache # https://gerrit.wikimedia.org/r/751697
  • 10:40 aokoth@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:37 aokoth@cumin1001: START - Cookbook sre.dns.netbox
  • 10:02 dcausse@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' .
  • 10:01 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 8137ffc: pwnwiki: Enable Growth features in dark mode (T298115; 3/3) (duration: 01m 07s)
  • 10:00 dcausse@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' .
  • 10:00 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 09:59 urbanecm@deploy1002: Synchronized wmf-config/config/pwnwiki.yaml: 8137ffc: pwnwiki: Enable Growth features in dark mode (T298115; 2/3) (duration: 01m 07s)
  • 09:59 dcausse@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'rdf-streaming-updater' for release 'main' .
  • 09:58 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 09:58 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 09:58 urbanecm@deploy1002: Synchronized dblists/growthexperiments.dblist: 8137ffc: pwnwiki: Enable Growth features in dark mode (T298115; 1/3) (duration: 01m 07s)
  • 09:57 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 09:53 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.13/extensions/GrowthExperiments/includes/Mentorship/Hooks/MentorFilterHooks.php: 24e15e1: MentorFilterHooks: Include only primary mentors (T298031) (duration: 01m 07s)
  • 09:48 aokoth@cumin1001: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts kubestage1001.eqiad.wmnet
  • 09:42 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 09:41 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 09:41 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 09:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 09:37 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.16/extensions/TrustedXFF/trusted-hosts.php: ab8fe98: Add Zscaler to list of trusted hosts for XFF (T298241) (duration: 01m 08s)
  • 09:35 urbanecm@deploy1002: Synchronized php-1.38.0-wmf.13/extensions/TrustedXFF/trusted-hosts.php: 010d96b: Add Zscaler to list of trusted hosts for XFF (T298241) (duration: 01m 09s)
  • 09:33 aokoth@cumin1001: START - Cookbook sre.hosts.decommission for hosts kubestage1001.eqiad.wmnet
  • 09:29 vgutierrez@cumin1001: START - Cookbook sre.hosts.reimage for host cp5005.eqsin.wmnet with OS buster
  • 09:24 vgutierrez: depool cp5005 to be reimaged as cache::upload_envoy - T271421
  • 08:57 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2087.codfw.wmnet with OS bullseye
  • 08:28 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db2087.codfw.wmnet with OS bullseye
  • 08:25 marostegui@cumin1001: dbctl commit (dc=all): 'Depool db2087:3316, db2087:3317 for Buster reimage T295965', diff saved to https://phabricator.wikimedia.org/P18399 and previous config saved to /var/cache/conftool/dbconfig/20220105-082529-marostegui.json
  • 08:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T297191)', diff saved to https://phabricator.wikimedia.org/P18398 and previous config saved to /var/cache/conftool/dbconfig/20220105-081600-marostegui.json
  • 08:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P18397 and previous config saved to /var/cache/conftool/dbconfig/20220105-080055-marostegui.json
  • 07:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316', diff saved to https://phabricator.wikimedia.org/P18396 and previous config saved to /var/cache/conftool/dbconfig/20220105-074551-marostegui.json
  • 07:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1096:3316 (T297191)', diff saved to https://phabricator.wikimedia.org/P18395 and previous config saved to /var/cache/conftool/dbconfig/20220105-073046-marostegui.json
  • 07:29 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1096:3316 (T297191)', diff saved to https://phabricator.wikimedia.org/P18394 and previous config saved to /var/cache/conftool/dbconfig/20220105-072937-marostegui.json
  • 07:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 07:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1096.eqiad.wmnet with reason: Maintenance
  • 02:20 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 02:19 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 02:19 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 02:18 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 02:13 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 02:13 Amir1: running foreachwikiindblist all maintenance/refreshImageMetadata.php --force --verbose --mediatype=OFFICE --oldimage (T298417)
  • 02:12 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 02:12 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 02:11 ladsgroup@deploy1002: Synchronized php-1.38.0-wmf.16/maintenance/refreshImageMetadata.php: Backport: maintenance: Add support for oldimage table metadata refresh (T298417) (duration: 01m 07s)
  • 02:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 02:09 ladsgroup@deploy1002: Synchronized php-1.38.0-wmf.13/maintenance/refreshImageMetadata.php: Backport: maintenance: Add support for oldimage table metadata refresh (T298417) (duration: 01m 08s)
  • 01:56 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 01:52 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 01:52 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 01:49 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Delete Tematica namespace (NS:104) in Italian Wikivoyage (T298315) (duration: 01m 07s)
  • 01:47 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 01:43 ebernhardson@deploy1002: Synchronized static/images/mobile/copyright/wikivoyage-wordmark-bn.svg: Config: Update bnwikivoyage wordmark logo (T298033) (duration: 01m 07s)
  • 01:42 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 01:41 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Update bnwikivoyage wordmark logo (T298033) (duration: 01m 07s)
  • 01:36 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 01:36 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 01:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 01:24 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 01:18 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 01:18 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 01:12 ebernhardson@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Move CirrusSearch more_like traffic to eqiad (duration: 01m 07s)
  • 01:11 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 01:06 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 01:05 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 01:05 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 01:04 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 01:01 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 34bf91e: GrowthExperiments: Add campaign pattern for JOSA (T298057) (duration: 01m 08s)
  • 00:59 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 00:58 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 00:58 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 00:57 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 00:53 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 7aff17f: Fix wordmark svgs for strategywiki, viwikibooks (T290091; 2/2) (duration: 01m 07s)
  • 00:52 urbanecm@deploy1002: Synchronized static/images/mobile/copyright/: 7aff17f: Fix wordmark svgs for strategywiki, viwikibooks (T290091; 1/2) (duration: 01m 07s)
  • 00:52 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 00:50 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 00:50 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 00:49 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 00:49 urbanecm@deploy1002: Synchronized wmf-config/InitialiseSettings.php: 6c220f0: Enable slow-parsoid logs (duration: 01m 08s)
  • 00:40 twentyafterfour@deploy1002: Synchronized php-1.38.0-wmf.16/includes/content/ContentModelChange.php: fix patch application failure (duration: 01m 07s)
  • 00:37 twentyafterfour@deploy1002: Synchronized php-1.38.0-wmf.16/extensions/VisualEditor/: fix patch application failure (duration: 01m 09s)

2022-01-04

  • 22:55 twentyafterfour@deploy1002: Finished scap: testwikis wikis to 1.38.0-wmf.16 refs T293957 (duration: 37m 56s)
  • 22:23 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 22:22 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 22:22 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 22:21 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 22:17 twentyafterfour@deploy1002: Started scap: testwikis wikis to 1.38.0-wmf.16 refs T293957
  • 21:15 eileen: process-control checkout revision (e58e4e50 -> eb83f208)
  • 21:02 eileen: process-control config 40467fc2 -> e58e4e50
  • 20:44 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:43 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:43 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 20:43 eileen: config b26653a4 -> 40467fc2 (latest)
  • 20:42 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 20:34 eileen: civicrm revision aaceb4ab -> 328c8542
  • 20:33 twentyafterfour_: MediaWiki train for 1.38.0-wmf.16 - ran `scap prep` T293957
  • 16:57 ebernhardson@deploy1002: Finished deploy [wikimedia/discovery/analytics@b38fb58]: Switch mjolnir norm_query_clustering to the shsaded refinery jar (duration: 02m 11s)
  • 16:55 ebernhardson@deploy1002: Started deploy [wikimedia/discovery/analytics@b38fb58]: Switch mjolnir norm_query_clustering to the shsaded refinery jar
  • 16:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T277354)', diff saved to https://phabricator.wikimedia.org/P18388 and previous config saved to /var/cache/conftool/dbconfig/20220104-160930-marostegui.json
  • 15:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P18387 and previous config saved to /var/cache/conftool/dbconfig/20220104-155425-marostegui.json
  • 15:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121', diff saved to https://phabricator.wikimedia.org/P18386 and previous config saved to /var/cache/conftool/dbconfig/20220104-153920-marostegui.json
  • 15:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1121 (T277354)', diff saved to https://phabricator.wikimedia.org/P18384 and previous config saved to /var/cache/conftool/dbconfig/20220104-152416-marostegui.json
  • 15:07 aokoth@deploy1002: helmfile [staging] Ran 'sync' command on namespace 'changeprop-jobqueue' for release 'staging' .
  • 14:34 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 14:31 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 14:31 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 14:29 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 14:21 oblivian@deploy1002: Synchronized docroot: Config: Make symlinks relative so they work on a local checkout too (T285232) (duration: 00m 57s)
  • 14:19 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 14:18 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 14:18 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 14:17 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 14:12 oblivian@deploy1002: Synchronized images: Config: Remove dead symlinks (T285232) (duration: 00m 58s)
  • 14:12 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 14:11 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 14:11 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 14:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 13:57 godog: bump prometheus k8s + ops space in eqiad
  • 13:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2087.codfw.wmnet with reason: Maintenance
  • 13:56 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2087.codfw.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1121 (T277354)', diff saved to https://phabricator.wikimedia.org/P18382 and previous config saved to /var/cache/conftool/dbconfig/20220104-134410-marostegui.json
  • 13:44 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db[1121,1155].eqiad.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db[1121,1155].eqiad.wmnet with reason: Maintenance
  • 13:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T277354)', diff saved to https://phabricator.wikimedia.org/P18381 and previous config saved to /var/cache/conftool/dbconfig/20220104-134359-marostegui.json
  • 13:28 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P18380 and previous config saved to /var/cache/conftool/dbconfig/20220104-132854-marostegui.json
  • 13:13 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141', diff saved to https://phabricator.wikimedia.org/P18379 and previous config saved to /var/cache/conftool/dbconfig/20220104-131349-marostegui.json
  • 13:08 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
  • 13:08 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1164.eqiad.wmnet with reason: Maintenance
  • 13:08 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T298316)', diff saved to https://phabricator.wikimedia.org/P18378 and previous config saved to /var/cache/conftool/dbconfig/20220104-130816-marostegui.json
  • 12:58 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1141 (T277354)', diff saved to https://phabricator.wikimedia.org/P18377 and previous config saved to /var/cache/conftool/dbconfig/20220104-125845-marostegui.json
  • 12:53 taavi: UTC morning deploys done
  • 12:53 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P18376 and previous config saved to /var/cache/conftool/dbconfig/20220104-125312-marostegui.json
  • 12:52 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: prod: WRITE_BOTH for centralauth hidden level migration (T289068) (duration: 00m 57s)
  • 12:44 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:42 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:42 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:41 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:38 marostegui@cumin1001: dbctl commit (dc=all): 'Remove recentchanges from s2 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P18375 and previous config saved to /var/cache/conftool/dbconfig/20220104-123845-marostegui.json
  • 12:38 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311', diff saved to https://phabricator.wikimedia.org/P18374 and previous config saved to /var/cache/conftool/dbconfig/20220104-123807-marostegui.json
  • 12:36 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:35 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:35 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:34 taavi@deploy1002: Synchronized php-1.38.0-wmf.13/extensions/LdapAuthentication/includes/LdapAuthenticationPlugin.php: Backport: Include ldap errno on account creation debug logs (T298508) (duration: 00m 58s)
  • 12:34 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:29 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:27 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:27 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:26 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3311 (T298316)', diff saved to https://phabricator.wikimedia.org/P18373 and previous config saved to /var/cache/conftool/dbconfig/20220104-122302-marostegui.json
  • 12:22 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Create autopatroller and patroller groups on bnwiktionary (T298187) (duration: 00m 57s)
  • 12:21 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:18 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:18 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3311 (T298316)', diff saved to https://phabricator.wikimedia.org/P18372 and previous config saved to /var/cache/conftool/dbconfig/20220104-121643-marostegui.json
  • 12:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 12:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 12:15 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Make reply tool available as opt-out on specieswiki (T297535) (duration: 00m 57s)
  • 12:15 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:13 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Make reply tool available as opt-out on metawiki (T297534) (duration: 00m 59s)
  • 12:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 12:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance
  • 12:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db[1106,1154].eqiad.wmnet with reason: Maintenance
  • 12:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db[1106,1154].eqiad.wmnet with reason: Maintenance
  • 12:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 15 hosts with reason: Maintenance
  • 12:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on 15 hosts with reason: Maintenance
  • 11:54 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 11:54 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 11:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 11:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1184.eqiad.wmnet with reason: Maintenance
  • 11:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 11:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298316)', diff saved to https://phabricator.wikimedia.org/P18370 and previous config saved to /var/cache/conftool/dbconfig/20220104-114503-marostegui.json
  • 11:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P18369 and previous config saved to /var/cache/conftool/dbconfig/20220104-112959-marostegui.json
  • 11:20 jayme@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 11:20 jayme@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 11:18 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 11:17 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311', diff saved to https://phabricator.wikimedia.org/P18368 and previous config saved to /var/cache/conftool/dbconfig/20220104-111454-marostegui.json
  • 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1099:3311 (T298316)', diff saved to https://phabricator.wikimedia.org/P18367 and previous config saved to /var/cache/conftool/dbconfig/20220104-105949-marostegui.json
  • 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1141 (T277354)', diff saved to https://phabricator.wikimedia.org/P18366 and previous config saved to /var/cache/conftool/dbconfig/20220104-105922-marostegui.json
  • 10:59 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 10:59 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1141.eqiad.wmnet with reason: Maintenance
  • 10:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T277354)', diff saved to https://phabricator.wikimedia.org/P18365 and previous config saved to /var/cache/conftool/dbconfig/20220104-105914-marostegui.json
  • 10:52 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1099:3311 (T298316)', diff saved to https://phabricator.wikimedia.org/P18364 and previous config saved to /var/cache/conftool/dbconfig/20220104-105244-marostegui.json
  • 10:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 10:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1099.eqiad.wmnet with reason: Maintenance
  • 10:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 10:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance
  • 10:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P18362 and previous config saved to /var/cache/conftool/dbconfig/20220104-104410-marostegui.json
  • 10:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 10:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance
  • 10:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 10:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1133.eqiad.wmnet with reason: Maintenance
  • 10:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 10:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1128.eqiad.wmnet with reason: Maintenance
  • 10:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 10:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 10:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142', diff saved to https://phabricator.wikimedia.org/P18360 and previous config saved to /var/cache/conftool/dbconfig/20220104-102905-marostegui.json
  • 10:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 10:26 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1140.eqiad.wmnet with reason: Maintenance
  • 10:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:20 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 10:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1142 (T277354)', diff saved to https://phabricator.wikimedia.org/P18359 and previous config saved to /var/cache/conftool/dbconfig/20220104-101400-marostegui.json
  • 09:49 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298316)', diff saved to https://phabricator.wikimedia.org/P18358 and previous config saved to /var/cache/conftool/dbconfig/20220104-094920-marostegui.json
  • 09:34 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P18357 and previous config saved to /var/cache/conftool/dbconfig/20220104-093415-marostegui.json
  • 09:19 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312', diff saved to https://phabricator.wikimedia.org/P18356 and previous config saved to /var/cache/conftool/dbconfig/20220104-091910-marostegui.json
  • 09:04 dcaro: start merging puppet cleanup patches
  • 09:04 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3312 (T298316)', diff saved to https://phabricator.wikimedia.org/P18355 and previous config saved to /var/cache/conftool/dbconfig/20220104-090406-marostegui.json
  • 08:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3312 (T298316)', diff saved to https://phabricator.wikimedia.org/P18354 and previous config saved to /var/cache/conftool/dbconfig/20220104-085127-marostegui.json
  • 08:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 08:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 08:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298316)', diff saved to https://phabricator.wikimedia.org/P18353 and previous config saved to /var/cache/conftool/dbconfig/20220104-085118-marostegui.json
  • 08:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P18352 and previous config saved to /var/cache/conftool/dbconfig/20220104-083613-marostegui.json
  • 08:26 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2094.codfw.wmnet with OS bullseye
  • 08:23 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1142 (T277354)', diff saved to https://phabricator.wikimedia.org/P18351 and previous config saved to /var/cache/conftool/dbconfig/20220104-082306-marostegui.json
  • 08:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 08:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1142.eqiad.wmnet with reason: Maintenance
  • 08:23 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T277354)', diff saved to https://phabricator.wikimedia.org/P18350 and previous config saved to /var/cache/conftool/dbconfig/20220104-082259-marostegui.json
  • 08:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312', diff saved to https://phabricator.wikimedia.org/P18349 and previous config saved to /var/cache/conftool/dbconfig/20220104-082109-marostegui.json
  • 08:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P18348 and previous config saved to /var/cache/conftool/dbconfig/20220104-080754-marostegui.json
  • 08:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3312 (T298316)', diff saved to https://phabricator.wikimedia.org/P18347 and previous config saved to /var/cache/conftool/dbconfig/20220104-080604-marostegui.json
  • 08:00 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3312 (T298316)', diff saved to https://phabricator.wikimedia.org/P18346 and previous config saved to /var/cache/conftool/dbconfig/20220104-080051-marostegui.json
  • 08:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 08:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 07:56 marostegui@cumin1001: START - Cookbook sre.hosts.reimage for host db2094.codfw.wmnet with OS bullseye
  • 07:56 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db[1155-1156].eqiad.wmnet with reason: Maintenance
  • 07:56 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db[1155-1156].eqiad.wmnet with reason: Maintenance
  • 07:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143', diff saved to https://phabricator.wikimedia.org/P18345 and previous config saved to /var/cache/conftool/dbconfig/20220104-075249-marostegui.json
  • 07:52 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 07:52 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 07:49 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 07:49 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 07:47 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 07:47 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance
  • 07:45 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 9 hosts with reason: Maintenance
  • 07:45 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on 9 hosts with reason: Maintenance
  • 07:44 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298316)', diff saved to https://phabricator.wikimedia.org/P18344 and previous config saved to /var/cache/conftool/dbconfig/20220104-074456-marostegui.json
  • 07:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1143 (T277354)', diff saved to https://phabricator.wikimedia.org/P18343 and previous config saved to /var/cache/conftool/dbconfig/20220104-073745-marostegui.json
  • 07:29 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P18342 and previous config saved to /var/cache/conftool/dbconfig/20220104-072951-marostegui.json
  • 07:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P18341 and previous config saved to /var/cache/conftool/dbconfig/20220104-071446-marostegui.json
  • 06:59 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298316)', diff saved to https://phabricator.wikimedia.org/P18340 and previous config saved to /var/cache/conftool/dbconfig/20220104-065942-marostegui.json
  • 06:37 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1105:3312 (T298316)', diff saved to https://phabricator.wikimedia.org/P18339 and previous config saved to /var/cache/conftool/dbconfig/20220104-063714-marostegui.json
  • 06:37 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 06:37 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 06:33 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 06:28 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 06:28 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1139.eqiad.wmnet with reason: Maintenance
  • 06:24 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 06:24 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance
  • 04:21 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1143 (T277354)', diff saved to https://phabricator.wikimedia.org/P18338 and previous config saved to /var/cache/conftool/dbconfig/20220104-042116-marostegui.json
  • 04:21 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 04:21 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1143.eqiad.wmnet with reason: Maintenance
  • 04:21 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18337 and previous config saved to /var/cache/conftool/dbconfig/20220104-042109-marostegui.json
  • 04:06 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P18335 and previous config saved to /var/cache/conftool/dbconfig/20220104-040604-marostegui.json
  • 04:05 ladsgroup@cumin1001: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2144.codfw.wmnet
  • 04:01 ladsgroup@cumin1001: START - Cookbook sre.mysql.upgrade for db2144.codfw.wmnet
  • 03:51 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P18334 and previous config saved to /var/cache/conftool/dbconfig/20220104-035059-marostegui.json
  • 03:50 ladsgroup@cumin1001: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for db2144.codfw.wmnet
  • 03:50 ladsgroup@cumin1001: START - Cookbook sre.mysql.upgrade for db2144.codfw.wmnet
  • 03:36 ladsgroup@cumin1001: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for db2144.codfw.wmnet
  • 03:36 ladsgroup@cumin1001: START - Cookbook sre.mysql.upgrade for db2144.codfw.wmnet
  • 03:35 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18333 and previous config saved to /var/cache/conftool/dbconfig/20220104-033555-marostegui.json
  • 02:30 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 02:29 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 02:29 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 02:28 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 02:08 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 02:07 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 02:07 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 02:06 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 01:51 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1144:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18332 and previous config saved to /var/cache/conftool/dbconfig/20220104-015125-marostegui.json
  • 01:51 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 01:51 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1144.eqiad.wmnet with reason: Maintenance
  • 01:25 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T297094)', diff saved to https://phabricator.wikimedia.org/P18331 and previous config saved to /var/cache/conftool/dbconfig/20220104-012506-marostegui.json
  • 01:10 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P18330 and previous config saved to /var/cache/conftool/dbconfig/20220104-011001-marostegui.json
  • 00:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317', diff saved to https://phabricator.wikimedia.org/P18329 and previous config saved to /var/cache/conftool/dbconfig/20220104-005456-marostegui.json
  • 00:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1098:3317 (T297094)', diff saved to https://phabricator.wikimedia.org/P18328 and previous config saved to /var/cache/conftool/dbconfig/20220104-003951-marostegui.json
  • 00:09 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 00:09 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1145.eqiad.wmnet with reason: Maintenance
  • 00:09 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18327 and previous config saved to /var/cache/conftool/dbconfig/20220104-000947-marostegui.json

2022-01-03

  • 23:54 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P18326 and previous config saved to /var/cache/conftool/dbconfig/20220103-235443-marostegui.json
  • 23:39 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314', diff saved to https://phabricator.wikimedia.org/P18325 and previous config saved to /var/cache/conftool/dbconfig/20220103-233938-marostegui.json
  • 23:24 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1146:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18324 and previous config saved to /var/cache/conftool/dbconfig/20220103-232433-marostegui.json
  • 21:50 cwhite: manually upgrade to grafana 8 on grafana-next (T282863)
  • 21:22 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1146:3314 (T277354)', diff saved to https://phabricator.wikimedia.org/P18323 and previous config saved to /var/cache/conftool/dbconfig/20220103-212216-marostegui.json
  • 21:22 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 21:22 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1146.eqiad.wmnet with reason: Maintenance
  • 21:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T277354)', diff saved to https://phabricator.wikimedia.org/P18322 and previous config saved to /var/cache/conftool/dbconfig/20220103-212209-marostegui.json
  • 21:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P18321 and previous config saved to /var/cache/conftool/dbconfig/20220103-210704-marostegui.json
  • 20:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P18320 and previous config saved to /var/cache/conftool/dbconfig/20220103-205159-marostegui.json
  • 20:36 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1147 (T277354)', diff saved to https://phabricator.wikimedia.org/P18319 and previous config saved to /var/cache/conftool/dbconfig/20220103-203654-marostegui.json
  • 18:53 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1147 (T277354)', diff saved to https://phabricator.wikimedia.org/P18318 and previous config saved to /var/cache/conftool/dbconfig/20220103-185305-marostegui.json
  • 18:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 18:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1147.eqiad.wmnet with reason: Maintenance
  • 18:52 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T277354)', diff saved to https://phabricator.wikimedia.org/P18317 and previous config saved to /var/cache/conftool/dbconfig/20220103-185257-marostegui.json
  • 18:37 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P18316 and previous config saved to /var/cache/conftool/dbconfig/20220103-183752-marostegui.json
  • 18:31 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1098:3317 (T297094)', diff saved to https://phabricator.wikimedia.org/P18315 and previous config saved to /var/cache/conftool/dbconfig/20220103-183130-marostegui.json
  • 18:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 18:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1098.eqiad.wmnet with reason: Maintenance
  • 18:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T297094)', diff saved to https://phabricator.wikimedia.org/P18314 and previous config saved to /var/cache/conftool/dbconfig/20220103-183122-marostegui.json
  • 18:22 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148', diff saved to https://phabricator.wikimedia.org/P18313 and previous config saved to /var/cache/conftool/dbconfig/20220103-182248-marostegui.json
  • 18:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P18312 and previous config saved to /var/cache/conftool/dbconfig/20220103-181617-marostegui.json
  • 18:07 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1148 (T277354)', diff saved to https://phabricator.wikimedia.org/P18311 and previous config saved to /var/cache/conftool/dbconfig/20220103-180743-marostegui.json
  • 18:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317', diff saved to https://phabricator.wikimedia.org/P18310 and previous config saved to /var/cache/conftool/dbconfig/20220103-180112-marostegui.json
  • 17:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1101:3317 (T297094)', diff saved to https://phabricator.wikimedia.org/P18309 and previous config saved to /var/cache/conftool/dbconfig/20220103-174608-marostegui.json
  • 17:13 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudbackup1003.eqiad.wmnet with OS buster
  • 16:57 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1088.eqiad.wmnet with OS buster
  • 16:54 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1086.eqiad.wmnet with OS buster
  • 16:46 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1101:3317 (T297094)', diff saved to https://phabricator.wikimedia.org/P18308 and previous config saved to /var/cache/conftool/dbconfig/20220103-164652-marostegui.json
  • 16:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 16:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1101.eqiad.wmnet with reason: Maintenance
  • 16:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T297094)', diff saved to https://phabricator.wikimedia.org/P18307 and previous config saved to /var/cache/conftool/dbconfig/20220103-164645-marostegui.json
  • 16:43 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster
  • 16:43 cmjohnson@cumin1001: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudbackup1003.eqiad.wmnet with OS buster
  • 16:40 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS buster
  • 16:37 ladsgroup@cumin1001: END (FAIL) - Cookbook sre.mysql.upgrade (exit_code=99) for db2144.codfw.wmnet
  • 16:37 ladsgroup@cumin1001: START - Cookbook sre.mysql.upgrade for db2144.codfw.wmnet
  • 16:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P18306 and previous config saved to /var/cache/conftool/dbconfig/20220103-163140-marostegui.json
  • 16:31 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1088.eqiad.wmnet with OS buster
  • 16:30 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1086.eqiad.wmnet with OS buster
  • 16:29 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1087.eqiad.wmnet with OS buster
  • 16:28 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1085.eqiad.wmnet with OS buster
  • 16:25 cmjohnson@cumin1001: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1084.eqiad.wmnet with OS buster
  • 16:22 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic1088.eqiad.wmnet with OS buster
  • 16:18 cmjohnson@cumin1001: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic1086.eqiad.wmnet with OS buster
  • 16:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127', diff saved to https://phabricator.wikimedia.org/P18305 and previous config saved to /var/cache/conftool/dbconfig/20220103-161635-marostegui.json
  • 16:12 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1148 (T277354)', diff saved to https://phabricator.wikimedia.org/P18304 and previous config saved to /var/cache/conftool/dbconfig/20220103-161232-marostegui.json
  • 16:12 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 16:12 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1148.eqiad.wmnet with reason: Maintenance
  • 16:12 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T277354)', diff saved to https://phabricator.wikimedia.org/P18303 and previous config saved to /var/cache/conftool/dbconfig/20220103-161224-marostegui.json
  • 16:06 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1088.eqiad.wmnet with OS buster
  • 16:05 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1087.eqiad.wmnet with OS buster
  • 16:04 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1086.eqiad.wmnet with OS buster
  • 16:04 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1085.eqiad.wmnet with OS buster
  • 16:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1127 (T297094)', diff saved to https://phabricator.wikimedia.org/P18302 and previous config saved to /var/cache/conftool/dbconfig/20220103-160131-marostegui.json
  • 16:00 cmjohnson@cumin1001: START - Cookbook sre.hosts.reimage for host elastic1084.eqiad.wmnet with OS buster
  • 15:58 vgutierrez: pool cp2029 - T298293
  • 15:57 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P18301 and previous config saved to /var/cache/conftool/dbconfig/20220103-155720-marostegui.json
  • 15:53 moritzm: installing publicsuffix 20211207.1025-0+deb11u1 on bullseye hosts
  • 15:50 moritzm: installing gmp security updates
  • 15:43 moritzm: installing datatables.js security updates
  • 15:42 vgutierrez@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on cp2029.codfw.wmnet with reason: Swapping faulty DIMM with B1
  • 15:42 vgutierrez@cumin1001: START - Cookbook sre.hosts.downtime for 0:30:00 on cp2029.codfw.wmnet with reason: Swapping faulty DIMM with B1
  • 15:42 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149', diff saved to https://phabricator.wikimedia.org/P18300 and previous config saved to /var/cache/conftool/dbconfig/20220103-154215-marostegui.json
  • 15:41 moritzm: installing edk2 security updates
  • 15:27 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1149 (T277354)', diff saved to https://phabricator.wikimedia.org/P18299 and previous config saved to /var/cache/conftool/dbconfig/20220103-152710-marostegui.json
  • 15:15 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1127 (T297094)', diff saved to https://phabricator.wikimedia.org/P18298 and previous config saved to /var/cache/conftool/dbconfig/20220103-151558-marostegui.json
  • 15:15 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 15:15 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1127.eqiad.wmnet with reason: Maintenance
  • 15:15 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T297094)', diff saved to https://phabricator.wikimedia.org/P18297 and previous config saved to /var/cache/conftool/dbconfig/20220103-151550-marostegui.json
  • 15:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P18296 and previous config saved to /var/cache/conftool/dbconfig/20220103-150045-marostegui.json
  • 15:00 hashar: Restarting Gerrit primary on gerrit1001
  • 14:59 hashar: Restarting Gerrit replica on gerrit2001
  • 14:46 jayme: published image docker-registry.discovery.wmnet/cfssl-issuer:0.2.0-1 - T294560
  • 14:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P18295 and previous config saved to /var/cache/conftool/dbconfig/20220103-144539-marostegui.json
  • 14:42 XioNoX: push CR744782 "Deprecate interface-range external" to all routers
  • 14:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T297094)', diff saved to https://phabricator.wikimedia.org/P18293 and previous config saved to /var/cache/conftool/dbconfig/20220103-143034-marostegui.json
  • 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1158 (T297094)', diff saved to https://phabricator.wikimedia.org/P18292 and previous config saved to /var/cache/conftool/dbconfig/20220103-140232-marostegui.json
  • 14:02 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db[1155,1158].eqiad.wmnet with reason: Maintenance
  • 14:02 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db[1155,1158].eqiad.wmnet with reason: Maintenance
  • 14:02 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T297094)', diff saved to https://phabricator.wikimedia.org/P18291 and previous config saved to /var/cache/conftool/dbconfig/20220103-140221-marostegui.json
  • 13:47 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P18290 and previous config saved to /var/cache/conftool/dbconfig/20220103-134716-marostegui.json
  • 13:42 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1149 (T277354)', diff saved to https://phabricator.wikimedia.org/P18289 and previous config saved to /var/cache/conftool/dbconfig/20220103-134227-marostegui.json
  • 13:42 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 13:42 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1149.eqiad.wmnet with reason: Maintenance
  • 13:32 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P18288 and previous config saved to /var/cache/conftool/dbconfig/20220103-133212-marostegui.json
  • 13:17 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T297094)', diff saved to https://phabricator.wikimedia.org/P18287 and previous config saved to /var/cache/conftool/dbconfig/20220103-131707-marostegui.json
  • 13:00 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host build2001.codfw.wmnet
  • 12:46 moritzm: installing openjdk-11 security updates on buster
  • 12:42 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:41 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:41 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:41 taavi: UTC morning deploys done
  • 12:41 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1170:3317 (T297094)', diff saved to https://phabricator.wikimedia.org/P18286 and previous config saved to /var/cache/conftool/dbconfig/20220103-124117-marostegui.json
  • 12:41 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 12:41 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 12:40 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:40 taavi@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: Use new class names for CentralAuth RC feed (duration: 00m 57s)
  • 12:35 taavi@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Add towiki.ru to the wgCopyUploadsDomains allowlist of Wikimedia Commons (T294190) (duration: 00m 57s)
  • 12:35 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:34 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:34 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:33 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:29 taavi@deploy1002: Synchronized wmf-config/logos.php: Config: Add a logo for amiwiki (T298439) (3/3) (duration: 00m 57s)
  • 12:28 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:28 taavi@deploy1002: Synchronized logos/config.yaml: Config: Add a logo for amiwiki (T298439) (2/3) (duration: 00m 57s)
  • 12:26 taavi@deploy1002: Synchronized static/images/project-logos: Config: Add a logo for amiwiki (T298439) (1/3) (duration: 00m 58s)
  • 12:25 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:25 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:22 taavi@deploy1002: Synchronized wmf-config/logos.php: Config: Add a logo for pwnwiki (T298438) (3/3) (duration: 00m 57s)
  • 12:22 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:21 taavi@deploy1002: Synchronized logos/config.yaml: Config: Add a logo for pwnwiki (T298438) (2/3) (duration: 00m 57s)
  • 12:20 taavi@deploy1002: Synchronized static/images/project-logos: Config: Add a logo for pwnwiki (T298438) (1/2) (duration: 00m 58s)
  • 12:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:15 kartik@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Set ContentTranslationContentImportForSectionTranslation for SX (T294642) (duration: 00m 59s)
  • 12:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 12:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 12:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 12:11 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T297094)', diff saved to https://phabricator.wikimedia.org/P18285 and previous config saved to /var/cache/conftool/dbconfig/20220103-121131-marostegui.json
  • 12:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 12:01 moritzm: installing wireshark security updates on stretch
  • 12:00 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 12:00 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 12:00 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T277354)', diff saved to https://phabricator.wikimedia.org/P18284 and previous config saved to /var/cache/conftool/dbconfig/20220103-120011-marostegui.json
  • 11:56 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P18283 and previous config saved to /var/cache/conftool/dbconfig/20220103-115627-marostegui.json
  • 11:54 marostegui@cumin1001: dbctl commit (dc=all): 'Remove watchlist from s2 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P18282 and previous config saved to /var/cache/conftool/dbconfig/20220103-115403-marostegui.json
  • 11:45 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P18281 and previous config saved to /var/cache/conftool/dbconfig/20220103-114507-marostegui.json
  • 11:41 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P18280 and previous config saved to /var/cache/conftool/dbconfig/20220103-114122-marostegui.json
  • 11:37 moritzm: rebalance row_A ganeti group in codfw (to allow to eventually free 2023 of instances)
  • 11:30 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P18279 and previous config saved to /var/cache/conftool/dbconfig/20220103-113002-marostegui.json
  • 11:29 elukey: restart cassandra-b on aqs1010 and aqs1015 (instances stuck / trashing, new cluster, not serving live traffic atm)
  • 11:27 oblivian@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1002.eqiad.wmnet
  • 11:26 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T297094)', diff saved to https://phabricator.wikimedia.org/P18278 and previous config saved to /var/cache/conftool/dbconfig/20220103-112617-marostegui.json
  • 11:19 oblivian@cumin2002: START - Cookbook sre.hosts.reboot-single for host deploy1002.eqiad.wmnet
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1174 (T297094)', diff saved to https://phabricator.wikimedia.org/P18277 and previous config saved to /var/cache/conftool/dbconfig/20220103-111638-marostegui.json
  • 11:16 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T297094)', diff saved to https://phabricator.wikimedia.org/P18276 and previous config saved to /var/cache/conftool/dbconfig/20220103-111631-marostegui.json
  • 11:14 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T277354)', diff saved to https://phabricator.wikimedia.org/P18275 and previous config saved to /var/cache/conftool/dbconfig/20220103-111457-marostegui.json
  • 11:01 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P18274 and previous config saved to /var/cache/conftool/dbconfig/20220103-110126-marostegui.json
  • 10:46 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P18273 and previous config saved to /var/cache/conftool/dbconfig/20220103-104621-marostegui.json
  • 10:41 oblivian@cumin1001: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2002.codfw.wmnet
  • 10:39 marostegui@cumin1001: dbctl commit (dc=all): 'Remove recentchangeslinked from s2 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P18272 and previous config saved to /var/cache/conftool/dbconfig/20220103-103909-marostegui.json
  • 10:32 oblivian@cumin1001: START - Cookbook sre.hosts.reboot-single for host deploy2002.codfw.wmnet
  • 10:31 marostegui@cumin1001: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T297094)', diff saved to https://phabricator.wikimedia.org/P18271 and previous config saved to /var/cache/conftool/dbconfig/20220103-103116-marostegui.json
  • 10:22 elukey: powercycle an-worker1114 (CPU soft lockup errors in mgmt console)
  • 10:20 elukey: powercycle an-worker1120 (CPU soft lockup errors in mgmt console)
  • 10:19 jmm@cumin2002: START - Cookbook sre.ganeti.makevm for new host build2001.codfw.wmnet
  • 10:11 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1181 (T297094)', diff saved to https://phabricator.wikimedia.org/P18270 and previous config saved to /var/cache/conftool/dbconfig/20220103-101116-marostegui.json
  • 10:11 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance
  • 10:11 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1181.eqiad.wmnet with reason: Maintenance
  • 09:59 moritzm: installing ruby2.3 security updates
  • 09:40 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 09:40 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance
  • 09:30 marostegui@cumin1001: dbctl commit (dc=all): 'Depooling db1160 (T277354)', diff saved to https://phabricator.wikimedia.org/P18269 and previous config saved to /var/cache/conftool/dbconfig/20220103-093003-marostegui.json
  • 09:29 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 09:29 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 09:24 moritzm: installing djvulibre security updates on buster
  • 09:05 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 09:05 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1001: dbctl commit (dc=all): 'Remove contributions and logpager from s2 eqiad T263127', diff saved to https://phabricator.wikimedia.org/P18268 and previous config saved to /var/cache/conftool/dbconfig/20220103-085824-marostegui.json
  • 08:54 marostegui@cumin1001: dbctl commit (dc=all): 'Remove special slaves from s2 codfw T263127', diff saved to https://phabricator.wikimedia.org/P18267 and previous config saved to /var/cache/conftool/dbconfig/20220103-085428-marostegui.json
  • 08:49 moritzm: installing libpcap security updates
  • 08:36 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 08:36 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 08:32 ayounsi@cumin1001: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:28 ayounsi@cumin1001: START - Cookbook sre.dns.netbox
  • 08:25 moritzm: installing zziplib security updates
  • 08:07 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2118.codfw.wmnet with reason: Maintenance
  • 08:07 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2118.codfw.wmnet with reason: Maintenance
  • 08:04 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 08:04 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 07:51 moritzm: draining primary and secondary instances off ganeti2023
  • 07:46 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 07:46 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 07:31 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 07:27 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2087.codfw.wmnet with reason: Maintenance
  • 07:27 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2087.codfw.wmnet with reason: Maintenance
  • 07:23 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2086.codfw.wmnet with reason: Maintenance
  • 07:23 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db2086.codfw.wmnet with reason: Maintenance
  • 07:17 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 07:13 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 07:13 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 07:09 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 07:04 mwdebug-deploy@deploy1002: helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 07:03 mwdebug-deploy@deploy1002: helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 07:03 mwdebug-deploy@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn
  • 07:02 ladsgroup@deploy1002: Synchronized wmf-config/InitialiseSettings.php: Config: Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708), Part I (duration: 01m 20s)
  • 07:01 mwdebug-deploy@deploy1002: helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn
  • 07:00 ladsgroup@deploy1002: Synchronized wmf-config/CommonSettings.php: Config: Full roll out of wgMaxExecutionTimeForExpensiveQueries (T297708), Part I (duration: 00m 58s)
  • 06:53 marostegui@cumin1001: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance
  • 06:53 marostegui@cumin1001: START - Cookbook sre.hosts.downtime for 6:00:00 on db[2077,2095].codfw.wmnet with reason: Maintenance
  • 04:21 Amir1: start of running populating actor in revision table on rest of sections. It will take two months to finish (T275246)


Archives

See Server Admin Log/Archives.