Server Admin Log
Appearance
2026-11-19
- 05:04 fab@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/research-tools: apply
- 05:03 fab@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/research-tools: apply
- 04:31 fab@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/research-tools: apply
- 04:30 fab@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/research-tools: apply
- 04:29 fab@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/research-tools: apply
- 04:29 fab@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/research-tools: apply
- 01:33 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough
- 01:13 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
2026-11-18
- 22:54 cooltey@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
- 22:53 cooltey@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
- 22:53 cooltey@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
- 22:53 cooltey@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
- 22:52 cooltey@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
- 22:52 cooltey@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
- 22:31 jforrester@deploy2003: Finished scap sync-world: Backport for Abstractwiki: Define virtual-abstractwiki DB pointer (T443295) (duration: 13m 19s)
- 22:25 jforrester@deploy2003: jforrester: Continuing with deployment
- 22:25 jforrester@deploy2003: jforrester: Backport for Abstractwiki: Define virtual-abstractwiki DB pointer (T443295) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 22:21 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_render_status.sql # T443522
- 22:18 jforrester@deploy2003: Started scap sync-world: Backport for Abstractwiki: Define virtual-abstractwiki DB pointer (T443295)
- 22:11 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
- 22:10 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
- 22:10 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
- 22:10 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy / 1371123
- 22:07 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy / 1371122 (duration: 02m 50s)
- 22:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
- 22:05 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
- 22:04 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
- 22:04 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy / 1371122
- 20:18 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
- 20:18 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
- 20:07 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
- 20:07 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
- 20:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
- 19:59 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
- 19:48 sukhe: sudo cumin -b1 -s10 "A:dnsbox" "run-puppet-agent --enable 'merging CR 1370534'"
- 19:41 sukhe: sudo cumin "A:dnsbox" "disable-puppet 'merging CR 1370534'"
- 19:20 brennen@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.5 refs T442223
- 19:07 brennen: 1.48.0-wmf.5 train status (T442223): no current blockers, rolling to group1
- 18:58 sukhe@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for role: aux_k8s::master@codfw
- 18:58 sukhe@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
- 18:57 sukhe@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
- 18:54 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::master@codfw
- 18:54 sukhe@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for role: aux_k8s::master@eqiad
- 18:54 sukhe@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
- 18:53 sukhe@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
- 18:50 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::master@eqiad
- 18:49 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: deployment_server::kubernetes@eqiad
- 18:48 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: deployment_server::kubernetes@eqiad
- 18:27 ladsgroup@deploy2003: Finished scap sync-world: Bump videoscalers (duration: 03m 32s)
- 18:24 ladsgroup@deploy2003: Started scap sync-world: Bump videoscalers
- 17:58 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T445307)', diff saved to https://phabricator.wikimedia.org/P97542 and previous config saved to /var/cache/conftool/dbconfig/20261118-175818-marostegui.json
- 17:44 cklimas@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
- 17:43 cklimas@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
- 17:43 cklimas@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
- 17:43 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P97541 and previous config saved to /var/cache/conftool/dbconfig/20261118-174316-marostegui.json
- 17:43 cklimas@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
- 17:41 cklimas@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
- 17:40 cklimas@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
- 17:28 kamila@deploy2003: Finished scap sync-world: Backport for Enroll 25% of client sessions in PHP 8.5 (T441689) (duration: 15m 56s)
- 17:28 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224', diff saved to https://phabricator.wikimedia.org/P97540 and previous config saved to /var/cache/conftool/dbconfig/20261118-172814-marostegui.json
- 17:19 kamila@deploy2003: kamila: Continuing with deployment
- 17:19 Mpostoronca-wmf: Updated WikimediaAntiAbusePrivate code
- 17:19 kamila@deploy2003: kamila: Backport for Enroll 25% of client sessions in PHP 8.5 (T441689) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 17:13 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2224 (T445307)', diff saved to https://phabricator.wikimedia.org/P97539 and previous config saved to /var/cache/conftool/dbconfig/20261118-171312-marostegui.json
- 17:12 kamila@deploy2003: Started scap sync-world: Backport for Enroll 25% of client sessions in PHP 8.5 (T441689)
- 17:10 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2224 (T445307)', diff saved to https://phabricator.wikimedia.org/P97538 and previous config saved to /var/cache/conftool/dbconfig/20261118-171042-marostegui.json
- 17:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2224.codfw.wmnet with reason: Maintenance
- 17:10 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T445307)', diff saved to https://phabricator.wikimedia.org/P97537 and previous config saved to /var/cache/conftool/dbconfig/20261118-171024-marostegui.json
- 16:55 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P97536 and previous config saved to /var/cache/conftool/dbconfig/20261118-165522-marostegui.json
- 16:51 robh@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on cirrussearch1069.eqiad.wmnet with reason: migration
- 16:49 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cloudcumin2001.codfw.wmnet
- 16:49 taavi@cumin1004: START - Cookbook sre.hosts.remove-downtime for cloudcumin2001.codfw.wmnet
- 16:48 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cloudcumin2001.codfw.wmnet,cloudcumin1001.eqiad.wmnet
- 16:48 taavi@cumin1004: START - Cookbook sre.hosts.remove-downtime for cloudcumin2001.codfw.wmnet,cloudcumin1001.eqiad.wmnet
- 16:46 taavi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on cloudcumin2001.codfw.wmnet,cloudcumin1001.eqiad.wmnet with reason: test
- 16:46 taavi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on cloudcumin2001.codfw.wmnet with reason: test
- 16:45 taavi@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:05:00 on cloudcumin2001.codfw.wmnet with reason: test
- 16:42 taavi@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:05:00 on cloudcumin2001.codfw.wmnet with reason: test
- 16:40 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P97535 and previous config saved to /var/cache/conftool/dbconfig/20261118-164020-marostegui.json
- 16:38 cwhite@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad
- 16:25 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T445307)', diff saved to https://phabricator.wikimedia.org/P97534 and previous config saved to /var/cache/conftool/dbconfig/20261118-162518-marostegui.json
- 16:24 robh@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on db1176.eqiad.wmnet with reason: migration
- 16:22 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2217 (T445307)', diff saved to https://phabricator.wikimedia.org/P97533 and previous config saved to /var/cache/conftool/dbconfig/20261118-162246-marostegui.json
- 16:22 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2217.codfw.wmnet with reason: Maintenance
- 16:22 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T445307)', diff saved to https://phabricator.wikimedia.org/P97532 and previous config saved to /var/cache/conftool/dbconfig/20261118-162229-marostegui.json
- 16:20 moritzm: installing apache2 security updates
- 16:19 cwhite@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad
- 16:17 cwhite@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw
- 16:09 robh@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on db1176.eqiad.wmnet with reason: migration
- 16:07 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P97531 and previous config saved to /var/cache/conftool/dbconfig/20261118-160727-marostegui.json
- 16:00 cwhite@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw
- 15:52 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P97530 and previous config saved to /var/cache/conftool/dbconfig/20261118-155225-marostegui.json
- 15:44 Emperor: thanos/search - swift delete t-309648 T309648
- 15:44 Emperor: thanos/search - swift delete snapperson T309648
- 15:43 Emperor: thanos/search - swift delete rando+segments T309648
- 15:42 Emperor: thanos/search - swift delete rando T309648
- 15:41 Emperor: thanos/search - swift delete elasticsearch-snapshot+segments T309648
- 15:41 Emperor: thanos/search - swift delete __dunder__ T309648
- 15:37 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T445307)', diff saved to https://phabricator.wikimedia.org/P97529 and previous config saved to /var/cache/conftool/dbconfig/20261118-153722-marostegui.json
- 15:34 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2214 (T445307)', diff saved to https://phabricator.wikimedia.org/P97528 and previous config saved to /var/cache/conftool/dbconfig/20261118-153452-marostegui.json
- 15:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2214.codfw.wmnet with reason: Maintenance
- 15:34 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2197.codfw.wmnet with reason: Maintenance
- 15:33 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T445307)', diff saved to https://phabricator.wikimedia.org/P97527 and previous config saved to /var/cache/conftool/dbconfig/20261118-153347-marostegui.json
- 15:18 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P97526 and previous config saved to /var/cache/conftool/dbconfig/20261118-151845-marostegui.json
- 15:03 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P97525 and previous config saved to /var/cache/conftool/dbconfig/20261118-150343-marostegui.json
- 14:51 awight: UTC afternoon deployment finished
- 14:49 awight@deploy2003: Finished scap sync-world: Backport for Disable hCaptcha on beta wikidata (T445298), Enable the new Wikidata mobile UI (MEX / wbui2025) on testwikidata (T445295) (duration: 16m 39s)
- 14:48 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T445307)', diff saved to https://phabricator.wikimedia.org/P97524 and previous config saved to /var/cache/conftool/dbconfig/20261118-144841-marostegui.json
- 14:46 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2193 (T445307)', diff saved to https://phabricator.wikimedia.org/P97523 and previous config saved to /var/cache/conftool/dbconfig/20261118-144619-marostegui.json
- 14:46 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2193.codfw.wmnet with reason: Maintenance
- 14:46 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T445307)', diff saved to https://phabricator.wikimedia.org/P97522 and previous config saved to /var/cache/conftool/dbconfig/20261118-144601-marostegui.json
- 14:45 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
- 14:45 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
- 14:44 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
- 14:44 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
- 14:43 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
- 14:43 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
- 14:42 awight@deploy2003: arthurtaylor, awight: Continuing with deployment
- 14:36 awight@deploy2003: arthurtaylor, awight: Backport for Disable hCaptcha on beta wikidata (T445298), Enable the new Wikidata mobile UI (MEX / wbui2025) on testwikidata (T445295) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:32 awight@deploy2003: Started scap sync-world: Backport for Disable hCaptcha on beta wikidata (T445298), Enable the new Wikidata mobile UI (MEX / wbui2025) on testwikidata (T445295)
- 14:31 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P97521 and previous config saved to /var/cache/conftool/dbconfig/20261118-143059-marostegui.json
- 14:30 awight@deploy2003: Finished scap sync-world: Backport for Partially revert I7c9b9305c0fc90058d148a44e9d738706a70a0e5 (T440856) (1.48.0-wmf.5) (duration: 09m 15s)
- 14:24 awight@deploy2003: awight: Continuing with deployment
- 14:23 awight@deploy2003: awight: Backport for Partially revert I7c9b9305c0fc90058d148a44e9d738706a70a0e5 (T440856) (1.48.0-wmf.5) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:21 awight@deploy2003: Started scap sync-world: Backport for Partially revert I7c9b9305c0fc90058d148a44e9d738706a70a0e5 (T440856) (1.48.0-wmf.5)
- 14:20 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
- 14:18 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
- 14:16 sbisson@deploy2003: Finished scap sync-world: Backport for Record the outline of articles saved from Article Guidance (T444536) (1.48.0-wmf.5) (duration: 11m 15s)
- 14:15 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P97519 and previous config saved to /var/cache/conftool/dbconfig/20261118-141557-marostegui.json
- 14:10 sbisson@deploy2003: sbisson: Continuing with deployment
- 14:09 hnowlan@dns1004: END - running authdns-update
- 14:07 sbisson@deploy2003: sbisson: Backport for Record the outline of articles saved from Article Guidance (T444536) (1.48.0-wmf.5) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:06 hnowlan@dns1004: START - running authdns-update
- 14:05 sbisson@deploy2003: Started scap sync-world: Backport for Record the outline of articles saved from Article Guidance (T444536) (1.48.0-wmf.5)
- 14:00 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T445307)', diff saved to https://phabricator.wikimedia.org/P97518 and previous config saved to /var/cache/conftool/dbconfig/20261118-140055-marostegui.json
- 13:58 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2180 (T445307)', diff saved to https://phabricator.wikimedia.org/P97517 and previous config saved to /var/cache/conftool/dbconfig/20261118-135831-marostegui.json
- 13:58 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2180.codfw.wmnet with reason: Maintenance
- 13:58 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T445307)', diff saved to https://phabricator.wikimedia.org/P97516 and previous config saved to /var/cache/conftool/dbconfig/20261118-135813-marostegui.json
- 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2219 (T441633)', diff saved to https://phabricator.wikimedia.org/P97515 and previous config saved to /var/cache/conftool/dbconfig/20261118-135726-ladsgroup.json
- 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2219.codfw.wmnet with reason: Maintenance
- 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T441633)', diff saved to https://phabricator.wikimedia.org/P97514 and previous config saved to /var/cache/conftool/dbconfig/20261118-135704-ladsgroup.json
- 13:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P97513 and previous config saved to /var/cache/conftool/dbconfig/20261118-134703-ladsgroup.json
- 13:43 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P97512 and previous config saved to /var/cache/conftool/dbconfig/20261118-134311-marostegui.json
- 13:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P97511 and previous config saved to /var/cache/conftool/dbconfig/20261118-133701-ladsgroup.json
- 13:28 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P97510 and previous config saved to /var/cache/conftool/dbconfig/20261118-132809-marostegui.json
- 13:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T441633)', diff saved to https://phabricator.wikimedia.org/P97509 and previous config saved to /var/cache/conftool/dbconfig/20261118-132659-ladsgroup.json
- 13:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2210 (T441633)', diff saved to https://phabricator.wikimedia.org/P97508 and previous config saved to /var/cache/conftool/dbconfig/20261118-131631-ladsgroup.json
- 13:16 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2210.codfw.wmnet with reason: Maintenance
- 13:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T441633)', diff saved to https://phabricator.wikimedia.org/P97507 and previous config saved to /var/cache/conftool/dbconfig/20261118-131610-ladsgroup.json
- 13:14 sfaci@deploy2003: Finished scap deploy-service test-kitchen: Updated to node26 (duration: 01m 06s)
- 13:13 marostegui@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T445307)', diff saved to https://phabricator.wikimedia.org/P97506 and previous config saved to /var/cache/conftool/dbconfig/20261118-131307-marostegui.json
- 13:12 sfaci@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
- 13:12 sfaci@deploy2003: Started scap deploy-service test-kitchen: Updated to node26
- 13:11 Dreamy_Jazz: Created `global_block_local_status` table in `centralauth` DB for T437037
- 13:10 marostegui@cumin1004: dbctl commit (dc=all): 'Depooling db2158 (T445307)', diff saved to https://phabricator.wikimedia.org/P97505 and previous config saved to /var/cache/conftool/dbconfig/20261118-131032-marostegui.json
- 13:10 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2158.codfw.wmnet with reason: Maintenance
- 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P97504 and previous config saved to /var/cache/conftool/dbconfig/20261118-130608-ladsgroup.json
- 13:06 sfaci@deploy2003: Finished scap deploy-service test-kitchen-next: Updated to node26 (duration: 02m 25s)
- 13:03 sfaci@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
- 13:03 sfaci@deploy2003: Started scap deploy-service test-kitchen-next: Updated to node26
- 13:02 Dreamy_Jazz: Running `/usr/local/bin/foreachwikiindblist "all.dblist - mediamoderation-continuous-scan.dblist - preinstall.dblist" extensions/MediaModeration/maintenance/scanFilesInScanTable.php --use-jobqueue --sleep=1 --poll-sleep=10 --verbose` for T439783
- 12:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P97503 and previous config saved to /var/cache/conftool/dbconfig/20261118-125606-ladsgroup.json
- 12:50 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 12:49 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 12:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T441633)', diff saved to https://phabricator.wikimedia.org/P97502 and previous config saved to /var/cache/conftool/dbconfig/20261118-124604-ladsgroup.json
- 12:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2206 (T441633)', diff saved to https://phabricator.wikimedia.org/P97501 and previous config saved to /var/cache/conftool/dbconfig/20261118-123830-ladsgroup.json
- 12:38 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2206.codfw.wmnet with reason: Maintenance
- 12:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T441633)', diff saved to https://phabricator.wikimedia.org/P97500 and previous config saved to /var/cache/conftool/dbconfig/20261118-123808-ladsgroup.json
- 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P97499 and previous config saved to /var/cache/conftool/dbconfig/20261118-122806-ladsgroup.json
- 12:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P97498 and previous config saved to /var/cache/conftool/dbconfig/20261118-121804-ladsgroup.json
- 12:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T441633)', diff saved to https://phabricator.wikimedia.org/P97497 and previous config saved to /var/cache/conftool/dbconfig/20261118-120802-ladsgroup.json
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 11:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2172 (T441633)', diff saved to https://phabricator.wikimedia.org/P97496 and previous config saved to /var/cache/conftool/dbconfig/20261118-115931-ladsgroup.json
- 11:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2172.codfw.wmnet with reason: Maintenance
- 11:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T441633)', diff saved to https://phabricator.wikimedia.org/P97495 and previous config saved to /var/cache/conftool/dbconfig/20261118-115909-ladsgroup.json
- 11:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P97494 and previous config saved to /var/cache/conftool/dbconfig/20261118-114907-ladsgroup.json
- 11:48 filippo@cumin1004: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for 8 hosts
- 11:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P97492 and previous config saved to /var/cache/conftool/dbconfig/20261118-113905-ladsgroup.json
- 11:33 jelto@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
- 11:32 jelto@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
- 11:30 jelto@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
- 11:30 jelto@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
- 11:30 daniel@deploy2003: Finished scap sync-world: Backport for REST: enable /api on testwiki (T433547) (duration: 12m 53s)
- 11:29 jelto@deploy2003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
- 11:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T441633)', diff saved to https://phabricator.wikimedia.org/P97491 and previous config saved to /var/cache/conftool/dbconfig/20261118-112903-ladsgroup.json
- 11:28 jelto@deploy2003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
- 11:23 daniel@deploy2003: daniel: Continuing with deployment
- 11:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2155 (T441633)', diff saved to https://phabricator.wikimedia.org/P97490 and previous config saved to /var/cache/conftool/dbconfig/20261118-112033-ladsgroup.json
- 11:20 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
- 11:19 daniel@deploy2003: daniel: Backport for REST: enable /api on testwiki (T433547) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:17 daniel@deploy2003: Started scap sync-world: Backport for REST: enable /api on testwiki (T433547)
- 11:10 filippo@cumin1004: START - Cookbook sre.mysql.multiinstance_reboot for 8 hosts
- 11:06 daniel@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
- 11:05 daniel@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
- 11:05 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-eqiad to 1.34.13
- 11:05 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
- 11:05 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
- 11:04 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
- 11:03 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
- 11:03 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
- 11:03 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
- 11:03 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
- 11:02 slyngshede@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp6003.drmrs.wmnet with OS trixie
- 11:02 filippo@cumin1004: END (PASS) - Cookbook sre.mysql.multiinstance_reboot (exit_code=0) for clouddb1025.eqiad.wmnet
- 11:02 daniel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
- 11:01 daniel@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
- 10:57 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
- 10:57 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
- 10:57 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
- 10:57 filippo@cumin1004: START - Cookbook sre.mysql.multiinstance_reboot for clouddb1025.eqiad.wmnet
- 10:57 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
- 10:56 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1004.eqiad.wmnet with OS trixie
- 10:56 filippo@cumin1004: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for clouddb1025.eqiad.wmnet
- 10:56 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cp6003.drmrs.wmnet (push config only)
- 10:56 slyngshede@cumin1004: START - Cookbook sre.hosts.dhcp for host cp6003.drmrs.wmnet (push config only)
- 10:56 filippo@cumin1004: START - Cookbook sre.mysql.multiinstance_reboot for clouddb1025.eqiad.wmnet
- 10:56 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
- 10:56 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
- 10:56 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
- 10:56 slyngshede@cumin1004: START - Cookbook sre.hosts.reimage for host cp6003.drmrs.wmnet with OS trixie
- 10:56 filippo@cumin1004: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for clouddb1025.eqiad.wmnet
- 10:56 filippo@cumin1004: START - Cookbook sre.mysql.multiinstance_reboot for clouddb1025.eqiad.wmnet
- 10:55 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
- 10:55 tappof@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 10:55 tappof@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 10:55 godog: remove stray sections from clouddb1025
- 10:51 slyngshede@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp6003.drmrs.wmnet with OS trixie
- 10:51 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cp6003.drmrs.wmnet (remove config only)
- 10:51 slyngshede@cumin1004: START - Cookbook sre.hosts.dhcp for host cp6003.drmrs.wmnet (remove config only)
- 10:51 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp7016.*} and A:cp
- 10:51 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7016.magru.wmnet
- 10:50 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
- 10:50 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-eqiad to 1.34.13
- 10:49 filippo@cumin1004: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for clouddb[1024-1025].eqiad.wmnet
- 10:48 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-eqiad to 1.34.13
- 10:48 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster1005.eqiad.wmnet
- 10:48 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster1005.eqiad.wmnet
- 10:48 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster1005.eqiad.wmnet
- 10:48 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster1005.eqiad.wmnet
- 10:48 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster1004.eqiad.wmnet
- 10:48 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster1004.eqiad.wmnet
- 10:47 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster1004.eqiad.wmnet
- 10:47 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster1004.eqiad.wmnet
- 10:47 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster1003.eqiad.wmnet
- 10:47 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster1003.eqiad.wmnet
- 10:46 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster1003.eqiad.wmnet
- 10:46 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster1003.eqiad.wmnet
- 10:46 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-eqiad to 1.34.13
- 10:43 filippo@cumin1004: START - Cookbook sre.mysql.multiinstance_reboot for clouddb[1024-1025].eqiad.wmnet
- 10:40 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp7016.*} and A:cp
- 10:39 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1004.eqiad.wmnet with reason: host reimage
- 10:39 filippo@cumin1004: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for clouddb[1024-1025].eqiad.wmnet
- 10:39 filippo@cumin1004: START - Cookbook sre.mysql.multiinstance_reboot for clouddb[1024-1025].eqiad.wmnet
- 10:36 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cp6003.drmrs.wmnet (push config only)
- 10:36 slyngshede@cumin1004: START - Cookbook sre.hosts.dhcp for host cp6003.drmrs.wmnet (push config only)
- 10:36 slyngshede@cumin1004: START - Cookbook sre.hosts.reimage for host cp6003.drmrs.wmnet with OS trixie
- 10:35 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp6003.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 10:35 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1004.eqiad.wmnet with reason: host reimage
- 10:33 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1004.eqiad.wmnet (remove config only)
- 10:33 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1004.eqiad.wmnet (remove config only)
- 10:33 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-eqiad to 1.34.13
- 10:33 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[1004-1005].eqiad.wmnet
- 10:33 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[1004-1005].eqiad.wmnet
- 10:24 daniel@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 10:24 daniel@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 10:23 daniel@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
- 10:23 daniel@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
- 10:23 slyngshede@cumin1004: START - Cookbook sre.hosts.provision for host cp6003.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 10:23 daniel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
- 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
- 10:23 slyngshede@cumin1004: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp6003.drmrs.wmnet with OS trixie
- 10:23 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cp6003.drmrs.wmnet (remove config only)
- 10:23 slyngshede@cumin1004: START - Cookbook sre.hosts.dhcp for host cp6003.drmrs.wmnet (remove config only)
- 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
- 10:22 daniel@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
- 10:22 daniel@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 10:21 filippo@cumin1004: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for clouddb[1022-1025].eqiad.wmnet
- 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1004.eqiad.wmnet (push config only)
- 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1004.eqiad.wmnet (push config only)
- 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov1004.eqiad.wmnet with OS trixie
- 10:17 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1005.eqiad.wmnet with OS trixie
- 10:14 daniel@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 10:10 filippo@cumin1004: START - Cookbook sre.mysql.multiinstance_reboot for clouddb[1022-1025].eqiad.wmnet
- 10:09 filippo@cumin1004: END (FAIL) - Cookbook sre.mysql.multiinstance_reboot (exit_code=99) for clouddb[1022-1025].eqiad.wmnet
- 10:09 filippo@cumin1004: START - Cookbook sre.mysql.multiinstance_reboot for clouddb[1022-1025].eqiad.wmnet
- 10:05 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cp6003.drmrs.wmnet (push config only)
- 10:05 slyngshede@cumin1004: START - Cookbook sre.hosts.dhcp for host cp6003.drmrs.wmnet (push config only)
- 10:04 slyngshede@cumin1004: START - Cookbook sre.hosts.reimage for host cp6003.drmrs.wmnet with OS trixie
- 10:03 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp6003.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 10:00 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1005.eqiad.wmnet with reason: host reimage
- 09:55 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1005.eqiad.wmnet with reason: host reimage
- 09:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1005.eqiad.wmnet (remove config only)
- 09:53 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1005.eqiad.wmnet (remove config only)
- 09:44 slyngshede@cumin1004: START - Cookbook sre.hosts.provision for host cp6003.mgmt.drmrs.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 09:42 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1005.eqiad.wmnet (push config only)
- 09:42 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1005.eqiad.wmnet (push config only)
- 09:41 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov1005.eqiad.wmnet with OS trixie
- 09:39 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1006.eqiad.wmnet with OS trixie
- 09:38 slyngshede@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp6003.drmrs.wmnet with reason: reprovision
- 09:36 slyngshede@puppetserver1001: conftool action : set/pooled=no; selector: name=cp6003.drmrs.wmnet
- 09:35 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[1004-1005].eqiad.wmnet
- 09:35 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[1004-1005].eqiad.wmnet
- 09:35 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-eqiad to 1.34.13
- 09:29 jayme@deploy2003: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
- 09:29 jayme@deploy2003: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
- 09:24 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
- 09:19 logmsgbot: @deploy2003 helmfile [codfw] DONE helmfile.d/services/mw-cron: apply
- 09:19 logmsgbot: @deploy2003 helmfile [codfw] START helmfile.d/services/mw-cron: apply
- 09:18 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
- 09:16 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1006.eqiad.wmnet (remove config only)
- 09:16 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1006.eqiad.wmnet (remove config only)
- 09:09 Msz2001: Private code deployment for SI finished
- 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1006.eqiad.wmnet (push config only)
- 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1006.eqiad.wmnet (push config only)
- 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS trixie
- 09:05 marostegui@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS trixie
- 09:04 marostegui@cumin1004: END (ERROR) - Cookbook sre.hosts.dhcp (exit_code=97) for host dbprov1006.eqiad.wmnet (remove config only)
- 09:04 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1006.eqiad.wmnet (remove config only)
- 08:54 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1006.eqiad.wmnet (push config only)
- 08:54 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1006.eqiad.wmnet (push config only)
- 08:53 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS trixie
- 08:53 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on dbprov1006.eqiad.wmnet with reason: Reimage
- 08:50 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1007.eqiad.wmnet with OS trixie
- 08:50 mszwarc@deploy2003: Finished scap sync-world: Backport for Lift Wikibase edit exemption on Beta and testwikidatawiki (T443420) (duration: 14m 22s)
- 08:47 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 08:44 mszwarc@deploy2003: mszwarc, mahmoud-abdelsattar: Continuing with deployment
- 08:43 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 08:42 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
- 08:39 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
- 08:38 mszwarc@deploy2003: mszwarc, mahmoud-abdelsattar: Backport for Lift Wikibase edit exemption on Beta and testwikidatawiki (T443420) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:35 mszwarc@deploy2003: Started scap sync-world: Backport for Lift Wikibase edit exemption on Beta and testwikidatawiki (T443420)
- 08:33 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1007.eqiad.wmnet with reason: host reimage
- 08:29 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1007.eqiad.wmnet with reason: host reimage
- 08:27 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1007.eqiad.wmnet (remove config only)
- 08:27 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1007.eqiad.wmnet (remove config only)
- 08:17 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1007.eqiad.wmnet (push config only)
- 08:17 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1007.eqiad.wmnet (push config only)
- 08:17 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov1007.eqiad.wmnet with OS trixie
- 08:16 marostegui@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1007.eqiad.wmnet with OS trixie
- 08:16 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1007.eqiad.wmnet (remove config only)
- 08:16 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1007.eqiad.wmnet (remove config only)
- 06:58 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov1007.eqiad.wmnet (push config only)
- 06:58 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov1007.eqiad.wmnet (push config only)
- 06:57 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov1007.eqiad.wmnet with OS trixie
- 06:57 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbprov1007.eqiad.wmnet with reason: Reimage
- 06:44 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
- 06:44 marostegui@cumin1004: Removing clouddb1017 from zarcillo T445177
- 06:44 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1017.eqiad.wmnet
- 06:44 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 06:44 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1017.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 06:44 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1017.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 06:40 marostegui@cumin1004: START - Cookbook sre.dns.netbox
- 06:39 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 06:39 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 06:35 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1017.eqiad.wmnet
- 06:35 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
- 06:21 rzl@cumin2003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching P{C:cassandra} and not A:cassandra-dev and not A:sessionstore: T445270 - rzl@cumin2003
- 04:46 kevinbazira@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
- 04:45 kevinbazira@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
- 02:07 ryankemper@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 02:04 ryankemper@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 01:58 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 01:39 ryankemper@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 01:39 ryankemper@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 01:31 ryankemper@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 01:30 ryankemper@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 01:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: Apply JVM mitigation for CVE-2026-114938 - ryankemper@cumin2003 - T445272
- 00:55 bking@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 00:52 ryankemper@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
- 00:51 ryankemper@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
- 00:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 00:04 bking@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 00:04 bking@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
2026-11-17
- 23:59 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 23:58 ryankemper@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: Apply JVM mitigation for CVE-2026-114938 - ryankemper@cumin2003 - T445272
- 23:56 bking@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 23:52 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply JVM config updates - bking@cumin2003 - T445272
- 23:45 rzl@cumin2003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:sessionstore: T445270 - rzl@cumin2003
- 23:12 rzl@cumin2003: START - Cookbook sre.cassandra.roll-restart for nodes matching P{C:cassandra} and not A:cassandra-dev and not A:sessionstore: T445270 - rzl@cumin2003
- 23:09 rzl@cumin2003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:sessionstore: T445270 - rzl@cumin2003
- 23:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-eqiad
- 23:08 ryankemper@cumin2003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-main-codfw
- 23:02 ryankemper@cumin2003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-jumbo-eqiad
- 23:01 ryankemper@cumin2003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad
- 23:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-codfw
- 23:00 ryankemper@cumin2003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad
- 22:58 rzl@cumin2003: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:cassandra-dev: T445270 - rzl@cumin2003
- 22:52 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply updates - bking@cumin2003 - T445272
- 22:50 ryankemper@cumin2003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad
- 22:49 ryankemper@cumin2003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-codfw
- 22:49 ryankemper@cumin2003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-jumbo-eqiad
- 22:48 ryankemper@cumin2003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-eqiad
- 22:48 ryankemper@cumin2003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-main-codfw
- 22:46 ryankemper@cumin2003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad
- 22:40 ryankemper@cumin2003: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-test-eqiad
- 22:39 ryankemper@cumin2003: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-test-eqiad
- 22:38 rzl@cumin2003: START - Cookbook sre.cassandra.roll-restart for nodes matching A:cassandra-dev: T445270 - rzl@cumin2003
- 22:36 bking@cumin2003: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply JVM config updates - bking@cumin2003 - T445272
- 22:34 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply JVM config updates - bking@cumin2003 - T445272
- 22:29 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: apply JVM config updates - bking@cumin2003 - T445272
- 22:28 bking@cumin2003: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply updates - bking@cumin2003 - T445272
- 22:13 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 22:12 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 22:12 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 22:03 cdanis@cumin1004: END (PASS) - Cookbook sre.gerrit.restart-gerrit (exit_code=0) Restarting Gerrit on gerrit1003
- 22:00 cdanis@cumin1004: START - Cookbook sre.gerrit.restart-gerrit Restarting Gerrit on gerrit1003
- 22:00 cdanis@cumin1004: END (ERROR) - Cookbook sre.gerrit.restart-gerrit (exit_code=97) Restarting Gerrit on gerrit2003
- 21:58 cdanis@cumin1004: START - Cookbook sre.gerrit.restart-gerrit Restarting Gerrit on gerrit2003
- 21:58 cdanis@cumin1004: END (PASS) - Cookbook sre.gerrit.restart-gerrit (exit_code=0) Restarting Gerrit on gerrit2002
- 21:57 cdanis@cumin1004: START - Cookbook sre.gerrit.restart-gerrit Restarting Gerrit on gerrit2002
- 21:53 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 21:42 bd808@deploy2003: Finished scap sync-world: Backport for VisualEditor: add image for Suggestion Mode welcome dialog (T442024) (duration: 11m 05s)
- 21:35 bd808@deploy2003: caro, bd808: Continuing with deployment
- 21:33 bd808@deploy2003: caro, bd808: Backport for VisualEditor: add image for Suggestion Mode welcome dialog (T442024) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:31 bd808@deploy2003: Started scap sync-world: Backport for VisualEditor: add image for Suggestion Mode welcome dialog (T442024)
- 20:57 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 20:48 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 20:47 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 20:47 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 20:41 thcipriani@deploy2003: rebuilt and synchronized wikiversions files: group0 to 1.48.0-wmf.5 refs T442223
- 20:41 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 20:40 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 20:36 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 20:35 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 20:33 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 20:32 gmodena@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 20:31 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 20:30 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 20:30 gmodena@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 20:24 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 20:24 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on dbstore1010.eqiad.wmnet with reason: Restoring analytics replica instances
- 20:20 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 20:18 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
- 20:18 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
- 20:12 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
- 20:12 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
- 18:58 lerickson@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 18:57 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 18:53 lerickson@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 18:52 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 18:37 lerickson@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 18:37 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 18:28 lerickson@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 18:23 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 16:16 Amir1: root@thanos-fe1004:~# swift delete elasticsearch-snapshot (T309648)
- 16:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
- 16:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T441633)', diff saved to https://phabricator.wikimedia.org/P97486 and previous config saved to /var/cache/conftool/dbconfig/20261117-161208-ladsgroup.json
- 16:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P97485 and previous config saved to /var/cache/conftool/dbconfig/20261117-160206-ladsgroup.json
- 15:56 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:55 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252', diff saved to https://phabricator.wikimedia.org/P97484 and previous config saved to /var/cache/conftool/dbconfig/20261117-155204-ladsgroup.json
- 15:50 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:50 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1252 (T441633)', diff saved to https://phabricator.wikimedia.org/P97483 and previous config saved to /var/cache/conftool/dbconfig/20261117-154202-ladsgroup.json
- 15:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1252 (T441633)', diff saved to https://phabricator.wikimedia.org/P97482 and previous config saved to /var/cache/conftool/dbconfig/20261117-153338-ladsgroup.json
- 15:33 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1252.eqiad.wmnet with reason: Maintenance
- 15:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T441633)', diff saved to https://phabricator.wikimedia.org/P97481 and previous config saved to /var/cache/conftool/dbconfig/20261117-153318-ladsgroup.json
- 15:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P97480 and previous config saved to /var/cache/conftool/dbconfig/20261117-152316-ladsgroup.json
- 15:20 ladsgroup@deploy2003: Finished scap sync-world: Backport for Claim transcodes atomically and simplify the queue split (T444235) (1.48.0-wmf.4) (duration: 11m 50s)
- 15:15 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet
- 15:14 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 15:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P97479 and previous config saved to /var/cache/conftool/dbconfig/20261117-151314-ladsgroup.json
- 15:13 ladsgroup@deploy2003: ladsgroup: Backport for Claim transcodes atomically and simplify the queue split (T444235) (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:11 taavi@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1002-dev.eqiad.wmnet
- 15:10 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet
- 15:08 ladsgroup@deploy2003: Started scap sync-world: Backport for Claim transcodes atomically and simplify the queue split (T444235) (1.48.0-wmf.4)
- 15:07 taavi@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudbackup1001-dev.eqiad.wmnet
- 15:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T441633)', diff saved to https://phabricator.wikimedia.org/P97478 and previous config saved to /var/cache/conftool/dbconfig/20261117-150312-ladsgroup.json
- 15:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2197.codfw.wmnet with OS trixie
- 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1249 (T441633)', diff saved to https://phabricator.wikimedia.org/P97476 and previous config saved to /var/cache/conftool/dbconfig/20261117-145431-ladsgroup.json
- 14:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1249.eqiad.wmnet with reason: Maintenance
- 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T441633)', diff saved to https://phabricator.wikimedia.org/P97475 and previous config saved to /var/cache/conftool/dbconfig/20261117-145411-ladsgroup.json
- 14:53 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
- 14:53 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
- 14:53 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
- 14:53 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
- 14:48 awight: UTC afternoon deployment finished
- 14:48 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
- 14:47 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
- 14:47 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
- 14:47 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
- 14:46 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
- 14:46 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
- 14:46 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
- 14:46 awight@deploy2003: Finished scap sync-world: Backport for Always use a fresh StatsFactory (T444626) (1.48.0-wmf.5) (duration: 12m 02s)
- 14:46 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
- 14:45 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
- 14:45 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
- 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P97474 and previous config saved to /var/cache/conftool/dbconfig/20261117-144409-ladsgroup.json
- 14:43 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2197.codfw.wmnet with reason: host reimage
- 14:40 awight@deploy2003: awight: Continuing with deployment
- 14:38 awight@deploy2003: awight: Backport for Always use a fresh StatsFactory (T444626) (1.48.0-wmf.5) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:37 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2197.codfw.wmnet with reason: host reimage
- 14:37 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
- 14:36 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
- 14:35 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2197.codfw.wmnet (remove config only)
- 14:34 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2197.codfw.wmnet (remove config only)
- 14:34 awight@deploy2003: Started scap sync-world: Backport for Always use a fresh StatsFactory (T444626) (1.48.0-wmf.5)
- 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P97473 and previous config saved to /var/cache/conftool/dbconfig/20261117-143408-ladsgroup.json
- 14:29 awight@deploy2003: Finished scap sync-world: Backport for hCaptcha: Enable hCaptcha for Wikibase edits on testwikidatawiki (T440845) (duration: 23m 30s)
- 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T441633)', diff saved to https://phabricator.wikimedia.org/P97472 and previous config saved to /var/cache/conftool/dbconfig/20261117-142405-ladsgroup.json
- 14:23 awight@deploy2003: awight, haak: Continuing with deployment
- 14:22 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2197.codfw.wmnet (push config only)
- 14:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2197.codfw.wmnet (push config only)
- 14:21 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2197.codfw.wmnet with OS trixie
- 14:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2197.codfw.wmnet with reason: Reimage
- 14:18 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2198.codfw.wmnet with OS trixie
- 14:16 moritzm: installing expat security updates
- 14:15 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
- 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1248 (T441633)', diff saved to https://phabricator.wikimedia.org/P97471 and previous config saved to /var/cache/conftool/dbconfig/20261117-141433-ladsgroup.json
- 14:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1248.eqiad.wmnet with reason: Maintenance
- 14:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Maint done', diff saved to https://phabricator.wikimedia.org/P97470 and previous config saved to /var/cache/conftool/dbconfig/20261117-141344-ladsgroup.json
- 14:08 awight@deploy2003: awight, haak: Backport for hCaptcha: Enable hCaptcha for Wikibase edits on testwikidatawiki (T440845) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:06 awight@deploy2003: Started scap sync-world: Backport for hCaptcha: Enable hCaptcha for Wikibase edits on testwikidatawiki (T440845)
- 14:05 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
- 13:54 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2198.codfw.wmnet with reason: host reimage
- 13:52 moritzm: installing pyopenssl security updates
- 13:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2198.codfw.wmnet with reason: host reimage
- 13:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2198.codfw.wmnet (remove config only)
- 13:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2198.codfw.wmnet (remove config only)
- 13:43 slyngshede@puppetserver1001: conftool action : set/pooled=yes; selector: name=cp7011.magru.wmnet
- 13:41 slyngshede@puppetserver1001: conftool action : set/weight=1; selector: name=cp7011.magru.wmnet
- 13:39 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7011.magru.wmnet with OS trixie
- 13:36 Tchanders: Adjusted WikimediaAntiAbuse AbuseReview private code
- 13:35 tchanders@deploy2003: Finished scap sync-world: Backport for WikimediaAntiAbuse: Map the virtual domain to x1 (T443475) (duration: 12m 10s)
- 13:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2198.codfw.wmnet (push config only)
- 13:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2198.codfw.wmnet (push config only)
- 13:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2198.codfw.wmnet with OS trixie
- 13:30 taavi@cumin1004: conftool action : set/pooled=yes; selector: name=cloudweb1003.wikimedia.org
- 13:29 tchanders@deploy2003: tchanders, mpostoronca: Continuing with deployment
- 13:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2198.codfw.wmnet with reason: Reimage
- 13:28 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudweb1003.wikimedia.org with OS trixie
- 13:28 tchanders@deploy2003: tchanders, mpostoronca: Backport for WikimediaAntiAbuse: Map the virtual domain to x1 (T443475) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 13:23 tchanders@deploy2003: Started scap sync-world: Backport for WikimediaAntiAbuse: Map the virtual domain to x1 (T443475)
- 13:19 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2199.codfw.wmnet with OS trixie
- 13:14 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7011.magru.wmnet with reason: host reimage
- 13:11 slyngshede@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7011.magru.wmnet with reason: host reimage
- 13:08 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudweb1003.wikimedia.org with reason: host reimage
- 13:07 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cp7011.magru.wmnet (remove config only)
- 13:07 slyngshede@cumin1004: START - Cookbook sre.hosts.dhcp for host cp7011.magru.wmnet (remove config only)
- 13:03 taavi@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudweb1003.wikimedia.org with reason: host reimage
- 13:01 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudweb1003.wikimedia.org (remove config only)
- 13:01 taavi@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudweb1003.wikimedia.org (remove config only)
- 13:00 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
- 13:00 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
- 12:59 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
- 12:59 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2199.codfw.wmnet with reason: host reimage
- 12:59 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
- 12:58 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
- 12:58 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
- 12:52 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2199.codfw.wmnet with reason: host reimage
- 12:51 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cp7011.magru.wmnet (push config only)
- 12:51 slyngshede@cumin1004: START - Cookbook sre.hosts.dhcp for host cp7011.magru.wmnet (push config only)
- 12:50 slyngshede@cumin1004: START - Cookbook sre.hosts.reimage for host cp7011.magru.wmnet with OS trixie
- 12:50 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2199.codfw.wmnet (remove config only)
- 12:50 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host cloudweb1003.wikimedia.org (push config only)
- 12:50 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2199.codfw.wmnet (remove config only)
- 12:50 taavi@cumin1004: START - Cookbook sre.hosts.dhcp for host cloudweb1003.wikimedia.org (push config only)
- 12:49 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7011.mgmt.magru.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 12:46 taavi@cumin1004: START - Cookbook sre.hosts.reimage for host cloudweb1003.wikimedia.org with OS trixie
- 12:41 moritzm: installing apache2 security updates
- 12:38 slyngshede@cumin1004: START - Cookbook sre.hosts.provision for host cp7011.mgmt.magru.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 12:36 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2199.codfw.wmnet (push config only)
- 12:36 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2199.codfw.wmnet (push config only)
- 12:35 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2199.codfw.wmnet with OS trixie
- 12:34 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2200.codfw.wmnet with OS trixie
- 12:32 slyngshede@puppetserver1001: conftool action : set/pooled=no; selector: name=cp7011.magru.wmnet
- 12:32 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2199.codfw.wmnet with reason: Reimage
- 12:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-esams (T444802)
- 12:21 sfaci@deploy2003: Finished scap deploy-service growthbook: Update GrowthBook to version that lets us disable Update button on a schedule (duration: 01m 40s)
- 12:19 sfaci@deploy2003: scap deploy-service growthbook: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying growthbook
- 12:19 sfaci@deploy2003: Started scap deploy-service growthbook: Update GrowthBook to version that lets us disable Update button on a schedule
- 12:16 sfaci@deploy2003: Finished scap deploy-service growthbook-next: Update GrowthBook to version that lets us disable Update button on a schedule (duration: 05m 07s)
- 12:13 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2200.codfw.wmnet with reason: host reimage
- 12:13 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2003.wikimedia.org
- 12:10 sfaci@deploy2003: scap deploy-service growthbook-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying growthbook-next
- 12:10 sfaci@deploy2003: Started scap deploy-service growthbook-next: Update GrowthBook to version that lets us disable Update button on a schedule
- 12:09 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc2003.wikimedia.org
- 12:08 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2200.codfw.wmnet with reason: host reimage
- 12:08 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting A:liberica-esams (T444802)
- 12:06 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2200.codfw.wmnet (remove config only)
- 12:06 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2200.codfw.wmnet (remove config only)
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:57 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-drmrs (T444802)
- 11:55 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 11:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1001.wikimedia.org
- 11:53 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 11:52 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2200.codfw.wmnet (push config only)
- 11:52 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2200.codfw.wmnet (push config only)
- 11:51 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2200.codfw.wmnet with OS trixie
- 11:50 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 11:50 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2200.codfw.wmnet with reason: Reimage
- 11:50 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 11:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2201.codfw.wmnet with OS trixie
- 11:45 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host clouddumps1001.wikimedia.org
- 11:41 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting A:liberica-drmrs (T444802)
- 11:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-eqsin (T444802)
- 11:35 blake@deploy2003: Finished scap sync-world: Backport for debug.json: Add k8s-mw-pretrain debug backends (T427666) (duration: 11m 01s)
- 11:30 btullis@puppetserver1001: conftool action : set/pooled=no; selector: cluster=dumps,dc=eqiad,name=clouddumps1001.wikimedia.org,service=dumps-nfs
- 11:29 blake@deploy2003: blake: Continuing with deployment
- 11:28 blake@deploy2003: blake: Backport for debug.json: Add k8s-mw-pretrain debug backends (T427666) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:27 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2201.codfw.wmnet with reason: host reimage
- 11:27 Amir1: requeueing errored transcodes of popular files in DASH (T373546)
- 11:24 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2201.codfw.wmnet with reason: host reimage
- 11:24 blake@deploy2003: Started scap sync-world: Backport for debug.json: Add k8s-mw-pretrain debug backends (T427666)
- 11:22 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2201.codfw.wmnet (remove config only)
- 11:22 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2201.codfw.wmnet (remove config only)
- 11:19 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting A:liberica-eqsin (T444802)
- 11:19 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-ulsfo (T444802)
- 11:19 ladsgroup@deploy2003: Finished scap sync-world: Backport for WebVideoTranscodeJob: Ignore size limit for DASH and rely on time limit (T357184) (1.48.0-wmf.4) (duration: 12m 03s)
- 11:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install1005.wikimedia.org
- 11:12 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 11:12 jmm@dns1004: END - running authdns-update
- 11:12 ladsgroup@deploy2003: ladsgroup: Backport for WebVideoTranscodeJob: Ignore size limit for DASH and rely on time limit (T357184) (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:10 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: cluster=dumps,dc=eqiad
- 11:10 jmm@dns1004: START - running authdns-update
- 11:08 klausman@deploy2003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
- 11:08 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2201.codfw.wmnet (push config only)
- 11:08 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2201.codfw.wmnet (push config only)
- 11:08 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
- 11:08 klausman@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
- 11:08 klausman@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
- 11:08 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
- 11:07 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2201.codfw.wmnet with OS trixie
- 11:07 klausman@deploy2003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
- 11:07 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install1005.wikimedia.org
- 11:07 klausman@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
- 11:07 klausman@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
- 11:07 ladsgroup@deploy2003: Started scap sync-world: Backport for WebVideoTranscodeJob: Ignore size limit for DASH and rely on time limit (T357184) (1.48.0-wmf.4)
- 11:06 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
- 11:05 fnegri@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
- 11:05 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db2201.codfw.wmnet with reason: Reimage
- 11:04 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
- 11:04 fnegri@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
- 11:04 fnegri@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
- 11:04 ladsgroup@deploy2003: Finished scap sync-world: Backport for Flip the condition on DASH (T357184) (1.48.0-wmf.5) (duration: 12m 52s)
- 11:03 fnegri@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
- 11:03 fnegri@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
- 11:03 fnegri@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
- 11:03 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting A:liberica-ulsfo (T444802)
- 11:02 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dborch1003.eqiad.wmnet
- 11:02 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 11:02 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dborch1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 11:01 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dborch1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 11:01 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: upgrade
- 10:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host clouddumps1002.wikimedia.org
- 10:57 marostegui@cumin1004: START - Cookbook sre.dns.netbox
- 10:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 10:55 ladsgroup@deploy2003: ladsgroup: Backport for Flip the condition on DASH (T357184) (1.48.0-wmf.5) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 10:54 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 10:52 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts dborch1003.eqiad.wmnet
- 10:52 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 10:51 ladsgroup@deploy2003: Started scap sync-world: Backport for Flip the condition on DASH (T357184) (1.48.0-wmf.5)
- 10:48 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 10:48 marostegui@cumin1004: END (ERROR) - Cookbook sre.mysql.decommission (exit_code=1)
- 10:48 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
- 10:47 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host clouddumps1002.wikimedia.org
- 10:46 btullis@puppetserver1001: conftool action : set/pooled=no; selector: cluster=dumps,dc=eqiad,name=clouddumps1002.wikimedia.org
- 10:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting A:liberica-magru (T444802)
- 10:33 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 10:31 brouberol@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host archiva1002.wikimedia.org
- 10:31 arnaudb@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: upgrade
- 10:30 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: upgrade
- 10:27 brouberol@cumin1004: START - Cookbook sre.hosts.reboot-single for host archiva1002.wikimedia.org
- 10:24 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.admin rebooting A:liberica-magru (T444802)
- 10:20 arnaudb@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: upgrade
- 10:18 arnaudb@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: upgrade
- 10:15 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install2005.wikimedia.org
- 10:09 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install2005.wikimedia.org
- 10:08 arnaudb@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: upgrade
- 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install3004.wikimedia.org
- 10:01 XioNoX: un-drain magru-codfw HE transport link
- 09:59 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install3004.wikimedia.org
- 09:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install4004.wikimedia.org
- 09:52 chlod@deploy2003: Finished scap sync-world: Backport for Lift IP cap for UKM workshop (T445180) (duration: 18m 49s)
- 09:49 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
- 09:47 moritzm: installing libdbi-perl security updates
- 09:46 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install4004.wikimedia.org
- 09:46 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
- 09:42 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install5004.wikimedia.org
- 09:41 chlod@deploy2003: chlod: Continuing with deployment
- 09:40 chlod@deploy2003: chlod: Backport for Lift IP cap for UKM workshop (T445180) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 09:36 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
- 09:33 chlod@deploy2003: Started scap sync-world: Backport for Lift IP cap for UKM workshop (T445180)
- 09:32 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install5004.wikimedia.org
- 09:32 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
- 09:31 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install6003.wikimedia.org
- 09:31 moritzm: installing vim security updates
- 09:27 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dborch1002.wikimedia.org
- 09:25 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install6003.wikimedia.org
- 09:23 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host dborch1002.wikimedia.org
- 09:22 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host install7002.wikimedia.org
- 09:16 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host install7002.wikimedia.org
- 09:15 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host contint2003.wikimedia.org
- 09:09 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host contint2003.wikimedia.org
- 09:08 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host contint1003.wikimedia.org
- 09:07 marostegui: Stop mariadb on clouddb1017 https://phabricator.wikimedia.org/T445177
- 09:07 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on clouddb1017.eqiad.wmnet with reason: Decommission
- 09:00 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host contint1003.wikimedia.org
- 08:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt1002.wikimedia.org
- 08:49 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host apt1002.wikimedia.org
- 08:49 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 140475
- 08:49 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 140475
- 08:16 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
- 08:15 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org
- 08:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test2005.wikimedia.org
- 08:09 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
- 08:09 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org
- 08:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test2005.wikimedia.org
- 08:07 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org
- 08:04 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
- 08:01 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org
- 07:59 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
- 07:58 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp-test1005.wikimedia.org
- 07:56 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm1001.wikimedia.org
- 07:54 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idp-test1005.wikimedia.org
- 07:52 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idm1001.wikimedia.org
- 07:46 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm2001.wikimedia.org
- 07:42 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idm2001.wikimedia.org
- 07:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idm-test1001.wikimedia.org
- 07:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host idm-test1001.wikimedia.org
- 07:33 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host apt2002.wikimedia.org
- 06:40 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
- 06:40 marostegui@cumin1004: Removing clouddb1016 from zarcillo T444585
- 06:40 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1016.eqiad.wmnet
- 06:40 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 06:40 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 06:39 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1016.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 06:36 marostegui@cumin1004: START - Cookbook sre.dns.netbox
- 06:29 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1016.eqiad.wmnet
- 06:29 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
- 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.48.0-wmf.1 (duration: 02m 26s)
- 04:43 mwpresync@deploy2003: Finished scap sync-world: testwikis to 1.48.0-wmf.5 refs T442223 (duration: 39m 47s)
- 04:03 mwpresync@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.5 refs T442223
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 29s)
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 01:18 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
2026-11-16
- 23:26 Southparkfan: Adjusted README in private/ (no-op, no code touched)
- 23:12 sbassett: Deployed security update for T192727
- 22:24 sbassett: Deployed security updates for T192727, T440125, T440783, T440386
- 21:40 southparkfan@deploy2003: Finished scap sync-world: Backport for LabsServices: Migrate cxserver to service RR (T436467), Add spenden.wikimedia.de to CSP for enwiki + dewiki (T444786) (duration: 17m 46s)
- 21:34 southparkfan@deploy2003: southparkfan, ejegg: Continuing with deployment
- 21:27 southparkfan@deploy2003: southparkfan, ejegg: Backport for LabsServices: Migrate cxserver to service RR (T436467), Add spenden.wikimedia.de to CSP for enwiki + dewiki (T444786) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:25 volans: run apt-get autoclean on config-master2001 (was at 8.9G 8.3G 91M 99% / )
- 21:22 southparkfan@deploy2003: Started scap sync-world: Backport for LabsServices: Migrate cxserver to service RR (T436467), Add spenden.wikimedia.de to CSP for enwiki + dewiki (T444786)
- 21:18 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
- 21:14 lerickson@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
- 21:11 lerickson@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
- 20:32 lerickson@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 20:29 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 20:28 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 20:25 lerickson@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 20:22 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 20:20 lerickson@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 20:17 lerickson@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 19:13 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on P{hcaptcha-proxy1001.wikimedia.org or hcaptcha-proxy1002.wikimedia.org or hcaptcha-proxy2001.wikimedia.org or hcaptcha-proxy4003.wikimedia.org or hcaptcha-proxy4004.wikimedia.org or hcaptcha-proxy5004.wikimedia.org or hcaptcha-proxy7001.wikimedia.org or hcaptcha-proxy7002.wikimedia.org} and A:hcaptcha-prox
- 19:04 sbassett: Deployed security fix for T445032
- 18:42 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum-eqsin
- 18:37 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on P{hcaptcha-proxy1001.wikimedia.org or hcaptcha-proxy1002.wikimedia.org or hcaptcha-proxy2001.wikimedia.org or hcaptcha-proxy4003.wikimedia.org or hcaptcha-proxy4004.wikimedia.org or hcaptcha-proxy5004.wikimedia.org or hcaptcha-proxy7001.wikimedia.org or hcaptcha-proxy7002.wikimedia.org} and A:hcaptcha-proxy
- 18:35 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and not A:eqsin and A:hcaptcha-proxy
- 18:34 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough
- 18:31 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum-eqsin
- 18:30 sukhe@cumin1004: END (ERROR) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=97) rolling reboot on A:durum-eqsin
- 18:30 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum-eqsin
- 18:25 volans@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2003.wikimedia.org
- 18:19 volans@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab2003.wikimedia.org
- 18:17 volans@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2003.wikimedia.org
- 18:10 volans@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit2003.wikimedia.org
- 18:07 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists1004.wikimedia.org
- 18:04 volans@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org
- 18:01 taavi@cumin1004: START - Cookbook sre.hosts.reboot-single for host lists1004.wikimedia.org
- 18:00 taavi@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lists2001.wikimedia.org
- 17:58 volans@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org
- 17:58 volans@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit1003.wikimedia.org
- 17:54 taavi@cumin1004: START - Cookbook sre.hosts.reboot-single for host lists2001.wikimedia.org
- 17:51 volans@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit1003.wikimedia.org
- 17:49 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
- 17:49 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
- 17:47 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:eqsin and A:hcaptcha-proxy
- 17:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve1005.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
- 17:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1005.eqiad.wmnet
- 17:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1005.eqiad.wmnet
- 17:44 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and not A:eqsin and A:hcaptcha-proxy
- 17:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1005.eqiad.wmnet
- 17:36 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and A:eqsin and A:hcaptcha-proxy
- 17:36 sukhe@cumin1004: END (ERROR) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=97) rolling reboot on A:hcaptcha-proxy or A:eqsin and A:hcaptcha-proxy
- 17:36 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy or A:eqsin and A:hcaptcha-proxy
- 17:35 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1005.eqiad.wmnet
- 17:35 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{ml-serve1005.eqiad.wmnet} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
- 17:33 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: sync
- 17:33 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: sync
- 17:32 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
- 17:31 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
- 17:31 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough
- 17:30 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy or A:eqiad or A:ulsfo or A:drmrs or A:magru and A:hcaptcha-proxy
- 17:25 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
- 17:25 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
- 17:23 sukhe@cumin1004: END (FAIL) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=1) rolling reboot on A:wikidough
- 17:23 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
- 17:23 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
- 17:22 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
- 17:22 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 17:22 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 17:22 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 17:22 volans: rebooting bast7002
- 17:21 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 17:19 sukhe@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-urldownloaders (exit_code=0) rolling reboot on A:url-downloader and A:bookworm and (A:url-downloader or A:url-downloader-eqiad or A:url-downloader-codfw)
- 17:00 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough
- 17:00 sukhe@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-reboot-urldownloaders rolling reboot on A:url-downloader and A:bookworm and (A:url-downloader or A:url-downloader-eqiad or A:url-downloader-codfw)
- 16:58 volans: rebooting gitlab2002 from console
- 16:55 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out1001.wikimedia.org with reason: new-kernel
- 16:55 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out2001.wikimedia.org with reason: new-kernel
- 16:53 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in1001.wikimedia.org with reason: new-kernel
- 16:53 volans: rebooting gitlab1003 from console
- 16:53 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in2001.wikimedia.org with reason: new-kernel
- 16:52 jhathaway@cumin1004: DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1:00:00 on mx-in2001.wikimedia.org with reason: new-kernel
- 16:29 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 16:29 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 15:53 jgiannelos@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
- 15:51 jgiannelos@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
- 15:50 jgiannelos@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
- 15:48 jgiannelos@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
- 15:44 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
- 15:43 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
- 15:43 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
- 15:43 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
- 15:41 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
- 15:41 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
- 15:35 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling reboot on A:durum
- 15:31 sukhe@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for role: ml_k8s::master@eqiad
- 15:31 sukhe@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
- 15:22 xcollazo@deploy2003: Finished deploy [analytics/refinery@2658f08]: Regular analytics weekly train [analytics/refinery@2658f088] (duration: 05m 48s)
- 15:17 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 15:17 xcollazo@deploy2003: Started deploy [analytics/refinery@2658f08]: Regular analytics weekly train [analytics/refinery@2658f088]
- 15:16 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 15:16 xcollazo@deploy2003: Finished deploy [analytics/refinery@2658f08] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2658f088] (duration: 00m 50s)
- 15:16 xcollazo@deploy2003: Started deploy [analytics/refinery@2658f08] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2658f088]
- 15:08 Lucas_WMDE: UTC afternoon backport+config window done
- 15:04 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for preview bugfixes (1.48.0-wmf.4) (duration: 19m 07s)
- 14:59 sukhe@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs
- 14:56 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: ml_k8s::master@eqiad
- 14:56 sukhe@cumin1004: END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for role: ml_k8s::master@codfw
- 14:56 sukhe@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
- 14:54 sukhe@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs
- 14:52 ejegg@deploy2003: ejegg: Continuing with deployment
- 14:50 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: ml_k8s::master@codfw
- 14:49 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for preview bugfixes (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:44 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for preview bugfixes (1.48.0-wmf.4)
- 14:34 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum
- 14:32 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for dewiki: Add als, bar and nds to import sources (T444759) (duration: 16m 14s)
- 14:26 moritzm: installing apache2 security updates
- 14:25 lucaswerkmeister-wmde@deploy2003: lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
- 14:20 lucaswerkmeister-wmde@deploy2003: lucaswerkmeister-wmde, tryvix1509: Backport for dewiki: Add als, bar and nds to import sources (T444759) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:16 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for dewiki: Add als, bar and nds to import sources (T444759)
- 14:03 krinkle@deploy2003: Finished scap sync-world: Backport for FileRepo: Switch getDescriptionShortUrl from curid to Special:Redirect (T428296) (1.48.0-wmf.4) (duration: 16m 35s)
- 13:53 krinkle@deploy2003: krinkle: Continuing with deployment
- 13:52 krinkle@deploy2003: krinkle: Backport for FileRepo: Switch getDescriptionShortUrl from curid to Special:Redirect (T428296) (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 13:50 Amir1: alter table if exists watchlist engine=innodb,force; on loginwiki (T442779)
- 13:46 krinkle@deploy2003: Started scap sync-world: Backport for FileRepo: Switch getDescriptionShortUrl from curid to Special:Redirect (T428296) (1.48.0-wmf.4)
- 13:39 taavi@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudcumin2001.codfw.wmnet
- 13:39 taavi@cumin1004: START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet
- 13:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P97467 and previous config saved to /var/cache/conftool/dbconfig/20261116-133630-ladsgroup.json
- 13:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P97466 and previous config saved to /var/cache/conftool/dbconfig/20261116-132628-ladsgroup.json
- 13:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T441633)', diff saved to https://phabricator.wikimedia.org/P97465 and previous config saved to /var/cache/conftool/dbconfig/20261116-131626-ladsgroup.json
- 13:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1247 (T441633)', diff saved to https://phabricator.wikimedia.org/P97464 and previous config saved to /var/cache/conftool/dbconfig/20261116-130852-ladsgroup.json
- 13:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance
- 13:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T441633)', diff saved to https://phabricator.wikimedia.org/P97463 and previous config saved to /var/cache/conftool/dbconfig/20261116-130832-ladsgroup.json
- 12:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P97462 and previous config saved to /var/cache/conftool/dbconfig/20261116-125830-ladsgroup.json
- 12:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P97461 and previous config saved to /var/cache/conftool/dbconfig/20261116-124828-ladsgroup.json
- 12:38 zabe@deploy2003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for some s2 wikis (T439753) (duration: 37m 12s)
- 12:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T441633)', diff saved to https://phabricator.wikimedia.org/P97460 and previous config saved to /var/cache/conftool/dbconfig/20261116-123826-ladsgroup.json
- 12:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1244 (T441633)', diff saved to https://phabricator.wikimedia.org/P97459 and previous config saved to /var/cache/conftool/dbconfig/20261116-123101-ladsgroup.json
- 12:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance
- 12:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T441633)', diff saved to https://phabricator.wikimedia.org/P97458 and previous config saved to /var/cache/conftool/dbconfig/20261116-123041-ladsgroup.json
- 12:23 kart_: Updated cxserver to 2026-11-03-143706-production (T428590, T442944)
- 12:22 kartik@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
- 12:22 zabe@deploy2003: zabe: Continuing with deployment
- 12:22 brett@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on cp3074.esams.wmnet with reason: troubleshooting cmos failure
- 12:22 kartik@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
- 12:22 zabe@deploy2003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for some s2 wikis (T439753) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 12:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P97457 and previous config saved to /var/cache/conftool/dbconfig/20261116-122040-ladsgroup.json
- 12:19 hashar: Restarting Gerrit (jvm update)
- 12:18 kartik@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
- 12:18 kartik@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
- 12:14 kartik@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
- 12:14 kartik@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
- 12:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P97455 and previous config saved to /var/cache/conftool/dbconfig/20261116-121038-ladsgroup.json
- 12:01 zabe@deploy2003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for some s2 wikis (T439753)
- 12:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T441633)', diff saved to https://phabricator.wikimedia.org/P97454 and previous config saved to /var/cache/conftool/dbconfig/20261116-120036-ladsgroup.json
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:59 moritzm: renew CA/SPICE certs on drmrs01 Ganeti cluster
- 11:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1242 (T441633)', diff saved to https://phabricator.wikimedia.org/P97453 and previous config saved to /var/cache/conftool/dbconfig/20261116-115151-ladsgroup.json
- 11:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1242.eqiad.wmnet with reason: Maintenance
- 11:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97452 and previous config saved to /var/cache/conftool/dbconfig/20261116-115131-ladsgroup.json
- 11:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P97451 and previous config saved to /var/cache/conftool/dbconfig/20261116-114129-ladsgroup.json
- 11:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P97450 and previous config saved to /var/cache/conftool/dbconfig/20261116-113127-ladsgroup.json
- 11:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97449 and previous config saved to /var/cache/conftool/dbconfig/20261116-112125-ladsgroup.json
- 11:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97448 and previous config saved to /var/cache/conftool/dbconfig/20261116-111253-ladsgroup.json
- 11:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance
- 11:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97447 and previous config saved to /var/cache/conftool/dbconfig/20261116-111232-ladsgroup.json
- 11:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P97446 and previous config saved to /var/cache/conftool/dbconfig/20261116-110230-ladsgroup.json
- 10:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P97445 and previous config saved to /var/cache/conftool/dbconfig/20261116-105228-ladsgroup.json
- 10:45 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1005.eqiad.wmnet with OS bookworm
- 10:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97444 and previous config saved to /var/cache/conftool/dbconfig/20261116-104226-ladsgroup.json
- 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97443 and previous config saved to /var/cache/conftool/dbconfig/20261116-103407-ladsgroup.json
- 10:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1028,1032].eqiad.wmnet,db[1155,1270].eqiad.wmnet with reason: Maintenance
- 10:33 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1221.eqiad.wmnet with reason: Maintenance
- 10:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T441633)', diff saved to https://phabricator.wikimedia.org/P97442 and previous config saved to /var/cache/conftool/dbconfig/20261116-103327-ladsgroup.json
- 10:29 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1005.eqiad.wmnet with reason: host reimage
- 10:26 klausman@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1005.eqiad.wmnet with reason: host reimage
- 10:23 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1005.eqiad.wmnet (remove config only)
- 10:23 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
- 10:23 klausman@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1005.eqiad.wmnet (remove config only)
- 10:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P97441 and previous config saved to /var/cache/conftool/dbconfig/20261116-102325-ladsgroup.json
- 10:22 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
- 10:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P97440 and previous config saved to /var/cache/conftool/dbconfig/20261116-101323-ladsgroup.json
- 10:05 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
- 10:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T441633)', diff saved to https://phabricator.wikimedia.org/P97439 and previous config saved to /var/cache/conftool/dbconfig/20261116-100321-ladsgroup.json
- 10:02 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
- 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1199 (T441633)', diff saved to https://phabricator.wikimedia.org/P97438 and previous config saved to /var/cache/conftool/dbconfig/20261116-095323-ladsgroup.json
- 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance
- 09:47 klausman@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ml-serve1005.eqiad.wmnet (push config only)
- 09:47 klausman@cumin1004: START - Cookbook sre.hosts.dhcp for host ml-serve1005.eqiad.wmnet (push config only)
- 09:46 klausman@cumin1004: START - Cookbook sre.hosts.reimage for host ml-serve1005.eqiad.wmnet with OS bookworm
- 09:43 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete --nowarn growthexperiments-homepage-suggestededits-preactivated # T444075
- 09:43 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php --delete growthexperiments-homepage-suggestededits-preactivated # T444075
- 09:43 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist growthexperiments userOptions.php -delete growthexperiments-homepage-suggestededits-preactivated # T444075
- 09:06 klausman@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8 days, 0:00:00 on ml-serve1005.eqiad.wmnet with reason: examining disk situation
- 08:48 fabfur: upgrade haproxykafka to 0.4.0 on A:cp (T443640)
- 08:32 tappof: alert* hosts reboot
- 08:31 jmm@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2003"
- 08:29 tappof@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host alert2002.wikimedia.org
- 08:29 tappof@cumin1004: START - Cookbook sre.hosts.reboot-single for host alert2002.wikimedia.org
- 08:20 jmm@cumin2003: START - Cookbook sre.dns.netbox
- 08:17 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2003.wikimedia.org
- 08:15 jmm@cumin2003: START - Cookbook sre.hosts.decommission for hosts testvm2001.codfw.wmnet
- 08:10 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit2003.wikimedia.org
- 08:10 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org
- 08:04 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org
- 08:04 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit1003.wikimedia.org
- 07:57 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host gerrit1003.wikimedia.org
- 07:52 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2003.wikimedia.org
- 07:46 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab2003.wikimedia.org
- 07:46 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org
- 07:39 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org
- 07:39 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org
- 07:31 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org
- 07:30 jelto@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org
- 07:24 jelto@cumin1004: START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org
- 03:48 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lists1004.wikimedia.org with reason: new-kernel
- 03:42 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lists2001.wikimedia.org with reason: new-kernel
- 03:38 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out2001.wikimedia.org with reason: new-kernel
- 03:37 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-out1001.wikimedia.org with reason: new-kernel
- 03:33 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in1001.wikimedia.org with reason: new-kernel
- 03:28 jhathaway@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx-in2001.wikimedia.org with reason: new-kernel
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 10s)
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
2026-11-15
- 22:20 fabfur@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy
- 22:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
- 22:16 fabfur@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough
- 22:13 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
- 22:08 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
- 22:07 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast7002.wikimedia.org
- 22:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
- 22:01 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast7002.wikimedia.org
- 21:31 fabfur@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-urldownloaders (exit_code=0) rolling reboot on A:url-downloader and A:bookworm and (A:url-downloader or A:url-downloader-eqiad or A:url-downloader-codfw)
- 21:14 fabfur@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy
- 21:13 fabfur@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-reboot-urldownloaders rolling reboot on A:url-downloader and A:bookworm and (A:url-downloader or A:url-downloader-eqiad or A:url-downloader-codfw)
- 21:10 fabfur@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 34s)
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
2026-11-14
- 15:44 fabfur@cumin1004: conftool action : tet; selector: cluster=cache
- 14:39 sukhe: sudo cumin -b1 -s30 "A:dnsbox and (A:eqiad or A:codfw)" "birdc restart bfd1"
- 14:34 sukhe: sudo cumin -b1 -s5 "A:wikidough and (A:eqiad or A:codfw)" "birdc restart bfd1": T434806
- 13:16 brett@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cp3074.esams.wmnet with reason: troubleshooting cmos failure
- 12:08 sukhe@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp3074.esams.wmnet with reason: host is depooled for HW failure, depooling
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:04 sukhe@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy (exit_code=0) rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy
- 11:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast5005.wikimedia.org
- 11:01 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough
- 11:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast4006.wikimedia.org
- 10:55 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast5005.wikimedia.org
- 10:55 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast4006.wikimedia.org
- 10:52 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
- 10:44 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
- 10:43 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast1004.wikimedia.org
- 10:39 sukhe@cumin1004: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-urldownloaders (exit_code=0) rolling reboot on A:url-downloader
- 10:37 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast1004.wikimedia.org
- 10:01 sukhe@cumin1004: START - Cookbook sre.misc-clusters.roll-restart-reboot-urldownloaders rolling reboot on A:url-downloader
- 09:58 sukhe@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-hcaptcha-proxy rolling reboot on A:hcaptcha-proxy and A:hcaptcha-proxy
- 09:58 sukhe@cumin1004: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling reboot on A:wikidough
- 05:23 vgutierrez@cumin1004: END (PASS) - Cookbook sre.dns.roll-reboot (exit_code=0) rolling reboot on A:dnsbox
- 05:23 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns7002.wikimedia.org
- 05:12 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns7002.wikimedia.org
- 04:57 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns7001.wikimedia.org
- 04:42 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns7001.wikimedia.org
- 04:36 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp3075.*} and A:cp
- 04:36 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3075.esams.wmnet
- 04:27 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns6002.wikimedia.org
- 04:25 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on P{cp3075.*} and A:cp
- 04:21 vgutierrez@cumin1004: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_esams
- 04:20 vgutierrez@cumin1004: END (ERROR) - Cookbook sre.cdn.roll-reboot (exit_code=97) rolling reboot on A:cp-text_eqiad
- 04:20 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad
- 04:06 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns6002.wikimedia.org
- 03:51 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns6001.wikimedia.org
- 03:37 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns6001.wikimedia.org
- 03:26 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_eqsin
- 03:26 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5026.eqsin.wmnet
- 03:22 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns5004.wikimedia.org
- 03:22 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_codfw
- 03:22 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2059.codfw.wmnet
- 03:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_eqiad
- 03:11 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1114.eqiad.wmnet
- 03:11 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_drmrs
- 03:11 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6016.drmrs.wmnet
- 03:10 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_magru
- 03:10 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7010.magru.wmnet
- 03:08 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-text_ulsfo
- 03:08 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4046.ulsfo.wmnet
- 03:06 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns5004.wikimedia.org
- 02:54 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5025.eqsin.wmnet
- 02:53 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2057.codfw.wmnet
- 02:51 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns5003.wikimedia.org
- 02:43 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns5003.wikimedia.org
- 02:41 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1112.eqiad.wmnet
- 02:41 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6015.drmrs.wmnet
- 02:40 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7009.magru.wmnet
- 02:39 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4045.ulsfo.wmnet
- 02:28 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns4004.wikimedia.org
- 02:23 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5024.eqsin.wmnet
- 02:22 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2055.codfw.wmnet
- 02:16 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3073.esams.wmnet
- 02:14 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns4004.wikimedia.org
- 02:11 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1110.eqiad.wmnet
- 02:10 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet
- 02:08 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7008.magru.wmnet
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 38s)
- 02:08 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4044.ulsfo.wmnet
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 01:59 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns4003.wikimedia.org
- 01:53 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2053.codfw.wmnet
- 01:50 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5023.eqsin.wmnet
- 01:45 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns4003.wikimedia.org
- 01:44 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3072.esams.wmnet
- 01:39 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1108.eqiad.wmnet
- 01:39 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet
- 01:38 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4043.ulsfo.wmnet
- 01:36 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7007.magru.wmnet
- 01:30 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns3004.wikimedia.org
- 01:24 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2051.codfw.wmnet
- 01:18 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5022.eqsin.wmnet
- 01:16 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns3004.wikimedia.org
- 01:14 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3071.esams.wmnet
- 01:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqsin
- 01:13 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5032.eqsin.wmnet
- 01:12 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_esams
- 01:12 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3081.esams.wmnet
- 01:10 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1106.eqiad.wmnet
- 01:08 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet
- 01:07 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4042.ulsfo.wmnet
- 01:07 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_drmrs
- 01:07 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6008.drmrs.wmnet
- 01:07 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_eqiad
- 01:07 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1115.eqiad.wmnet
- 01:05 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7006.magru.wmnet
- 01:03 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_magru
- 01:03 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7016.magru.wmnet
- 01:01 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns3003.wikimedia.org
- 00:59 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_ulsfo
- 00:59 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4052.ulsfo.wmnet
- 00:54 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on A:cp-upload_codfw
- 00:54 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2058.codfw.wmnet
- 00:53 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2049.codfw.wmnet
- 00:51 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns3003.wikimedia.org
- 00:42 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3070.esams.wmnet
- 00:41 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5031.eqsin.wmnet
- 00:41 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5021.eqsin.wmnet
- 00:40 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3080.esams.wmnet
- 00:40 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1104.eqiad.wmnet
- 00:38 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4041.ulsfo.wmnet
- 00:37 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet
- 00:37 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1113.eqiad.wmnet
- 00:37 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6007.drmrs.wmnet
- 00:36 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns2006.wikimedia.org
- 00:33 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7005.magru.wmnet
- 00:32 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7015.magru.wmnet
- 00:29 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4051.ulsfo.wmnet
- 00:26 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2056.codfw.wmnet
- 00:24 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2047.codfw.wmnet
- 00:22 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns2006.wikimedia.org
- 00:11 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3069.esams.wmnet
- 00:10 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1103.eqiad.wmnet
- 00:09 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5030.eqsin.wmnet
- 00:09 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5020.eqsin.wmnet
- 00:09 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3079.esams.wmnet
- 00:07 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1111.eqiad.wmnet
- 00:07 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns2005.wikimedia.org
- 00:06 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6010.drmrs.wmnet
- 00:06 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet
- 00:05 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4040.ulsfo.wmnet
- 00:02 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7004.magru.wmnet
- 00:00 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7014.magru.wmnet
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
2026-11-13
- 23:59 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4050.ulsfo.wmnet
- 23:59 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns2005.wikimedia.org
- 23:57 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2054.codfw.wmnet
- 23:56 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet
- 23:44 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns2004.wikimedia.org
- 23:39 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3078.esams.wmnet
- 23:39 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3068.esams.wmnet
- 23:38 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1102.eqiad.wmnet
- 23:38 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1109.eqiad.wmnet
- 23:37 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5029.eqsin.wmnet
- 23:36 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5019.eqsin.wmnet
- 23:36 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6009.drmrs.wmnet
- 23:35 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet
- 23:34 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4039.ulsfo.wmnet
- 23:30 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4049.ulsfo.wmnet
- 23:30 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7003.magru.wmnet
- 23:30 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns2004.wikimedia.org
- 23:29 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7013.magru.wmnet
- 23:29 vgutierrez@cumin1004: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir
- 23:28 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2052.codfw.wmnet
- 23:27 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet
- 23:15 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns1006.wikimedia.org
- 23:08 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1101.eqiad.wmnet
- 23:08 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1107.eqiad.wmnet
- 23:07 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3067.esams.wmnet
- 23:07 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3077.esams.wmnet
- 23:05 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5028.eqsin.wmnet
- 23:05 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6002.drmrs.wmnet
- 23:04 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5018.eqsin.wmnet
- 23:04 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet
- 23:02 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4038.ulsfo.wmnet
- 23:01 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns1006.wikimedia.org
- 23:00 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4048.ulsfo.wmnet
- 23:00 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7002.magru.wmnet
- 22:59 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2050.codfw.wmnet
- 22:58 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2044.codfw.wmnet
- 22:58 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7012.magru.wmnet
- 22:46 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns1005.wikimedia.org
- 22:38 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1105.eqiad.wmnet
- 22:38 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp1100.eqiad.wmnet
- 22:37 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3076.esams.wmnet
- 22:35 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp3066.esams.wmnet
- 22:34 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6001.drmrs.wmnet
- 22:33 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet
- 22:33 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5027.eqsin.wmnet
- 22:32 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp5017.eqsin.wmnet
- 22:32 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns1005.wikimedia.org
- 22:31 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2048.codfw.wmnet
- 22:30 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4037.ulsfo.wmnet
- 22:30 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp2043.codfw.wmnet
- 22:29 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp4047.ulsfo.wmnet
- 22:28 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqiad
- 22:28 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqiad
- 22:28 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7001.magru.wmnet
- 22:27 vgutierrez@cumin1004: cookbooks.sre.cdn.roll-reboot finished rebooting cp7011.magru.wmnet
- 22:25 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_esams
- 22:25 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_esams
- 22:24 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_drmrs
- 22:24 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_drmrs
- 22:22 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_codfw
- 22:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_codfw
- 22:21 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_eqsin
- 22:20 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_eqsin
- 22:19 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_ulsfo
- 22:18 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_ulsfo
- 22:17 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-upload_magru
- 22:17 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot finished rebooting dns1004.wikimedia.org
- 22:17 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-reboot rolling reboot on A:cp-text_magru
- 22:14 vgutierrez@cumin1004: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling reboot on A:ncredir
- 22:08 vgutierrez@cumin1004: cookbooks.sre.dns.roll-reboot begin reboot of dns1004.wikimedia.org
- 22:08 vgutierrez@cumin1004: START - Cookbook sre.dns.roll-reboot rolling reboot on A:dnsbox
- 21:49 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp2005.wikimedia.org
- 21:45 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp2005.wikimedia.org
- 21:39 slyngshede@dns1004: END - running authdns-update
- 21:36 slyngshede@dns1004: START - running authdns-update
- 21:35 slyngshede@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host idp1005.wikimedia.org
- 21:31 slyngshede@cumin1004: START - Cookbook sre.hosts.reboot-single for host idp1005.wikimedia.org
- 19:33 dancy@deploy2003: Installation of scap version "4.314.0" completed for 154 hosts
- 19:29 dancy@deploy2003: Installing scap version "4.314.0" for 154 host(s)
- 18:43 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2055.codfw.wmnet with OS bookworm
- 18:43 jhancock@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 18:42 jhancock@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 18:39 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2056.codfw.wmnet with OS bookworm
- 18:39 jhancock@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 18:38 jhancock@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 18:35 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2054.codfw.wmnet with OS bookworm
- 18:35 jhancock@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 18:35 jhancock@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 18:24 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2055.codfw.wmnet with reason: host reimage
- 18:21 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2056.codfw.wmnet with reason: host reimage
- 18:18 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2054.codfw.wmnet with reason: host reimage
- 18:15 jhancock@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2056.codfw.wmnet with reason: host reimage
- 18:15 jhancock@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2055.codfw.wmnet with reason: host reimage
- 18:14 jhancock@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2054.codfw.wmnet with reason: host reimage
- 18:13 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2056.codfw.wmnet (remove config only)
- 18:13 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2056.codfw.wmnet (remove config only)
- 18:12 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2055.codfw.wmnet (remove config only)
- 18:12 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2055.codfw.wmnet (remove config only)
- 18:11 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2054.codfw.wmnet (remove config only)
- 18:11 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2054.codfw.wmnet (remove config only)
- 18:03 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2056.codfw.wmnet (push config only)
- 18:03 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2056.codfw.wmnet (push config only)
- 18:03 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host ganeti2056.codfw.wmnet with OS bookworm
- 18:03 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2055.codfw.wmnet (push config only)
- 18:03 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2055.codfw.wmnet (push config only)
- 18:03 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host ganeti2055.codfw.wmnet with OS bookworm
- 18:02 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2054.codfw.wmnet (push config only)
- 18:02 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2054.codfw.wmnet (push config only)
- 18:02 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host ganeti2054.codfw.wmnet with OS bookworm
- 15:46 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
- 15:42 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
- 15:41 ebysans@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
- 15:41 ebysans@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
- 15:40 ebysans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
- 15:40 ebysans@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
- 15:39 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 15:36 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 15:22 brouberol@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
- 15:22 brouberol@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
- 15:21 brouberol@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
- 15:21 brouberol@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
- 15:21 brouberol@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
- 15:20 brouberol@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
- 15:19 brouberol@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
- 15:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
- 15:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
- 15:08 brouberol@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply
- 15:08 brouberol@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-main: apply
- 15:08 brouberol@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
- 15:07 brouberol@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
- 15:07 brouberol@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-main: apply
- 15:07 brouberol@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-main: apply
- 15:07 brouberol@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply
- 15:06 brouberol@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply
- 15:06 brouberol@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply
- 15:06 brouberol@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply
- 15:05 brouberol@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply
- 15:05 brouberol@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply
- 15:05 brouberol@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply
- 15:04 brouberol@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply
- 15:04 brouberol@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply
- 15:03 brouberol@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply
- 15:03 brouberol@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply
- 15:03 brouberol@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply
- 15:03 brouberol@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
- 15:02 brouberol@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
- 15:01 brouberol@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
- 15:00 brouberol@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
- 14:58 brouberol@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
- 14:58 brouberol@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
- 14:51 brouberol@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
- 13:58 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
- 13:57 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
- 13:57 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
- 13:57 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
- 13:56 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
- 13:55 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
- 13:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
- 13:53 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
- 13:44 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2056.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 13:44 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2055.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 13:44 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2056.codfw.wmnet (remove config only)
- 13:44 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2056.codfw.wmnet (remove config only)
- 13:44 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2055.codfw.wmnet (remove config only)
- 13:44 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2054.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 13:44 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2055.codfw.wmnet (remove config only)
- 13:43 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2054.codfw.wmnet (remove config only)
- 13:43 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2054.codfw.wmnet (remove config only)
- 13:37 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2056.codfw.wmnet (push config only)
- 13:37 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2056.codfw.wmnet (push config only)
- 13:37 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host ganeti2056.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 13:37 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2055.codfw.wmnet (push config only)
- 13:37 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2055.codfw.wmnet (push config only)
- 13:36 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host ganeti2055.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 13:36 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2054.codfw.wmnet (push config only)
- 13:36 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2054.codfw.wmnet (push config only)
- 13:36 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host ganeti2054.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 13:34 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2053.codfw.wmnet with OS bookworm
- 13:34 jhancock@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 13:33 jhancock@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 13:32 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2052.codfw.wmnet with OS bookworm
- 13:32 jhancock@cumin2003: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 13:32 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2051.codfw.wmnet with OS bookworm
- 13:32 jhancock@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 13:31 jhancock@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 13:26 jhancock@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2003"
- 13:16 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2053.codfw.wmnet with reason: host reimage
- 13:13 dpogorzelski@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: sync
- 13:13 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2052.codfw.wmnet with reason: host reimage
- 13:12 dpogorzelski@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: sync
- 13:09 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti2051.codfw.wmnet with reason: host reimage
- 13:06 cgoubert@deploy2003: Finished scap sync-world: Backport for Lift IP cap for WikiIndaba 2026 (T444794) (duration: 09m 28s)
- 13:05 jhancock@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2053.codfw.wmnet with reason: host reimage
- 13:05 jhancock@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2052.codfw.wmnet with reason: host reimage
- 13:05 jhancock@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2051.codfw.wmnet with reason: host reimage
- 13:02 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2053.codfw.wmnet (remove config only)
- 13:02 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2053.codfw.wmnet (remove config only)
- 13:02 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2052.codfw.wmnet (remove config only)
- 13:02 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2051.codfw.wmnet (remove config only)
- 13:02 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2052.codfw.wmnet (remove config only)
- 13:02 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2051.codfw.wmnet (remove config only)
- 13:00 cgoubert@deploy2003: cgoubert: Continuing with deployment
- 12:59 cgoubert@deploy2003: cgoubert: Backport for Lift IP cap for WikiIndaba 2026 (T444794) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 12:57 cgoubert@deploy2003: Started scap sync-world: Backport for Lift IP cap for WikiIndaba 2026 (T444794)
- 12:53 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2053.codfw.wmnet (push config only)
- 12:53 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2053.codfw.wmnet (push config only)
- 12:53 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2052.codfw.wmnet (push config only)
- 12:53 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2052.codfw.wmnet (push config only)
- 12:53 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host ganeti2053.codfw.wmnet with OS bookworm
- 12:53 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host ganeti2052.codfw.wmnet with OS bookworm
- 12:53 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2051.codfw.wmnet (push config only)
- 12:53 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2051.codfw.wmnet (push config only)
- 12:52 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host ganeti2051.codfw.wmnet with OS bookworm
- 12:49 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2053.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:49 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2052.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:49 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2051.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:49 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2053.codfw.wmnet (remove config only)
- 12:49 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2053.codfw.wmnet (remove config only)
- 12:49 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2052.codfw.wmnet (remove config only)
- 12:49 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2052.codfw.wmnet (remove config only)
- 12:49 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2051.codfw.wmnet (remove config only)
- 12:49 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2051.codfw.wmnet (remove config only)
- 12:42 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2053.codfw.wmnet (push config only)
- 12:42 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2053.codfw.wmnet (push config only)
- 12:42 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host ganeti2053.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:42 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2052.codfw.wmnet (push config only)
- 12:42 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2052.codfw.wmnet (push config only)
- 12:42 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host ganeti2051.codfw.wmnet (push config only)
- 12:42 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host ganeti2051.codfw.wmnet (push config only)
- 12:42 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host ganeti2052.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:41 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host ganeti2051.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/liftwing-studio: apply
- 11:20 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/liftwing-studio: apply
- 11:04 javiermonton@deploy2003: Finished scap deploy-service mw-content-history-reconcile-enrich: Deploying due to pending changes from SRE on Envoy (duration: 03m 02s)
- 11:01 javiermonton@deploy2003: scap deploy-service mw-content-history-reconcile-enrich: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying mw-content-history-reconcile-enrich
- 11:01 javiermonton@deploy2003: Started scap deploy-service mw-content-history-reconcile-enrich: Deploying due to pending changes from SRE on Envoy
- 10:59 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
- 10:59 javiermonton@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-content-history-reconcile-enrich-next: apply
- 10:56 javiermonton@deploy2003: Finished scap deploy-service webrequest-pageview: Deploying due to pending changes from SRE on Envoy (duration: 03m 50s)
- 10:55 javiermonton@deploy2003: Finished scap deploy-service mw-page-html-content-change-enrich: Deploying due to pending changes from SRE on Envoy (duration: 03m 11s)
- 10:52 javiermonton@deploy2003: scap deploy-service webrequest-pageview: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying webrequest-pageview
- 10:52 javiermonton@deploy2003: Started scap deploy-service webrequest-pageview: Deploying due to pending changes from SRE on Envoy
- 10:52 javiermonton@deploy2003: scap deploy-service mw-page-html-content-change-enrich: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying mw-page-html-content-change-enrich
- 10:52 javiermonton@deploy2003: Started scap deploy-service mw-page-html-content-change-enrich: Deploying due to pending changes from SRE on Envoy
- 10:51 javiermonton@deploy2003: Finished scap deploy-service mw-page-html-feature-counts-change-enrich: Deploying due to pending changes from SRE on Envoy (duration: 06m 40s)
- 10:45 javiermonton@deploy2003: scap deploy-service mw-page-html-feature-counts-change-enrich: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying mw-page-html-feature-counts-change-enrich
- 10:45 javiermonton@deploy2003: Started scap deploy-service mw-page-html-feature-counts-change-enrich: Deploying due to pending changes from SRE on Envoy
- 10:43 hashar@deploy2003: Finished deploy [integration/docroot@069e018]: build: Updating postcss-selector-parser to 6.1.6 (duration: 00m 14s)
- 10:43 hashar@deploy2003: Started deploy [integration/docroot@069e018]: build: Updating postcss-selector-parser to 6.1.6
- 10:42 javiermonton@deploy2003: Finished scap deploy-service pageview-trending-relative: Deploying due to pending changes from SRE on Envoy (duration: 06m 08s)
- 10:42 javiermonton@deploy2003: Finished scap deploy-service pageview-trending-relative-next: Deploying due to pending changes from SRE on Envoy (duration: 05m 30s)
- 10:37 javiermonton@deploy2003: scap deploy-service pageview-trending-relative-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative-next
- 10:37 javiermonton@deploy2003: Started scap deploy-service pageview-trending-relative-next: Deploying due to pending changes from SRE on Envoy
- 10:36 javiermonton@deploy2003: scap deploy-service pageview-trending-relative: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying pageview-trending-relative
- 10:36 javiermonton@deploy2003: Started scap deploy-service pageview-trending-relative: Deploying due to pending changes from SRE on Envoy
- 09:08 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
- 09:08 jhancock@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 09:08 jhancock@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserve2005 to codfw - jhancock@cumin2003"
- 09:08 jhancock@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding puppetserve2005 to codfw - jhancock@cumin2003"
- 09:08 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
- 09:07 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
- 09:06 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
- 09:04 jhancock@cumin2003: START - Cookbook sre.dns.netbox
- 09:01 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host puppetserve2006.codfw.wmnet with OS bookworm
- 09:01 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (remove config only)
- 09:01 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (remove config only)
- 09:00 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host puppetserve2005.codfw.wmnet with OS bookworm
- 09:00 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (remove config only)
- 09:00 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (remove config only)
- 08:58 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (push config only)
- 08:58 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (push config only)
- 08:58 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host puppetserve2006.codfw.wmnet with OS bookworm
- 08:57 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (push config only)
- 08:57 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (push config only)
- 08:57 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host puppetserve2005.codfw.wmnet with OS bookworm
- 08:42 kevinbazira@deploy2003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
- 08:41 kevinbazira@deploy2003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
- 08:39 kevinbazira@deploy2003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
- 08:17 jhancock@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 08:15 jhancock@cumin2003: START - Cookbook sre.dns.netbox
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
2026-11-12
- 23:10 catrope@deploy2003: Finished scap sync-world: Backport for REST: Correct test server hostname to test.wikipedia.org (T408816) (duration: 16m 14s)
- 23:04 catrope@deploy2003: kineticpelagic, catrope: Continuing with deployment
- 22:56 catrope@deploy2003: kineticpelagic, catrope: Backport for REST: Correct test server hostname to test.wikipedia.org (T408816) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 22:54 catrope@deploy2003: Started scap sync-world: Backport for REST: Correct test server hostname to test.wikipedia.org (T408816)
- 21:55 dhardy@deploy2003: Finished scap sync-world: Backport for PersonalDashboard: Clean up config and set QA values for test and beta wikis (T441729), IS: Enable watchstar popover everywhere (T417847) (duration: 10m 57s)
- 21:49 dhardy@deploy2003: samtar, dhardy: Continuing with deployment
- 21:46 dhardy@deploy2003: samtar, dhardy: Backport for PersonalDashboard: Clean up config and set QA values for test and beta wikis (T441729), IS: Enable watchstar popover everywhere (T417847) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:44 dhardy@deploy2003: Started scap sync-world: Backport for PersonalDashboard: Clean up config and set QA values for test and beta wikis (T441729), IS: Enable watchstar popover everywhere (T417847)
- 21:42 cklimas@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
- 21:41 cklimas@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
- 21:41 cklimas@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
- 21:40 cklimas@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
- 21:39 cklimas@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
- 21:39 cklimas@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
- 21:33 sbisson@deploy2003: Finished scap sync-world: Backport for CX: Skip unregistered VE plugin modules (T444489) (1.48.0-wmf.4) (duration: 10m 39s)
- 21:27 sbisson@deploy2003: sbisson: Continuing with deployment
- 21:24 sbisson@deploy2003: sbisson: Backport for CX: Skip unregistered VE plugin modules (T444489) (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:22 sbisson@deploy2003: Started scap sync-world: Backport for CX: Skip unregistered VE plugin modules (T444489) (1.48.0-wmf.4)
- 21:15 sbassett@deploy2003: Finished scap sync-world: Backport for Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424) (duration: 08m 53s)
- 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
- 21:08 sbassett@deploy2003: sbassett: Backport for Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:06 sbassett@deploy2003: Started scap sync-world: Backport for Content Security Policy: add stricter report-only policy (T419612 T427557 T431144 T327588 T245424)
- 21:02 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
- 21:02 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
- 21:00 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
- 21:00 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
- 20:59 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
- 20:57 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
- 20:45 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 20:45 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 17:30 fnegri@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
- 17:29 fnegri@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
- 17:29 fnegri@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
- 17:28 fnegri@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
- 17:16 fnegri@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
- 17:16 fnegri@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
- 17:03 inflatador: bking@localhost unban cirrussearch2081 (hot shards) T444363
- 15:15 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 15:14 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 15:04 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 15:00 matthiasmullie: UTC afternoon backport window done
- 15:00 mlitn@deploy2003: Finished scap sync-world: Backport for Move exposure logging back to JS (T444205) (1.48.0-wmf.4) (duration: 10m 43s)
- 14:54 mlitn@deploy2003: mlitn: Continuing with deployment
- 14:53 mlitn@deploy2003: mlitn: Backport for Move exposure logging back to JS (T444205) (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:53 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 14:49 mlitn@deploy2003: Started scap sync-world: Backport for Move exposure logging back to JS (T444205) (1.48.0-wmf.4)
- 14:48 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
- 14:48 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
- 14:46 seanleong-wmde@deploy2003: Finished scap sync-world: Backport for Switch site lookup to sitematrix lookup (T438982) (duration: 12m 19s)
- 14:42 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 14:41 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 14:40 seanleong-wmde@deploy2003: seanleong-wmde: Continuing with deployment
- 14:36 seanleong-wmde@deploy2003: seanleong-wmde: Backport for Switch site lookup to sitematrix lookup (T438982) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:34 seanleong-wmde@deploy2003: Started scap sync-world: Backport for Switch site lookup to sitematrix lookup (T438982)
- 14:27 sergi0: UTC afternoon backport window done
- 14:26 sgimeno@deploy2003: Finished scap sync-world: Backport for GrowthExperiments: Scale early onboarding on ar/en/es/frwiki (duration: 08m 41s)
- 14:20 sgimeno@deploy2003: sgimeno: Continuing with deployment
- 14:20 sgimeno@deploy2003: sgimeno: Backport for GrowthExperiments: Scale early onboarding on ar/en/es/frwiki synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:17 sgimeno@deploy2003: Started scap sync-world: Backport for GrowthExperiments: Scale early onboarding on ar/en/es/frwiki
- 14:09 seanleong-wmde@deploy2003: Failed scap sync-world: Backport for Switch site lookup to sitematrix lookup (T438982) (duration: 05m 10s)
- 14:09 seanleong-wmde@deploy2003: Rolling back deployment
- 14:06 seanleong-wmde@deploy2003: seanleong-wmde: Backport for Switch site lookup to sitematrix lookup (T438982) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:04 seanleong-wmde@deploy2003: Started scap sync-world: Backport for Switch site lookup to sitematrix lookup (T438982)
- 14:04 sfaci@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
- 14:03 sfaci@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
- 13:57 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host puppetserve2006.codfw.wmnet with OS bookworm
- 13:57 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host puppetserve2005.codfw.wmnet with OS bookworm
- 13:37 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'llmisvc' .
- 13:37 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm-batch' for release 'main' .
- 13:30 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
- 13:29 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
- 13:05 eevans@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1018.eqiad.wmnet
- 12:58 eevans@cumin1004: START - Cookbook sre.hosts.reboot-single for host aqs1018.eqiad.wmnet
- 12:54 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (remove config only)
- 12:54 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (remove config only)
- 12:54 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (remove config only)
- 12:53 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (remove config only)
- 12:51 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
- 12:51 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
- 12:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
- 12:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
- 12:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
- 12:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
- 12:47 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
- 12:47 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
- 12:47 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
- 12:46 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
- 12:46 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
- 12:46 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (push config only)
- 12:46 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (push config only)
- 12:46 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
- 12:45 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host puppetserve2006.codfw.wmnet with OS bookworm
- 12:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
- 12:45 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (push config only)
- 12:45 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (push config only)
- 12:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
- 12:45 jhancock@cumin2003: START - Cookbook sre.hosts.reimage for host puppetserve2005.codfw.wmnet with OS bookworm
- 12:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
- 12:45 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
- 12:45 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
- 12:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
- 12:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
- 12:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
- 12:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
- 12:43 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
- 12:43 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
- 12:43 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
- 12:43 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 12:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 12:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
- 12:42 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host puppetserve2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:42 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host puppetserve2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:42 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (remove config only)
- 12:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
- 12:42 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (remove config only)
- 12:42 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (remove config only)
- 12:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
- 12:42 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (remove config only)
- 12:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
- 12:35 urandom: rolling Cassandra resart, aqs1018 — T444597
- 12:35 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2006.codfw.wmnet (push config only)
- 12:35 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2006.codfw.wmnet (push config only)
- 12:35 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host puppetserve2005.codfw.wmnet (push config only)
- 12:35 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host puppetserve2006.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:35 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host puppetserve2005.codfw.wmnet (push config only)
- 12:34 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host puppetserve2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:32 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pki2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:32 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host pki2004.codfw.wmnet (remove config only)
- 12:32 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host pki2004.codfw.wmnet (remove config only)
- 12:26 inflatador: bking@localhost ban cirrussearch2081 (hot shards) T444363
- 12:21 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host pki2004.codfw.wmnet (push config only)
- 12:21 jhancock@cumin2003: START - Cookbook sre.hosts.dhcp for host pki2004.codfw.wmnet (push config only)
- 12:21 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host pki2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 12:17 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetserve2006
- 12:15 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host puppetserve2006
- 12:14 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetserve2005
- 12:14 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host puppetserve2005
- 12:14 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pki2004
- 12:14 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host pki2004
- 12:14 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2073
- 12:14 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2073
- 12:13 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2072
- 12:13 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2072
- 12:13 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2071
- 12:13 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2071
- 12:13 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2070
- 12:12 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2070
- 12:12 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2069
- 12:12 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2069
- 12:12 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2068
- 12:11 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2068
- 12:11 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2067
- 12:11 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2067
- 12:11 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2066
- 12:11 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2066
- 12:10 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2065
- 12:10 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2065
- 12:10 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2064
- 12:10 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2064
- 12:10 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2062
- 12:10 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2062
- 12:09 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2061
- 12:09 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2061
- 12:09 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2059
- 12:09 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2059
- 12:09 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2058
- 12:09 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2058
- 12:09 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2057
- 12:09 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2057
- 12:08 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host mc2056
- 12:08 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host mc2056
- 12:08 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2056
- 12:08 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ganeti2056
- 12:08 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2055
- 12:08 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ganeti2055
- 12:07 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2054
- 12:07 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ganeti2054
- 12:07 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2053
- 12:07 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ganeti2053
- 12:07 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2052
- 12:07 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ganeti2052
- 12:06 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2051
- 12:06 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host ganeti2051
- 12:06 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2120
- 12:06 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2120
- 12:06 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2119
- 12:05 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2119
- 12:05 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2118
- 12:05 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2118
- 12:05 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2117
- 12:05 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2117
- 12:04 jhancock@cumin2003: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2116
- 12:04 jhancock@cumin2003: START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2116
- 12:04 jhancock@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 12:04 jhancock@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding several servers to codfw - jhancock@cumin2003"
- 12:04 jhancock@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding several servers to codfw - jhancock@cumin2003"
- 12:00 jhancock@cumin2003: START - Cookbook sre.dns.netbox
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:50 dreamyjazz@deploy2003: Finished scap sync-world: Backport for Enable CheckUser AbuseFilter variables on enwiki (T234155) (duration: 11m 44s)
- 11:44 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
- 11:41 dreamyjazz@deploy2003: dreamyjazz: Backport for Enable CheckUser AbuseFilter variables on enwiki (T234155) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:39 dreamyjazz@deploy2003: Started scap sync-world: Backport for Enable CheckUser AbuseFilter variables on enwiki (T234155)
- 11:32 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
- 11:02 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
- 10:59 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 10:38 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 10:32 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
- 10:31 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
- 10:31 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply
- 10:31 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply
- 10:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/turnilo: apply
- 10:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/turnilo: apply
- 10:12 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 10:11 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 10:09 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 10:08 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 10:07 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 10:03 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:56 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:55 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:55 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:54 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:51 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:50 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:48 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:48 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:46 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:46 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:46 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 09:41 tappof@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:41 tappof@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/bento-mw-accesslog-metrics: apply
- 09:40 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: sync
- 09:40 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: sync
- 09:40 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: sync
- 09:40 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: sync
- 09:33 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 09:28 marostegui@cumin1004: END (FAIL) - Cookbook sre.mysql.decommission (exit_code=99)
- 09:28 marostegui@cumin1004: Removing clouddb1015 from zarcillo T443768
- 09:28 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1015.eqiad.wmnet
- 09:28 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 09:28 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 09:27 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 09:26 aklapper@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.4 refs T442222
- 09:24 marostegui@cumin1004: START - Cookbook sre.dns.netbox
- 09:18 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1015.eqiad.wmnet
- 09:18 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
- 08:58 dcausse@deploy2003: Finished scap sync-world: Backport for Revert "search: add pool counter rules and dedicated pool for WDQS" (duration: 08m 58s)
- 08:52 dcausse@deploy2003: dcausse, trainbranchbot: Continuing with deployment
- 08:51 dcausse@deploy2003: dcausse, trainbranchbot: Backport for Revert "search: add pool counter rules and dedicated pool for WDQS" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:49 dcausse@deploy2003: Started scap sync-world: Backport for Revert "search: add pool counter rules and dedicated pool for WDQS"
- 08:42 dcausse@deploy2003: Failed scap sync-world: Backport for search: add pool counter rules and dedicated pool for WDQS (T442824) (duration: 06m 29s)
- 08:41 dcausse@deploy2003: dcausse: Rolling back deployment
- 08:38 dcausse@deploy2003: dcausse: Backport for search: add pool counter rules and dedicated pool for WDQS (T442824) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:35 dcausse@deploy2003: Started scap sync-world: Backport for search: add pool counter rules and dedicated pool for WDQS (T442824)
- 08:31 kharlan@deploy2003: Finished scap sync-world: Backport for hCaptcha: Enable on French Wikipedia for API editing (T444509) (duration: 09m 27s)
- 08:24 kharlan@deploy2003: dreamyjazz, kharlan: Continuing with deployment
- 08:23 kharlan@deploy2003: dreamyjazz, kharlan: Backport for hCaptcha: Enable on French Wikipedia for API editing (T444509) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:21 kharlan@deploy2003: Started scap sync-world: Backport for hCaptcha: Enable on French Wikipedia for API editing (T444509)
- 08:19 dcausse@deploy2003: Finished scap sync-world: Backport for cirrus: Provide short cluster names (T440376) (duration: 11m 06s)
- 08:13 dcausse@deploy2003: ebernhardson, dcausse: Continuing with deployment
- 08:10 dcausse@deploy2003: ebernhardson, dcausse: Backport for cirrus: Provide short cluster names (T440376) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:08 dcausse@deploy2003: Started scap sync-world: Backport for cirrus: Provide short cluster names (T440376)
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 49s)
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
2026-11-11
- 23:41 brett@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp3074.esams.wmnet with reason: troubleshooting cmos failure
- 21:16 matthiasmullie: UTC late backport window done
- 21:15 mlitn@deploy2003: Finished scap sync-world: Backport for Do not Implement UserThankedListener (1.48.0-wmf.4), Squashed diff to master (1.48.0-wmf.4) (duration: 09m 15s)
- 21:09 mlitn@deploy2003: mlitn: Continuing with deployment
- 21:08 mlitn@deploy2003: mlitn: Backport for Do not Implement UserThankedListener (1.48.0-wmf.4), Squashed diff to master (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:06 mlitn@deploy2003: Started scap sync-world: Backport for Do not Implement UserThankedListener (1.48.0-wmf.4), Squashed diff to master (1.48.0-wmf.4)
- 19:47 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/echoserver: apply
- 19:47 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/echoserver: apply
- 19:46 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/analytics-test: apply
- 19:46 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/analytics-test: apply
- 19:46 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wmde: apply
- 19:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wmde: apply
- 19:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-wikidata: apply
- 19:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-wikidata: apply
- 19:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
- 19:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
- 19:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-sre: apply
- 19:43 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-sre: apply
- 19:43 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-search: apply
- 19:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-search: apply
- 19:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply
- 19:41 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply
- 19:41 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-platform-eng: apply
- 19:40 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-platform-eng: apply
- 19:40 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
- 19:39 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
- 19:39 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
- 19:38 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
- 19:38 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-fr-tech: apply
- 19:37 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-fr-tech: apply
- 19:37 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
- 19:36 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-experiment-platform: apply
- 19:36 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 19:36 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 19:36 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply
- 19:35 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply
- 19:35 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-product: apply
- 19:33 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-product: apply
- 19:33 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply
- 19:32 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply
- 19:32 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
- 19:31 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
- 19:31 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
- 19:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
- 19:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
- 19:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
- 19:30 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
- 19:29 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
- 19:28 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
- 19:28 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
- 19:28 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
- 19:28 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
- 19:27 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
- 19:27 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
- 19:27 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
- 19:26 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
- 19:21 sfaci@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
- 19:20 sfaci@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
- 19:14 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
- 19:14 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
- 19:12 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
- 19:10 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
- 16:34 inflatador: bking@deploy2003 `apt-get install python3-swiftclient` T434970
- 15:33 urbanecm@deploy2003: Finished scap sync-world: Backport for cswiki: Remove celebration logo (T442387), GrowthExperiments: Remove the $wgGECampaignTopics setting (T444124) (duration: 09m 26s)
- 15:27 urbanecm@deploy2003: urbanecm: Continuing with deployment
- 15:26 urbanecm@deploy2003: urbanecm: Backport for cswiki: Remove celebration logo (T442387), GrowthExperiments: Remove the $wgGECampaignTopics setting (T444124) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:23 urbanecm@deploy2003: Started scap sync-world: Backport for cswiki: Remove celebration logo (T442387), GrowthExperiments: Remove the $wgGECampaignTopics setting (T444124)
- 15:18 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Deploy evaluators config to silence logs from incoming calls (1363475) (duration: 02m 20s)
- 15:18 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
- 15:17 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
- 15:16 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
- 15:16 jforrester@deploy2003: Started scap deploy-service wikifunctions: Deploy evaluators config to silence logs from incoming calls (1363475)
- 15:13 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy for orchestrator (1368639) (duration: 02m 05s)
- 15:12 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
- 15:11 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
- 15:11 urbanecm@deploy2003: Finished scap sync-world: Backport for Revert^2 "Growth: enable home beta on pilot and innovation wikis" (T443176) (duration: 12m 51s)
- 15:11 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
- 15:11 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy for orchestrator (1368639)
- 15:07 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy for evaluators (1368638) (duration: 03m 05s)
- 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
- 15:05 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
- 15:05 urbanecm@deploy2003: urbanecm: Continuing with deployment
- 15:04 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
- 15:04 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy for evaluators (1368638)
- 15:02 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
- 15:01 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
- 15:00 urbanecm@deploy2003: urbanecm: Backport for Revert^2 "Growth: enable home beta on pilot and innovation wikis" (T443176) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:58 urbanecm@deploy2003: Started scap sync-world: Backport for Revert^2 "Growth: enable home beta on pilot and innovation wikis" (T443176)
- 14:57 dzahn@dns1004: END - running authdns-update
- 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for Hooks: Do not disable exempt beta features when auto-enroll is on (T444361) (1.48.0-wmf.4), Handle multiple updates in case creation atomic section (T444117) (1.48.0-wmf.3, 1.48.0-wmf.4) (duration: 14m 34s)
- 14:55 dzahn@dns1004: START - running authdns-update
- 14:51 urbanecm@deploy2003: stran, urbanecm: Continuing with deployment
- 14:45 urbanecm@deploy2003: stran, urbanecm: Backport for Hooks: Do not disable exempt beta features when auto-enroll is on (T444361) (1.48.0-wmf.4), Handle multiple updates in case creation atomic section (T444117) (1.48.0-wmf.3, 1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:42 urbanecm@deploy2003: Started scap sync-world: Backport for Hooks: Do not disable exempt beta features when auto-enroll is on (T444361) (1.48.0-wmf.4), Handle multiple updates in case creation atomic section (T444117) (1.48.0-wmf.3, 1.48.0-wmf.4)
- 14:31 stran@deploy2003: Finished scap sync-world: Backport for Enable Special:SuggestedInvestigations on ptwiki (T442582) (duration: 09m 38s)
- 14:25 stran@deploy2003: stran: Continuing with deployment
- 14:24 stran@deploy2003: stran: Backport for Enable Special:SuggestedInvestigations on ptwiki (T442582) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:21 stran@deploy2003: Started scap sync-world: Backport for Enable Special:SuggestedInvestigations on ptwiki (T442582)
- 14:19 Tran: Deployed patch for T442582, T443646
- 13:59 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
- 13:59 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
- 13:54 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 13:53 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 13:52 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 13:51 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 13:14 dreamyjazz@deploy2003: Finished scap sync-world: Backport for hCaptcha: Stop collecting risk scores for IP block notices (T444473) (duration: 09m 24s)
- 13:07 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
- 13:06 dreamyjazz@deploy2003: dreamyjazz: Backport for hCaptcha: Stop collecting risk scores for IP block notices (T444473) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 13:04 dreamyjazz@deploy2003: Started scap sync-world: Backport for hCaptcha: Stop collecting risk scores for IP block notices (T444473)
- 13:03 mvolz@deploy2003: Finished scap deploy-service citoid: PDF fallback (duration: 21m 04s)
- 12:47 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
- 12:46 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
- 12:42 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
- 12:42 mvolz@deploy2003: Started scap deploy-service citoid: PDF fallback
- 12:26 dreamyjazz@deploy2003: Finished scap sync-world: Backport for hCaptcha: Drop CommunityRequests exemption (T426897) (duration: 11m 00s)
- 12:20 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
- 12:17 dreamyjazz@deploy2003: dreamyjazz: Backport for hCaptcha: Drop CommunityRequests exemption (T426897) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 12:15 dreamyjazz@deploy2003: Started scap sync-world: Backport for hCaptcha: Drop CommunityRequests exemption (T426897)
- 12:12 dreamyjazz@deploy2003: Finished scap sync-world: Backport for hCaptcha: Require hCaptcha for all API edits on testwiki (T444470) (duration: 11m 30s)
- 12:06 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
- 12:04 dreamyjazz@deploy2003: dreamyjazz: Backport for hCaptcha: Require hCaptcha for all API edits on testwiki (T444470) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 12:01 dreamyjazz@deploy2003: Started scap sync-world: Backport for hCaptcha: Require hCaptcha for all API edits on testwiki (T444470)
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:36 zabe@deploy2003: Finished scap sync-world: Backport for Add script to remove own user page watches for users without edits (T442779) (1.48.0-wmf.4) (duration: 11m 05s)
- 11:30 zabe@deploy2003: zabe: Continuing with deployment
- 11:29 zabe@deploy2003: zabe: Backport for Add script to remove own user page watches for users without edits (T442779) (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:25 zabe@deploy2003: Started scap sync-world: Backport for Add script to remove own user page watches for users without edits (T442779) (1.48.0-wmf.4)
- 11:01 kevinbazira@deploy2003: helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
- 11:00 kevinbazira@deploy2003: helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' .
- 10:56 kevinbazira@deploy2003: helmfile [ml-staging-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' .
- 10:52 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging CDobbins out of all services on: 2439 hosts
- 10:43 dpogorzelski@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
- 10:42 dpogorzelski@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
- 10:41 ammarpad@deploy2003: mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki mediawikiwiki 'Language Converter' 'Language converter' Ammarpad --reason 'Requested at T444079'
- 10:40 dpogorzelski@deploy2003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'.
- 10:38 dpogorzelski@deploy2003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'.
- 10:36 dpogorzelski@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'.
- 10:35 dpogorzelski@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'.
- 10:30 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
- 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
- 10:03 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 09:58 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 09:40 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
- 09:19 aklapper@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.4 refs T442222
- 09:10 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
- 09:09 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
- 08:59 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
- 08:58 jelto@cumin1004: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
- 08:47 jelto@cumin1004: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
- 08:31 wmde-fisch@deploy2003: Finished scap sync-world: Backport for Drop TemplateData EventStreams/EventLogging config (T258917) (duration: 11m 08s)
- 08:25 wmde-fisch@deploy2003: wmde-fisch, reedy: Continuing with deployment
- 08:24 wmde-fisch@deploy2003: wmde-fisch, reedy: Backport for Drop TemplateData EventStreams/EventLogging config (T258917) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:20 wmde-fisch@deploy2003: Started scap sync-world: Backport for Drop TemplateData EventStreams/EventLogging config (T258917)
- 08:15 wmde-fisch@deploy2003: Finished scap sync-world: Backport for Enable Citoid PDF fallback on testwiki (T443469) (duration: 10m 59s)
- 08:09 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
- 08:07 wmde-fisch@deploy2003: wmde-fisch: Backport for Enable Citoid PDF fallback on testwiki (T443469) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:04 wmde-fisch@deploy2003: Started scap sync-world: Backport for Enable Citoid PDF fallback on testwiki (T443469)
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
2026-11-10
- 23:52 dreamyjazz@deploy2003: Finished scap sync-world: Backport for Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' (T444367) (duration: 10m 02s)
- 23:46 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
- 23:45 dreamyjazz@deploy2003: dreamyjazz: Backport for Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' (T444367) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 23:42 dreamyjazz@deploy2003: Started scap sync-world: Backport for Set $wgGlobalBlockingCentralWikiContentLanguage to 'en' (T444367)
- 23:38 zabe@deploy2003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for wikidatawiki (T439753) (duration: 10m 18s)
- 23:32 zabe@deploy2003: zabe: Continuing with deployment
- 23:30 zabe@deploy2003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for wikidatawiki (T439753) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 23:28 zabe@deploy2003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for wikidatawiki (T439753)
- 23:18 cooltey@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
- 23:17 cooltey@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
- 23:17 cooltey@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
- 23:17 cooltey@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
- 23:16 cooltey@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
- 23:16 cooltey@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
- 23:03 inflatador: bking@cumin2003 `curl -XDELETE "https://search.svc.codfw.wmnet:9242/enwiki_general_1794311678" T444363
- 22:26 cjming@deploy2003: Finished scap sync-world: Backport for Force absent request headers to null (T444205) (1.48.0-wmf.3, 1.48.0-wmf.4) (duration: 09m 03s)
- 22:20 cjming@deploy2003: cjming: Continuing with deployment
- 22:19 cjming@deploy2003: cjming: Backport for Force absent request headers to null (T444205) (1.48.0-wmf.3, 1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 22:17 cjming@deploy2003: Started scap sync-world: Backport for Force absent request headers to null (T444205) (1.48.0-wmf.3, 1.48.0-wmf.4)
- 21:37 tchin@deploy2003: Finished deploy [analytics/refinery@23a3975]: Regular analytics weekly train [analytics/refinery@23a39754] (duration: 10m 33s)
- 21:26 tchin@deploy2003: Started deploy [analytics/refinery@23a3975]: Regular analytics weekly train [analytics/refinery@23a39754]
- 21:26 tchin@deploy2003: Finished deploy [analytics/refinery@23a3975] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@23a39754] (duration: 00m 45s)
- 21:25 jsn@deploy2003: Finished scap sync-world: Backport for Enable PersonalDashboard on Home pilot wikis (T444355), Enable DBSC report-only mode on all Wikimedia wikis (T393212), VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki (T442767), Implement multiple proposals for Wikimedia Outreach (T444368) (duration: 16m 5
- 21:25 tchin@deploy2003: Started deploy [analytics/refinery@23a3975] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@23a39754]
- 21:19 jsn@deploy2003: jsn, sbassett, codenamenoreste, kemayo: Continuing with deployment
- 21:11 jsn@deploy2003: jsn, sbassett, codenamenoreste, kemayo: Backport for Enable PersonalDashboard on Home pilot wikis (T444355), Enable DBSC report-only mode on all Wikimedia wikis (T393212), VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki (T442767), Implement multiple proposals for Wikimedia Outreach (T444368) sy
- 21:09 jsn@deploy2003: Started scap sync-world: Backport for Enable PersonalDashboard on Home pilot wikis (T444355), Enable DBSC report-only mode on all Wikimedia wikis (T393212), VisualEditor: Precompute edit suggestion counts on 100% of enwiki, jawiki, arwiki (T442767), Implement multiple proposals for Wikimedia Outreach (T444368)
- 20:11 kemayo@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
- 20:10 kemayo@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
- 20:09 kemayo@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
- 20:08 kemayo@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
- 20:08 kemayo@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
- 20:08 kemayo@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
- 20:01 kemayo@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
- 20:01 kemayo@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
- 19:58 kemayo@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
- 19:57 kemayo@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
- 19:57 kemayo@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
- 19:56 kemayo@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
- 19:52 kemayo@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
- 19:52 kemayo@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
- 18:44 tchin@deploy2003: Finished scap sync-world: Backport for EventStreamConfig: Refine app_base as medium job (T444095) (duration: 13m 41s)
- 18:38 tchin@deploy2003: tchin, aqu: Continuing with deployment
- 18:32 tchin@deploy2003: tchin, aqu: Backport for EventStreamConfig: Refine app_base as medium job (T444095) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 18:30 tchin@deploy2003: Started scap sync-world: Backport for EventStreamConfig: Refine app_base as medium job (T444095)
- 17:31 urbanecm@deploy2003: Finished scap sync-world: Backport for beta: Enable PersonalDashboard on all wikis (T444353), [Growth] beta: Enable Special:Home as a beta feature on all beta wikis (T443176) (duration: 11m 08s)
- 17:25 urbanecm@deploy2003: urbanecm: Continuing with deployment
- 17:25 urbanecm@deploy2003: urbanecm: Backport for beta: Enable PersonalDashboard on all wikis (T444353), [Growth] beta: Enable Special:Home as a beta feature on all beta wikis (T443176) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 17:20 urbanecm@deploy2003: Started scap sync-world: Backport for beta: Enable PersonalDashboard on all wikis (T444353), [Growth] beta: Enable Special:Home as a beta feature on all beta wikis (T443176)
- 16:35 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns3.*
- 16:33 sukhe@dns1004: END - running authdns-update
- 16:31 sukhe@dns1004: START - running authdns-update
- 16:31 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns3.*,service=authdns-update
- 16:22 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
- 16:19 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
- 16:17 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
- 16:13 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
- 16:11 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: network upgrade, T440349]
- 16:11 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool esams [reason: network upgrade, T440349]
- 15:30 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
- 15:29 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
- 15:24 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
- 15:23 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
- 14:40 Lucas_WMDE: UTC afternoon backport+config window done
- 14:40 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis (T440359) (duration: 14m 22s)
- 14:34 lucaswerkmeister-wmde@deploy2003: lucaswerkmeister-wmde, daimona: Continuing with deployment
- 14:32 lucaswerkmeister-wmde@deploy2003: lucaswerkmeister-wmde, daimona: Backport for CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis (T440359) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:25 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for CommonSettings: set $wgCampaignEventsArticleQualitySupportedWikis (T440359)
- 14:20 dreamyjazz@deploy2003: Finished scap sync-world: Backport for Follow-ups for CheckUser AbuseFilter variables (T234155) (1.48.0-wmf.4), Enable CheckUser AbuseFilter variables on testwiki (T234155) (duration: 13m 37s)
- 14:14 dreamyjazz@deploy2003: dreamyjazz: Continuing with deployment
- 14:11 dreamyjazz@deploy2003: dreamyjazz: Backport for Follow-ups for CheckUser AbuseFilter variables (T234155) (1.48.0-wmf.4), Enable CheckUser AbuseFilter variables on testwiki (T234155) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:06 dreamyjazz@deploy2003: Started scap sync-world: Backport for Follow-ups for CheckUser AbuseFilter variables (T234155) (1.48.0-wmf.4), Enable CheckUser AbuseFilter variables on testwiki (T234155)
- 13:46 ladsgroup@deploy2003: Finished scap sync-world: Backport for DASH: Add "auto" resolution as default and still show other resolutions (T443728) (1.48.0-wmf.4, 1.48.0-wmf.3) (duration: 18m 18s)
- 13:40 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 13:32 ladsgroup@deploy2003: ladsgroup: Backport for DASH: Add "auto" resolution as default and still show other resolutions (T443728) (1.48.0-wmf.4, 1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 13:28 ladsgroup@deploy2003: Started scap sync-world: Backport for DASH: Add "auto" resolution as default and still show other resolutions (T443728) (1.48.0-wmf.4, 1.48.0-wmf.3)
- 12:58 kharlan@deploy2003: Finished scap sync-world: Backport for AbuseReview: Delay the link spam model check job (T442170) (1.48.0-wmf.3) (duration: 12m 21s)
- 12:52 kharlan@deploy2003: kharlan: Continuing with deployment
- 12:50 kharlan@deploy2003: kharlan: Backport for AbuseReview: Delay the link spam model check job (T442170) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 12:46 kharlan@deploy2003: Started scap sync-world: Backport for AbuseReview: Delay the link spam model check job (T442170) (1.48.0-wmf.3)
- 12:05 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 12:05 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 12:05 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 12:05 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 12:04 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 12:04 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 12:03 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp3074.mgmt.esams.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:55 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 11:55 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 11:49 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host cp3074.mgmt.esams.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 11:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 11:42 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new cr1-esams - ayounsi@cumin1004"
- 11:41 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: new cr1-esams - ayounsi@cumin1004"
- 11:35 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
- 10:57 ayounsi@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
- 10:54 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
- 10:37 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-bw27-esams,asw1-bw27-esams IPv6,asw1-bw27-esams.mgmt with reason: Switch maintenance
- 10:35 ayounsi@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 39 hosts with reason: Switch maintenance
- 09:28 aklapper@deploy2003: rebuilt and synchronized wikiversions files: group0 to 1.48.0-wmf.4 refs T442222
- 09:22 XioNoX: set graceful shutdown on cr1-esams transit/ix bgp - T440349
- 09:02 matthiasmullie: UTC morning backport window done
- 09:01 mlitn@deploy2003: Finished scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3) (duration: 13m 06s)
- 08:55 mlitn@deploy2003: mlitn: Continuing with deployment
- 08:55 mlitn@deploy2003: mlitn: Backport for Squashed diff to master (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:48 mlitn@deploy2003: Started scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3)
- 08:44 chlod@deploy2003: Finished scap sync-world: Backport for tcywiki: revert 10 anniversary logo (T434176), bjnwiktionary: update logos (T444085) (duration: 15m 43s)
- 08:43 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: network upgrade, T440349]
- 08:43 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: network upgrade, T440349]
- 08:42 ayounsi@cumin1004: END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool esams [reason: network upgrade, T440349]
- 08:42 ayounsi@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool esams [reason: network upgrade, T440349]
- 08:41 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 08:41 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 08:36 chlod@deploy2003: chlod, anzx: Continuing with deployment
- 08:35 chlod@deploy2003: chlod, anzx: Backport for tcywiki: revert 10 anniversary logo (T434176), bjnwiktionary: update logos (T444085) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:28 chlod@deploy2003: Started scap sync-world: Backport for tcywiki: revert 10 anniversary logo (T434176), bjnwiktionary: update logos (T444085)
- 08:15 jdrewniak@deploy2003: Finished scap sync-world: Backport for Bumping portals to master (T128546) (duration: 18m 26s)
- 08:04 jdrewniak@deploy2003: jdrewniak: Continuing with deployment
- 08:03 jdrewniak@deploy2003: jdrewniak: Backport for Bumping portals to master (T128546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 07:56 jdrewniak@deploy2003: Started scap sync-world: Backport for Bumping portals to master (T128546)
- 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.24 (duration: 02m 26s)
- 04:42 mwpresync@deploy2003: Finished scap sync-world: testwikis to 1.48.0-wmf.4 refs T442222 (duration: 39m 18s)
- 04:03 mwpresync@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.4 refs T442222
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 35s)
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 01:06 zabe: run populateLuAttachmentMethod.php # T442322
- 00:54 zabe@deploy2003: Finished scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for enwiki (T439753) (duration: 11m 47s)
- 00:47 zabe@deploy2003: zabe: Continuing with deployment
- 00:47 zabe@deploy2003: zabe: Backport for Move reads/writes of wbc_entity_usage to x1 for enwiki (T439753) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 00:42 zabe@deploy2003: Started scap sync-world: Backport for Move reads/writes of wbc_entity_usage to x1 for enwiki (T439753)
- 00:25 zabe@deploy2003: Finished scap sync-world: Backport for Disable automatic user page watching on loginwiki (T442779) (duration: 18m 24s)
- 00:16 zabe@deploy2003: zabe: Continuing with deployment
- 00:15 zabe@deploy2003: zabe: Backport for Disable automatic user page watching on loginwiki (T442779) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 00:07 zabe@deploy2003: Started scap sync-world: Backport for Disable automatic user page watching on loginwiki (T442779)
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
2026-11-09
- 23:11 maryum: Deployed security fix for T443539, T440392, T435882, T442600
- 21:23 akhatun@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
- 21:22 akhatun@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
- 21:19 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
- 21:19 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
- 21:18 egardner@deploy2003: Finished scap sync-world: Backport for Enable the ShareCard feature on all Wikipedias (Minerva only) (T439162) (duration: 12m 48s)
- 21:17 akhatun@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
- 21:16 akhatun@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
- 21:12 egardner@deploy2003: egardner: Continuing with deployment
- 21:11 egardner@deploy2003: egardner: Backport for Enable the ShareCard feature on all Wikipedias (Minerva only) (T439162) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:05 egardner@deploy2003: Started scap sync-world: Backport for Enable the ShareCard feature on all Wikipedias (Minerva only) (T439162)
- 20:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 20:19 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 19:44 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 19:44 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 19:01 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 19:00 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 18:06 cgoubert@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
- 18:06 cgoubert@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
- 18:06 cgoubert@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 18:06 cgoubert@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 17:54 cgoubert@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 17:54 cgoubert@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 17:49 cgoubert@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
- 17:48 cgoubert@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
- 17:45 trueg@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
- 17:40 ebernhardson: T444015 start cirrussearch full-cluster reindex in codfw and cloudelastic
- 17:36 trueg@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
- 17:35 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 17:28 ebernhardson: T444015 start full-cluster reindex in eqiad
- 17:25 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 17:23 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 17:08 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 16:45 klausman@deploy2003: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
- 16:43 klausman@deploy2003: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
- 16:15 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
- 16:10 klausman@deploy2003: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
- 16:08 klausman@deploy2003: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
- 16:07 otto@deploy2003: Finished scap sync-world: Backport for [Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta (T434143) (duration: 11m 59s)
- 16:06 trueg@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
- 16:05 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
- 16:03 trueg@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
- 16:02 klausman@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
- 16:01 klausman@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
- 16:00 otto@deploy2003: tchin, otto: Continuing with deployment
- 15:59 otto@deploy2003: tchin, otto: Backport for [Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta (T434143) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:55 otto@deploy2003: Started scap sync-world: Backport for [Beta EventBus] Add revision_tags_change and log_tags_change dev0 streams, enable in beta (T434143)
- 15:48 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply
- 15:48 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-airflow-dumps: apply
- 15:35 ladsgroup@deploy2003: Finished scap sync-world: Backport for Enable MPEG-DASH everywhere (T373546) (duration: 08m 48s)
- 15:33 btullis@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:32 btullis@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:29 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 15:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 267201
- 15:28 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 267201
- 15:28 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
- 15:28 ladsgroup@deploy2003: ladsgroup: Backport for Enable MPEG-DASH everywhere (T373546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:28 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
- 15:28 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
- 15:27 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
- 15:27 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
- 15:27 klausman@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
- 15:27 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
- 15:26 ladsgroup@deploy2003: Started scap sync-world: Backport for Enable MPEG-DASH everywhere (T373546)
- 15:23 otto@deploy2003: Finished scap sync-world: Backport for Prefetch edit_suggestions_counts with EventBus LinkedArtifacts (T443442) (duration: 08m 40s)
- 15:22 klausman@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
- 15:17 otto@deploy2003: otto: Continuing with deployment
- 15:16 otto@deploy2003: otto: Backport for Prefetch edit_suggestions_counts with EventBus LinkedArtifacts (T443442) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:14 otto@deploy2003: Started scap sync-world: Backport for Prefetch edit_suggestions_counts with EventBus LinkedArtifacts (T443442)
- 15:11 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
- 15:10 otto@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
- 15:10 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
- 15:08 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
- 15:07 otto@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
- 15:07 ottomata: deploying changeprop-jobqueue to pick up new queue for T443442
- 15:07 otto@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
- 14:59 krinkle@deploy2003: Finished scap sync-world: Backport for Remove unused "TitleBlacklist-cache" channel (T85428 T441284) (duration: 11m 20s)
- 14:53 krinkle@deploy2003: krinkle: Continuing with deployment
- 14:52 krinkle@deploy2003: krinkle: Backport for Remove unused "TitleBlacklist-cache" channel (T85428 T441284) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:48 krinkle@deploy2003: Started scap sync-world: Backport for Remove unused "TitleBlacklist-cache" channel (T85428 T441284)
- 14:47 krinkle@deploy2003: sync-world aborted: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3) (duration: 00m 41s)
- 14:46 krinkle@deploy2003: Started scap sync-world: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3)
- 14:44 krinkle@deploy2003: Finished scap sync-world: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3), Always return bundled from getLexers, remove "fetch_lexers" shellbox call (T292663) (1.48.0-wmf.3) (duration: 13m 51s)
- 14:39 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1049.eqiad.wmnet
- 14:39 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1049.eqiad.wmnet
- 14:38 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2012.codfw.wmnet
- 14:38 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2012.codfw.wmnet
- 14:38 krinkle@deploy2003: krinkle: Continuing with deployment
- 14:34 krinkle@deploy2003: krinkle: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3), Always return bundled from getLexers, remove "fetch_lexers" shellbox call (T292663) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:30 krinkle@deploy2003: Started scap sync-world: Backport for RawAction: Enable full CDN caching for canonical CSS/JSON/JS URLs (T279120) (1.48.0-wmf.3), Always return bundled from getLexers, remove "fetch_lexers" shellbox call (T292663) (1.48.0-wmf.3)
- 14:22 jdrewniak@deploy2003: Finished scap sync-world: Backport for Enable ReadingLists custom lists on all Wikipedias. (T441372) (duration: 18m 18s)
- 14:16 jdrewniak@deploy2003: jdrewniak: Continuing with deployment
- 14:11 jdrewniak@deploy2003: jdrewniak: Backport for Enable ReadingLists custom lists on all Wikipedias. (T441372) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:08 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet
- 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet
- 14:04 jdrewniak@deploy2003: Started scap sync-world: Backport for Enable ReadingLists custom lists on all Wikipedias. (T441372)
- 14:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1049.eqiad.wmnet
- 14:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2012.codfw.wmnet
- 12:19 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet with OS bookworm
- 12:17 Emperor: VACUUM 27G container database on thanos-be2008 T443888
- 12:11 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet with OS bookworm
- 12:08 Emperor: stop swift on thanos-be2006 to VACUUM 27G container DB T443888
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:59 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1049.eqiad.wmnet with reason: host reimage
- 11:55 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1049.eqiad.wmnet with reason: host reimage
- 11:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2012.codfw.wmnet with reason: host reimage
- 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet (remove config only)
- 11:52 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1049.eqiad.wmnet (remove config only)
- 11:48 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2012.codfw.wmnet with reason: host reimage
- 11:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet (remove config only)
- 11:45 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2012.codfw.wmnet (remove config only)
- 11:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1049.eqiad.wmnet (push config only)
- 11:34 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1049.eqiad.wmnet (push config only)
- 11:34 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1049.eqiad.wmnet with OS bookworm
- 11:33 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['dse-k8s-worker1049.eqiad.wmnet']
- 11:31 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2012.codfw.wmnet (push config only)
- 11:31 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2012.codfw.wmnet (push config only)
- 11:30 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2012.codfw.wmnet with OS bookworm
- 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from dse-k8s-wdqs-test2001 to dse-k8s-worker2012
- 11:26 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dse-k8s-worker1049.eqiad.wmnet']
- 11:26 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2012
- 11:25 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2012
- 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2012 on all recursors
- 11:25 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2012 on all recursors
- 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test2001 to dse-k8s-worker2012 - btullis@cumin1004"
- 11:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from dse-k8s-wdqs-test1001 to dse-k8s-worker1049
- 11:25 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test2001 to dse-k8s-worker2012 - btullis@cumin1004"
- 11:24 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1049
- 11:24 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1049
- 11:24 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1049 on all recursors
- 11:24 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1049 on all recursors
- 11:24 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 11:24 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test1001 to dse-k8s-worker1049 - btullis@cumin1004"
- 11:22 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming dse-k8s-wdqs-test1001 to dse-k8s-worker1049 - btullis@cumin1004"
- 11:21 btullis@cumin1004: START - Cookbook sre.dns.netbox
- 11:21 btullis@cumin1004: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
- 11:16 btullis@cumin1004: START - Cookbook sre.dns.netbox
- 11:10 btullis@cumin1004: START - Cookbook sre.dns.netbox
- 11:10 btullis@cumin1004: START - Cookbook sre.hosts.rename from dse-k8s-wdqs-test2001 to dse-k8s-worker2012
- 11:10 btullis@cumin1004: START - Cookbook sre.hosts.rename from dse-k8s-wdqs-test1001 to dse-k8s-worker1049
- 10:59 urbanecm@deploy2003: Finished scap sync-world: Backport for GrowthExperiments: Remove settings that nothing reads (T443648) (duration: 12m 54s)
- 10:58 btullis@cumin1004: conftool action : set/pooled=inactive; selector: name=dse-k8s-wdqs-test2001.codfw.wmnet
- 10:58 btullis@cumin1004: conftool action : set/pooled=inactive; selector: name=dse-k8s-wdqs-test1001.eqiad.wmnet
- 10:53 urbanecm@deploy2003: urbanecm: Continuing with deployment
- 10:53 urbanecm@deploy2003: urbanecm: Backport for GrowthExperiments: Remove settings that nothing reads (T443648) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 10:46 urbanecm@deploy2003: Started scap sync-world: Backport for GrowthExperiments: Remove settings that nothing reads (T443648)
- 10:42 Tchanders: Updated WikimediaAntiAbuse AbuseReview private code
- 10:41 tchanders@deploy2003: Finished scap sync-world: Backport for Take Special:AbuseReview vandalism queue out of alpha on frwiki (T443815) (duration: 15m 29s)
- 10:33 tchanders@deploy2003: tchanders: Continuing with deployment
- 10:32 tchanders@deploy2003: tchanders: Backport for Take Special:AbuseReview vandalism queue out of alpha on frwiki (T443815) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 10:25 tchanders@deploy2003: Started scap sync-world: Backport for Take Special:AbuseReview vandalism queue out of alpha on frwiki (T443815)
- 10:01 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 10:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 09:21 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 09:21 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 09:20 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 09:19 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 09:10 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 09:10 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 08:41 mlitn@deploy2003: Started scap sync-world: Backport for Enable en on bdwikimedia (T444031)
- 08:39 mlitn@deploy2003: Finished scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3) (duration: 36m 37s)
- 08:24 mlitn@deploy2003: mlitn: Continuing with deployment
- 08:23 mlitn@deploy2003: mlitn: Backport for Squashed diff to master (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:02 mlitn@deploy2003: Started scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3)
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 01s)
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
2026-11-08
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
- 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
2026-11-07
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 45s)
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 01:19 jasmine@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
- 01:18 jasmine@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
- 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/sophroid: apply
- 00:58 rzl@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/sophroid: apply
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
2026-11-06
- 18:30 aokoth@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply
- 18:29 aokoth@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply
- 18:29 aokoth@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/services/miscweb: apply
- 18:28 aokoth@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/services/miscweb: apply
- 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{kubestagemaster2005.codfw.wmnet} and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
- 17:10 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
- 17:10 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
- 17:05 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
- 17:05 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
- 17:05 jayme@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{kubestagemaster2005.codfw.wmnet} and (A:wikikube-staging-master-codfw or A:wikikube-staging-worker-codfw)
- 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
- 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
- 16:43 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
- 16:43 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
- 16:38 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
- 16:37 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
- 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=0) Upgrade kubelet of staging-codfw to 1.34.13
- 16:32 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2005.codfw.wmnet
- 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2005.codfw.wmnet
- 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2005.codfw.wmnet
- 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2005.codfw.wmnet
- 16:31 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2004.codfw.wmnet
- 16:31 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2004.codfw.wmnet
- 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2004.codfw.wmnet
- 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2004.codfw.wmnet
- 16:30 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster2003.codfw.wmnet
- 16:30 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster2003.codfw.wmnet
- 16:29 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster2003.codfw.wmnet
- 16:29 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster2003.codfw.wmnet
- 16:29 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.13
- 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.13
- 16:18 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
- 16:18 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
- 16:09 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
- 16:09 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
- 16:08 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
- 16:08 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=97) Upgrade apiservers of staging-codfw to 1.34.13
- 16:07 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
- 16:07 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
- 16:07 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
- 16:07 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.13
- 16:07 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
- 16:03 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
- 16:03 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.13
- 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet
- 15:41 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker2013.codfw.wmnet
- 15:41 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker2013.codfw.wmnet
- 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
- 15:41 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
- 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1048.eqiad.wmnet
- 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1046.eqiad.wmnet
- 15:40 btullis@puppetserver1001: conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
- 15:40 btullis@puppetserver1001: conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1047.eqiad.wmnet
- 15:38 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet
- 15:31 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1048.eqiad.wmnet
- 15:28 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet
- 15:21 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1047.eqiad.wmnet
- 15:11 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
- 15:11 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
- 15:09 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet
- 15:01 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1046.eqiad.wmnet
- 15:00 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 15:00 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 14:30 jgiannelos@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 14:30 jgiannelos@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 14:07 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
- 13:48 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
- 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
- 13:44 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker2013.codfw.wmnet with reason: host reimage
- 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
- 13:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (remove config only)
- 13:42 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (remove config only)
- 13:37 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
- 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
- 13:25 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker2013.codfw.wmnet (push config only)
- 13:25 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker2013.codfw.wmnet (push config only)
- 13:25 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker2013.codfw.wmnet with OS bookworm
- 13:22 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
- 13:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
- 13:18 moritzm: installing linux 6.12.115 on Trixie hosts
- 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1048.eqiad.wmnet with reason: host reimage
- 13:15 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1047.eqiad.wmnet with reason: host reimage
- 13:14 btullis@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dse-k8s-worker1046.eqiad.wmnet with reason: host reimage
- 13:13 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
- 13:13 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (remove config only)
- 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
- 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (remove config only)
- 13:12 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
- 13:12 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (remove config only)
- 13:08 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
- 13:07 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
- 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 13:02 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
- 13:02 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
- 13:00 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 13:00 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1048.eqiad.wmnet (push config only)
- 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1048.eqiad.wmnet (push config only)
- 12:55 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1047.eqiad.wmnet (push config only)
- 12:55 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1047.eqiad.wmnet (push config only)
- 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1048.eqiad.wmnet with OS bookworm
- 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1047.eqiad.wmnet with OS bookworm
- 12:54 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dse-k8s-worker1046.eqiad.wmnet (push config only)
- 12:54 btullis@cumin1004: START - Cookbook sre.hosts.dhcp for host dse-k8s-worker1046.eqiad.wmnet (push config only)
- 12:54 btullis@cumin1004: START - Cookbook sre.hosts.reimage for host dse-k8s-worker1046.eqiad.wmnet with OS bookworm
- 12:54 btullis@cumin1004: START - Cookbook sre.hosts.provision for host dse-k8s-worker2013.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 12:47 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1031 to dse-k8s-worker1047
- 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1047
- 12:46 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs2025 to dse-k8s-worker2013
- 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker2013
- 12:44 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker2013
- 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker2013 on all recursors
- 12:44 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker2013 on all recursors
- 12:44 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 12:42 btullis@cumin1004: START - Cookbook sre.dns.netbox
- 12:41 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1047
- 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1047 on all recursors
- 12:41 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1047 on all recursors
- 12:41 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1032 to dse-k8s-worker1048
- 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1048
- 12:39 btullis@cumin1004: START - Cookbook sre.dns.netbox
- 12:38 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1048
- 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1048 on all recursors
- 12:38 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1048 on all recursors
- 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 12:38 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004"
- 12:38 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1032 to dse-k8s-worker1048 - btullis@cumin1004"
- 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from wdqs1030 to dse-k8s-worker1046
- 12:34 btullis@cumin1004: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dse-k8s-worker1046
- 12:33 btullis@cumin1004: START - Cookbook sre.dns.netbox
- 12:33 btullis@cumin1004: START - Cookbook sre.network.configure-switch-interfaces for host dse-k8s-worker1046
- 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) dse-k8s-worker1046 on all recursors
- 12:33 btullis@cumin1004: START - Cookbook sre.dns.wipe-cache dse-k8s-worker1046 on all recursors
- 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004"
- 12:33 btullis@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming wdqs1030 to dse-k8s-worker1046 - btullis@cumin1004"
- 12:31 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs2025 to dse-k8s-worker2013
- 12:30 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1032 to dse-k8s-worker1048
- 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1031 to dse-k8s-worker1047
- 12:29 btullis@cumin1004: START - Cookbook sre.dns.netbox
- 12:29 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
- 12:29 btullis@cumin1004: START - Cookbook sre.hosts.rename from wdqs1030 to dse-k8s-worker1046
- 12:12 moritzm: imported openjdk-21 21.0.13+9-1~deb13u2~wmf12u1 to component/jdk21 for bookworm-wikimedia
- 12:03 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
- 12:03 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
- 12:02 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1032.eqiad.wmnet
- 11:58 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1032.eqiad.wmnet
- 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'.
- 11:52 cgoubert@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'.
- 11:51 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
- 11:49 cgoubert@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
- 11:48 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet
- 11:47 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet
- 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet
- 11:43 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet
- 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet
- 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet
- 11:39 fabfur: installing haproxykafka 0.4.0 on cp7001 for testing (T443632) (T443640)
- 11:36 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s)
- 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet
- 11:32 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet
- 11:31 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
- 11:31 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127]
- 11:30 a-pizzata@deploy2003: Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s)
- 11:30 a-pizzata@deploy2003: Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127]
- 11:19 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet
- 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet
- 11:18 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet
- 11:09 btullis@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet
- 11:09 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet
- 11:08 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet
- 11:00 btullis@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet
- 10:49 jiji@cumin1004: END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance
- 10:44 jiji@cumin1004: START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance
- 10:42 effie: depool kartotherian from eqiad (ie maps) - T443438
- 10:34 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix T443357 (duration: 03m 39s)
- 10:33 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
- 10:31 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
- 10:31 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix T443357
- 10:29 marostegui: Limit icinga db user connections to 10 in ms* and pc* T442995
- 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
- 10:16 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
- 10:16 ayounsi@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
- 10:11 fabfur@cumin1004: conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox
- 10:08 ayounsi@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
- 09:04 mvernon@cumin1004: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe
- 08:57 jelto@deploy2003: Finished scap deploy-service miscweb-aux: Deploy to fix T443826 (duration: 04m 11s)
- 08:56 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb
- 08:53 jelto@deploy2003: scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb
- 08:53 jelto@deploy2003: Started scap deploy-service miscweb-aux: Deploy to fix T443826
- 08:52 mvernon@cumin2003: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
- 08:48 mvernon@cumin1004: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe
- 08:47 mvernon@cumin2003: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
- 06:52 XioNoX: push pfw policies - T443852
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
2026-11-05
- 23:30 mutante: codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout.
- 23:18 maryum: Deployed security fix for T443424, T443467, T443036
- 23:10 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
- 23:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json
- 23:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json
- 22:51 sbassett: Deployed security fixes for T439878, T440323, T440657, T440656, T440655
- 22:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json
- 22:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json
- 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json
- 22:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
- 22:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json
- 22:31 egardner@deploy2003: Finished scap sync-world: Backport for ShareCard: Log client errors with the topic "error.sharecard" (T439163) (1.48.0-wmf.3), ShareCard: Show animated GIFs as a still image (T442968) (1.48.0-wmf.3) (duration: 11m 46s)
- 22:24 egardner@deploy2003: egardner: Continuing with deployment
- 22:23 egardner@deploy2003: egardner: Backport for ShareCard: Log client errors with the topic "error.sharecard" (T439163) (1.48.0-wmf.3), ShareCard: Show animated GIFs as a still image (T442968) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 22:22 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
- 22:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json
- 22:22 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
- 22:19 egardner@deploy2003: Started scap sync-world: Backport for ShareCard: Log client errors with the topic "error.sharecard" (T439163) (1.48.0-wmf.3), ShareCard: Show animated GIFs as a still image (T442968) (1.48.0-wmf.3)
- 22:15 egardner@deploy2003: Finished scap sync-world: Backport for Add Creative Commons license icons for the share card (T439162) (duration: 14m 19s)
- 22:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json
- 22:08 egardner@deploy2003: egardner: Continuing with deployment
- 22:07 egardner@deploy2003: egardner: Backport for Add Creative Commons license icons for the share card (T439162) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 22:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json
- 22:01 egardner@deploy2003: Started scap sync-world: Backport for Add Creative Commons license icons for the share card (T439162)
- 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json
- 21:54 mutante: ci / contint1003/contint2003 - if "sudo docker info -f 'Template:.Driver'" says "overlay2" it's the classic way. after restart when it says "overlayfs2" that will be the NEW way! (T440668)
- 21:54 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
- 21:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json
- 21:51 mutante: ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4 (T440668)
- 21:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json
- 21:34 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
- 21:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json
- 21:33 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
- 21:30 topranks: re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add T440859
- 21:28 kemayo@deploy2003: mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix # T430731
- 21:26 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet
- 21:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json
- 21:24 kemayo@deploy2003: Finished scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767), Add new namespace for zh_min_nanwiki (T430731) (duration: 16m 22s)
- 21:19 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet
- 21:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json
- 21:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
- 21:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json
- 21:15 kemayo@deploy2003: pppery, kemayo: Continuing with deployment
- 21:14 kemayo@deploy2003: pppery, kemayo: Backport for VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767), Add new namespace for zh_min_nanwiki (T430731) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:08 kemayo@deploy2003: Started scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767), Add new namespace for zh_min_nanwiki (T430731)
- 21:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json
- 20:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json
- 20:56 James_F: mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # T443295
- 20:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json
- 20:44 topranks: disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former T440859
- 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json
- 20:39 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
- 20:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json
- 20:39 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints
- 20:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json
- 20:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json
- 20:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json
- 20:06 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
- 20:05 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
- 20:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json
- 20:02 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance
- 20:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json
- 19:56 bking@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply
- 19:55 bking@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply
- 19:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json
- 19:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json
- 19:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json
- 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json
- 19:24 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
- 19:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json
- 19:14 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
- 19:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json
- 19:13 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
- 19:12 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
- 19:12 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
- 19:11 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
- 19:10 bd808@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
- 19:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json
- 18:57 dancy@deploy2003: Finished scap sync-world: Testing T411063 (duration: 44m 09s)
- 18:54 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-b3-eqiad
- 18:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97383 and previous config saved to /var/cache/conftool/dbconfig/20261105-185411-ladsgroup.json
- 18:54 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-b3-eqiad
- 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97382 and previous config saved to /var/cache/conftool/dbconfig/20261105-184550-ladsgroup.json
- 18:45 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
- 18:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T441633)', diff saved to https://phabricator.wikimedia.org/P97381 and previous config saved to /var/cache/conftool/dbconfig/20261105-184529-ladsgroup.json
- 18:42 dancy@deploy2003: dancy: Continuing with deployment
- 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97380 and previous config saved to /var/cache/conftool/dbconfig/20261105-183527-ladsgroup.json
- 18:35 dancy@deploy2003: dancy: Testing T411063 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 18:32 bd808@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
- 18:32 bd808@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
- 18:32 bd808@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
- 18:31 bd808@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
- 18:31 bd808@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
- 18:31 bd808@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
- 18:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P97379 and previous config saved to /var/cache/conftool/dbconfig/20261105-182526-ladsgroup.json
- 18:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T441633)', diff saved to https://phabricator.wikimedia.org/P97378 and previous config saved to /var/cache/conftool/dbconfig/20261105-181524-ladsgroup.json
- 18:13 dancy@deploy2003: Started scap sync-world: Testing T411063
- 18:08 ladsgroup@deploy2003: Finished scap sync-world: Push small changes for error pages (duration: 05m 08s)
- 18:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2154 (T441633)', diff saved to https://phabricator.wikimedia.org/P97377 and previous config saved to /var/cache/conftool/dbconfig/20261105-180812-ladsgroup.json
- 18:08 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
- 18:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 18:04 ladsgroup@deploy2003: ladsgroup: Push small changes for error pages synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 18:03 ladsgroup@deploy2003: Started scap sync-world: Push small changes for error pages
- 17:59 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad IPv6,lsw1-b3-eqiad.mgmt with reason: need to test things on switch
- 17:53 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
- 17:52 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/services/miscweb: apply
- 17:52 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
- 17:51 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
- 17:46 hnowlan@deploy1003: helmfile [codfw] DONE helmfile.d/admin 'apply'.
- 17:45 jayme@cumin2003: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parsoidtest1001.eqiad.wmnet
- 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 17:45 jayme@cumin2003: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003"
- 17:45 hnowlan@deploy1003: helmfile [codfw] START helmfile.d/admin 'apply'.
- 17:45 jayme@cumin2003: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: parsoidtest1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin2003"
- 17:43 hnowlan@deploy1003: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
- 17:43 hnowlan@deploy1003: helmfile [eqiad] START helmfile.d/admin 'apply'.
- 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
- 17:40 jayme@cumin2003: START - Cookbook sre.dns.netbox
- 17:40 hnowlan@deploy1003: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
- 17:39 topranks: reboot lsw1-b3-eqiad to downgrade to SR-Linux 26.3.3 T443687
- 17:35 hnowlan@deploy1003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
- 17:35 hnowlan@deploy1003: helmfile [staging] START helmfile.d/services/miscweb: apply
- 17:34 jayme@cumin2003: START - Cookbook sre.hosts.decommission for hosts parsoidtest1001.eqiad.wmnet
- 17:33 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on ssw1-a[1,8]-eqiad with reason: need to test things on switch
- 17:32 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on lsw1-b3-eqiad with reason: need to test things on switch
- 17:31 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on sretest1006.eqiad.wmnet with reason: need to test things on switch
- 17:04 fabfur: installing haproxykafka 0.4.0 on cp7002 for testing (T443632) (T443640)
- 16:48 sukhe@cumin1004: END (FAIL) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=99) for role: aux_k8s::worker@eqiad
- 16:47 sukhe@cumin1004: START - Cookbook sre.loadbalancer.migrate-service-ipip for role: aux_k8s::worker@eqiad
- 16:44 kamila@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
- 16:43 kamila@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
- 16:42 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
- 16:41 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
- 16:40 moritzm: installing krb5 security updates
- 16:40 kamila@deploy2003: Finished scap sync-world: switch mw-parsoid to php 8.5 - T439955 (duration: 03m 03s)
- 16:37 kamila@deploy2003: Started scap sync-world: switch mw-parsoid to php 8.5 - T439955
- 16:35 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
- 16:35 moritzm: installing krb5 security updates
- 16:34 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
- 15:28 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
- 15:28 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool magru [reason: done testing DNS admin cookbook changes, no task ID specified]
- 15:27 sukhe@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
- 15:27 sukhe@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool magru [reason: testing DNS admin cookbook changes, no task ID specified]
- 15:14 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2013.codfw.wmnet} and A:lvs
- 15:13 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2013.codfw.wmnet} and A:lvs
- 15:13 vgutierrez@cumin1004: END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs
- 15:12 vgutierrez@cumin1004: START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on P{lvs2014.codfw.wmnet} and A:lvs
- 15:10 stran@deploy2003: Finished scap sync-world: Backport for Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449), Enable Special:SuggestedInvestigations for nlwiki (T443646) (duration: 12m 28s)
- 15:08 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 15:04 stran@deploy2003: stran, codenamenoreste: Continuing with deployment
- 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 15:02 stran@deploy2003: stran, codenamenoreste: Backport for Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449), Enable Special:SuggestedInvestigations for nlwiki (T443646) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:02 jhancock@cumin2003: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 15:02 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 14:58 stran@deploy2003: Started scap sync-world: Backport for Round 2 on disabling createpage for anonymous users on English Wikibooks (T443449), Enable Special:SuggestedInvestigations for nlwiki (T443646)
- 14:56 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
- 14:56 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
- 14:48 dbrant@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
- 14:48 dbrant@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
- 14:47 dbrant@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
- 14:47 mlitn@deploy2003: Finished scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3) (duration: 12m 38s)
- 14:47 dbrant@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
- 14:46 akhatun@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
- 14:46 akhatun@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
- 14:44 dbrant@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
- 14:44 dbrant@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
- 14:41 mlitn@deploy2003: mlitn: Continuing with deployment
- 14:38 mlitn@deploy2003: mlitn: Backport for Squashed diff to master (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:34 mlitn@deploy2003: Started scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.3)
- 14:32 Tran: Deployed patch for T443598
- 14:17 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
- 14:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 (T441633)', diff saved to https://phabricator.wikimedia.org/P97374 and previous config saved to /var/cache/conftool/dbconfig/20261105-141655-ladsgroup.json
- 14:15 stran@deploy2003: Finished scap sync-world: Backport for Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147) (duration: 11m 34s)
- 14:11 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
- 14:09 stran@deploy2003: stran: Continuing with deployment
- 14:08 stran@deploy2003: stran: Backport for Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97373 and previous config saved to /var/cache/conftool/dbconfig/20261105-140653-ladsgroup.json
- 14:03 stran@deploy2003: Started scap sync-world: Backport for Enable SuggestedInvestigations for nlwiki (hidden), enwikivoyage (T443646 T443147)
- 13:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286', diff saved to https://phabricator.wikimedia.org/P97372 and previous config saved to /var/cache/conftool/dbconfig/20261105-135651-ladsgroup.json
- 13:53 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
- 13:48 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
- 13:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1286 (T441633)', diff saved to https://phabricator.wikimedia.org/P97371 and previous config saved to /var/cache/conftool/dbconfig/20261105-134649-ladsgroup.json
- 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
- 13:45 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
- 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1286 (T441633)', diff saved to https://phabricator.wikimedia.org/P97370 and previous config saved to /var/cache/conftool/dbconfig/20261105-134145-ladsgroup.json
- 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1286.eqiad.wmnet with reason: Maintenance
- 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 (T441633)', diff saved to https://phabricator.wikimedia.org/P97369 and previous config saved to /var/cache/conftool/dbconfig/20261105-134125-ladsgroup.json
- 13:35 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
- 13:35 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
- 13:35 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
- 13:35 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
- 13:35 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
- 13:34 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
- 13:33 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97368 and previous config saved to /var/cache/conftool/dbconfig/20261105-133123-ladsgroup.json
- 13:26 cmooney@cumin1004: START - Cookbook sre.hosts.provision for host sretest1006.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL
- 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281', diff saved to https://phabricator.wikimedia.org/P97367 and previous config saved to /var/cache/conftool/dbconfig/20261105-132122-ladsgroup.json
- 13:20 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
- 13:19 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
- 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
- 13:19 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
- 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 13:19 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
- 13:19 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
- 13:18 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
- 13:14 cmooney@cumin1004: START - Cookbook sre.dns.netbox
- 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1281 (T441633)', diff saved to https://phabricator.wikimedia.org/P97366 and previous config saved to /var/cache/conftool/dbconfig/20261105-131120-ladsgroup.json
- 13:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1281 (T441633)', diff saved to https://phabricator.wikimedia.org/P97365 and previous config saved to /var/cache/conftool/dbconfig/20261105-130607-ladsgroup.json
- 13:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
- 13:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1281.eqiad.wmnet with reason: Maintenance
- 13:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97364 and previous config saved to /var/cache/conftool/dbconfig/20261105-130524-ladsgroup.json
- 12:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97363 and previous config saved to /var/cache/conftool/dbconfig/20261105-125522-ladsgroup.json
- 12:52 kamila@deploy2003: Finished scap sync-world: Backport for Enroll 5% of client sessions in PHP 8.5 (T441689) (duration: 10m 50s)
- 12:46 kamila@deploy2003: kamila: Continuing with deployment
- 12:46 kamila@deploy2003: kamila: Backport for Enroll 5% of client sessions in PHP 8.5 (T441689) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 12:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P97362 and previous config saved to /var/cache/conftool/dbconfig/20261105-124521-ladsgroup.json
- 12:41 kamila@deploy2003: Started scap sync-world: Backport for Enroll 5% of client sessions in PHP 8.5 (T441689)
- 12:37 moritzm: installing alsa-lib security updates
- 12:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97361 and previous config saved to /var/cache/conftool/dbconfig/20261105-123519-ladsgroup.json
- 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
- 12:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97360 and previous config saved to /var/cache/conftool/dbconfig/20261105-122614-ladsgroup.json
- 12:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
- 12:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T441633)', diff saved to https://phabricator.wikimedia.org/P97359 and previous config saved to /var/cache/conftool/dbconfig/20261105-122554-ladsgroup.json
- 12:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97358 and previous config saved to /var/cache/conftool/dbconfig/20261105-121552-ladsgroup.json
- 12:13 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
- 12:11 moritzm: installing alsa-lib security updates
- 12:09 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
- 12:07 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
- 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
- 12:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P97357 and previous config saved to /var/cache/conftool/dbconfig/20261105-120551-ladsgroup.json
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T441633)', diff saved to https://phabricator.wikimedia.org/P97356 and previous config saved to /var/cache/conftool/dbconfig/20261105-115549-ladsgroup.json
- 11:52 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
- 11:52 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
- 11:51 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
- 11:49 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
- 11:49 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
- 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1214 (T441633)', diff saved to https://phabricator.wikimedia.org/P97355 and previous config saved to /var/cache/conftool/dbconfig/20261105-114845-ladsgroup.json
- 11:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
- 11:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T441633)', diff saved to https://phabricator.wikimedia.org/P97354 and previous config saved to /var/cache/conftool/dbconfig/20261105-114825-ladsgroup.json
- 11:47 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1006.eqiad.wmnet with OS trixie
- 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
- 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 11:47 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
- 11:47 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
- 11:47 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
- 11:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97353 and previous config saved to /var/cache/conftool/dbconfig/20261105-113823-ladsgroup.json
- 11:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209', diff saved to https://phabricator.wikimedia.org/P97352 and previous config saved to /var/cache/conftool/dbconfig/20261105-112821-ladsgroup.json
- 11:27 btullis@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad
- 11:22 btullis@cumin1004: END (ERROR) - Cookbook sre.hadoop.reboot-workers (exit_code=97) for Hadoop analytics cluster
- 11:20 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1262: Maintenance
- 11:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1209 (T441633)', diff saved to https://phabricator.wikimedia.org/P97350 and previous config saved to /var/cache/conftool/dbconfig/20261105-111819-ladsgroup.json
- 11:17 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
- 11:16 cmooney@cumin1004: START - Cookbook sre.dns.netbox
- 11:16 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-ml: apply
- 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-ml: apply
- 11:14 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
- 11:13 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
- 11:12 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
- 11:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1209 (T441633)', diff saved to https://phabricator.wikimedia.org/P97349 and previous config saved to /var/cache/conftool/dbconfig/20261105-111113-ladsgroup.json
- 11:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1209.eqiad.wmnet with reason: Maintenance
- 11:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T441633)', diff saved to https://phabricator.wikimedia.org/P97348 and previous config saved to /var/cache/conftool/dbconfig/20261105-111053-ladsgroup.json
- 11:09 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2250.codfw.wmnet with OS trixie
- 11:05 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 11:04 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-eqiad
- 11:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1002.eqiad.wmnet
- 11:01 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1002.eqiad.wmnet
- 11:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97346 and previous config saved to /var/cache/conftool/dbconfig/20261105-110051-ladsgroup.json
- 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1002.eqiad.wmnet
- 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1002.eqiad.wmnet
- 10:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl1001.eqiad.wmnet
- 10:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl1001.eqiad.wmnet
- 10:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl1001.eqiad.wmnet
- 10:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl1001.eqiad.wmnet
- 10:51 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-eqiad
- 10:51 btullis@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad
- 10:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P97344 and previous config saved to /var/cache/conftool/dbconfig/20261105-105049-ladsgroup.json
- 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-master-codfw
- 10:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2002.codfw.wmnet
- 10:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2002.codfw.wmnet
- 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2002.codfw.wmnet
- 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2002.codfw.wmnet
- 10:43 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-ctrl2001.codfw.wmnet
- 10:43 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-ctrl2001.codfw.wmnet
- 10:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T441633)', diff saved to https://phabricator.wikimedia.org/P97342 and previous config saved to /var/cache/conftool/dbconfig/20261105-104047-ladsgroup.json
- 10:38 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-ctrl2001.codfw.wmnet
- 10:38 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-ctrl2001.codfw.wmnet
- 10:38 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
- 10:38 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-master-codfw
- 10:35 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
- 10:34 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on db2250.codfw.wmnet with reason: host reimage
- 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1192 (T441633)', diff saved to https://phabricator.wikimedia.org/P97340 and previous config saved to /var/cache/conftool/dbconfig/20261105-103234-ladsgroup.json
- 10:32 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
- 10:32 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97339 and previous config saved to /var/cache/conftool/dbconfig/20261105-103215-ladsgroup.json
- 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (remove config only)
- 10:32 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (remove config only)
- 10:32 moritzm: installing libgit2 security updates
- 10:27 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
- 10:27 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
- 10:26 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
- 10:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97338 and previous config saved to /var/cache/conftool/dbconfig/20261105-102213-ladsgroup.json
- 10:21 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host db2250.codfw.wmnet (push config only)
- 10:21 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host db2250.codfw.wmnet (push config only)
- 10:20 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host db2250.codfw.wmnet with OS trixie
- 10:19 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2250.codfw.wmnet with reason: Reimage
- 10:16 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on sretest1006.eqiad.wmnet with reason: test on other vlan
- 10:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P97337 and previous config saved to /var/cache/conftool/dbconfig/20261105-101211-ladsgroup.json
- 10:06 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1006.wikimedia.org
- 10:04 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1262: Maintenance
- 10:04 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1262: Maintenance
- 10:02 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1006.wikimedia.org
- 10:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97336 and previous config saved to /var/cache/conftool/dbconfig/20261105-100209-ladsgroup.json
- 09:57 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica1005.wikimedia.org
- 09:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97335 and previous config saved to /var/cache/conftool/dbconfig/20261105-095350-ladsgroup.json
- 09:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
- 09:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica1005.wikimedia.org
- 09:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
- 09:45 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
- 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jynus
- 09:36 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jynus
- 09:36 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user jmoore111
- 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user jmoore111
- 09:35 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user andreawest
- 09:35 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user andreawest
- 09:31 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user suecarmol
- 09:31 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user suecarmol
- 09:24 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
- 09:23 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
- 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user harrayo-wmf
- 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user harrayo-wmf
- 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user atieno
- 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user atieno
- 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user ksiebert
- 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user ksiebert
- 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user dmaza
- 09:17 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user dmaza
- 09:17 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user hmonroy
- 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user hmonroy
- 09:16 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user kgraessle
- 09:16 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user kgraessle
- 09:12 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group2 to 1.48.0-wmf.3 refs T442221
- 09:09 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user nettrom
- 09:08 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user nettrom
- 08:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply
- 08:42 moritzm: restarting FPM on phab1005 to pick up Expat security updates
- 08:42 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply
- 08:22 dcausse@deploy2003: Finished scap sync-world: Backport for Restore the semantic highlighting batch ext (1.48.0-wmf.3) (duration: 16m 04s)
- 08:14 dcausse@deploy2003: dcausse: Continuing with deployment
- 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.decommission (exit_code=0)
- 08:11 marostegui@cumin1004: Removing clouddb1014 from zarcillo T441395
- 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts clouddb1014.eqiad.wmnet
- 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 08:11 marostegui@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: clouddb1014.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1004"
- 08:10 dcausse@deploy2003: dcausse: Backport for Restore the semantic highlighting batch ext (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:07 marostegui@cumin1004: START - Cookbook sre.dns.netbox
- 08:06 dcausse@deploy2003: Started scap sync-world: Backport for Restore the semantic highlighting batch ext (1.48.0-wmf.3)
- 08:02 marostegui@cumin1004: START - Cookbook sre.hosts.decommission for hosts clouddb1014.eqiad.wmnet
- 08:01 marostegui@cumin1004: START - Cookbook sre.mysql.decommission
- 07:59 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast6003.wikimedia.org
- 07:56 moritzm: installing expat security updates
- 07:53 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast6003.wikimedia.org
- 07:37 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast2003.wikimedia.org
- 07:30 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast2003.wikimedia.org
- 07:26 marostegui: Drop thread_reaction tables T442289
- 07:23 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2007.codfw.wmnet with OS trixie
- 07:07 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
- 07:03 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2007.codfw.wmnet with reason: host reimage
- 07:01 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (remove config only)
- 07:01 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (remove config only)
- 06:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2007.codfw.wmnet (push config only)
- 06:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2007.codfw.wmnet (push config only)
- 06:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2007.codfw.wmnet with OS trixie
- 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-eqiad
- 05:55 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1044.eqiad.wmnet
- 05:55 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1044.eqiad.wmnet
- 05:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1044.eqiad.wmnet
- 05:21 denisse@cumin1004: DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Cfoster out of all services on: 4 hosts
- 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1044.eqiad.wmnet
- 05:18 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1043.eqiad.wmnet
- 05:18 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1043.eqiad.wmnet
- 05:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1043.eqiad.wmnet
- 05:08 denisse@cumin1004: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Cfoster out of all services on: 4 hosts
- 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1043.eqiad.wmnet
- 04:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1042.eqiad.wmnet
- 04:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1042.eqiad.wmnet
- 04:33 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2020.codfw.wmnet, repooling both afterwards
- 04:33 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1042.eqiad.wmnet
- 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1042.eqiad.wmnet
- 04:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1041.eqiad.wmnet
- 04:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1041.eqiad.wmnet
- 04:27 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling both afterwards
- 04:26 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2020.codfw.wmnet, repooling both afterwards
- 04:26 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2019.codfw.wmnet, repooling both afterwards
- 04:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1041.eqiad.wmnet
- 04:19 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1026.eqiad.wmnet, repooling both afterwards
- 04:19 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards
- 04:18 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2019.codfw.wmnet, repooling both afterwards
- 04:18 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling both afterwards
- 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1025.eqiad.wmnet, repooling both afterwards
- 04:11 ryankemper@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, copy the reloaded categories journal to wdqs-internal-main (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2018.codfw.wmnet, repooling both afterwards
- 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1041.eqiad.wmnet
- 03:56 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1040.eqiad.wmnet
- 03:56 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1040.eqiad.wmnet
- 03:46 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1040.eqiad.wmnet
- 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1040.eqiad.wmnet
- 03:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1039.eqiad.wmnet
- 03:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1039.eqiad.wmnet
- 03:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1039.eqiad.wmnet
- 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1039.eqiad.wmnet
- 03:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1028.eqiad.wmnet
- 03:10 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1028.eqiad.wmnet
- 03:02 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet
- 02:35 dani@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
- 02:35 dani@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
- 02:35 dani@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
- 02:35 dani@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
- 02:34 dani@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
- 02:34 dani@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
- 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet
- 02:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1027.eqiad.wmnet
- 02:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1027.eqiad.wmnet
- 02:24 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1027.eqiad.wmnet
- 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1027.eqiad.wmnet
- 02:23 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1026.eqiad.wmnet
- 02:23 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1026.eqiad.wmnet
- 02:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T441633)', diff saved to https://phabricator.wikimedia.org/P97334 and previous config saved to /var/cache/conftool/dbconfig/20261105-022159-ladsgroup.json
- 02:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97333 and previous config saved to /var/cache/conftool/dbconfig/20261105-021157-ladsgroup.json
- 02:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1026.eqiad.wmnet
- 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1026.eqiad.wmnet
- 02:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1025.eqiad.wmnet
- 02:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1025.eqiad.wmnet
- 02:09 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
- 02:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P97332 and previous config saved to /var/cache/conftool/dbconfig/20261105-020155-ladsgroup.json
- 02:01 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 02:01 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1025.eqiad.wmnet
- 01:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2228 (T441633)', diff saved to https://phabricator.wikimedia.org/P97331 and previous config saved to /var/cache/conftool/dbconfig/20261105-015153-ladsgroup.json
- 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1025.eqiad.wmnet
- 01:31 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1024.eqiad.wmnet
- 01:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1024.eqiad.wmnet
- 01:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet
- 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
- 00:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
- 00:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
- 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2228 (T441633)', diff saved to https://phabricator.wikimedia.org/P97330 and previous config saved to /var/cache/conftool/dbconfig/20261105-005129-ladsgroup.json
- 00:51 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance
- 00:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97329 and previous config saved to /var/cache/conftool/dbconfig/20261105-005107-ladsgroup.json
- 00:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
- 00:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97328 and previous config saved to /var/cache/conftool/dbconfig/20261105-004105-ladsgroup.json
- 00:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P97327 and previous config saved to /var/cache/conftool/dbconfig/20261105-003103-ladsgroup.json
- 00:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97326 and previous config saved to /var/cache/conftool/dbconfig/20261105-002101-ladsgroup.json
- 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97325 and previous config saved to /var/cache/conftool/dbconfig/20261105-001358-ladsgroup.json
- 00:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2222.codfw.wmnet with reason: Maintenance
- 00:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97324 and previous config saved to /var/cache/conftool/dbconfig/20261105-001336-ladsgroup.json
- 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
- 00:11 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
- 00:11 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
- 00:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
- 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
- 00:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
- 00:04 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
- 00:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97323 and previous config saved to /var/cache/conftool/dbconfig/20261105-000334-ladsgroup.json
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
2026-11-04
- 23:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1021.eqiad.wmnet
- 23:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P97322 and previous config saved to /var/cache/conftool/dbconfig/20261104-235332-ladsgroup.json
- 23:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97321 and previous config saved to /var/cache/conftool/dbconfig/20261104-234330-ladsgroup.json
- 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2221 (T441633)', diff saved to https://phabricator.wikimedia.org/P97320 and previous config saved to /var/cache/conftool/dbconfig/20261104-233620-ladsgroup.json
- 23:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2221.codfw.wmnet with reason: Maintenance
- 23:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T441633)', diff saved to https://phabricator.wikimedia.org/P97319 and previous config saved to /var/cache/conftool/dbconfig/20261104-233559-ladsgroup.json
- 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
- 23:27 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
- 23:27 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
- 23:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97318 and previous config saved to /var/cache/conftool/dbconfig/20261104-232557-ladsgroup.json
- 23:24 jdlrobson@deploy2003: Finished scap sync-world: Backport for UnsaveDialog: Don't render in place in bottom sheet mode (T443481) (1.48.0-wmf.3) (duration: 16m 34s)
- 23:16 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
- 23:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P97317 and previous config saved to /var/cache/conftool/dbconfig/20261104-231555-ladsgroup.json
- 23:15 jdlrobson@deploy2003: jdlrobson: Backport for UnsaveDialog: Don't render in place in bottom sheet mode (T443481) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 23:08 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
- 23:07 jdlrobson@deploy2003: Started scap sync-world: Backport for UnsaveDialog: Don't render in place in bottom sheet mode (T443481) (1.48.0-wmf.3)
- 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
- 23:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
- 23:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
- 23:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T441633)', diff saved to https://phabricator.wikimedia.org/P97316 and previous config saved to /var/cache/conftool/dbconfig/20261104-230554-ladsgroup.json
- 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2220 (T441633)', diff saved to https://phabricator.wikimedia.org/P97315 and previous config saved to /var/cache/conftool/dbconfig/20261104-225843-ladsgroup.json
- 22:58 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2220.codfw.wmnet with reason: Maintenance
- 22:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T441633)', diff saved to https://phabricator.wikimedia.org/P97314 and previous config saved to /var/cache/conftool/dbconfig/20261104-225821-ladsgroup.json
- 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97312 and previous config saved to /var/cache/conftool/dbconfig/20261104-224819-ladsgroup.json
- 22:43 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1019.eqiad.wmnet
- 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
- 22:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
- 22:41 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
- 22:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P97311 and previous config saved to /var/cache/conftool/dbconfig/20261104-223817-ladsgroup.json
- 22:28 btullis@cumin1004: END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1018.eqiad.wmnet
- 22:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T441633)', diff saved to https://phabricator.wikimedia.org/P97310 and previous config saved to /var/cache/conftool/dbconfig/20261104-222815-ladsgroup.json
- 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
- 22:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1017.eqiad.wmnet
- 22:26 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1017.eqiad.wmnet
- 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2218 (T441633)', diff saved to https://phabricator.wikimedia.org/P97309 and previous config saved to /var/cache/conftool/dbconfig/20261104-222124-ladsgroup.json
- 22:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2218.codfw.wmnet with reason: Maintenance
- 22:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T441633)', diff saved to https://phabricator.wikimedia.org/P97308 and previous config saved to /var/cache/conftool/dbconfig/20261104-222102-ladsgroup.json
- 22:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1017.eqiad.wmnet
- 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1017.eqiad.wmnet
- 22:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
- 22:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
- 22:17 ejegg@deploy2003: Finished scap sync-world: Backport for Update CentralNotice for new mixin (1.48.0-wmf.1, 1.48.0-wmf.3) (duration: 41m 33s)
- 22:13 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
- 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
- 22:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
- 22:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
- 22:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97307 and previous config saved to /var/cache/conftool/dbconfig/20261104-221101-ladsgroup.json
- 22:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
- 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
- 22:05 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
- 22:05 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
- 22:02 ejegg@deploy2003: ejegg: Continuing with deployment
- 22:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P97306 and previous config saved to /var/cache/conftool/dbconfig/20261104-220059-ladsgroup.json
- 22:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
- 21:57 ejegg@deploy2003: ejegg: Backport for Update CentralNotice for new mixin (1.48.0-wmf.1, 1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T441633)', diff saved to https://phabricator.wikimedia.org/P97305 and previous config saved to /var/cache/conftool/dbconfig/20261104-215057-ladsgroup.json
- 21:48 tsev@deploy2003: mwscript-k8s job started: purgeList.php # T443456
- 21:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2208 (T441633)', diff saved to https://phabricator.wikimedia.org/P97304 and previous config saved to /var/cache/conftool/dbconfig/20261104-214349-ladsgroup.json
- 21:43 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2208.codfw.wmnet with reason: Maintenance
- 21:41 tsev@deploy2003: mwscript-k8s job started: purgeList.php # T443456
- 21:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
- 21:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T441633)', diff saved to https://phabricator.wikimedia.org/P97303 and previous config saved to /var/cache/conftool/dbconfig/20261104-213547-ladsgroup.json
- 21:35 ejegg@deploy2003: Started scap sync-world: Backport for Update CentralNotice for new mixin (1.48.0-wmf.1, 1.48.0-wmf.3)
- 21:30 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
- 21:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
- 21:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
- 21:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97302 and previous config saved to /var/cache/conftool/dbconfig/20261104-212545-ladsgroup.json
- 21:22 tsev@deploy2003: Finished scap sync-world: Backport for Allow ReadingLists path in Apple App Site Association File (T443456) (duration: 11m 26s)
- 21:16 tsev@deploy2003: tsev: Continuing with deployment
- 21:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P97301 and previous config saved to /var/cache/conftool/dbconfig/20261104-211544-ladsgroup.json
- 21:13 tsev@deploy2003: tsev: Backport for Allow ReadingLists path in Apple App Site Association File (T443456) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 19:37 mutante: !log is not !logging
- 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1018.eqiad.wmnet, repooling both afterwards
- 19:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling both afterwards
- 19:28 ladsgroup@deploy2003: Finished scap sync-world: Backport for Redesign 404 pages (T129433) (duration: 09m 46s)
- 19:22 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Continuing with deployment
- 19:22 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1018.eqiad.wmnet, repooling both afterwards
- 19:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling both afterwards
- 19:20 ladsgroup@deploy2003: ladsgroup, ladsgroup-claude: Backport for Redesign 404 pages (T129433) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 19:18 ladsgroup@deploy2003: Started scap sync-world: Backport for Redesign 404 pages (T129433)
- 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling both afterwards
- 19:16 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling both afterwards
- 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2013.codfw.wmnet, repooling both afterwards
- 19:08 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1017.eqiad.wmnet, repooling both afterwards
- 19:08 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling both afterwards
- 19:07 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling both afterwards
- 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
- 19:06 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
- 19:06 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
- 19:00 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling both afterwards
- 18:59 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2012.codfw.wmnet, repooling both afterwards
- 18:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
- 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling both afterwards
- 18:53 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1014.eqiad.wmnet, repooling both afterwards
- 18:48 ladsgroup@deploy2003: Finished scap sync-world: Pushing 503 redesigns (T129433) (duration: 09m 40s)
- 18:46 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2011.codfw.wmnet, repooling both afterwards
- 18:45 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1014.eqiad.wmnet, repooling both afterwards
- 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling both afterwards
- 18:45 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1013.eqiad.wmnet, repooling both afterwards
- 18:42 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 18:40 ladsgroup@deploy2003: ladsgroup: Pushing 503 redesigns (T129433) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 18:39 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns (T129433)
- 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2010.codfw.wmnet, repooling both afterwards
- 18:37 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1013.eqiad.wmnet, repooling both afterwards
- 18:35 ladsgroup@deploy2003: Stopping before sync operations
- 18:34 ladsgroup@deploy2003: Started scap sync-world: Pushing 503 redesigns (T129433)
- 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
- 18:29 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
- 18:29 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
- 18:27 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2007.codfw.wmnet, repooling both afterwards
- 18:26 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling both afterwards
- 18:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
- 18:19 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed codfw with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs2007.codfw.wmnet, repooling both afterwards
- 18:18 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs1011.eqiad.wmnet -> wdqs1012.eqiad.wmnet, repooling both afterwards
- 18:12 krinkle@deploy2003: Finished scap sync-world: Backport for Remove Mathoid SVG option from wgMathValidModes (T271001) (duration: 13m 43s)
- 18:11 mutante: gerrit2003/gerrit2002 - systemctl restart gerrit to deploy gerrit:1308081 to cap index query results to 5000 (T431398)
- 18:06 krinkle@deploy2003: krinkle: Continuing with deployment
- 18:01 krinkle@deploy2003: krinkle: Backport for Remove Mathoid SVG option from wgMathValidModes (T271001) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 17:58 krinkle@deploy2003: Started scap sync-world: Backport for Remove Mathoid SVG option from wgMathValidModes (T271001)
- 17:54 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
- 17:53 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
- 17:53 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
- 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
- 17:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
- 17:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
- 17:52 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
- 17:52 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
- 17:51 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
- 17:51 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
- 17:51 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
- 17:49 ariel@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
- 17:48 ariel@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
- 17:48 ariel@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
- 17:47 ariel@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
- 17:46 ariel@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
- 17:45 ariel@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
- 17:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
- 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
- 17:44 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
- 17:44 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
- 17:37 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
- 17:31 jhancock@cumin2003: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 17:26 fabfur: apt: included haproxykafka 0.3.18 into trixie-wikimedia (T443632)
- 17:26 jhancock@cumin2003: START - Cookbook sre.hosts.provision for host wikikube-worker2337.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART
- 17:13 cmooney@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lsw1-b3-eqiad,ssw1-a[1,8]-eqiad with reason: desperate troubleshooting measure
- 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
- 17:07 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
- 17:07 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
- 17:02 btullis@cumin1004: START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster
- 17:02 ladsgroup@deploy2003: Finished scap sync-world: Backport for Redesign 503 pages (T129433) (duration: 11m 19s)
- 16:59 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
- 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
- 16:58 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
- 16:58 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
- 16:56 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 16:53 ladsgroup@deploy2003: ladsgroup: Backport for Redesign 503 pages (T129433) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 16:51 ladsgroup@deploy2003: Started scap sync-world: Backport for Redesign 503 pages (T129433)
- 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
- 16:47 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
- 16:47 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
- 16:46 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
- 16:36 gmodena@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
- 16:36 gmodena@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
- 16:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1002.eqiad.wmnet
- 16:32 gmodena@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
- 16:31 gmodena@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
- 16:29 bking@cumin2003: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs1011.eqiad.wmnet, repooling both afterwards
- 16:21 bking@cumin2003: START - Cookbook sre.wdqs.data-transfer (T441682, seed eqiad with the reloaded categories journal (fills the 10-15 to 10-28 Commons gap)) xfer categories from wdqs2008.codfw.wmnet -> wdqs1011.eqiad.wmnet, repooling both afterwards
- 16:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97287 and previous config saved to /var/cache/conftool/dbconfig/20261104-161948-ladsgroup.json
- 16:19 gmodena@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 16:18 gmodena@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 16:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97286 and previous config saved to /var/cache/conftool/dbconfig/20261104-160946-ladsgroup.json
- 16:06 kamila@deploy2003: Finished scap sync-world: Backport for Enroll 1% of client sessions in PHP 8.5 (T441689) (duration: 10m 02s)
- 16:03 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqsin [reason: no reason specified, T416563]
- 16:03 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: pool eqsin [reason: no reason specified, T416563]
- 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
- 16:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
- 16:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
- 16:00 kamila@deploy2003: kamila: Continuing with deployment
- 15:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238', diff saved to https://phabricator.wikimedia.org/P97285 and previous config saved to /var/cache/conftool/dbconfig/20261104-155943-ladsgroup.json
- 15:59 pt1979@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
- 15:59 pt1979@cumin1004: START - Cookbook sre.hosts.remove-downtime for cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt
- 15:58 kamila@deploy2003: kamila: Backport for Enroll 1% of client sessions in PHP 8.5 (T441689) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:56 kamila@deploy2003: Started scap sync-world: Backport for Enroll 1% of client sessions in PHP 8.5 (T441689)
- 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 15:51 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004"
- 15:51 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove old cr2-eqord lo IPs - ayounsi@cumin1004"
- 15:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97284 and previous config saved to /var/cache/conftool/dbconfig/20261104-154941-ladsgroup.json
- 15:49 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
- 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
- 15:48 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
- 15:48 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
- 15:46 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
- 15:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2238 (T441633)', diff saved to https://phabricator.wikimedia.org/P97283 and previous config saved to /var/cache/conftool/dbconfig/20261104-154218-ladsgroup.json
- 15:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2238.codfw.wmnet with reason: Maintenance
- 15:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97282 and previous config saved to /var/cache/conftool/dbconfig/20261104-154156-ladsgroup.json
- 15:41 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
- 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
- 15:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1004.eqiad.wmnet
- 15:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1004.eqiad.wmnet
- 15:40 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply
- 15:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply
- 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1004.eqiad.wmnet
- 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1004.eqiad.wmnet
- 15:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
- 15:33 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
- 15:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97281 and previous config saved to /var/cache/conftool/dbconfig/20261104-153154-ladsgroup.json
- 15:31 kharlan@deploy2003: Finished scap sync-world: Backport for AbuseReview: Add undo and rollback links to all queues (T443591) (1.48.0-wmf.3) (duration: 09m 14s)
- 15:28 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
- 15:26 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
- 15:25 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
- 15:25 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
- 15:25 kharlan@deploy2003: kharlan: Continuing with deployment
- 15:24 kharlan@deploy2003: kharlan: Backport for AbuseReview: Add undo and rollback links to all queues (T443591) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:22 kharlan@deploy2003: Started scap sync-world: Backport for AbuseReview: Add undo and rollback links to all queues (T443591) (1.48.0-wmf.3)
- 15:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226', diff saved to https://phabricator.wikimedia.org/P97280 and previous config saved to /var/cache/conftool/dbconfig/20261104-152152-ladsgroup.json
- 15:20 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
- 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
- 15:19 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
- 15:19 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
- 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts db1176.eqiad.wmnet
- 15:18 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host db1176.eqiad.wmnet
- 15:13 sbisson@deploy2003: Finished scap sync-world: Backport for ArticleGuidance: Enable the reference risk model in production (T442164), Load ArticleGuidance on all Wikipedias (T442035) (duration: 12m 29s)
- 15:12 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
- 15:12 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
- 15:11 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-eqiad
- 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2226 (T441633)', diff saved to and previous config saved to /var/cache/conftool/dbconfig/20261104-151151-ladsgroup.json
- 15:10 cwilliams@cumin1004: START - Cookbook sre.hosts.reboot-single for host db1176.eqiad.wmnet
- 15:08 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second (1363248) (duration: 01m 46s)
- 15:08 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
- 15:07 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
- 15:07 sbisson@deploy2003: sbisson: Continuing with deployment
- 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
- 15:06 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
- 15:06 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-orchestrator second (1363248)
- 15:05 sbisson@deploy2003: sbisson: Backport for ArticleGuidance: Enable the reference risk model in production (T442164), Load ArticleGuidance on all Wikipedias (T442035) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:05 papaul: start maintenance on cr2-eqsin
- 15:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2226 (T441633)', diff saved to https://phabricator.wikimedia.org/P97279 and previous config saved to /var/cache/conftool/dbconfig/20261104-150502-ladsgroup.json
- 15:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2226.codfw.wmnet with reason: Maintenance
- 15:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T441633)', diff saved to https://phabricator.wikimedia.org/P97278 and previous config saved to /var/cache/conftool/dbconfig/20261104-150440-ladsgroup.json
- 15:03 jforrester@deploy2003: Finished scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first (1363247) (duration: 02m 28s)
- 15:03 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 8 hosts with reason: Testing
- 15:03 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] codfw Deploying wikifunctions
- 15:02 jforrester@deploy2003: scap deploy-service wikifunctions: [PRODUCTION/DEFAULT] eqiad Deploying wikifunctions
- 15:01 jforrester@deploy2003: scap deploy-service wikifunctions: [STAGING/DEFAULT] staging-eqiad Deploying wikifunctions
- 15:01 jforrester@deploy2003: Started scap deploy-service wikifunctions: Weekly service deploy, function-evaluators first (1363247)
- 15:00 sbisson@deploy2003: Started scap sync-world: Backport for ArticleGuidance: Enable the reference risk model in production (T442164), Load ArticleGuidance on all Wikipedias (T442035)
- 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary # T440137
- 15:00 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --add-prefix=BROKEN --fix # T440137
- 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary # T440137
- 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary --fix # T440137
- 14:59 urbanecm@deploy2003: mwscript-k8s job started: namespaceDupes.php --wiki=madwiktionary # T440137
- 14:57 urbanecm@deploy2003: Finished scap sync-world: Backport for Rename project namespace on Madurese Wiktionary (T440137) (duration: 11m 14s)
- 14:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97277 and previous config saved to /var/cache/conftool/dbconfig/20261104-145438-ladsgroup.json
- 14:49 urbanecm@deploy2003: urbanecm, amire80: Continuing with deployment
- 14:49 cwilliams@cumin1004: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts db1176.eqiad.wmnet
- 14:48 urbanecm@deploy2003: urbanecm, amire80: Backport for Rename project namespace on Madurese Wiktionary (T440137) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:46 urbanecm@deploy2003: Started scap sync-world: Backport for Rename project namespace on Madurese Wiktionary (T440137)
- 14:45 urbanecm@deploy2003: Finished scap sync-world: Backport for Re-enable NewUserMessage for viwikivoyage (T442485), Configure cookie-based enrollment in PHP 8.5 (T441689) (duration: 29m 32s)
- 14:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225', diff saved to https://phabricator.wikimedia.org/P97275 and previous config saved to /var/cache/conftool/dbconfig/20261104-144436-ladsgroup.json
- 14:43 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
- 14:42 pt1979@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr2-eqsin,cr2-eqsin IPv6,cr2-eqsin.mgmt with reason: Junos upgrade
- 14:39 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 14:38 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2225 (T441633)', diff saved to https://phabricator.wikimedia.org/P97274 and previous config saved to /var/cache/conftool/dbconfig/20261104-143434-ladsgroup.json
- 14:33 cwilliams@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts db1176.eqiad.wmnet
- 14:32 pt1979@cumin1004: END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool eqsin [reason: router upgrade, T416563]
- 14:32 pt1979@cumin1004: START - Cookbook sre.dns.admin DNS admin: depool eqsin [reason: router upgrade, T416563]
- 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2225 (T441633)', diff saved to https://phabricator.wikimedia.org/P97273 and previous config saved to /var/cache/conftool/dbconfig/20261104-142729-ladsgroup.json
- 14:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2225.codfw.wmnet with reason: Maintenance
- 14:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97272 and previous config saved to /var/cache/conftool/dbconfig/20261104-142707-ladsgroup.json
- 14:19 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Continuing with deployment
- 14:18 urbanecm@deploy2003: kamila, nvdtn19, urbanecm: Backport for Re-enable NewUserMessage for viwikivoyage (T442485), Configure cookie-based enrollment in PHP 8.5 (T441689) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97270 and previous config saved to /var/cache/conftool/dbconfig/20261104-141705-ladsgroup.json
- 14:15 urbanecm@deploy2003: Started scap sync-world: Backport for Re-enable NewUserMessage for viwikivoyage (T442485), Configure cookie-based enrollment in PHP 8.5 (T441689)
- 14:15 urbanecm@deploy2003: urbanecm, nvdtn19: Continuing with deployment
- 14:08 urbanecm@deploy2003: urbanecm, nvdtn19: Backport for Re-enable NewUserMessage for viwikivoyage (T442485) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 14:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P97269 and previous config saved to /var/cache/conftool/dbconfig/20261104-140703-ladsgroup.json
- 14:06 kamila@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
- 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
- 14:05 kamila@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-web: apply
- 14:03 urbanecm@deploy2003: Started scap sync-world: Backport for Re-enable NewUserMessage for viwikivoyage (T442485)
- 14:01 jelto@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/etherpad: apply
- 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:dse-k8s-worker-codfw
- 14:00 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2011.codfw.wmnet
- 14:00 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2011.codfw.wmnet
- 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97268 and previous config saved to /var/cache/conftool/dbconfig/20261104-135701-ladsgroup.json
- 13:53 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2011.codfw.wmnet
- 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2011.codfw.wmnet
- 13:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2010.codfw.wmnet
- 13:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2010.codfw.wmnet
- 13:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97267 and previous config saved to /var/cache/conftool/dbconfig/20261104-134951-ladsgroup.json
- 13:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: Maintenance
- 13:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2010.codfw.wmnet
- 13:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2197.codfw.wmnet with reason: Maintenance
- 13:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T441633)', diff saved to https://phabricator.wikimedia.org/P97266 and previous config saved to /var/cache/conftool/dbconfig/20261104-134347-ladsgroup.json
- 13:41 jelto@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/etherpad: apply
- 13:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2010.codfw.wmnet
- 13:39 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2009.codfw.wmnet
- 13:39 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2009.codfw.wmnet
- 13:33 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97265 and previous config saved to /var/cache/conftool/dbconfig/20261104-133345-ladsgroup.json
- 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2009.codfw.wmnet
- 13:32 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2009.codfw.wmnet
- 13:32 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2008.codfw.wmnet
- 13:31 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2008.codfw.wmnet
- 13:26 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2008.codfw.wmnet
- 13:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P97264 and previous config saved to /var/cache/conftool/dbconfig/20261104-132343-ladsgroup.json
- 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2008.codfw.wmnet
- 13:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2007.codfw.wmnet
- 13:21 mszwarc@deploy2003: Finished scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see (T439885) (1.48.0-wmf.1, 1.48.0-wmf.3) (duration: 11m 06s)
- 13:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2007.codfw.wmnet
- 13:17 moritzm: installing expat security updates
- 13:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2007.codfw.wmnet
- 13:15 mszwarc@deploy2003: mszwarc: Continuing with deployment
- 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2007.codfw.wmnet
- 13:15 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2006.codfw.wmnet
- 13:15 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2006.codfw.wmnet
- 13:14 mszwarc@deploy2003: mszwarc: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see (T439885) (1.48.0-wmf.1, 1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 13:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T441633)', diff saved to https://phabricator.wikimedia.org/P97263 and previous config saved to /var/cache/conftool/dbconfig/20261104-131341-ladsgroup.json
- 13:10 mszwarc@deploy2003: Started scap sync-world: Backport for SECURITY: AbuseLogLookup: Don't count logs the user can't see (T439885) (1.48.0-wmf.1, 1.48.0-wmf.3)
- 13:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2006.codfw.wmnet
- 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2006.codfw.wmnet
- 13:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2005.codfw.wmnet
- 13:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2005.codfw.wmnet
- 13:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2005.codfw.wmnet
- 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2005.codfw.wmnet
- 13:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2004.codfw.wmnet
- 13:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2004.codfw.wmnet
- 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-eqiad
- 13:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1006.eqiad.wmnet
- 13:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1006.eqiad.wmnet
- 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 13:01 ayounsi@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004"
- 13:01 ayounsi@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cr3-esams mgmt - ayounsi@cumin1004"
- 12:58 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
- 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2004.codfw.wmnet
- 12:57 ayounsi@cumin1004: START - Cookbook sre.dns.netbox
- 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2004.codfw.wmnet
- 12:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2003.codfw.wmnet
- 12:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2003.codfw.wmnet
- 12:56 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
- 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1006.eqiad.wmnet
- 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1006.eqiad.wmnet
- 12:54 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1005.eqiad.wmnet
- 12:54 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1005.eqiad.wmnet
- 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2003.codfw.wmnet
- 12:51 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2003.codfw.wmnet
- 12:51 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2002.codfw.wmnet
- 12:50 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2002.codfw.wmnet
- 12:50 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
- 12:47 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
- 12:47 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1005.eqiad.wmnet
- 12:46 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 12:46 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1005.eqiad.wmnet
- 12:46 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1004.eqiad.wmnet
- 12:46 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1004.eqiad.wmnet
- 12:45 claime: donne with poolcounter reboots
- 12:45 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2002.codfw.wmnet
- 12:44 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653) (duration: 10m 58s)
- 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2002.codfw.wmnet
- 12:40 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker2001.codfw.wmnet
- 12:40 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker2001.codfw.wmnet
- 12:39 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1004.eqiad.wmnet
- 12:38 cgoubert@deploy2003: cgoubert: Continuing with deployment
- 12:37 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1004.eqiad.wmnet
- 12:34 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage1003.eqiad.wmnet
- 12:34 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage1003.eqiad.wmnet
- 12:33 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker2001.codfw.wmnet
- 12:33 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Repool poolcounter2006 after reboot (#5/5) (T441653)
- 12:31 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2006.codfw.wmnet
- 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker2001.codfw.wmnet
- 12:28 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage1003.eqiad.wmnet
- 12:28 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test2001.codfw.wmnet
- 12:28 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test2001.codfw.wmnet
- 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage1003.eqiad.wmnet
- 12:27 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-eqiad
- 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-staging-worker-codfw
- 12:27 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2004.codfw.wmnet
- 12:27 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2004.codfw.wmnet
- 12:27 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2006.codfw.wmnet
- 12:26 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653) (duration: 10m 32s)
- 12:22 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test2001.codfw.wmnet
- 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test2001.codfw.wmnet
- 12:21 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2005.codfw.wmnet
- 12:21 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2005.codfw.wmnet
- 12:21 cgoubert@deploy2003: cgoubert: Continuing with deployment
- 12:20 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2004.codfw.wmnet
- 12:20 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 12:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 12:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 12:16 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Repool poolcounter2005, depool poolcounter2006 (#4/5) (T441653)
- 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2005.codfw.wmnet
- 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2005.codfw.wmnet
- 12:16 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2004.codfw.wmnet
- 12:16 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2004.codfw.wmnet
- 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2004.codfw.wmnet
- 12:15 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2003.codfw.wmnet
- 12:15 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2003.codfw.wmnet
- 12:14 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter2005.codfw.wmnet
- 12:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2189 (T441633)', diff saved to https://phabricator.wikimedia.org/P97262 and previous config saved to /var/cache/conftool/dbconfig/20261104-121318-ladsgroup.json
- 12:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2189.codfw.wmnet with reason: Maintenance
- 12:13 moritzm: installing apt security updates
- 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T441633)', diff saved to https://phabricator.wikimedia.org/P97261 and previous config saved to /var/cache/conftool/dbconfig/20261104-121256-ladsgroup.json
- 12:12 mvolz@deploy2003: Finished scap deploy-service citoid: Update user-agent (duration: 05m 26s)
- 12:10 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter2005.codfw.wmnet
- 12:10 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] codfw Deploying citoid
- 12:10 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2004.codfw.wmnet
- 12:10 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653) (duration: 15m 19s)
- 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2004.codfw.wmnet
- 12:09 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2003.codfw.wmnet
- 12:09 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2003.codfw.wmnet
- 12:09 mvolz@deploy2003: scap deploy-service citoid: [PRODUCTION/DEFAULT] eqiad Deploying citoid
- 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2003.codfw.wmnet
- 12:08 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2003.codfw.wmnet
- 12:08 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2002.codfw.wmnet
- 12:07 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2002.codfw.wmnet
- 12:07 mvolz@deploy2003: scap deploy-service citoid: [STAGING/DEFAULT] staging-eqiad Deploying citoid
- 12:07 mvolz@deploy2003: Started scap deploy-service citoid: Update user-agent
- 12:04 cgoubert@deploy2003: cgoubert: Continuing with deployment
- 12:04 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2003.codfw.wmnet
- 12:03 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2003.codfw.wmnet
- 12:03 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2002.codfw.wmnet
- 12:03 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2002.codfw.wmnet
- 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97260 and previous config saved to /var/cache/conftool/dbconfig/20261104-120254-ladsgroup.json
- 12:02 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet
- 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet
- 12:01 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet
- 12:01 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet
- 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:59 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2002.codfw.wmnet
- 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2002.codfw.wmnet
- 11:57 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs2001.codfw.wmnet
- 11:57 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs2001.codfw.wmnet
- 11:55 blake@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
- 11:55 blake@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
- 11:55 blake@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-staging-worker-codfw
- 11:54 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Repool poolcounter1007, depool poolcounter2005 (#3/5) (T441653)
- 11:53 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1007.eqiad.wmnet
- 11:52 btullis@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs2001.codfw.wmnet
- 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P97258 and previous config saved to /var/cache/conftool/dbconfig/20261104-115252-ladsgroup.json
- 11:52 btullis@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs2001.codfw.wmnet
- 11:52 btullis@cumin1004: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker-codfw
- 11:49 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1007.eqiad.wmnet
- 11:48 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653), ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653) (duration: 10m 24s)
- 11:44 tappof: putting metamonitoring checks into "maintenance mode without sending a notification" to update the metamonitoring script T443406
- 11:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T441633)', diff saved to https://phabricator.wikimedia.org/P97257 and previous config saved to /var/cache/conftool/dbconfig/20261104-114251-ladsgroup.json
- 11:42 cgoubert@deploy2003: cgoubert: Continuing with deployment
- 11:42 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1002.eqiad.wmnet
- 11:41 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653), ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:37 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653), ProductionServices: Repool poolcounter1006, depool poolcounter1007 (#2/5) (T441653)
- 11:35 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1002.eqiad.wmnet
- 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db2175 (T441633)', diff saved to https://phabricator.wikimedia.org/P97256 and previous config saved to /var/cache/conftool/dbconfig/20261104-113517-ladsgroup.json
- 11:35 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2175.codfw.wmnet with reason: Maintenance
- 11:28 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1262.eqiad.wmnet with reason: Reboot
- 11:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host poolcounter1006.eqiad.wmnet
- 11:25 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host poolcounter1006.eqiad.wmnet
- 11:24 cgoubert@deploy2003: Finished scap sync-world: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653) (duration: 08m 39s)
- 11:21 btullis@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-mariadb1001.eqiad.wmnet
- 11:20 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1262.eqiad.wmnet with reason: Reboot
- 11:19 cgoubert@deploy2003: cgoubert: Continuing with deployment
- 11:18 cgoubert@deploy2003: cgoubert: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:17 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 11:16 cgoubert@deploy2003: Started scap sync-world: Backport for ProductionServices: Depool poolcounter1006 for reboot (#1/5) (T441653)
- 11:15 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 11:15 btullis@cumin1004: START - Cookbook sre.hosts.reboot-single for host an-mariadb1001.eqiad.wmnet
- 11:14 claime: Starting poolcounter reboot sequence
- 11:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for Better browser support for DASH (T442988) (1.48.0-wmf.3) (duration: 12m 51s)
- 11:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 11:04 ladsgroup@deploy2003: ladsgroup: Backport for Better browser support for DASH (T442988) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:02 fabfur: enabling puppet again on A:cp to apply fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1363067
- 10:59 ladsgroup@deploy2003: Started scap sync-world: Backport for Better browser support for DASH (T442988) (1.48.0-wmf.3)
- 10:52 fabfur: temporary disabling puppet on A:cp to fix https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328543
- 10:49 moritzm: installing vim security updates
- 10:34 cmooney@cumin1004: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host lvs1020.eqiad.wmnet
- 10:32 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2004.codfw.wmnet with OS trixie
- 10:28 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
- 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
- 10:28 klausman@cumin1004: END (ERROR) - Cookbook sre.hadoop.check-user-leftovers (exit_code=97) Checking for leftover data for user akosiaris
- 10:28 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
- 10:27 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
- 10:27 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
- 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
- 10:19 gkyziridis@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
- 10:18 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
- 10:17 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
- 10:14 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
- 10:12 cmooney@cumin1004: START - Cookbook sre.hosts.reboot-single for host lvs1020.eqiad.wmnet
- 10:11 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
- 10:10 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
- 10:10 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
- 10:09 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
- 10:06 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2004.codfw.wmnet with reason: host reimage
- 10:03 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (remove config only)
- 10:03 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (remove config only)
- 09:59 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
- 09:59 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
- 09:58 klausman@cumin1004: END (PASS) - Cookbook sre.hadoop.check-user-leftovers (exit_code=0) Checking for leftover data for user akosiaris
- 09:58 klausman@cumin1004: START - Cookbook sre.hadoop.check-user-leftovers Checking for leftover data for user akosiaris
- 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
- 09:54 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
- 09:53 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool es2036: Maintenance
- 09:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
- 09:49 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2004.codfw.wmnet (push config only)
- 09:49 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2004.codfw.wmnet (push config only)
- 09:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
- 09:48 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2004.codfw.wmnet with OS trixie
- 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
- 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/ferretdb-growthbook: apply
- 09:45 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthbook: apply
- 09:44 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook: apply
- 09:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2006.codfw.wmnet with OS trixie
- 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
- 09:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
- 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook: apply
- 09:14 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook: apply
- 09:13 jnuche@deploy2003: rebuilt and synchronized wikiversions files: group1 to 1.48.0-wmf.3 refs T442221
- 09:12 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
- 09:08 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool es2036: Maintenance
- 09:07 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2006.codfw.wmnet with reason: host reimage
- 09:07 arnaudb@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on gerrit2002.wikimedia.org with reason: T440177 offline reindex test
- 09:05 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (remove config only)
- 09:05 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (remove config only)
- 08:51 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2006.codfw.wmnet (push config only)
- 08:51 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2006.codfw.wmnet (push config only)
- 08:50 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2006.codfw.wmnet with OS trixie
- 08:15 Tran: Deployed patch for T441775
- 08:11 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov2005.codfw.wmnet with OS trixie
- 08:10 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host bast3007.wikimedia.org
- 08:06 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host bast3007.wikimedia.org
- 08:00 marostegui@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1274: after reboot
- 07:53 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
- 07:47 marostegui@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov2005.codfw.wmnet with reason: host reimage
- 07:45 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (remove config only)
- 07:45 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (remove config only)
- 07:31 marostegui@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host dbprov2005.codfw.wmnet (push config only)
- 07:31 marostegui@cumin1004: START - Cookbook sre.hosts.dhcp for host dbprov2005.codfw.wmnet (push config only)
- 07:30 marostegui@cumin1004: START - Cookbook sre.hosts.reimage for host dbprov2005.codfw.wmnet with OS trixie
- 07:15 marostegui@cumin1004: START - Cookbook sre.mysql.pool pool db1274: after reboot
- 07:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
- 07:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 (T441633)', diff saved to https://phabricator.wikimedia.org/P97242 and previous config saved to /var/cache/conftool/dbconfig/20261104-070600-ladsgroup.json
- 06:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97241 and previous config saved to /var/cache/conftool/dbconfig/20261104-065558-ladsgroup.json
- 06:55 marostegui@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Server didn't come back
- 06:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288', diff saved to https://phabricator.wikimedia.org/P97240 and previous config saved to /var/cache/conftool/dbconfig/20261104-064556-ladsgroup.json
- 06:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1288 (T441633)', diff saved to https://phabricator.wikimedia.org/P97239 and previous config saved to /var/cache/conftool/dbconfig/20261104-063554-ladsgroup.json
- 06:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1288 (T441633)', diff saved to https://phabricator.wikimedia.org/P97238 and previous config saved to /var/cache/conftool/dbconfig/20261104-063055-ladsgroup.json
- 06:30 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1288.eqiad.wmnet with reason: Maintenance
- 06:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1285.eqiad.wmnet with reason: Maintenance
- 06:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 (T441633)', diff saved to https://phabricator.wikimedia.org/P97237 and previous config saved to /var/cache/conftool/dbconfig/20261104-062712-ladsgroup.json
- 06:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97236 and previous config saved to /var/cache/conftool/dbconfig/20261104-061710-ladsgroup.json
- 06:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284', diff saved to https://phabricator.wikimedia.org/P97235 and previous config saved to /var/cache/conftool/dbconfig/20261104-060708-ladsgroup.json
- 05:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1284 (T441633)', diff saved to https://phabricator.wikimedia.org/P97234 and previous config saved to /var/cache/conftool/dbconfig/20261104-055706-ladsgroup.json
- 05:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1284 (T441633)', diff saved to https://phabricator.wikimedia.org/P97233 and previous config saved to /var/cache/conftool/dbconfig/20261104-055201-ladsgroup.json
- 05:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1284.eqiad.wmnet with reason: Maintenance
- 05:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 (T441633)', diff saved to https://phabricator.wikimedia.org/P97232 and previous config saved to /var/cache/conftool/dbconfig/20261104-055141-ladsgroup.json
- 05:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97231 and previous config saved to /var/cache/conftool/dbconfig/20261104-054139-ladsgroup.json
- 05:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273', diff saved to https://phabricator.wikimedia.org/P97230 and previous config saved to /var/cache/conftool/dbconfig/20261104-053137-ladsgroup.json
- 05:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1273 (T441633)', diff saved to https://phabricator.wikimedia.org/P97229 and previous config saved to /var/cache/conftool/dbconfig/20261104-052135-ladsgroup.json
- 05:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1273 (T441633)', diff saved to https://phabricator.wikimedia.org/P97228 and previous config saved to /var/cache/conftool/dbconfig/20261104-051423-ladsgroup.json
- 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
- 05:14 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1273.eqiad.wmnet with reason: Maintenance
- 05:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T441633)', diff saved to https://phabricator.wikimedia.org/P97227 and previous config saved to /var/cache/conftool/dbconfig/20261104-051351-ladsgroup.json
- 05:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97226 and previous config saved to /var/cache/conftool/dbconfig/20261104-050349-ladsgroup.json
- 04:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P97225 and previous config saved to /var/cache/conftool/dbconfig/20261104-045347-ladsgroup.json
- 04:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1253 (T441633)', diff saved to https://phabricator.wikimedia.org/P97224 and previous config saved to /var/cache/conftool/dbconfig/20261104-044345-ladsgroup.json
- 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1253 (T441633)', diff saved to https://phabricator.wikimedia.org/P97223 and previous config saved to /var/cache/conftool/dbconfig/20261104-043639-ladsgroup.json
- 04:36 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1253.eqiad.wmnet with reason: Maintenance
- 04:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T441633)', diff saved to https://phabricator.wikimedia.org/P97222 and previous config saved to /var/cache/conftool/dbconfig/20261104-043619-ladsgroup.json
- 04:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97221 and previous config saved to /var/cache/conftool/dbconfig/20261104-042617-ladsgroup.json
- 04:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P97220 and previous config saved to /var/cache/conftool/dbconfig/20261104-041615-ladsgroup.json
- 04:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T441633)', diff saved to https://phabricator.wikimedia.org/P97219 and previous config saved to /var/cache/conftool/dbconfig/20261104-040613-ladsgroup.json
- 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1231 (T441633)', diff saved to https://phabricator.wikimedia.org/P97218 and previous config saved to /var/cache/conftool/dbconfig/20261104-035922-ladsgroup.json
- 03:59 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1231.eqiad.wmnet with reason: Maintenance
- 03:59 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T441633)', diff saved to https://phabricator.wikimedia.org/P97217 and previous config saved to /var/cache/conftool/dbconfig/20261104-035901-ladsgroup.json
- 03:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97216 and previous config saved to /var/cache/conftool/dbconfig/20261104-034859-ladsgroup.json
- 03:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P97215 and previous config saved to /var/cache/conftool/dbconfig/20261104-033857-ladsgroup.json
- 03:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T441633)', diff saved to https://phabricator.wikimedia.org/P97214 and previous config saved to /var/cache/conftool/dbconfig/20261104-032855-ladsgroup.json
- 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1227 (T441633)', diff saved to https://phabricator.wikimedia.org/P97213 and previous config saved to /var/cache/conftool/dbconfig/20261104-032149-ladsgroup.json
- 03:21 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1227.eqiad.wmnet with reason: Maintenance
- 03:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T441633)', diff saved to https://phabricator.wikimedia.org/P97212 and previous config saved to /var/cache/conftool/dbconfig/20261104-032129-ladsgroup.json
- 03:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97211 and previous config saved to /var/cache/conftool/dbconfig/20261104-031127-ladsgroup.json
- 03:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P97210 and previous config saved to /var/cache/conftool/dbconfig/20261104-030125-ladsgroup.json
- 02:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T441633)', diff saved to https://phabricator.wikimedia.org/P97209 and previous config saved to /var/cache/conftool/dbconfig/20261104-025123-ladsgroup.json
- 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1202 (T441633)', diff saved to https://phabricator.wikimedia.org/P97208 and previous config saved to /var/cache/conftool/dbconfig/20261104-024423-ladsgroup.json
- 02:44 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance
- 02:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T441633)', diff saved to https://phabricator.wikimedia.org/P97207 and previous config saved to /var/cache/conftool/dbconfig/20261104-024403-ladsgroup.json
- 02:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97206 and previous config saved to /var/cache/conftool/dbconfig/20261104-023401-ladsgroup.json
- 02:23 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P97205 and previous config saved to /var/cache/conftool/dbconfig/20261104-022358-ladsgroup.json
- 02:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T441633)', diff saved to https://phabricator.wikimedia.org/P97204 and previous config saved to /var/cache/conftool/dbconfig/20261104-021356-ladsgroup.json
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 33s)
- 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1194 (T441633)', diff saved to https://phabricator.wikimedia.org/P97203 and previous config saved to /var/cache/conftool/dbconfig/20261104-020545-ladsgroup.json
- 02:05 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance
- 02:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T441633)', diff saved to https://phabricator.wikimedia.org/P97202 and previous config saved to /var/cache/conftool/dbconfig/20261104-020525-ladsgroup.json
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 01:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97201 and previous config saved to /var/cache/conftool/dbconfig/20261104-015523-ladsgroup.json
- 01:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P97200 and previous config saved to /var/cache/conftool/dbconfig/20261104-014521-ladsgroup.json
- 01:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T441633)', diff saved to https://phabricator.wikimedia.org/P97199 and previous config saved to /var/cache/conftool/dbconfig/20261104-013519-ladsgroup.json
- 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1191 (T441633)', diff saved to https://phabricator.wikimedia.org/P97198 and previous config saved to /var/cache/conftool/dbconfig/20261104-012829-ladsgroup.json
- 01:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1191.eqiad.wmnet with reason: Maintenance
- 01:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T441633)', diff saved to https://phabricator.wikimedia.org/P97197 and previous config saved to /var/cache/conftool/dbconfig/20261104-012809-ladsgroup.json
- 01:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97196 and previous config saved to /var/cache/conftool/dbconfig/20261104-011807-ladsgroup.json
- 01:08 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P97195 and previous config saved to /var/cache/conftool/dbconfig/20261104-010805-ladsgroup.json
- 00:58 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T441633)', diff saved to https://phabricator.wikimedia.org/P97194 and previous config saved to /var/cache/conftool/dbconfig/20261104-005803-ladsgroup.json
- 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1190 (T441633)', diff saved to https://phabricator.wikimedia.org/P97193 and previous config saved to /var/cache/conftool/dbconfig/20261104-004935-ladsgroup.json
- 00:49 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1190.eqiad.wmnet with reason: Maintenance
- 00:49 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97192 and previous config saved to /var/cache/conftool/dbconfig/20261104-004926-ladsgroup.json
- 00:39 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97191 and previous config saved to /var/cache/conftool/dbconfig/20261104-003924-ladsgroup.json
- 00:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
- 00:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
- 00:29 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P97190 and previous config saved to /var/cache/conftool/dbconfig/20261104-002922-ladsgroup.json
- 00:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/zotero: apply
- 00:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/zotero: apply
- 00:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
- 00:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
- 00:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
- 00:19 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97189 and previous config saved to /var/cache/conftool/dbconfig/20261104-001920-ladsgroup.json
- 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
- 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
- 00:15 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
- 00:15 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
- 00:14 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
- 00:14 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/toolhub: apply
- 00:14 ladsgroup@deploy2003: Finished scap sync-world: Backport for Avoid unnecessary/aborted requests in DASH mode (T442988) (1.48.0-wmf.3), Enable chunked videos on testwiki (T373546) (duration: 13m 02s)
- 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/toolhub: apply
- 00:13 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
- 00:13 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
- 00:12 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
- 00:12 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
- 00:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97188 and previous config saved to /var/cache/conftool/dbconfig/20261104-001120-ladsgroup.json
- 00:11 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/sessionstore: apply
- 00:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1181.eqiad.wmnet with reason: Maintenance
- 00:10 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/sessionstore: apply
- 00:10 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/recommendation-api: apply
- 00:09 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/recommendation-api: apply
- 00:09 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/push-notifications: apply
- 00:08 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/push-notifications: apply
- 00:07 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 00:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance
- 00:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T441633)', diff saved to https://phabricator.wikimedia.org/P97187 and previous config saved to /var/cache/conftool/dbconfig/20261104-000542-ladsgroup.json
- 00:05 ladsgroup@deploy2003: ladsgroup: Backport for Avoid unnecessary/aborted requests in DASH mode (T442988) (1.48.0-wmf.3), Enable chunked videos on testwiki (T373546) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 00:01 ladsgroup@deploy2003: Started scap sync-world: Backport for Avoid unnecessary/aborted requests in DASH mode (T442988) (1.48.0-wmf.3), Enable chunked videos on testwiki (T373546)
- 00:00 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/proton: apply
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
2026-11-03
- 23:59 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/proton: apply
- 23:58 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
- 23:58 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/page-analytics: apply
- 23:57 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
- 23:56 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mobileapps: apply
- 23:56 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
- 23:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97186 and previous config saved to /var/cache/conftool/dbconfig/20261103-235540-ladsgroup.json
- 23:54 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
- 23:54 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
- 23:53 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/media-analytics: apply
- 23:52 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/machinetranslation: apply
- 23:46 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/machinetranslation: apply
- 23:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P97185 and previous config saved to /var/cache/conftool/dbconfig/20261103-234538-ladsgroup.json
- 23:45 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/lac-lambdas: apply
- 23:44 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/lac-lambdas: apply
- 23:44 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/kartotherian: apply
- 23:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T441633)', diff saved to https://phabricator.wikimedia.org/P97184 and previous config saved to /var/cache/conftool/dbconfig/20261103-233536-ladsgroup.json
- 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/kartotherian: apply
- 23:34 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
- 23:34 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
- 23:33 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
- 23:33 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
- 23:33 musikanimal@deploy2003: Finished scap sync-world: Backport for CommonSettings: remove redundant CodeMirror config settings (T440077), metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015) (duration: 14m 56s)
- 23:32 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
- 23:32 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
- 23:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/editcheck-headless: apply
- 23:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/editcheck-headless: apply
- 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
- 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
- 23:30 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/echostore: apply
- 23:30 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/echostore: apply
- 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/device-analytics: apply
- 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/device-analytics: apply
- 23:29 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
- 23:29 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/developer-portal: apply
- 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/data-gateway: apply
- 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/data-gateway: apply
- 23:28 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cxserver: apply
- 23:28 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cxserver: apply
- 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/commons-impact-analytics: apply
- 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/commons-impact-analytics: apply
- 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/citoid: apply
- 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/citoid: apply
- 23:27 musikanimal@deploy2003: musikanimal, amdrel: Continuing with deployment
- 23:27 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
- 23:27 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
- 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1170 (T441633)', diff saved to https://phabricator.wikimedia.org/P97183 and previous config saved to /var/cache/conftool/dbconfig/20261103-232657-ladsgroup.json
- 23:26 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1170.eqiad.wmnet with reason: Maintenance
- 23:26 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/chart-renderer: apply
- 23:26 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T441633)', diff saved to https://phabricator.wikimedia.org/P97182 and previous config saved to /var/cache/conftool/dbconfig/20261103-232637-ladsgroup.json
- 23:26 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/chart-renderer: apply
- 23:24 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/apertium: apply
- 23:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/apertium: apply
- 23:22 musikanimal@deploy2003: musikanimal, amdrel: Backport for CommonSettings: remove redundant CodeMirror config settings (T440077), metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 23:18 musikanimal@deploy2003: Started scap sync-world: Backport for CommonSettings: remove redundant CodeMirror config settings (T440077), metawiki: Disable wish submissions on 2026-11-09 (17:00 UTC) (T442015)
- 23:16 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97181 and previous config saved to /var/cache/conftool/dbconfig/20261103-231635-ladsgroup.json
- 23:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P97180 and previous config saved to /var/cache/conftool/dbconfig/20261103-230633-ladsgroup.json
- 22:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T441633)', diff saved to https://phabricator.wikimedia.org/P97179 and previous config saved to /var/cache/conftool/dbconfig/20261103-225631-ladsgroup.json
- 22:53 ryankemper@cumin2003: END (PASS) - Cookbook sre.wdqs.categories-reload (exit_code=0) reloading categories to wdqs2008.codfw.wmnet
- 22:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1158 (T441633)', diff saved to https://phabricator.wikimedia.org/P97178 and previous config saved to /var/cache/conftool/dbconfig/20261103-224816-ladsgroup.json
- 22:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1158.eqiad.wmnet with reason: Maintenance
- 22:44 kemayo@deploy2003: Finished scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767) (duration: 10m 49s)
- 22:38 kemayo@deploy2003: kemayo: Continuing with deployment
- 22:38 kemayo@deploy2003: kemayo: Backport for VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 22:33 kemayo@deploy2003: Started scap sync-world: Backport for VisualEditor: Precompute edit suggestion counts on 5% of jawiki (T442767)
- 22:27 jdlrobson@deploy2003: Finished scap sync-world: Backport for styles: Restore style to Special:MobileOptions (T363032) (1.48.0-wmf.3), Revert "footer: Hide separator before a hidden last footer place" (T387797) (1.48.0-wmf.3), downloadPageAction: Add the download link before the bookmark (T443387) (1.48.0-wmf.3) (duration: 13m 25s)
- 22:21 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
- 22:18 jdlrobson@deploy2003: jdlrobson: Backport for styles: Restore style to Special:MobileOptions (T363032) (1.48.0-wmf.3), Revert "footer: Hide separator before a hidden last footer place" (T387797) (1.48.0-wmf.3), downloadPageAction: Add the download link before the bookmark (T443387) (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki
- 22:14 jdlrobson@deploy2003: Started scap sync-world: Backport for styles: Restore style to Special:MobileOptions (T363032) (1.48.0-wmf.3), Revert "footer: Hide separator before a hidden last footer place" (T387797) (1.48.0-wmf.3), downloadPageAction: Add the download link before the bookmark (T443387) (1.48.0-wmf.3)
- {{safesubst:SAL entry|1=21:42 ebernhardson@deploy2003: Finished scap sync-world: Backport for eswiki: add editprotected to botadmin (T443272), liwiktionary: Create Appendix namespace (T443269), Stop exempting item/property creation from hCaptcha (T440832), Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054), [[gerrit:1362185|Revert "Reapply "cirrus: Send more_lik}}
- 21:42 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1274 (T441633)', diff saved to https://phabricator.wikimedia.org/P97176 and previous config saved to /var/cache/conftool/dbconfig/20261103-214206-ladsgroup.json
- 21:42 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1274.eqiad.wmnet with reason: Maintenance
- 21:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T441633)', diff saved to https://phabricator.wikimedia.org/P97175 and previous config saved to /var/cache/conftool/dbconfig/20261103-214146-ladsgroup.json
- 21:36 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Continuing with deployment
- 21:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97174 and previous config saved to /var/cache/conftool/dbconfig/20261103-213145-ladsgroup.json
- {{safesubst:SAL entry|1=21:28 ebernhardson@deploy2003: ebernhardson, ameisenigel, thiemowmde, mahmoud-abdelsattar: Backport for eswiki: add editprotected to botadmin (T443272), liwiktionary: Create Appendix namespace (T443269), Stop exempting item/property creation from hCaptcha (T440832), Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054), [[gerrit:1362185|Reve}}
- {{safesubst:SAL entry|1=21:24 ebernhardson@deploy2003: Started scap sync-world: Backport for eswiki: add editprotected to botadmin (T443272), liwiktionary: Create Appendix namespace (T443269), Stop exempting item/property creation from hCaptcha (T440832), Remove WikibaseTmpMobileEditingUIBetaFeature config (T437054), [[gerrit:1362185|Revert "Reapply "cirrus: Send more_like}}
- 21:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P97173 and previous config saved to /var/cache/conftool/dbconfig/20261103-212143-ladsgroup.json
- 21:16 sbassett@deploy2003: Finished scap sync-world: Backport for Enable DBSC report-only mode on group0 and test2wiki (T393212) (duration: 11m 41s)
- 21:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T441633)', diff saved to https://phabricator.wikimedia.org/P97172 and previous config saved to /var/cache/conftool/dbconfig/20261103-211141-ladsgroup.json
- 21:09 sbassett@deploy2003: sbassett: Continuing with deployment
- 21:09 sbassett@deploy2003: sbassett: Backport for Enable DBSC report-only mode on group0 and test2wiki (T393212) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:05 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1230 (T441633)', diff saved to https://phabricator.wikimedia.org/P97171 and previous config saved to /var/cache/conftool/dbconfig/20261103-210458-ladsgroup.json
- 21:05 sbassett@deploy2003: Started scap sync-world: Backport for Enable DBSC report-only mode on group0 and test2wiki (T393212)
- 21:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1230.eqiad.wmnet with reason: Maintenance
- 21:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97170 and previous config saved to /var/cache/conftool/dbconfig/20261103-210435-ladsgroup.json
- 20:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97169 and previous config saved to /var/cache/conftool/dbconfig/20261103-205434-ladsgroup.json
- 20:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P97168 and previous config saved to /var/cache/conftool/dbconfig/20261103-204432-ladsgroup.json
- 20:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97167 and previous config saved to /var/cache/conftool/dbconfig/20261103-203430-ladsgroup.json
- 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1207 (T441633)', diff saved to https://phabricator.wikimedia.org/P97166 and previous config saved to /var/cache/conftool/dbconfig/20261103-202749-ladsgroup.json
- 20:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
- 20:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T441633)', diff saved to https://phabricator.wikimedia.org/P97165 and previous config saved to /var/cache/conftool/dbconfig/20261103-202739-ladsgroup.json
- 20:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97164 and previous config saved to /var/cache/conftool/dbconfig/20261103-201737-ladsgroup.json
- 20:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P97163 and previous config saved to /var/cache/conftool/dbconfig/20261103-200735-ladsgroup.json
- 20:00 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
- 19:59 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
- 19:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T441633)', diff saved to https://phabricator.wikimedia.org/P97162 and previous config saved to /var/cache/conftool/dbconfig/20261103-195734-ladsgroup.json
- 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1200 (T441633)', diff saved to https://phabricator.wikimedia.org/P97161 and previous config saved to /var/cache/conftool/dbconfig/20261103-195043-ladsgroup.json
- 19:50 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1200.eqiad.wmnet with reason: Maintenance
- 19:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T441633)', diff saved to https://phabricator.wikimedia.org/P97160 and previous config saved to /var/cache/conftool/dbconfig/20261103-195023-ladsgroup.json
- 19:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97159 and previous config saved to /var/cache/conftool/dbconfig/20261103-194021-ladsgroup.json
- 19:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P97158 and previous config saved to /var/cache/conftool/dbconfig/20261103-193019-ladsgroup.json
- 19:29 dancy@deploy2003: Installation of scap version "4.313.0" completed for 2 hosts
- 19:27 dancy@deploy2003: Installing scap version "4.313.0" for 2 host(s)
- 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 19:25 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 19:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T441633)', diff saved to https://phabricator.wikimedia.org/P97157 and previous config saved to /var/cache/conftool/dbconfig/20261103-192017-ladsgroup.json
- 19:19 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 19:18 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 19:17 dancy@deploy2003: Installing scap version "4.313.0" for 1 host(s)
- 19:16 dancy@deploy2003: Installing scap version "4.313.0" for 1 host(s)
- 19:14 dancy@deploy2003: Installing scap version "4.313.0" for 2 host(s)
- 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1185 (T441633)', diff saved to https://phabricator.wikimedia.org/P97156 and previous config saved to /var/cache/conftool/dbconfig/20261103-191327-ladsgroup.json
- 19:13 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance
- 19:13 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T441633)', diff saved to https://phabricator.wikimedia.org/P97155 and previous config saved to /var/cache/conftool/dbconfig/20261103-191307-ladsgroup.json
- 19:11 dancy@deploy2003: Installing scap version "4.313.0" for 2 host(s)
- 19:09 ryankemper@cumin2003: START - Cookbook sre.wdqs.categories-reload reloading categories to wdqs2008.codfw.wmnet
- 19:03 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97154 and previous config saved to /var/cache/conftool/dbconfig/20261103-190305-ladsgroup.json
- 18:53 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P97153 and previous config saved to /var/cache/conftool/dbconfig/20261103-185303-ladsgroup.json
- 18:51 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7002.* [reason: [end] testing pdns-rec 5.4.9]
- 18:50 atsuko@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 18:49 atsuko@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 18:49 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7002.* [reason: testing pdns-rec 5.4.9]
- 18:46 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy1003.eqiad.wmnet
- 18:43 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T441633)', diff saved to https://phabricator.wikimedia.org/P97152 and previous config saved to /var/cache/conftool/dbconfig/20261103-184301-ladsgroup.json
- 18:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1161 (T441633)', diff saved to https://phabricator.wikimedia.org/P97151 and previous config saved to /var/cache/conftool/dbconfig/20261103-183459-ladsgroup.json
- 18:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1161.eqiad.wmnet with reason: Maintenance
- 18:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T441633)', diff saved to https://phabricator.wikimedia.org/P97150 and previous config saved to /var/cache/conftool/dbconfig/20261103-183439-ladsgroup.json
- 18:34 kamila@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2003.codfw.wmnet
- 18:33 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy1003.eqiad.wmnet
- 18:31 cjming@deploy2003: Finished scap deploy-service test-kitchen: deploy v2.3.0 to TK production (duration: 02m 12s)
- 18:29 cjming@deploy2003: scap deploy-service test-kitchen: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen
- 18:29 cjming@deploy2003: Started scap deploy-service test-kitchen: deploy v2.3.0 to TK production
- 18:28 eevans@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
- 18:28 eevans@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
- 18:28 eevans@deploy2003: helmfile [codfw] DONE helmfile.d/services/linked-artifacts: apply
- 18:28 eevans@deploy2003: helmfile [codfw] START helmfile.d/services/linked-artifacts: apply
- 18:27 eevans@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
- 18:27 eevans@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
- 18:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97149 and previous config saved to /var/cache/conftool/dbconfig/20261103-182437-ladsgroup.json
- 18:23 kamila@cumin1004: START - Cookbook sre.hosts.reboot-single for host deploy2003.codfw.wmnet
- 18:22 akhatun@deploy2003: Finished scap sync-world: Backport for revision feature counts: increase timeout 20s to 30s (T443482) (duration: 10m 40s)
- 18:18 cjming@deploy2003: Finished scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging (duration: 01m 03s)
- 18:17 cjming@deploy2003: scap deploy-service test-kitchen-next: [PRODUCTION/DEFAULT] dse-k8s-eqiad Deploying test-kitchen-next
- 18:17 cjming@deploy2003: Started scap deploy-service test-kitchen-next: deploy v2.3.0 to TK staging
- 18:16 akhatun@deploy2003: akhatun: Continuing with deployment
- 18:16 akhatun@deploy2003: akhatun: Backport for revision feature counts: increase timeout 20s to 30s (T443482) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 18:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159', diff saved to https://phabricator.wikimedia.org/P97148 and previous config saved to /var/cache/conftool/dbconfig/20261103-181435-ladsgroup.json
- 18:11 akhatun@deploy2003: Started scap sync-world: Backport for revision feature counts: increase timeout 20s to 30s (T443482)
- 18:05 tchanders@deploy2003: Finished scap sync-world: Backport for Enable link spam queue on Special:AbuseReview for alpha testing (T443432) (duration: 19m 00s)
- 18:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1159 (T441633)', diff saved to https://phabricator.wikimedia.org/P97147 and previous config saved to /var/cache/conftool/dbconfig/20261103-180432-ladsgroup.json
- 17:59 tchanders@deploy2003: tchanders: Continuing with deployment
- 17:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1159 (T441633)', diff saved to https://phabricator.wikimedia.org/P97146 and previous config saved to /var/cache/conftool/dbconfig/20261103-175629-ladsgroup.json
- 17:56 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1159.eqiad.wmnet with reason: Maintenance
- 17:50 tchanders@deploy2003: tchanders: Backport for Enable link spam queue on Special:AbuseReview for alpha testing (T443432) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 17:46 tchanders@deploy2003: Started scap sync-world: Backport for Enable link spam queue on Special:AbuseReview for alpha testing (T443432)
- 17:03 kamila@deploy2003: Finished scap deploy-service shellbox: bump also shellbox score to php 8.5 (duration: 04m 51s)
- 17:00 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
- 16:59 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
- 16:58 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
- 16:58 kamila@deploy2003: Started scap deploy-service shellbox: bump also shellbox score to php 8.5
- 16:40 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netbox-dev2003.codfw.wmnet
- 16:39 otto@deploy2003: Finished scap deploy-service eventgate-all: v1.38.0 for T440968 (duration: 14m 43s)
- 16:36 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netbox-dev2003.codfw.wmnet
- 16:36 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
- 16:28 dancy@deploy2003: Installation of scap version "4.312.0" completed for 2 hosts
- 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
- 16:27 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
- 16:26 dancy@deploy2003: Installing scap version "4.312.0" for 2 host(s)
- 16:26 otto@deploy2003: scap deploy-service eventgate-all: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
- 16:24 otto@deploy2003: scap deploy-service eventgate-all: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics, eventgate-analytics-external, eventgate-logging-external, eventgate-main
- 16:24 otto@deploy2003: Started scap deploy-service eventgate-all: v1.38.0 for T440968
- 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{ml-serve101[2-5]*} and (A:ml-serve-master-eqiad or A:ml-serve-worker-eqiad)
- 16:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1015.eqiad.wmnet
- 16:16 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1015.eqiad.wmnet
- 16:13 ozge@deploy2003: helmfile [codfw] DONE helmfile.d/services/linkrecommendation: apply
- 16:11 ozge@deploy2003: helmfile [codfw] START helmfile.d/services/linkrecommendation: apply
- 16:10 ozge@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
- 16:10 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1015.eqiad.wmnet
- 16:08 ozge@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
- 16:06 ozge@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
- 16:05 ozge@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
- 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1015.eqiad.wmnet
- 16:05 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1014.eqiad.wmnet
- 16:05 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1014.eqiad.wmnet
- 15:53 otto@deploy2003: Finished scap deploy-service eventgate-analytics: v1.38.0 for T440968 (duration: 11m 24s)
- 15:53 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] codfw Deploying eventgate-analytics
- 15:46 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2003-2005].codfw.wmnet
- 15:46 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2003-2005].codfw.wmnet
- 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/DEFAULT] eqiad Deploying eventgate-analytics
- 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] codfw Deploying eventgate-analytics
- 15:44 otto@deploy2003: scap deploy-service eventgate-analytics: [PRODUCTION/CANARY] eqiad Deploying eventgate-analytics
- 15:42 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
- 15:42 otto@deploy2003: Started scap deploy-service eventgate-analytics: v1.38.0 for T440968
- 15:40 dancy@deploy2003: Installation of scap version "4.311.0" completed for 155 hosts
- 15:39 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.31.14
- 15:36 dancy@deploy2003: Installing scap version "4.311.0" for 155 host(s)
- 15:36 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
- 15:36 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
- 15:35 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.31.14
- 15:35 samtar@deploy2003: Finished scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.1, 1.48.0-wmf.3), Revert "Enable NewUserMessage for viwikivoyage" (duration: 33m 48s)
- 15:32 jelto@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-kubelets-inplace (exit_code=1) Upgrade kubelet of staging-codfw to 1.34.12
- 15:23 samtar@deploy2003: samtar, mlitn: Continuing with deployment
- 15:23 samtar@deploy2003: samtar, mlitn: Backport for Squashed diff to master (1.48.0-wmf.1, 1.48.0-wmf.3), Revert "Enable NewUserMessage for viwikivoyage" synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:20 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:19 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 15:16 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1014.eqiad.wmnet
- 15:12 cwilliams@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for es2035.codfw.wmnet
- 15:12 cwilliams@cumin1004: START - Cookbook sre.hosts.remove-downtime for es2035.codfw.wmnet
- 15:10 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2036.codfw.wmnet with reason: Maintenance
- 15:09 slyngshede@cumin1004: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) pool all active/active services in eqiad: Datacenter services switchover - T439676
- 15:07 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es2035.codfw.wmnet with reason: Maintenance
- 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host ml-serve1014.eqiad.wmnet
- 15:06 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host ml-serve1013.eqiad.wmnet
- 15:06 klausman@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host ml-serve1013.eqiad.wmnet
- 15:02 samtar@deploy2003: Started scap sync-world: Backport for Squashed diff to master (1.48.0-wmf.1, 1.48.0-wmf.3), Revert "Enable NewUserMessage for viwikivoyage"
- 15:00 klausman@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host ml-serve1013.eqiad.wmnet
- 14:59 samtar@deploy2003: Finished scap sync-world: Backport for Add Draft/Draft_talk namespace aliases on wikis using ns 118 as draft namespace (T440179), Add missing groupOverrides for sysop and bot on ltwiki. (T440216) (duration: 21m 43s)
- 14:00 jelto@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2001.codfw.wmnet
- 14:00 jelto@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2001.codfw.wmnet
- 13:59 jelto@cumin1004: START - Cookbook sre.k8s.upgrade-kubelets-inplace Upgrade kubelet of staging-codfw to 1.34.12
- 13:55 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271', diff saved to https://phabricator.wikimedia.org/P97137 and previous config saved to /var/cache/conftool/dbconfig/20261103-135521-ladsgroup.json
- 13:47 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
- 13:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1271 (T441633)', diff saved to https://phabricator.wikimedia.org/P97136 and previous config saved to /var/cache/conftool/dbconfig/20261103-134519-ladsgroup.json
- 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=0) Upgrade apiservers of staging-codfw to 1.34.12
- 13:42 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:42 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1271 (T441633)', diff saved to https://phabricator.wikimedia.org/P97135 and previous config saved to /var/cache/conftool/dbconfig/20261103-134118-ladsgroup.json
- 13:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 7 hosts with reason: Maintenance
- 13:40 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1271.eqiad.wmnet with reason: Maintenance
- 13:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 (T441633)', diff saved to https://phabricator.wikimedia.org/P97134 and previous config saved to /var/cache/conftool/dbconfig/20261103-134036-ladsgroup.json
- 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:34 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
- 13:34 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:34 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:33 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:33 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:31 jayme@cumin1004: END (FAIL) - Cookbook sre.k8s.upgrade-apiservers-inplace (exit_code=99) Upgrade apiservers of staging-codfw to 1.34.12
- 13:31 jayme@cumin1004: END (ERROR) - Cookbook sre.k8s.pool-depool-node (exit_code=97) depool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97133 and previous config saved to /var/cache/conftool/dbconfig/20261103-133034-ladsgroup.json
- 13:29 jnuche@deploy2003: Started scap sync-world: testwikis to 1.48.0-wmf.3 refs T442221
- 13:28 atsuko@deploy1003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-dumps: apply
- 13:27 atsuko@deploy1003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-dumps: apply
- 13:27 kamila@deploy2003: Finished scap deploy-service shellbox: update to 8.5 + new image (duration: 05m 00s)
- 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
- 13:27 bwojtowicz@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
- 13:25 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] codfw Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
- 13:23 kamila@deploy2003: scap deploy-service shellbox: [PRODUCTION/DEFAULT] eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
- 13:22 kamila@deploy2003: scap deploy-service shellbox: [STAGING/DEFAULT] staging-eqiad Deploying shellbox, shellbox-constraints, shellbox-media, shellbox-syntaxhighlight, shellbox-timeline, shellbox-video
- 13:22 kamila@deploy2003: Started scap deploy-service shellbox: update to 8.5 + new image
- 13:20 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host kubestagemaster[2004-2005].codfw.wmnet
- 13:20 jayme@cumin1004: START - Cookbook sre.k8s.upgrade-apiservers-inplace Upgrade apiservers of staging-codfw to 1.34.12
- 13:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P97132 and previous config saved to /var/cache/conftool/dbconfig/20261103-132031-ladsgroup.json
- 13:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1259 (T441633)', diff saved to https://phabricator.wikimedia.org/P97131 and previous config saved to /var/cache/conftool/dbconfig/20261103-131029-ladsgroup.json
- 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1259 (T441633)', diff saved to https://phabricator.wikimedia.org/P97130 and previous config saved to /var/cache/conftool/dbconfig/20261103-130419-ladsgroup.json
- 13:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
- 13:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 (T441633)', diff saved to https://phabricator.wikimedia.org/P97129 and previous config saved to /var/cache/conftool/dbconfig/20261103-130409-ladsgroup.json
- 12:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97128 and previous config saved to /var/cache/conftool/dbconfig/20261103-125407-ladsgroup.json
- 12:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P97127 and previous config saved to /var/cache/conftool/dbconfig/20261103-124405-ladsgroup.json
- 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1254 (T441633)', diff saved to https://phabricator.wikimedia.org/P97126 and previous config saved to /var/cache/conftool/dbconfig/20261103-123403-ladsgroup.json
- 12:32 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1003.wikimedia.org
- 12:31 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1006.eqiad.wmnet with OS trixie
- 12:28 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host irc1003.wikimedia.org
- 12:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1254 (T441633)', diff saved to https://phabricator.wikimedia.org/P97125 and previous config saved to /var/cache/conftool/dbconfig/20261103-122803-ladsgroup.json
- 12:28 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
- 12:22 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
- 12:22 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T441633)', diff saved to https://phabricator.wikimedia.org/P97124 and previous config saved to /var/cache/conftool/dbconfig/20261103-122223-ladsgroup.json
- 12:12 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
- 12:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97123 and previous config saved to /var/cache/conftool/dbconfig/20261103-121221-ladsgroup.json
- 12:07 cmooney@cumin1004: START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1006.eqiad.wmnet with reason: host reimage
- 12:05 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (remove config only)
- 12:05 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (remove config only)
- 12:02 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P97122 and previous config saved to /var/cache/conftool/dbconfig/20261103-120219-ladsgroup.json
- 11:57 ladsgroup@deploy2003: Finished scap sync-world: Backport for Add --listfile to transcode maint scripts (T435744 T373546) (1.48.0-wmf.1) (duration: 10m 58s)
- 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) sretest1006.eqiad.wmnet on all recursors
- 11:56 cmooney@cumin1004: START - Cookbook sre.dns.wipe-cache sretest1006.eqiad.wmnet on all recursors
- 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
- 11:56 cmooney@cumin1004: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
- 11:56 cmooney@cumin1004: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update entry for sretest1006 - cmooney@cumin1004"
- 11:52 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T441633)', diff saved to https://phabricator.wikimedia.org/P97121 and previous config saved to /var/cache/conftool/dbconfig/20261103-115217-ladsgroup.json
- 11:51 cmooney@cumin1004: START - Cookbook sre.dns.netbox
- 11:51 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 11:50 ladsgroup@deploy2003: ladsgroup: Backport for Add --listfile to transcode maint scripts (T435744 T373546) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:50 cmooney@cumin1004: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host sretest1006.eqiad.wmnet (push config only)
- 11:50 cmooney@cumin1004: START - Cookbook sre.hosts.dhcp for host sretest1006.eqiad.wmnet (push config only)
- 11:49 cmooney@cumin1004: START - Cookbook sre.hosts.reimage for host sretest1006.eqiad.wmnet with OS trixie
- 11:46 ladsgroup@deploy2003: Started scap sync-world: Backport for Add --listfile to transcode maint scripts (T435744 T373546) (1.48.0-wmf.1)
- 11:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1233 (T441633)', diff saved to https://phabricator.wikimedia.org/P97120 and previous config saved to /var/cache/conftool/dbconfig/20261103-114609-ladsgroup.json
- 11:46 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
- 11:45 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T441633)', diff saved to https://phabricator.wikimedia.org/P97119 and previous config saved to /var/cache/conftool/dbconfig/20261103-114548-ladsgroup.json
- 11:35 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97118 and previous config saved to /var/cache/conftool/dbconfig/20261103-113546-ladsgroup.json
- 11:30 cmooney@cumin1004: END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-d6-eqiad
- 11:29 cmooney@cumin1004: START - Cookbook sre.network.tls for network device lsw1-d6-eqiad
- 11:25 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P97117 and previous config saved to /var/cache/conftool/dbconfig/20261103-112544-ladsgroup.json
- 11:21 cwilliams@cumin1004: END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1228: Repooling
- 11:15 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T441633)', diff saved to https://phabricator.wikimedia.org/P97115 and previous config saved to /var/cache/conftool/dbconfig/20261103-111542-ladsgroup.json
- 11:12 moritzm: installing chromium security updates
- 11:09 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1229 (T441633)', diff saved to https://phabricator.wikimedia.org/P97114 and previous config saved to /var/cache/conftool/dbconfig/20261103-110944-ladsgroup.json
- 11:09 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
- 11:04 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
- 11:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97112 and previous config saved to /var/cache/conftool/dbconfig/20261103-110410-ladsgroup.json
- 10:54 ladsgroup@deploy2003: helmfile [eqiad] DONE helmfile.d/services/thumbor: apply
- 10:54 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97111 and previous config saved to /var/cache/conftool/dbconfig/20261103-105408-ladsgroup.json
- 10:53 jmm@dns1004: END - running authdns-update
- 10:53 ladsgroup@deploy2003: helmfile [eqiad] START helmfile.d/services/thumbor: apply
- 10:51 jmm@dns1004: START - running authdns-update
- 10:46 ladsgroup@deploy2003: helmfile [codfw] DONE helmfile.d/services/thumbor: apply
- 10:44 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P97109 and previous config saved to /var/cache/conftool/dbconfig/20261103-104406-ladsgroup.json
- 10:43 ladsgroup@deploy2003: helmfile [codfw] START helmfile.d/services/thumbor: apply
- 10:42 ladsgroup@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 10:42 ladsgroup@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 10:36 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Repooling
- 10:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97107 and previous config saved to /var/cache/conftool/dbconfig/20261103-103404-ladsgroup.json
- 10:33 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
- 10:33 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
- 10:31 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
- 10:31 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=eswiki --olderThan=1790175600 --verbose # T438366
- 10:31 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
- 10:30 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366 (dblist: https://phabricator.wikimedia.org/P97106)
- 10:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
- 10:30 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
- 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
- 10:29 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
- 10:27 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
- 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1222 (T441633)', diff saved to https://phabricator.wikimedia.org/P97105 and previous config saved to /var/cache/conftool/dbconfig/20261103-102742-ladsgroup.json
- 10:27 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance
- 10:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T441633)', diff saved to https://phabricator.wikimedia.org/P97104 and previous config saved to /var/cache/conftool/dbconfig/20261103-102721-ladsgroup.json
- 10:23 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
- 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=itwiki --olderThan=1790175600 --verbose # T438366
- 10:23 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
- 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=frwiki --olderThan=1790175600 --verbose # T438366
- 10:23 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
- 10:23 urbanecm@deploy2003: mwscript-k8s job started: GrowthExperiments:revalidateLinkRecommendations.php --wiki=enwiki --olderThan=1790175600 --verbose # T438366
- 10:22 moritzm: installing git-lfs security updates
- 10:22 urbanecm@deploy2003: mwscript-k8s job started: foreachwikiindblist mwscript.dblist GrowthExperiments:revalidateLinkRecommendations.php --olderThan=1790175600 --verbose # T438366 (dblist: https://phabricator.wikimedia.org/P97103)
- 10:22 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
- 10:22 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
- 10:22 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
- 10:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97102 and previous config saved to /var/cache/conftool/dbconfig/20261103-101719-ladsgroup.json
- 10:15 moritzm: installing imagemagick security updates
- 10:07 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P97101 and previous config saved to /var/cache/conftool/dbconfig/20261103-100717-ladsgroup.json
- 10:04 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
- 10:00 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
- 09:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T441633)', diff saved to https://phabricator.wikimedia.org/P97099 and previous config saved to /var/cache/conftool/dbconfig/20261103-095715-ladsgroup.json
- 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1197 (T441633)', diff saved to https://phabricator.wikimedia.org/P97098 and previous config saved to /var/cache/conftool/dbconfig/20261103-094843-ladsgroup.json
- 09:48 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
- 09:48 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T441633)', diff saved to https://phabricator.wikimedia.org/P97097 and previous config saved to /var/cache/conftool/dbconfig/20261103-094823-ladsgroup.json
- 09:38 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97096 and previous config saved to /var/cache/conftool/dbconfig/20261103-093820-ladsgroup.json
- 09:28 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P97095 and previous config saved to /var/cache/conftool/dbconfig/20261103-092818-ladsgroup.json
- 09:21 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'llm' .
- 09:18 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T441633)', diff saved to https://phabricator.wikimedia.org/P97094 and previous config saved to /var/cache/conftool/dbconfig/20261103-091816-ladsgroup.json
- 09:12 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2003.wikimedia.org
- 09:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1188 (T441633)', diff saved to https://phabricator.wikimedia.org/P97093 and previous config saved to /var/cache/conftool/dbconfig/20261103-091203-ladsgroup.json
- 09:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
- 09:08 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2003.wikimedia.org
- 09:03 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host urldownloader2004.wikimedia.org
- 09:02 Tran: Deployed patch for T441675
- 08:58 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host urldownloader2004.wikimedia.org
- 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
- 08:50 dpogorzelski@deploy2003: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
- 08:24 denisse: increasing disk size in arclamp1001
- 08:14 denisse: deleting stale .tmp logs from 2025 in arclamp2001
- 08:13 denisse: incresing disk in arclamp2001
- 05:02 mwpresync@deploy2003: Pruned MediaWiki: 1.47.0-wmf.23 (duration: 02m 26s)
- 04:40 krinkle@deploy2003: Finished scap sync-world: Backport for MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001) (duration: 24m 34s)
- 04:33 krinkle@deploy2003: krinkle: Continuing with deployment
- 04:24 krinkle@deploy2003: krinkle: Backport for MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 04:16 krinkle@deploy2003: Started scap sync-world: Backport for MathML+MathJax rollout to phase 6 (remaining large Wikipedias) (T271001)
- 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
- 01:30 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/zotero: apply
- 01:30 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
- 01:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
- 01:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
- 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
- 01:28 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
- 01:28 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
- 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/toolhub: apply
- 01:27 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/toolhub: apply
- 01:27 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
- 01:26 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
- 01:26 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
- 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
- 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply
- 01:25 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-video: apply
- 01:25 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply
- 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply
- 01:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply
- 01:24 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-media: apply
- 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply
- 01:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply
- 01:23 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: apply
- 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: apply
- 01:22 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/sessionstore: apply
- 01:22 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/sessionstore: apply
- 01:21 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/recommendation-api: apply
- 01:21 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/recommendation-api: apply
- 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/push-notifications: apply
- 01:20 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/push-notifications: apply
- 01:20 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/proton: apply
- 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/proton: apply
- 01:18 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
- 01:18 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
- 01:17 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
- 01:16 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
- 01:16 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
- 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
- 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
- 01:13 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
- 01:13 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/machinetranslation: apply
- 01:05 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/machinetranslation: apply
- 01:05 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linkrecommendation: apply
- 01:04 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linkrecommendation: apply
- 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/linked-artifacts: apply
- 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/linked-artifacts: apply
- 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/lac-lambdas: apply
- 01:03 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/lac-lambdas: apply
- 01:03 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/kartotherian: apply
- 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/kartotherian: apply
- 01:01 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
- 01:01 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
- 01:00 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
- 01:00 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
- 00:57 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
- 00:56 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
- 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
- 00:55 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
- 00:55 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/editcheck-headless: apply
- 00:54 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/editcheck-headless: apply
- 00:54 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
- 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
- 00:53 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/echostore: apply
- 00:53 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/echostore: apply
- 00:51 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/device-analytics: apply
- 00:51 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/device-analytics: apply
- 00:50 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
- 00:50 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
- 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/data-gateway: apply
- 00:49 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/data-gateway: apply
- 00:49 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cxserver: apply
- 00:48 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cxserver: apply
- 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/commons-impact-analytics: apply
- 00:47 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/commons-impact-analytics: apply
- 00:47 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
- 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/citoid: apply
- 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
- 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
- 00:46 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/chart-renderer: apply
- 00:46 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/chart-renderer: apply
- 00:41 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
- 00:41 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/apertium: apply
- 00:10 maryum: Deployed security fix for T440469, T441384, T440735, T440501, T439716
2026-11-02
- 23:24 maryum: Deployed security fix for T440502, T440330, T442830, T442861
- 22:35 cscott@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
- 22:35 cscott@deploy2003: helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
- 22:25 krinkle@deploy2003: Finished scap sync-world: Backport for Handle \mathcal in mmlFilter instead of the font (1.48.0-wmf.1), Keep \mathrm upright inside \mathbf (T442312) (1.48.0-wmf.1), Draw no outer line for a nested \hline (1.48.0-wmf.1) (duration: 13m 50s)
- 22:19 krinkle@deploy2003: krinkle: Continuing with deployment
- 22:15 krinkle@deploy2003: krinkle: Backport for Handle \mathcal in mmlFilter instead of the font (1.48.0-wmf.1), Keep \mathrm upright inside \mathbf (T442312) (1.48.0-wmf.1), Draw no outer line for a nested \hline (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 22:11 krinkle@deploy2003: Started scap sync-world: Backport for Handle \mathcal in mmlFilter instead of the font (1.48.0-wmf.1), Keep \mathrm upright inside \mathbf (T442312) (1.48.0-wmf.1), Draw no outer line for a nested \hline (1.48.0-wmf.1)
- 22:05 jdlrobson@deploy2003: Finished scap sync-world: Backport for Enable Vector CTA on all wikipedias (T441705) (duration: 13m 05s)
- 21:57 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
- 21:56 jdlrobson@deploy2003: jdlrobson: Backport for Enable Vector CTA on all wikipedias (T441705) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 21:52 jdlrobson@deploy2003: Started scap sync-world: Backport for Enable Vector CTA on all wikipedias (T441705)
- 21:50 jdlrobson@deploy2003: Finished scap sync-world: Backport for Skin: Update skin-responsive logic (T439273) (1.48.0-wmf.1), Skin: skin--responsive class no longer varies on user (T439273) (1.48.0-wmf.1), Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906) (1.48.0-wmf.1) (duration: 35m 12s)
- 21:38 jdlrobson@deploy2003: jdlrobson: Continuing with deployment
- 21:36 jdlrobson@deploy2003: jdlrobson: Backport for Skin: Update skin-responsive logic (T439273) (1.48.0-wmf.1), Skin: skin--responsive class no longer varies on user (T439273) (1.48.0-wmf.1), Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes
- 21:15 jdlrobson@deploy2003: Started scap sync-world: Backport for Skin: Update skin-responsive logic (T439273) (1.48.0-wmf.1), Skin: skin--responsive class no longer varies on user (T439273) (1.48.0-wmf.1), Fast forward wmf.1 branch to 0626e6 (T440916 T441259 T441260 T441477 T442906) (1.48.0-wmf.1)
- 20:43 cscott@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
- 20:43 cscott@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
- 20:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/miscweb: apply
- 20:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/miscweb: apply
- 20:02 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/zotero: apply
- 20:01 rzl@deploy2003: helmfile [staging] START helmfile.d/services/zotero: apply
- 20:01 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
- 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifunctions: apply
- 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
- 20:00 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikifeeds: apply
- 20:00 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
- 19:59 rzl@deploy2003: helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
- 19:59 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/toolhub: apply
- 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/toolhub: apply
- 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/thumbor: apply
- 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/thumbor: apply
- 19:58 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
- 19:58 rzl@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
- 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
- 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
- 19:57 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-video: apply
- 19:57 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-video: apply
- 19:56 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply
- 19:56 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: apply
- 19:55 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-media: apply
- 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-media: apply
- 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply
- 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-constraints: apply
- 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: apply
- 19:54 rzl@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: apply
- 19:54 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
- 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/sessionstore: apply
- 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/rest-gateway: apply
- 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/rest-gateway: apply
- 19:53 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/recommendation-api: apply
- 19:53 rzl@deploy2003: helmfile [staging] START helmfile.d/services/recommendation-api: apply
- 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/push-notifications: apply
- 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/push-notifications: apply
- 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/proton: apply
- 19:52 rzl@deploy2003: helmfile [staging] START helmfile.d/services/proton: apply
- 19:52 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
- 19:51 rzl@deploy2003: helmfile [staging] START helmfile.d/services/page-analytics: apply
- 19:51 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
- 19:50 rzl@deploy2003: helmfile [staging] START helmfile.d/services/mobileapps: apply
- 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
- 19:45 rzl@deploy2003: helmfile [staging] START helmfile.d/services/media-analytics: apply
- 19:45 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
- 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/machinetranslation: apply
- 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linkrecommendation: apply
- 19:41 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linkrecommendation: apply
- 19:41 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/linked-artifacts: apply
- 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/linked-artifacts: apply
- 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/lac-lambdas: apply
- 19:40 rzl@deploy2003: helmfile [staging] START helmfile.d/services/lac-lambdas: apply
- 19:40 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/kartotherian: apply
- 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/kartotherian: apply
- 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
- 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/geo-analytics: apply
- 19:39 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
- 19:39 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
- 19:38 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
- 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
- 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
- 19:37 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editor-analytics: apply
- 19:37 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/editcheck-headless: apply
- 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/editcheck-headless: apply
- 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
- 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/edit-analytics: apply
- 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/echostore: apply
- 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/echostore: apply
- 19:36 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/device-analytics: apply
- 19:36 rzl@deploy2003: helmfile [staging] START helmfile.d/services/device-analytics: apply
- 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
- 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/developer-portal: apply
- 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/data-gateway: apply
- 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/data-gateway: apply
- 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cxserver: apply
- 19:35 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cxserver: apply
- 19:35 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/commons-impact-analytics: apply
- 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/commons-impact-analytics: apply
- 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/citoid: apply
- 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/citoid: apply
- 19:34 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply
- 19:34 rzl@deploy2003: helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply
- 19:33 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/chart-renderer: apply
- 19:33 rzl@deploy2003: helmfile [staging] START helmfile.d/services/chart-renderer: apply
- 19:30 rzl@deploy2003: helmfile [staging] DONE helmfile.d/services/apertium: apply
- 19:29 rzl@deploy2003: helmfile [staging] START helmfile.d/services/apertium: apply
- 18:57 dancy@deploy2003: Installation of scap version "4.310.0" completed for 2 hosts
- 18:55 dancy@deploy2003: Installing scap version "4.310.0" for 2 host(s)
- 18:54 akhatun@deploy2003: Finished scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on all wikipedias (duration: 10m 43s)
- 18:48 akhatun@deploy2003: akhatun: Continuing with deployment
- 18:47 akhatun@deploy2003: akhatun: Backport for Prefetch the LAC revision_feature_counts artifact on all wikipedias synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 18:43 akhatun@deploy2003: Started scap sync-world: Backport for Prefetch the LAC revision_feature_counts artifact on all wikipedias
- 18:37 rsilvola@deploy2003: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
- 18:36 rsilvola@deploy2003: helmfile [eqiad] START helmfile.d/services/miscweb: apply
- 18:36 rsilvola@deploy2003: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
- 18:36 rsilvola@deploy2003: helmfile [codfw] START helmfile.d/services/miscweb: apply
- 18:31 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply
- 18:31 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply
- 18:30 btullis@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-presto1015.eqiad.wmnet with reason: Recreate data RAID without failed disk
- 18:29 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-videoscaler: apply
- 18:29 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-videoscaler: apply
- 18:25 rzl@deploy2003: helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply
- 18:24 rzl@deploy2003: helmfile [codfw] START helmfile.d/services/rest-gateway: apply
- 18:24 rzl@deploy2003: helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply
- 18:23 rzl@deploy2003: helmfile [eqiad] START helmfile.d/services/rest-gateway: apply
- 18:18 rzl@deploy2003: Finished scap sync-world: https://gerrit.wikimedia.org/r/1360005 T436338 (duration: 07m 48s)
- 18:11 rzl@deploy2003: Started scap sync-world: https://gerrit.wikimedia.org/r/1360005 T436338
- 18:03 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
- 18:03 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
- 18:03 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
- 18:02 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
- 18:02 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
- 18:02 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
- 17:59 ladsgroup@deploy2003: Finished scap sync-world: Backport for Adjust circuit breaking thresholds (T443274), rdbms: Allow overriding of load monitor per lb cluster (T443274) (1.48.0-wmf.1) (duration: 11m 51s)
- 17:54 jgiannelos@deploy1003: helmfile [codfw] DONE helmfile.d/services/tegola-vector-tiles: apply
- 17:53 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 17:53 jgiannelos@deploy1003: helmfile [codfw] START helmfile.d/services/tegola-vector-tiles: apply
- 17:53 jgiannelos@deploy1003: helmfile [eqiad] DONE helmfile.d/services/tegola-vector-tiles: apply
- 17:52 jgiannelos@deploy1003: helmfile [eqiad] START helmfile.d/services/tegola-vector-tiles: apply
- 17:51 ladsgroup@deploy2003: ladsgroup: Backport for Adjust circuit breaking thresholds (T443274), rdbms: Allow overriding of load monitor per lb cluster (T443274) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 17:50 otto@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
- 17:49 otto@deploy2003: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
- 17:49 otto@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
- 17:48 otto@deploy2003: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
- 17:47 ladsgroup@deploy2003: Started scap sync-world: Backport for Adjust circuit breaking thresholds (T443274), rdbms: Allow overriding of load monitor per lb cluster (T443274) (1.48.0-wmf.1)
- 17:46 jgiannelos@deploy1003: helmfile [staging] DONE helmfile.d/services/tegola-vector-tiles: apply
- 17:46 jgiannelos@deploy1003: helmfile [staging] START helmfile.d/services/tegola-vector-tiles: apply
- 17:45 dancy: Investigating spiderpig
- 17:41 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
- 17:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 (T441633)', diff saved to https://phabricator.wikimedia.org/P97090 and previous config saved to /var/cache/conftool/dbconfig/20261102-174043-ladsgroup.json
- 17:32 otto@deploy2003: scap deploy-service eventgate-analytics: [STAGING/DEFAULT] staging-eqiad Deploying eventgate-analytics
- 17:32 otto@deploy2003: Started scap deploy-service eventgate-analytics: Time to persist metric - T440968
- 17:30 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97089 and previous config saved to /var/cache/conftool/dbconfig/20261102-173041-ladsgroup.json
- 17:20 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289', diff saved to https://phabricator.wikimedia.org/P97088 and previous config saved to /var/cache/conftool/dbconfig/20261102-172039-ladsgroup.json
- 17:12 ladsgroup@deploy2003: Finished scap sync-world: Backport for Temporarily increase circuit breaking (T443274) (duration: 13m 39s)
- 17:10 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1289 (T441633)', diff saved to https://phabricator.wikimedia.org/P97087 and previous config saved to /var/cache/conftool/dbconfig/20261102-171037-ladsgroup.json
- 17:07 cwilliams@cumin1004: END (ERROR) - Cookbook sre.mysql.pool (exit_code=97) pool db1228: Maintenance
- 17:07 cwilliams@cumin1004: START - Cookbook sre.mysql.pool pool db1228: Maintenance
- 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1289 (T441633)', diff saved to https://phabricator.wikimedia.org/P97086 and previous config saved to /var/cache/conftool/dbconfig/20261102-170629-ladsgroup.json
- 17:06 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1289.eqiad.wmnet with reason: Maintenance
- 17:06 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 (T441633)', diff saved to https://phabricator.wikimedia.org/P97085 and previous config saved to /var/cache/conftool/dbconfig/20261102-170609-ladsgroup.json
- 17:06 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 17:03 ladsgroup@deploy2003: ladsgroup: Backport for Temporarily increase circuit breaking (T443274) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 16:59 ladsgroup@deploy2003: Started scap sync-world: Backport for Temporarily increase circuit breaking (T443274)
- 16:56 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97084 and previous config saved to /var/cache/conftool/dbconfig/20261102-165607-ladsgroup.json
- 16:46 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280', diff saved to https://phabricator.wikimedia.org/P97083 and previous config saved to /var/cache/conftool/dbconfig/20261102-164605-ladsgroup.json
- 16:36 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1280 (T441633)', diff saved to https://phabricator.wikimedia.org/P97082 and previous config saved to /var/cache/conftool/dbconfig/20261102-163602-ladsgroup.json
- 16:32 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1253.eqiad.wmnet
- 16:32 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1253.eqiad.wmnet
- 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1280 (T441633)', diff saved to https://phabricator.wikimedia.org/P97081 and previous config saved to /var/cache/conftool/dbconfig/20261102-163154-ladsgroup.json
- 16:31 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1280.eqiad.wmnet with reason: Maintenance
- 16:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 (T441633)', diff saved to https://phabricator.wikimedia.org/P97080 and previous config saved to /var/cache/conftool/dbconfig/20261102-163134-ladsgroup.json
- 16:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97079 and previous config saved to /var/cache/conftool/dbconfig/20261102-162132-ladsgroup.json
- 16:19 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1253.eqiad.wmnet
- 16:18 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1253.eqiad.wmnet
- 16:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272', diff saved to https://phabricator.wikimedia.org/P97078 and previous config saved to /var/cache/conftool/dbconfig/20261102-161129-ladsgroup.json
- 16:04 volans@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2036.codfw.wmnet with reason: host crashed
- 16:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1272 (T441633)', diff saved to https://phabricator.wikimedia.org/P97077 and previous config saved to /var/cache/conftool/dbconfig/20261102-160127-ladsgroup.json
- 15:57 tchin@deploy2003: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
- 15:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1272 (T441633)', diff saved to https://phabricator.wikimedia.org/P97076 and previous config saved to /var/cache/conftool/dbconfig/20261102-155719-ladsgroup.json
- 15:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1272.eqiad.wmnet with reason: Maintenance
- 15:57 tchin@deploy2003: helmfile [codfw] START helmfile.d/services/eventstreams: apply
- 15:56 tchin@deploy2003: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
- 15:56 tchin@deploy2003: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
- 15:54 tchin@deploy2003: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
- 15:54 tchin@deploy2003: helmfile [staging] START helmfile.d/services/eventstreams: apply
- 15:53 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1265.eqiad.wmnet with reason: Maintenance
- 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/eventstreams-internal: apply
- 15:51 tchin@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/eventstreams-internal: apply
- 15:50 volans@cumin1004: END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool es2036: Host crashed
- 15:50 volans@cumin1004: START - Cookbook sre.mysql.depool depool es2036: Host crashed
- 15:47 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
- 15:47 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T441633)', diff saved to https://phabricator.wikimedia.org/P97074 and previous config saved to /var/cache/conftool/dbconfig/20261102-154733-ladsgroup.json
- 15:37 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97073 and previous config saved to /var/cache/conftool/dbconfig/20261102-153731-ladsgroup.json
- 15:31 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6] (duration: 06m 29s)
- 15:27 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P97072 and previous config saved to /var/cache/conftool/dbconfig/20261102-152728-ladsgroup.json
- 15:24 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a]: Regular analytics weekly train [analytics/refinery@bbf3d3a6]
- 15:18 aqu@deploy2003: Finished deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6] (duration: 00m 44s)
- 15:17 aqu@deploy2003: Started deploy [analytics/refinery@bbf3d3a] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bbf3d3a6]
- 15:17 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T441633)', diff saved to https://phabricator.wikimedia.org/P97070 and previous config saved to /var/cache/conftool/dbconfig/20261102-151726-ladsgroup.json
- 15:13 Lucas_WMDE: UTC afternoon backport+config window done
- 15:13 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for eswiki: Add protect user right to botadmin group (T443124), GrowthExperiments: enable SpecialHome on testwiki (T442733) (duration: 14m 35s)
- 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1223 (T441633)', diff saved to https://phabricator.wikimedia.org/P97069 and previous config saved to /var/cache/conftool/dbconfig/20261102-151119-ladsgroup.json
- 15:11 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1223.eqiad.wmnet with reason: Maintenance
- 15:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T441633)', diff saved to https://phabricator.wikimedia.org/P97068 and previous config saved to /var/cache/conftool/dbconfig/20261102-151059-ladsgroup.json
- 15:05 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Continuing with deployment
- 15:03 lucaswerkmeister-wmde@deploy2003: sgimeno, lucaswerkmeister-wmde, tryvix1509: Backport for eswiki: Add protect user right to botadmin group (T443124), GrowthExperiments: enable SpecialHome on testwiki (T442733) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 15:00 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97067 and previous config saved to /var/cache/conftool/dbconfig/20261102-150057-ladsgroup.json
- 14:59 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for eswiki: Add protect user right to botadmin group (T443124), GrowthExperiments: enable SpecialHome on testwiki (T442733)
- 14:50 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P97066 and previous config saved to /var/cache/conftool/dbconfig/20261102-145054-ladsgroup.json
- {{safesubst:SAL entry|1=14:47 lucaswerkmeister-wmde@deploy2003: Finished scap sync-world: Backport for Set license acknowledgement cookie for MEX cypress tests (T441916) (1.48.0-wmf.1), Fix incomplete settings in NewEntitySchemaTest (T441916) (1.48.0-wmf.1), Set license acknowledgement cookie for MEX cypress tests (T441916) (1.48.0-wmf.1), [[gerrit:1361378|Set license acknowledgement cookie}}
- 14:40 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T441633)', diff saved to https://phabricator.wikimedia.org/P97065 and previous config saved to /var/cache/conftool/dbconfig/20261102-144052-ladsgroup.json
- 14:36 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1252.eqiad.wmnet
- 14:36 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1252.eqiad.wmnet
- 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1212 (T441633)', diff saved to https://phabricator.wikimedia.org/P97064 and previous config saved to /var/cache/conftool/dbconfig/20261102-143454-ladsgroup.json
- 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance
- 14:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance
- 14:34 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Continuing with deployment
- 14:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T441633)', diff saved to https://phabricator.wikimedia.org/P97063 and previous config saved to /var/cache/conftool/dbconfig/20261102-143413-ladsgroup.json
- {{safesubst:SAL entry|1=14:29 lucaswerkmeister-wmde@deploy2003: mlitn, audreypenven, lucaswerkmeister-wmde: Backport for Set license acknowledgement cookie for MEX cypress tests (T441916) (1.48.0-wmf.1), Fix incomplete settings in NewEntitySchemaTest (T441916) (1.48.0-wmf.1), Set license acknowledgement cookie for MEX cypress tests (T441916) (1.48.0-wmf.1), [[gerrit:1361378|Set license ackno}}
- 14:24 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97062 and previous config saved to /var/cache/conftool/dbconfig/20261102-142411-ladsgroup.json
- 14:14 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P97061 and previous config saved to /var/cache/conftool/dbconfig/20261102-141408-ladsgroup.json
- 14:11 sukhe@puppetserver1001: conftool action : set/pooled=yes; selector: name=dns7001.* [reason: [end] testing pdns-rec 5.4.9]
- {{safesubst:SAL entry|1=14:10 lucaswerkmeister-wmde@deploy2003: Started scap sync-world: Backport for Set license acknowledgement cookie for MEX cypress tests (T441916) (1.48.0-wmf.1), Fix incomplete settings in NewEntitySchemaTest (T441916) (1.48.0-wmf.1), Set license acknowledgement cookie for MEX cypress tests (T441916) (1.48.0-wmf.1), [[gerrit:1361378|Set license acknowledgement cookie f}}
- 14:07 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1252.eqiad.wmnet
- 14:06 sukhe@puppetserver1001: conftool action : set/pooled=no; selector: name=dns7001.* [reason: testing pdns-rec 5.4.9]
- 14:06 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1252.eqiad.wmnet
- 14:04 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T441633)', diff saved to https://phabricator.wikimedia.org/P97060 and previous config saved to /var/cache/conftool/dbconfig/20261102-140406-ladsgroup.json
- 14:02 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1251.eqiad.wmnet
- 14:02 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1251.eqiad.wmnet
- 13:59 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-codfw
- 13:57 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1198 (T441633)', diff saved to https://phabricator.wikimedia.org/P97059 and previous config saved to /var/cache/conftool/dbconfig/20261102-135744-ladsgroup.json
- 13:57 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1198.eqiad.wmnet with reason: Maintenance
- 13:55 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe] (duration: 04m 46s)
- 13:52 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance
- 13:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97057 and previous config saved to /var/cache/conftool/dbconfig/20261102-135145-ladsgroup.json
- 13:50 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f]: Regular analytics weekly train [analytics/refinery@2226a9fe]
- 13:50 aqu@deploy2003: Finished deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe] (duration: 00m 47s)
- 13:49 aqu@deploy2003: Started deploy [analytics/refinery@2226a9f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2226a9fe]
- 13:45 cmooney@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker1251.eqiad.wmnet
- 13:44 cmooney@cumin1004: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker1251.eqiad.wmnet
- 13:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97056 and previous config saved to /var/cache/conftool/dbconfig/20261102-134143-ladsgroup.json
- 13:31 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P97055 and previous config saved to /var/cache/conftool/dbconfig/20261102-133140-ladsgroup.json
- 13:26 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb1003.eqiad.wmnet
- 13:22 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb1003.eqiad.wmnet
- 13:21 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97054 and previous config saved to /var/cache/conftool/dbconfig/20261102-132138-ladsgroup.json
- 13:12 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97052 and previous config saved to /var/cache/conftool/dbconfig/20261102-131205-ladsgroup.json
- 13:12 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance
- 13:11 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T441633)', diff saved to https://phabricator.wikimedia.org/P97051 and previous config saved to /var/cache/conftool/dbconfig/20261102-131155-ladsgroup.json
- 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'llmisvc' .
- 13:07 ozge@deploy2003: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'llm' for release 'main' .
- 13:05 cwilliams@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1217.eqiad.wmnet with reason: Maintenance
- 13:02 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-codfw
- 13:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
- 13:01 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97050 and previous config saved to /var/cache/conftool/dbconfig/20261102-130153-ladsgroup.json
- 12:56 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
- 12:53 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
- 12:51 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P97049 and previous config saved to /var/cache/conftool/dbconfig/20261102-125151-ladsgroup.json
- 12:48 cgoubert@deploy2003: Unlocked for deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot (duration: 14m 21s)
- 12:47 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
- 12:47 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode2001.codfw.wmnet
- 12:46 jayme@cumin1004: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-main-eqiad
- 12:43 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode2001.codfw.wmnet
- 12:41 ladsgroup@cumin1004: dbctl commit (dc=all): 'Repooling after maintenance db1157 (T441633)', diff saved to https://phabricator.wikimedia.org/P97048 and previous config saved to /var/cache/conftool/dbconfig/20261102-124149-ladsgroup.json
- 12:40 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2002.codfw.wmnet
- 12:38 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dragonfly-supernode1001.eqiad.wmnet
- 12:38 cmooney@cumin1004: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
- 12:37 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
- 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on A:wikikube-master-eqiad
- 12:37 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1006.eqiad.wmnet
- 12:37 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1006.eqiad.wmnet
- 12:34 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2002.codfw.wmnet
- 12:34 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf2001.codfw.wmnet
- 12:34 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host dragonfly-supernode1001.eqiad.wmnet
- 12:34 ladsgroup@cumin1004: dbctl commit (dc=all): 'Depooling db1157 (T441633)', diff saved to https://phabricator.wikimedia.org/P97046 and previous config saved to /var/cache/conftool/dbconfig/20261102-123404-ladsgroup.json
- 12:34 ladsgroup@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
- 12:33 cgoubert@deploy2003: Locking from deployment [ALL REPOSITORIES]: Dragonfly supernodes reboot
- 12:31 cmooney@cumin1004: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
- 12:31 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
- 12:30 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
- 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1006.eqiad.wmnet
- 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1006.eqiad.wmnet
- 12:30 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
- 12:30 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1005.eqiad.wmnet
- 12:30 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1005.eqiad.wmnet
- 12:30 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
- 12:30 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
- 12:29 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
- 12:29 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=eqiad
- 12:28 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum1001.eqiad.wmnet
- 12:27 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf2001.codfw.wmnet
- 12:24 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum1001.eqiad.wmnet
- 12:23 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=eqiad
- 12:23 cgoubert@cumin2003: conftool action : set/pooled=true; selector: dnsdisc=helm-charts.*,name=codfw
- 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host chartmuseum2001.codfw.wmnet
- 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1005.eqiad.wmnet
- 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1005.eqiad.wmnet
- 12:23 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1004.eqiad.wmnet
- 12:23 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1004.eqiad.wmnet
- 12:20 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
- 12:19 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1002.eqiad.wmnet
- 12:19 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
- 12:19 cgoubert@cumin2003: START - Cookbook sre.hosts.reboot-single for host chartmuseum2001.codfw.wmnet
- 12:19 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
- 12:19 cgoubert@cumin2003: conftool action : set/pooled=false; selector: dnsdisc=helm-charts.*,name=codfw
- 12:18 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
- 12:18 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
- 12:18 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
- 12:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host rpki1001.eqiad.wmnet with OS trixie
- 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1004.eqiad.wmnet
- 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1004.eqiad.wmnet
- 12:15 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1003.eqiad.wmnet
- 12:15 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1003.eqiad.wmnet
- 12:15 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1002.eqiad.wmnet
- 12:14 jayme@cumin1004: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker1250.eqiad.wmnet
- 12:14 jayme@cumin1004: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker1250.eqiad.wmnet
- 12:12 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1002.eqiad.wmnet
- 12:12 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-wf1001.eqiad.wmnet
- 12:09 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1002.eqiad.wmnet
- 12:09 jiji@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mc-misc1001.eqiad.wmnet
- 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1003.eqiad.wmnet
- 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1003.eqiad.wmnet
- 12:08 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-ctrl1002.eqiad.wmnet
- 12:08 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-ctrl1002.eqiad.wmnet
- 12:06 blake@deploy2003: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
- 12:06 blake@deploy2003: helmfile [eqiad] START helmfile.d/services/changeprop: apply
- 12:06 blake@deploy2003: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
- 12:06 blake@deploy2003: helmfile [codfw] START helmfile.d/services/changeprop: apply
- 12:05 blake@deploy2003: helmfile [staging] DONE helmfile.d/services/changeprop: apply
- 12:05 blake@deploy2003: helmfile [staging] START helmfile.d/services/changeprop: apply
- 12:04 jayme@cumin1004: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-main-eqiad
- 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-wf1001.eqiad.wmnet
- 12:04 jiji@cumin1004: START - Cookbook sre.hosts.reboot-single for host mc-misc1001.eqiad.wmnet
- 12:03 cmooney@cumin1004: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
- 12:02 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
- 12:00 cgoubert@cumin2003: END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-ctrl1002.eqiad.wmnet
- 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-ctrl1002.eqiad.wmnet
- 12:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 cgoubert@cumin2003: START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:wikikube-master-eqiad
- 12:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 11:59 blake@deploy2003: Finished scap sync-world: Non-build deploy to bump chart version (T434925) (duration: 07m 48s)
- 11:57 jmm@cumin2003: START - Cookbook sre.hosts.downtime for 2:00:00 on rpki1001.eqiad.wmnet with reason: host reimage
- 11:56 blake@deploy2003: Started scap sync-world: Non-build deploy to bump chart version (T434925)
- 11:55 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (remove config only)
- 11:55 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (remove config only)
- 11:49 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.dhcp (exit_code=0) for host rpki1001.eqiad.wmnet (push config only)
- 11:49 jmm@cumin2003: START - Cookbook sre.hosts.dhcp for host rpki1001.eqiad.wmnet (push config only)
- 11:48 jmm@cumin2003: START - Cookbook sre.hosts.reimage for host rpki1001.eqiad.wmnet with OS trixie
- 11:47 ladsgroup@deploy2003: Finished scap sync-world: Backport for Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885) (1.48.0-wmf.1) (duration: 10m 56s)
- 11:44 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-gutter-eqiad
- 11:41 ladsgroup@deploy2003: ladsgroup, krinkle: Continuing with deployment
- 11:41 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for kafka-main1008.eqiad.wmnet
- 11:41 jayme@cumin1004: START - Cookbook sre.hosts.remove-downtime for kafka-main1008.eqiad.wmnet
- 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts kafka-main1008.eqiad.wmnet
- 11:40 jayme@cumin1004: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-main1008.eqiad.wmnet
- 11:40 ladsgroup@deploy2003: ladsgroup, krinkle: Backport for Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 11:39 Amir1: dropped links tables from db1160 (s4 master in eqiad): T437278
- 11:36 ladsgroup@deploy2003: Started scap sync-world: Backport for Rdbms: Bump the LoadMonitor APcu ttl to 15s (from 10s) (T440885) (1.48.0-wmf.1)
- 11:35 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-gutter-eqiad
- 11:33 cgoubert@cumin2003: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wikikube-worker2337.codfw.wmnet with reason: hardware/networking issues
- 11:27 jayme@cumin1004: START - Cookbook sre.hosts.reboot-single for host kafka-main1008.eqiad.wmnet
- 11:27 jayme@cumin1004: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts kafka-main1008.eqiad.wmnet
- 11:16 jayme@cumin1004: DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on kafka-main1008.eqiad.wmnet with reason: firmware upgrade
- 10:57 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 4788
- 10:56 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 4788
- 10:56 ladsgroup@deploy2003: Finished scap sync-world: Backport for Populate LinkCache when linking referenced entities (T428512) (1.48.0-wmf.1) (duration: 15m 43s)
- 10:55 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 13359
- 10:55 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 13359
- 10:54 ayounsi@cumin1004: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 40317
- 10:54 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 40317
- 10:51 ayounsi@cumin1004: END (FAIL) - Cookbook sre.network.peering (exit_code=99) with action 'email' for AS: 396993
- 10:51 ayounsi@cumin1004: START - Cookbook sre.network.peering with action 'email' for AS: 396993
- 10:50 ladsgroup@deploy2003: ladsgroup: Continuing with deployment
- 10:44 ladsgroup@deploy2003: ladsgroup: Backport for Populate LinkCache when linking referenced entities (T428512) (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 10:40 ladsgroup@deploy2003: Started scap sync-world: Backport for Populate LinkCache when linking referenced entities (T428512) (1.48.0-wmf.1)
- 10:01 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netboxdb2003.codfw.wmnet
- 09:57 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host netboxdb2003.codfw.wmnet
- 09:53 jiji@cumin1004: END (PASS) - Cookbook sre.memcached.roll-reboot-restart (exit_code=0) rolling reboot on A:memcached-eqiad
- 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-main: apply
- 09:50 aqu@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-main: apply
- 09:25 arthurtaylor@deploy2003: Finished scap sync-world: Backport for Turn on Wikibase hCaptcha flag for Beta (T440832) (duration: 30m 34s)
- 09:21 jiji@cumin1004: START - Cookbook sre.memcached.roll-reboot-restart rolling reboot on A:memcached-eqiad
- 09:17 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1006.eqiad.wmnet
- 09:17 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Continuing with deployment
- 09:14 jmm@cumin2003: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-replica2008.wikimedia.org
- 09:11 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host sretest1006.eqiad.wmnet
- 09:10 brouberol@dns1004: END - running authdns-update
- 09:10 jmm@cumin2003: START - Cookbook sre.hosts.reboot-single for host ldap-replica2008.wikimedia.org
- 09:08 brouberol@dns1004: START - running authdns-update
- 09:00 arthurtaylor@deploy2003: arthurtaylor, mahmoud-abdelsattar: Backport for Turn on Wikibase hCaptcha flag for Beta (T440832) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:54 arthurtaylor@deploy2003: Started scap sync-world: Backport for Turn on Wikibase hCaptcha flag for Beta (T440832)
- 08:51 brouberol@deploy2003: helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
- 08:50 brouberol@deploy2003: helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
- 08:50 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
- 08:49 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
- 08:49 wmde-fisch@deploy2003: Finished scap sync-world: Backport for Enable sub-references globally (T440304) (duration: 36m 41s)
- 08:36 wmde-fisch@deploy2003: wmde-fisch: Continuing with deployment
- 08:33 wmde-fisch@deploy2003: wmde-fisch: Backport for Enable sub-references globally (T440304) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
- 08:21 brouberol@deploy2003: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
- 08:19 brouberol@deploy2003: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
- 08:12 wmde-fisch@deploy2003: Started scap sync-world: Backport for Enable sub-references globally (T440304)
- 07:58 jmm@cumin2003: DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Elukey out of all services on: 2438 hosts
- 07:29 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
- 07:29 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
- 07:28 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
- 07:27 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
- 07:27 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
- 07:27 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
- 07:24 arthurtaylor@deploy2003: helmfile [eqiad] DONE helmfile.d/services/termbox: apply
- 07:24 arthurtaylor@deploy2003: helmfile [eqiad] START helmfile.d/services/termbox: apply
- 07:24 arthurtaylor@deploy2003: helmfile [codfw] DONE helmfile.d/services/termbox: apply
- 07:24 arthurtaylor@deploy2003: helmfile [codfw] START helmfile.d/services/termbox: apply
- 07:23 arthurtaylor@deploy2003: helmfile [staging] DONE helmfile.d/services/termbox: apply
- 07:23 arthurtaylor@deploy2003: helmfile [staging] START helmfile.d/services/termbox: apply
- 05:49 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
- 05:49 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
- 05:48 ssastry@deploy2003: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
- 05:47 ssastry@deploy2003: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
- 02:08 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 08m 07s)
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
2026-11-01
- 12:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 12:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 12:01 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: sync
- 12:01 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 12:00 root@deploy2003: helmfile [eqiad] DONE helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [eqiad] START helmfile.d/services/shellbox: sync
- 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] DONE helmfile.d/aux-k8s-services/jaeger: sync
- 12:00 root@deploy2003: helmfile [aux-k8s-eqiad] START helmfile.d/aux-k8s-services/jaeger: sync
- 02:07 mwpresync@deploy2003: Finished scap build-images: Publishing wmf/next image (duration: 07m 31s)
- 02:00 mwpresync@deploy2003: Started scap build-images: Publishing wmf/next image
- 00:01 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync
- 00:01 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync
- 00:01 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox-timeline: sync
- 00:01 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox-timeline: sync
- 00:00 root@deploy2003: helmfile [codfw] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [codfw] START helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] DONE helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [staging] START helmfile.d/services/shellbox: sync
- 00:00 root@deploy2003: helmfile [aux-k8s-codfw] DONE helmfile.d/aux-k8s-services/jaeger: sync
- 00:00 root@deploy2003: helmfile [aux-k8s-codfw] START helmfile.d/aux-k8s-services/jaeger: sync