You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Server Admin Log

From Wikitech-static
Revision as of 00:21, 3 January 2019 by imported>Stashbot (mutante: notebook1004 - started nagios-nrpe-server one more time)
Jump to navigation Jump to search

2019-01-03

  • 00:21 mutante: notebook1004 - started nagios-nrpe-server one more time

2019-01-02

  • 23:59 mutante: notebook1004 still keeps running out of memory from some user actions and that kills nagios-nrpe-server and that causes a bunch of Icinga alerts
  • 23:39 mutante: notebook1004 - systemctl start nagios-nrpe-server
  • 23:39 mutante: notebook1004 - systemctl status nagios-nrpe-server
  • 20:59 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=parsoid,service=parsoid,name=wtp1028.eqiad.wmnet
  • 20:59 herron: repooling wtp1028 T212624
  • 20:52 herron: rebooting wtp1028 — looking for POST errors T212624
  • 20:05 Krinkle: mwmaint1002: foreachwikiindblist s5 deleteEqualMessages.php
  • 20:04 Krinkle: mwmaint1002: foreachwikiindblist s2 deleteEqualMessages.php
  • 18:35 volans: restarting icinga on icinga1001 T212669
  • 16:50 XioNoX: create BGP sessions to AS3214 in AMS-IX
  • 16:46 XioNoX: remove BGP sessions to AS42949 in AMS-IX (leaving the IX)
  • 16:43 XioNoX: remove BGP sessions to AS6866 in AMS-IX (leaving the IX)
  • 16:33 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1090:3317 after schema change - T85757 (duration: 00m 46s)
  • 16:30 arturo: reimaging cloudvirt1030 with stretch, server cleanup after puppet refactoring
  • 16:29 moritzm: restarting Superset to pick up openssl security update
  • 16:25 moritzm: restarting Hue to pick up openssl security update
  • 16:23 arturo: T212302 re-enable puppet in all {cloud,lab}virt* servers, all was fine
  • 16:22 banyek: repooling db1090:3317 after schema change (T85757)
  • 16:11 arturo: T212302 disable puppet in all {cloud,lab}virt* servers to merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/481194/
  • 15:39 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1090:3317 for schema change - T85757 (duration: 00m 44s)
  • 15:34 moritzm: installing OpenSSL security updates
  • 15:31 banyek: depooling db1090:3317 for schema change (T85757)
  • 15:13 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: repool db1086 after schema change - T85757 (duration: 00m 44s)
  • 15:07 banyek: repooling db1086 after schema change (T85757)
  • 14:49 banyek: executing schema change on db1086 - T85757
  • 14:48 moritzm: installing ghostscript security update for jessie
  • 14:47 banyek@deploy1001: Synchronized wmf-config/db-eqiad.php: depool db1086 for schema change - T85757 (duration: 00m 45s)
  • 14:38 banyek: depooling db1086 for schema change (T85757)
  • 14:15 ema: cp hosts: upgrade OpenSSL from 1.1.0f to 1.1.0j
  • 13:39 moritzm: installing ghostscript update for stretch
  • 13:33 moritzm: installing libav security updates
  • 13:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 T86338 T202167 (duration: 00m 44s)
  • 13:17 moritzm: installing openjpeg2 security updates
  • 13:17 banyek: executing schema change on db2040 (s7 codfw master) replication lag could be expected on codfw - T85757
  • 13:13 banyek: stopping replication on db2077 prior to executing schema change on codfw s7 master (db2040) - T85757
  • 13:06 marostegui: Deploy schema change on db1119 - T86338 T202167
  • 13:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 T86338 T202167 (duration: 00m 45s)
  • 13:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 T86338 T202167 (duration: 00m 47s)
  • 12:00 moritzm: rebooting labtestpuppetmaster2001 for kernel security update
  • 11:53 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe1006.eqiad.wmnet
  • 11:51 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1006.eqiad.wmnet
  • 11:50 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe1006.codfw.wmnet
  • 11:46 ema: replace TLS certificates on ms-fe eqiad hosts T212215
  • 11:41 moritzm: rebooting labtestweb2001 for kernel security update
  • 11:24 marostegui: Deploy schema change on db1099:3311 - T86338 T202167
  • 11:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 T86338 T202167 (duration: 00m 45s)
  • 11:17 ema@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ms-fe2006.codfw.wmnet
  • 11:10 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=ms-fe2006.codfw.wmnet
  • 10:59 ema: replace TLS certificates on ms-fe codfw hosts T212215
  • 10:52 moritzm: rebooting centrallog1001 for kernel security update
  • 10:48 volans: testing the new spicerack package on cumin2001, in the unlikely event you need to use spicerack cookbooks today please use cumin1001
  • 10:45 godog: ms-be2018 Flashing Smart Array P840 in Slot 3 [ 3.00 -> 6.60 ]
  • 10:43 moritzm: removed labvirt1013 from debmonitor, got renamed in T212513
  • 10:42 volans: uploaded spicerack_0.0.10-1_amd64.deb to apt.wikimedia.org stretch-wikimedia
  • 10:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2096 (duration: 00m 44s)
  • 09:50 marostegui: Stop MySQL on db2096 for kernel and mysql upgrade
  • 09:49 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2096 (duration: 00m 45s)
  • 09:48 marostegui@deploy1001: sync-file aborted: Depool db2096 (duration: 00m 01s)
  • 09:18 moritzm: installing c3p0 security updates
  • 09:07 Zoranzoki21: Drop valid_tag from s8 by Marostegui - T212254
  • 09:06 godog: eqiad-prod: final weight for ms-be10[44-50].eqiad.wmnet - T209618
  • 08:56 moritzm: installing libarchive security updates
  • 07:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 - T212692 (duration: 00m 46s)
  • 07:30 marostegui: Fix login.logging table on db1078 - T212692
  • 07:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 - T212692 (duration: 00m 47s)
  • 07:01 marostegui: Deploy schema change on s1 codfw master (lag will be generated on s1 codfw) - T202167 T86338
  • 06:54 marostegui: Drop empty valid_tag table from labswiki labtestwiki - T212254
  • 06:49 marostegui: Drop empty valid_tag table from s5 - T212254
  • 06:25 marostegui: Drop valid_tag from s6 - T212254
  • 06:15 marostegui: Fix last chunks on db1124:338 - T212574


Archives

See Server admin log/Archives.