You are browsing a read-only backup copy of Wikitech. The live site can be found at

Incident documentation meeting/QR201407/group2/notes

From Wikitech-static
< Incident documentation meeting/QR201407‎ | group2
Revision as of 22:13, 2 April 2020 by imported>Krinkle (Krinkle moved page Incident documentation/QR201407/group2/notes to Incident documentation meeting/QR201407/group2/notes: rename so that prefixindex/from works without these showing up as sorting "after" 2020)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

See also:


Bryan, Reedy

  • PrivateSettings.php should be in a repo so we can be sure what's changed.
    • (Not Gerrit obvs, maybe living same place as private ops repo or similar?) (reedy)
      • Local on tin in a sub directory I guess. Properly backed up (hashar)
    • AdminSettings.php has the same problem bd808 thinks (bryan)
  • Changes made should go out immediately as they do for all configuration files.
  • Db user and password settings should perhaps go into PrivateSettings (and not be removed from AdminSettings until anyone relying on that file has converted their jobs).
    • Migration
      • Move AdminSettings.php contents into PrivateSettings.php
      • Symlink AdminSettings.php to PrivateSettings.php (saves keeping 2 copies of passwords around, but keeps target file in place till known to be perfectly unused)
    • Only (obvious) usage would seem to be snapshots - "Fixed" in (stuff needs moving to PrivateSettings.php first)

reedy@ubuntu64-web-esxi:~/git/operations/puppet$ grep -R AdminSettings * modules/snapshot/templates/ modules/snapshot/templates/wikidump.conf.erb:adminsettings=wmf-config/AdminSettings.php modules/snapshot/templates/addschanges.conf.erb:adminsettings=wmf-config/AdminSettings.php modules/snapshot/templates/wq.conf.erb:adminsettings=wmf-config/AdminSettings.php

  • Better coordination (dunno how to implement this one).
    • Header comment in PrivateSettings to tell people to commit (when it's in a repo) and SYNC after changing


Bryan, Reedy (bd808) basically had to run scap twice. updated

  • go back to the testwikipedia removal discussion (GREG)
    • bug 43722


Antoine, Faidon

  • $wgSquidServersNoPurge needed to include the varnish hosts for images
  • due to a change by Brandon ($wgSquidServersNoPurge now uses ranges instead of unique IP), the only times this should need to be updated will be with new rows/DCs/etc, hopefully
  • we still need some sort of auto-graph-watching that notices anomalies/slope changes/etc



  • Tracking bug for the outage: bug 65424
  • Monitor for anomalies/spikes in read failures of memcached
    • bugzilla:67817


Ori, Reedy

  • we really need to get rid of wikimedia-task-appserver debian package in favor of puppet
  • Reedy will update status, if this even makes sense any more ;)


Andrew O. Most things are done. Two monitoring related RTs to resolve:



  • add schema change test on beta cluster bug
  • still need an arch review?


Andrew B. / Marc-André

  • Need RT filled to add an icinga monitoring of puppet status on wikitech host
    • RT 7842