You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
Memcached for MediaWiki/mcrouter: Difference between revisions
imported>Krinkle No edit summary |
imported>Jelto No edit summary |
||
| Line 15: | Line 15: | ||
* run <code>sudo /usr/local/sbin/mcrouter_generate_certs --generate --add <FQDN1> <FDN2>...</code> Please make sure all FQDNs already resolve correctly to the server's IP address. | * run <code>sudo /usr/local/sbin/mcrouter_generate_certs --generate --add <FQDN1> <FDN2>...</code> Please make sure all FQDNs already resolve correctly to the server's IP address. | ||
* Review and commit the contents of <code>/srv/private/modules/secret/secrets/mcrouter</code> | * Review and commit the contents of <code>/srv/private/modules/secret/secrets/mcrouter</code> | ||
* Add similar fake certs for your hosts to labs/private (just copy the fake directory of another host). | * Add similar fake certs for your hosts to [[gerrit:plugins/gitiles/labs/private|labs/private]] (just copy the fake directory of another host). | ||
**<code>hostname="mw1442.eqiad.wmnet" ; mkdir ./modules/secret/secrets/mcrouter/$hostname ; touch ./modules/secret/secrets/mcrouter/$hostname/$hostname.crt.pem ; touch ./modules/secret/secrets/mcrouter/$hostname/$hostname.key.private.pem</code> | |||
== Renew CA and certificates == | == Renew CA and certificates == | ||
Revision as of 10:16, 23 July 2021
Generate certs for a new host
Why
When adding a new Mediawiki (mw/mwmaint/mwdebug/deploy) host to the site, before adding the puppet role a cert for mcrouter needs to be generated and added to the private puppet repository.
Without these files in the repo puppet will throw an error similar to Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, secret(): invalid secret mcrouter/mwmaint1002.eqiad.wmnet/mwmaint1002.eqiad.wmnet.crt.pem.
The generated files need to be added and git committed in /srv/private/modules/secret/secrets/mcrouter on a puppetmaster (puppetmaster1001) before the first puppet run happens.
HowTo
Based on the chat excerpt below:
- ssh to a puppetmaster (puppetmaster1001)
- save a copy of /etc/cergen/mcrouter.manifests.d/mediawiki-hosts.certs.yaml for good measure. (If this file does not exist for some reason you can recreate it by running the script below without the "--add <FQDN>" part.)
- run
sudo /usr/local/sbin/mcrouter_generate_certs --generate --add <FQDN1> <FDN2>...Please make sure all FQDNs already resolve correctly to the server's IP address. - Review and commit the contents of
/srv/private/modules/secret/secrets/mcrouter - Add similar fake certs for your hosts to labs/private (just copy the fake directory of another host).
hostname="mw1442.eqiad.wmnet" ; mkdir ./modules/secret/secrets/mcrouter/$hostname ; touch ./modules/secret/secrets/mcrouter/$hostname/$hostname.crt.pem ; touch ./modules/secret/secrets/mcrouter/$hostname/$hostname.key.private.pem
Renew CA and certificates
- Disable Puppet on mcrouter hosts.
cumin1001:~$ sudo cumin C:mcrouter "disable-puppet 'mcrouter cert renewal -YOUR_NAME'" - Renew the certs on the puppetmaster. The script uses sudo, so it will prompt for your password. The source is at renew_mcrouter_certs.py.
puppetmaster1001:~$ renew_mcrouter_certs
- Re-enable Puppet.
cumin1001:~$ sudo cumin C:mcrouter "enable-puppet 'mcrouter cert renewal -YOUR_NAME'"
As Puppet runs over the next 30 minutes, new certificates and CA will be stored. The mcrouter process will get inotified and will reload them itself; there is no need for a restart.
If you don't disable and re-enable Puppet, it might happen to run at exactly the wrong time during the renewal, in which case it will fail. The next time it runs on that machine, it will succeed and pick up the new certs -- the transient failure is harmless. Thus, if you're feeling bold, you can skip steps 1 and 3 above.