You are browsing a read-only backup copy of Wikitech. The live site can be found at


From Wikitech
Jump to navigation Jump to search

Generate certs for a new host


When adding a new Mediawiki (mw/mwmaint/mwdebug/deploy) host to the site, before adding the puppet role a cert for mcrouter needs to be generated and added to the private puppet repository.

Without these files in the repo puppet will throw an error similar to Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, secret(): invalid secret mcrouter/mwmaint1002.eqiad.wmnet/mwmaint1002.eqiad.wmnet.crt.pem.

The generated files need to be added and git committed in /srv/private/modules/secret/secrets/mcrouter on a puppetmaster (puppetmaster1001) before the first puppet run happens.


Based on the chat excerpt below:

  • ssh to a puppetmaster (puppetmaster1001)
  • save a copy of /etc/cergen/mcrouter.manifests.d/mediawiki-hosts.certs.yaml for good measure. (If this file does not exist for some reason you can recreate it by running the script below without the "--add <FQDN>" part.)
  • run sudo /usr/local/sbin/mcrouter_generate_certs --generate --add <FQDN1> <FDN2>... Please make sure all FQDNs already resolve correctly to the server's IP address.
  • Review and commit the contents of /srv/private/modules/secret/secrets/mcrouter
  • Add similar fake certs for your hosts to labs/private (just copy the fake directory of another host).

Renew CA and certificates

ssh to a puppetmaster and then do the following:

# Save a copy of the current mcrouter tree
cp -r /srv/private/modules/secret/secrets/mcrouter/ mcrouter
# Delete the current public certs
find /srv/private/modules/secret/secrets/mcrouter/ -type f -name "*.crt.pem" -delete
# Re-generate the certs
sudo cergen --base-path /srv/private/modules/secret/secrets/mcrouter/  --generate /etc/cergen/mcrouter.manifests.d
# Verify the old CA cert can validate the new certs
openssl verify -verbose -CAFile ./mcrouter/mcrouter_ca/ca.crt.pem /srv/private/modules/secret/secrets/mcrouter/mw1295.eqiad.wmnet/mw1295.eqiad.wmnet.crt.pem
# Verify the new CA cert can validate old certs
openssl verify -verbose -CAFile  /srv/private/modules/secret/secrets/mcrouter/mcrouter_ca/ca.crt.pem ./mcrouter/mw1295.eqiad.wmnet/mw1295.eqiad.wmnet.crt.pem
# Verify the expiry of the new certs
openssl x509 -noout -dates -in /srv/private/modules/secret/secrets/mcrouter/mcrouter_ca/ca.crt.pem
openssl x509 -noout -dates -in  /srv/private/modules/secret/secrets/mcrouter/mw1295.eqiad.wmnet/mw1295.eqiad.wmnet.crt.pem
# Commit your changes
cd /srv/private && git add modules/secret/secrets/mcrouter/
## sudo -i ; git commit

remember to disable puppet on mcrouter hosts, then after running puppet agent again new certificates and ca will be stored in the path and mcrouter process will get inotified and will reload them itself, there is no need for a restart.