You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org
This page contains information on how to administer and replace SSL certificates for Toolforge.
Certificates are usually valid for 1 year, and they should be renewed at least 2 weeks prior to expiration date.
There are currently 2 certificates in use.
- *.wmflabs.org (also know as star.wmflabs.org)
- *.tools.wmflabs.org (also known as star.tools.wmflabs.org)
This certificate is in use by Toolforge web proxies and by other web proxies in WMCS, and is physically deployed in several servers:
In the case of Toolforge servers, the private key is hosted in the project puppetmaster (tools-puppetmaster-*).
In the case of proxy servers in the project-proxy tenant, the private key is hosted directly in the server (non-puppeticed).
This is used for the k8s master and docker registry servers in Toolforge:
These aren't centrally managed yet, but should be! TODO: What does this mean?
Renewing a certificate
The process for both certificates is very similar, and also could be complex.
- request certificate renewal, usually with DCops and/or Traffic team. Involves purchase approval, etc. May take a couple of weeks.
- the private key is added to the /srv/private repo in a prod puppetmaster without replacing the old one. Usually with a new. prefix in the filename. This is usually done by whoever purchases the certificate.
- once the public key is received, an operations/puppet.git repo patch should be stagged into gerrit. This is usually done by whoever purchases the certificate. This patch is not merged yet.
- ensure affected servers/services are working correctly previous to further operations.
- disable/stop puppet agent in all affected servers, i.e, the servers running the webservers using the certificate to be renewed.
- merge public key patch into the operations/puppet.git repo.
- replace the old private key with the new one in the /srv/private repo in production puppetmaster.
- refresh/rebase Toolforge puppetmaster repos (use git pull --rebase): /var/lib/git/operations/puppet and /var/lib/git/labs/private
- manually copy (scp) the private key from /srv/private from a production puppetmaster (puppetmaster1001.eqiad.wmnet for example) into Toolforge own puppetmaster
- replace the old private key with the new one in the private repo in Toolforge puppetmaster (/var/lib/git/labs/private), and do a local git commit (tag it with [local] in the commit msg). Check owner and permissions.
- if required by the certificate you are renewing, scp private key to nova-proxy servers and put it into /etc/ssl/private.
- enable and run puppet agent in one of the server to see if all public/private keys are in place. Restart nginx to see if it can start with no issues with the new certificate.
- if all was fine in the previous test, continue to all other servers.
- in the case of k8s master, restart kube-apiserver as well.
TODO: generate copy-paste commands for simplicity!
A collection of example Phabricator tasks:
- T206223 *.wmflabs.org cert needs renewing
- T160187 ssl certificate/key update: *.tools.wmflabs.org (expires on 2017-03-24)
- T174611 update *.wmflabs.org by 2017-10-16