You are browsing a read-only backup copy of Wikitech. The live site can be found at

Portal:Toolforge/Admin/SSL certificates

From Wikitech-static
< Portal:Toolforge‎ | Admin
Revision as of 12:46, 5 November 2018 by imported>Arturo Borrero Gonzalez (→‎Renewing a certificate: more public vs private repo)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

This page contains information on how to administer and replace SSL certificates for Toolforge.

Certificates are usually valid for 1 year, and they should be renewed at least 2 weeks prior to expiration date.


There are currently 2 certificates in use.

  • * (also know as
  • * (also known as


This certificate is in use by Toolforge web proxies and by other web proxies in WMCS, and is physically deployed in several servers:

  • tools-static-*.tools.eqiad.wmflabs
  • tools-proxy-*.tools.eqiad.wmflabs
  • novaproxy-*.project-proxy.eqiad.wmflabs

In the case of Toolforge servers, the private key is hosted in the project puppetmaster (tools-puppetmaster-*).

In the case of proxy servers in the project-proxy tenant, the private key is hosted directly in the server (non-puppeticed).


This is used for the k8s master and docker registry servers in Toolforge:

  • tools-k8s-master-*.tools.eqiad.wmflabs
  • tools-docker-registry-*.tools.eqiad.wmflabs

These aren't centrally managed yet, but should be! TODO: What does this mean?

Renewing a certificate

The process for both certificates is very similar, and also could be complex.

  1. request certificate renewal, usually with DCops and/or Traffic team. Involves purchase approval, etc. May take a couple of weeks.
  2. the private key is added to the /srv/private repo in a prod puppetmaster without replacing the old one. Usually with a new. prefix in the filename. This is usually done by whoever purchases the certificate.
  3. once the public key is received, an operations/puppet.git repo patch should be stagged into gerrit. This is usually done by whoever purchases the certificate. This patch is not merged yet.
  4. ensure affected servers/services are working correctly previous to further operations.
  5. disable/stop puppet agent in all affected servers, i.e, the servers running the webservers using the certificate to be renewed.
  6. merge public key patch into the operations/puppet.git repo.
  7. replace the old private key with the new one in the /srv/private repo in production puppetmaster.
  8. refresh/rebase Toolforge puppetmaster repos (use git pull --rebase): /var/lib/git/operations/puppet and /var/lib/git/labs/private
  9. manually copy (scp) the private key from /srv/private from a production puppetmaster (puppetmaster1001.eqiad.wmnet for example) into Toolforge own puppetmaster
  10. replace the old private key with the new one in the private repo in Toolforge puppetmaster (/var/lib/git/labs/private), and do a local git commit (tag it with [local] in the commit msg). Check owner and permissions.
  11. if required by the certificate you are renewing, scp private key to nova-proxy servers and put it into /etc/ssl/private.
  12. enable and run puppet agent in one of the server to see if all public/private keys are in place. Restart nginx to see if it can start with no issues with the new certificate.
  13. if all was fine in the previous test, continue to all other servers.
  14. in the case of k8s master, restart kube-apiserver as well.

TODO: generate copy-paste commands for simplicity!


A collection of example Phabricator tasks:

See also