Acme-chief

Acme-chief is a central service for requesting TLS certificates from ACME servers and then distributing the material to authorized API users.

Acme-chief was originally developed by Alex Monk at the 2018 Barcelona Wikimedia Hackathon based on discussions with the Wikimedia Traffic team. It aims to:

Issue our certificates centrally, distributing private key material and certificates as appropriate.
Use DNS to respond to Acme challenges, enabling use of wildcards.
- Integrate with gdnsd (production) and OpenStack Designate (Cloud VPS) to do this.
Generate RSA and ECDSA variants of the same certificates.
Expose new certificates after a staging time to allow for outdated client clocks.

See Acme-chief/Cloud VPS setup for how to set this up for a Cloud VPS project - particularly the service account creation subtask which needs to be performed by the cloud administrators.

In production this is already set up to manage production DNS, most people probably just want to know to find the certificate configuration in the hieradata/role/common/acme_chief.yaml file in operations/puppet .

Architecture

Acme-chief is comprised of binaries, services, scripts, and mounts that coordinate together.

System

acme-chief.service: The central management service ( acme-chief-backend being the binary name) running on the active master host and inactive/masked on passive hosts. It handles ACME protocol sessions with the Certificate Authority, coordinates validation challenges (e.g. DNS-01/HTTP-01), tracks expiration schedules, manages retry backoffs, and writes issued keys/certs to the master certificate store ( /var/lib/acme-chief/certs/ ).

uWSGI app server: REST API for acme-chief on the active server. Runs under uwsgi-acme-chief.service and behind an nginx reverse proxy for TLS termination. Receives administrative CLI commands via CLI tools (sometimes collectively referred to as acme-chief-cli ) and receives challenge webhooks.

acmedata mount: The Puppet file-serving mount point configured on the master server. It exports published certificate bundles ( /var/lib/acme-chief/certs/published/ on the active acme-chief) to client nodes during routine Puppet runs.

CLI utilities

A pile of executables, collectively referred to as acme-chief-cli , are installed on acme-chief hosts. Despite their name suggesting use for one-off manual commands, they're essential for the continued running of acme-chief. They each have their own systemd service/timer units for periodic execution against the uWSGI API server. For information on each utility, refer to the codebase documentation. A few important ones include:

acme-chief-certs-sync: Synchronizes acme-chief state and certificate data across active/passive acme-chief infrastructure hosts.

acme-chief-designate-sync
acme-chief-gdnsd-sync: Creates up DNS-01 challenge TXT records for gdnsd (production) or OpenStack designate (cloud).

Active vs passive

acme-chief's production environment is composed of one active instance and at least one passive instance. The hiera key profile::acme_chief::active flags an instance an active while passive instances are listed in an array called profile::acme_chief::passive .

The active instance is responsible of running both the acme-chief backend and the puppet file API service. A passive instance idly runs the puppet file API service but not the backend. TLS material is synchronized between instances regularly via acme-chief-certs-sync .

Certificate lifecycle

Definition

acme-chief's certificates are listed in the hieradata file . Blocks are grouped into a maximum of 40 domains per certificate; Splitting these many domains into smaller, more numerous certificates prevents the x509 certificates from growing to a size detrimental to our performance. This avoids potential extra round-trip delays due to overly-large certificate SANs .

Issuance

When new certificates are to be issued, clients locally generate a " snake-oil " certificate for immediate use.

While generating a self-signed certificate is useful to prevent software start failures, unexpected self-signed cert errors will occur until legitimate certificates are fetched.

Meanwhile, the active acme-chief server communicates with the Certificate Authority, performs validation challenges, and then writes the resulting keypair and certificate chain into its master storage directory ( /var/lib/acme-chief/certs/ ).

Client sync

Client nodes download new certificates via Puppet's file-serving mount point ( puppet://${acmechief_host}/acmedata/[…] ) during standard Puppet runs and store them in their configured certs directory (by default, /etc/acmecerts ).

An example client (in this case, an apt server) may have an certs directory that looks like this:

/etc/acmecerts/apt/
├── abc123abc123abc123abc123abc123ab
│   ├── ec-prime256v1.alt.chain.crt
│   ├── ec-prime256v1.alt.chained.crt
│   ├── ec-prime256v1.alt.chained.crt.key
│   ├── ec-prime256v1.chain.crt
│   ├── ec-prime256v1.chained.crt
│   ├── ec-prime256v1.chained.crt.key
│   ├── ec-prime256v1.crt
│   ├── ec-prime256v1.crt.key
│   └── ec-prime256v1.key
├── def456def456def456def456def456de
│   ├── ec-prime256v1.alt.chain.crt
│   ├── ec-prime256v1.alt.chained.crt
│   ├── ec-prime256v1.alt.chained.crt.key
│   ├── ec-prime256v1.chain.crt
│   ├── ec-prime256v1.chained.crt
│   ├── ec-prime256v1.chained.crt.key
│   ├── ec-prime256v1.crt
│   ├── ec-prime256v1.crt.key
│   └── ec-prime256v1.key
[…]
├── live -> /etc/acmecerts/apt/abc123abc123abc123abc123abc123ab
└── new -> /etc/acmecerts/apt/def456def456def456def456def456de

Certificates reside in titled directories (in this case, /etc/acmecerts/apt ). Within these individual cert directories, uniquely-hashed subdirectories are populated (in this case we have two but there may be many).

Atomic rollout

Two symlinks coordinate atomic rollouts:

new: Points to the most recently issued or synced certificate directory.
live: Points to the active certificate directory currently loaded and served by local daemons (e.g., HAProxy, Varnish, Icinga).

In the above example, the def456[…] certificate was recently issued but hasn't been put into production quite yet. The incumbent certificate is the abc123[…] certificate. When Puppet next runs, it will repoint the live symlink to the def456[…] hash directory, which is poised to take over.

Renewal

The active acme-chief daemon continuously tracks expiration dates across all managed certificates. When a certificate enters its renewal window (typically 30 days prior to expiration), acme-chief automatically initiates an ACME challenge with the Certificate Authority.

If a challenge fails or CA rate limits are hit, acme-chief executes an exponential backoff cooldown before retrying, ensuring sufficient lead time before expiration.

Cleanup

Superseded certificate directories (including the initial self-signed placeholder) remain unreferenced until Puppet's directory cleanup removes them from disk on subsequent runs.

Management

Switching the active instance

Disable puppet on the old active instance and on every acme-chief client:

$ sudo -i cumin 'R:acme_chief::cert' "disable-puppet 'acmechief maintenance'"

Stop acme-chief.service on the old active instance
Set the new instance as active in profile::acme_chief::active and remove it from profile::acme_chief::passive
Execute run-puppet-agent on the new active instance. After Puppet is done acme-chief.service should be up and running in the new instance.

Re-enable puppet on the acme-chief clients:

$ sudo -i cumin 'R:acme_chief::cert' "enable-puppet 'acmechief maintenance'"

Replacing the active instance

Create the new Ganeti VM.
Set its role as acme_chief on site.pp and make sure that's listed as a passive instance on profile::acme_chief::passive
Once the new instance is up and runnning arm keyholder for SSH access
Run run-puppet-agent on the current active instance
Trigger the TLS material sync service on the active instance — either directly on the host or via Cumin :

$ sudo systemctl start acme-chief-certs-sync.service
The new instance should now have a current copy of the TLS material in /var/lib/acme-chief/certs . You can verify any of the certificates. For example:
```
# openssl x509 -noout -dates -issuer -in /var/lib/acme-chief/certs/unified/live/ec-prime256v1.crt
```
Switch the active instance to the new host.
Decommission the old instance using the sre.hosts.decommission cookbook

Service restarts

See Service restarts#acmechief hosts

Cloud VPS setup

You will need:

a project-local puppetmaster (i.e. your own puppetmaster, not just relying on the central one at puppetmaster.cloudinfra.wmflabs.org)
a friendly production root to give your DNS management service user special permissions in Keystone and safelist it for access from within the Cloud VPS address range
the domain for which you want to issue certs as a zone in Designate, with delegation set up correctly

To deploy a new instance in your project:

Create an instance named <project>-acme-chief01 and do the usual dance with puppet to get them signed puppet certs.
Create a puppet prefix config in Horizon for <project>-acme-chief . Copy/create hiera values to mimic those of other project prefixes, substituting their values with your own.
Use the acme_chief::cert resource on your TLS termination VMs to get a certificate, with a name matching what you have in the hiera config.

Troubleshooting

If a cert has strangely expired, you may have hit a known issue in acme chief where it doesn't respond to HUP quite right. Restarting acme-chief should work. Otherwise, see Acme-chief#Monitoring
If client hosts complain about 'unable to get local issuer certificate' you may need to restart nginx on the acme-chief host, or restart the puppetserver.
If acme-chief is having issues, you should also check the Let's Encrypt status page or Google PKI status page to make sure they aren't having an outage or maintenance.