You are browsing a read-only backup copy of Wikitech. The primary site can be found at wikitech.wikimedia.org

Cergen: Difference between revisions

From Wikitech-static
Jump to navigation Jump to search
imported>Ema
imported>Filippo Giunchedi
(→‎Cheatsheet: mention automatic public key search for sslcert::certificate https://phabricator.wikimedia.org/T290261)
 
(12 intermediate revisions by 9 users not shown)
Line 30: Line 30:
</syntaxhighlight>
</syntaxhighlight>


With the CA and certificates declared, we can now use cergen to check the status of our certificates.  We keep generated files in <tt>/srv/private/modules/secret/secrets/certificates</tt>, so we need to provide that as the <code>--base-path</code> option.
With the CA and certificates declared, we can now use cergen to check the status of our certificates.  We keep generated files in <tt>/srv/private/modules/secret/secrets/certificates</tt>, so we need to provide that as the <code>--base-path=</code> option.


<syntaxhighlight lang="shell">
<syntaxhighlight lang="shell">
$ cergen --base-path /srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d
$ cergen --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d


Status of certificates ['kafka_jumbo-eqiad_broker']
Status of certificates ['kafka_jumbo-eqiad_broker']
Line 60: Line 60:
On the puppetmaster (eg: puppetmaster1001.eqiad.wmnet), create <code>/srv/private/modules/secret/secrets/certificates/certificate.manifests.d/SERVICENAME.certs.yaml</code> as follows:
On the puppetmaster (eg: puppetmaster1001.eqiad.wmnet), create <code>/srv/private/modules/secret/secrets/certificates/certificate.manifests.d/SERVICENAME.certs.yaml</code> as follows:


<syntaxhighlight>
<syntaxhighlight lang="yaml">
SERVICENAME.discovery.wmnet:
SERVICENAME.discovery.wmnet:
   authority: puppet_ca
   authority: puppet_ca
   expiry: null
   expiry: null
   alt_names: ["SERVICENAME.discovery.wmnet","*.whatever.org", ...]
   alt_names: ["SERVICENAME.svc.eqiad.wmnet","*.whatever.org", ...]
   key:
   key:
     password: YOURSECRETPASSWORD
     password: YOURSECRETPASSWORD
Line 70: Line 70:
</syntaxhighlight>
</syntaxhighlight>


Note: make sure that SERVICENAME.discovery.wmnet is listed under '''alt_names''' as in the example above. If not, the hostname will be listed under '''Subject''' only and not in the '''Subject Alternative Name''' section of the certificate.
Note: the service name / common name (i.e. SERVICENAME.discovery.wmnet) will be implicitly included in alt_names, no need to duplicate it there as well.


<syntaxhighlight>
By setting a key.password, cergen will output encrypted private key files. '''If you need an unencrypted private key file you can omit the key.password and skip the openssl command below.'''
chmod 400 /srv/private/modules/secret/secrets/certificates/certificate.manifests.d/SERVICENAME.certs.yaml
 
cergen -c 'SERVICENAME.*' --generate --base-path /srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d
<syntaxhighlight lang="bash">
cergen -c 'SERVICENAME.*' --generate --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d
 
# For certificates with key.password, run:
openssl ec -in modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.key.private.pem -out /srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key
openssl ec -in modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.key.private.pem -out /srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key
# openssl will ask for YOURSECRETPASSWORD at this point
# openssl will ask for YOURSECRETPASSWORD at this point
# For certificates without key.password, just copy the private key over:
cp ./modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.key.private.pem /srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key
git add /modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key
</syntaxhighlight>
</syntaxhighlight>


If everything went well, the public key is now under <code>/srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.crt.pem</code>. Copy it to the <code>operations/puppet</code> repo under <code>files/ssl/SERVICENAME.discovery.wmnet.crt</code>.
If everything went well, the public key is now under <code>/srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.crt.pem</code>. If your service is using <code>sslcert::certificate</code> with "use_cergen" then there's nothing else to do other than commit your changes to <code>/srv/private</code>.
 
Otherwise copy the public key to the <code>operations/puppet</code> repository under <code>files/ssl/SERVICENAME.discovery.wmnet.crt</code> Notice the extension difference! The extension is <code>.crt.pem</code> on the puppetmaster, but it needs to be just <code>.crt</code> in <code>operations/puppet</code>.


The private key is <code>/srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key</code>. There's nothing else to do other than commit your changes to <code>/srv/private</code>.
The private key is <code>/srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key</code>. There's nothing else to do other than commit your changes to <code>/srv/private</code>.
Line 85: Line 94:
In order for [[Help:Puppet-compiler|pcc]] to work properly, a dummy secret key (NOT the real one) needs to be added to the <code>labs/private</code> repository under <code>modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key</code>. Please note that the <code>labs/private</code> repository is NOT private as the name might suggest, it is public. The key you add should look like this:
In order for [[Help:Puppet-compiler|pcc]] to work properly, a dummy secret key (NOT the real one) needs to be added to the <code>labs/private</code> repository under <code>modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key</code>. Please note that the <code>labs/private</code> repository is NOT private as the name might suggest, it is public. The key you add should look like this:


<syntaxhighlight>
<syntaxhighlight lang="text">
-----BEGIN RSA PRIVATE KEY-----
-----BEGIN RSA PRIVATE KEY-----
dummy
dummy
Line 91: Line 100:
</syntaxhighlight>
</syntaxhighlight>
See https://gerrit.wikimedia.org/r/#/c/labs/private/+/523929/ as an example.
See https://gerrit.wikimedia.org/r/#/c/labs/private/+/523929/ as an example.
=== Update a certificate ===
* Update the YAML file <code>/srv/private/modules/secret/secrets/certificates/certificate.manifests.d/SERVICENAME.certs.yaml</code>
* Remove the old certificate from the Puppet CA
  puppet cert clean <common_name>
* Remove the following files from <code>/srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/</code>:
<code>SERVICENAME.discovery.wmnet.crt.pem SERVICENAME.discovery.wmnet.csr.pem SERVICENAME.discovery.wmnet.keystore.jks SERVICENAME.discovery.wmnet.keystore.p12 truststore.jks</code>
* Update the certificate with:
<code>cergen -c 'SERVICENAME.*' --generate  --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d</code>
* The new cert should be under <code>/srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.crt.pem</code>.
* Add the updated certificate to puppet under <code>files/ssl/SERVICENAME.discovery.wmnet.crt</code>.


== Distribution via Puppet ==
== Distribution via Puppet ==
Line 120: Line 145:


The original Phabricator ticket tracking this work was [[phab:T166167|T166167]].
The original Phabricator ticket tracking this work was [[phab:T166167|T166167]].
[[Category:TLS]]

Latest revision as of 08:57, 8 February 2022

cergen is a Python command line tool and library to help with managing and generating asymmetric key pairs and x509 certificates for use with TLS/SSL encryption and authentication.

The openssl CLI is useful and very featureful, but fairly complicated. The Java keytool CLI serves a similar purpose, but outputs files in a different format than openssl. cergen allows you to declare desired certificates and keys in a YAML manifest, and then idempotently generate them into files of various formats for use with different technologies.

cergen was originally based on cassandra-ca-manager, and expanded to generate more file formats and to integrate with custom CA implementations, including Puppet CA.

cergen is installed on puppetmaster frontends that are also CAs. It is intended to be used there to generate certificates signed by our Puppet CA, and committed to the ops puppet private repository's secret module.

For more information, see the cergen README.

Usage on Puppet CA host

cergen production certificate manifest files are checked into Puppet private. On puppetmaster1001, these can be found at /srv/private/modules/secret/secrets/certificates/certificate.manifests.d/. *.certs.yaml files in this directory declare certificates and keys that should be generated and managed, as well as any CAs that will be used to sign those certificates.

Most likely, you'll be using the Puppet CA to sign your certificates. The puppet_ca signer is declared in puppet_ca.certs.yaml:

# This is our Puppet CA.  It will be used to sign other certificates.
puppet_ca:
  class_name: puppet
  hostname: puppetmaster1001.eqiad.wmnet

Here puppet_ca is the name of the authority. You can refer to puppet_ca as the authority for the certificates you want to generate and have signed by our Puppet CA. In kafka_jumbo.certs.yaml, we declare a certificate for use by Kafka brokers in the jumbo-eqiad cluster:

# Certificates for the kafka-jumbo cluster.
kafka_jumbo-eqiad_broker:
  authority: puppet_ca
  key:
    password: 'XXXXXXXX'
    algorithm: ec

With the CA and certificates declared, we can now use cergen to check the status of our certificates. We keep generated files in /srv/private/modules/secret/secrets/certificates, so we need to provide that as the --base-path= option.

$ cergen --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d

Status of certificates ['kafka_jumbo-eqiad_broker']

Certificate(kafka_jumbo-eqiad_broker, authorities=[PuppetCA(puppetmaster1001.eqiad.wmnet_8140)]):
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.key.private.pem: PRESENT (mtime: 2017-12-05T14:47:45.105453)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.key.public.pem: PRESENT (mtime: 2017-12-05T14:47:45.105453)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.crt.pem: PRESENT (mtime: 2017-12-05T14:47:46.417451)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.keystore.p12: PRESENT (mtime: 2017-12-05T14:47:46.433451)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/kafka_jumbo-eqiad_broker.keystore.jks: PRESENT (mtime: 2017-12-05T14:47:47.261450)
	/srv/private/modules/secret/secrets/certificates/kafka_jumbo-eqiad_broker/truststore.jks: PRESENT (mtime: 2017-12-05T14:47:47.649449)

Without the --generate flag, cergen will just print out a list of certificates it is managing, and the status of the expected files. By providing --generate, cergen will attempt to generate any files for certificates declared in its manifests that are not PRESENT. For these certificates, the key.private.pem and the crt.pem files are considered the canonical sources of data. The other files are subordinate to those. E.g., if the keystore.jks file is missing, it will be re-generated from the key.private.pem and crt.pem files. However, if either crt.pem or key.private.pem is missing, you will need to manually generate a brand new key and certificate by removing the files and running again with --generate.

NOTE: If you are regenerating a Puppet signed certificate, you must first remove the certificate from the Puppet CA. puppet cert clean <common_name> should do it.

Certificates that declare authority: puppet_ca will be auto-signed by our Puppet CA. If you need to remove a Puppet signed cert, remember to also destroy that cert using the puppet cert CLI, e.g. puppet cert destroy kafka_jumbo-eqiad_broker.

To avoid messing with certificates you are not concerned with, you can instruct cergen to only select specific certificates by using the -c flag. E.g.

cergen --generate -c 'kafka.*' ... might select all certificates who's names start with 'kafka'. See cergen --helpfor more options.

Cheatsheet

On the puppetmaster (eg: puppetmaster1001.eqiad.wmnet), create /srv/private/modules/secret/secrets/certificates/certificate.manifests.d/SERVICENAME.certs.yaml as follows:

SERVICENAME.discovery.wmnet:
  authority: puppet_ca
  expiry: null
  alt_names: ["SERVICENAME.svc.eqiad.wmnet","*.whatever.org", ...]
  key:
    password: YOURSECRETPASSWORD
    algorithm: ec

Note: the service name / common name (i.e. SERVICENAME.discovery.wmnet) will be implicitly included in alt_names, no need to duplicate it there as well.

By setting a key.password, cergen will output encrypted private key files. If you need an unencrypted private key file you can omit the key.password and skip the openssl command below.

cergen -c 'SERVICENAME.*' --generate --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d

# For certificates with key.password, run:
openssl ec -in modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.key.private.pem -out /srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key
# openssl will ask for YOURSECRETPASSWORD at this point

# For certificates without key.password, just copy the private key over:
cp ./modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.key.private.pem /srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key
git add /modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key

If everything went well, the public key is now under /srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.crt.pem. If your service is using sslcert::certificate with "use_cergen" then there's nothing else to do other than commit your changes to /srv/private.

Otherwise copy the public key to the operations/puppet repository under files/ssl/SERVICENAME.discovery.wmnet.crt Notice the extension difference! The extension is .crt.pem on the puppetmaster, but it needs to be just .crt in operations/puppet.

The private key is /srv/private/modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key. There's nothing else to do other than commit your changes to /srv/private.

In order for pcc to work properly, a dummy secret key (NOT the real one) needs to be added to the labs/private repository under modules/secret/secrets/ssl/SERVICENAME.discovery.wmnet.key. Please note that the labs/private repository is NOT private as the name might suggest, it is public. The key you add should look like this:

-----BEGIN RSA PRIVATE KEY-----
dummy
-----END RSA PRIVATE KEY-----

See https://gerrit.wikimedia.org/r/#/c/labs/private/+/523929/ as an example.

Update a certificate

  • Update the YAML file /srv/private/modules/secret/secrets/certificates/certificate.manifests.d/SERVICENAME.certs.yaml
  • Remove the old certificate from the Puppet CA
  puppet cert clean <common_name>
  • Remove the following files from /srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/:

SERVICENAME.discovery.wmnet.crt.pem SERVICENAME.discovery.wmnet.csr.pem SERVICENAME.discovery.wmnet.keystore.jks SERVICENAME.discovery.wmnet.keystore.p12 truststore.jks

  • Update the certificate with:

cergen -c 'SERVICENAME.*' --generate --base-path=/srv/private/modules/secret/secrets/certificates /srv/private/modules/secret/secrets/certificates/certificate.manifests.d

  • The new cert should be under /srv/private/modules/secret/secrets/certificates/SERVICENAME.discovery.wmnet/SERVICENAME.discovery.wmnet.crt.pem.
  • Add the updated certificate to puppet under files/ssl/SERVICENAME.discovery.wmnet.crt.

Distribution via Puppet

Once you've generated your certificate and key files using cergen, you should commit them, as well as any changes you've made to certificate.manifests.d/. Then you can use the puppet secret template function to render the contents of the files into locations for use by your production service.

file { "/path/to/my/my_certificate_keystore.jks":
    content => secret('certificates/my_certificate/my_certificate.keystore.jks'),
}

See also the profile::kafka::broker class for another example.

Future work

It would be very convenient if the manual steps of declaring a certificate, generating it, and distributing the desired files were all abstracted away into a single handy puppet define. This might be possible, but is complicated. It might look something like this:

cergen::certificate { 'kafka_jumbo-eqiad_broker':
    destination => '/etc/kafka/ssl',
    properties  => {
      'authority' => 'puppet_ca',
      'key' => {
          'algorithm' => 'rsa',
          'password'  => hiera('passwords::blabla'),
      }
    }
}

This would allow us to declare a certificate in the class/node/location where it should be deployed. This is easier said than done, as Puppet signed certificates must be created and signed on the same host as the Puppet CA (you can't just use the Puppet CA HTTP API). Creating a define like above would require

  • declare the certificate on the host
  • Using exported resources, realize the rendered cergen manifest.yaml to the Puppet CA host (e.g. puppetmaster1001)
  • Use the generate() puppet function to run the cergen command on the puppetmaster and commit the resulting certificate files public Puppet, and private key files to the puppet private repo.
  • Pull the certificate files using source and key files from puppet private using the secret() function.

I wrote a PoC to do this, but it is totally untested and probably would not work as is.

The original Phabricator ticket tracking this work was T166167.