You are browsing a read-only backup copy of Wikitech. The live site can be found at wikitech.wikimedia.org

PKI/Clients

From Wikitech-static
< PKI
Jump to navigation Jump to search

We currently use CFSSL to provide and manage PKI solutions. clients are able to make use of the CFSSL API end point available at https://pki.discovery.wmnet/. however this end point is protected via TLS Client authentications and currently requires users to using the puppet agent certificate. The puppet agent certificate was chosen as it is something we already issue when a host is provisioned or re-imaged, however we can change this at a later date or configure additional intermediate CAs which have different authentication mechanisms. Further to the client auth requirement API request also need to be signed with a hmac using a secret key (available in the puppet private repo)

Puppet integration

Puppet hosts can start using the PKI infrastructure by adding the profile::pki::client to their role. This will configure the host with the required CFSSL configuration to be able request new certificates for the configured intermediate certificates.

Creating certificates

When a user wants to create a new cerrtificate the will first need to decide which intermediate CA they want to use for issuing the certificates. Note that each realm of operations should have its own intermediate CA and you should not use cross functional CA's e.g. debmonitor, DB connections, conftool should all have there own intermediated CA's. If you need to create a new intermediate CA Please check the documentation on the CA Operations Page. Also unless the defaults have been changed all certificates will be issued with an expiry of 96h and set to auto renew 24hours before the certificate expires.

Once you know which intermediate CA you want to use for issuing your certificate you will need to know which profile to use. By default intermediate certificates are create with Two profiles which are useful for this consideration. The default profile which should be used for client auth and a server profile which can be used for daemon services. If you have created additional profiles for you CA then you will need to use them below. For the below example i am going to assume we want to create two certificates in the "WMF testing CA", on in using the default profile and one using the server profile.

profile::pki::get_cert

The simplest interface to use to create certificates is the profile::pki::get_cert functions. This function has a similar signature to the cfssl::cert resource, however it also returns the path of the cert, private key and CA bundle (if requested). As you most often need to have theses paths for other parts of puppet configurations this is likely the most useful interface.

class profile::foobar () {
    $ssl_paths = profile::pki::get_cert('WMF_testing_CA')
    $config = {
      ssl_cert = $ssl_paths['cert']
      ssl_key = $ssl_paths['key']
    }
    file {'/etc/ssl_config.json':
      ensure => file,
      content = $config.to_json
    }
 }

Also fetch the CA bundle

profile::pki::get_cert('WMF_testing_CA', $facts['fqdn']) >> {
  'cert' => '/etc/cfssl/ssl/WMF_testing_CA/WMF_testing_CA_$fqdn.pem',     # owned by root
  'key' => '/etc/cfssl/ssl/WMF_testing_CA/WMF_testing_CA_$fqdn-key.pem',  # owned by root
  'ca' => '/etc/cfssl/ssl/WMF_testing_CA/WMF_testing_CA_chain.pem',       # owned by root
}

Also fetch the CA bundle, change the outputdir and user

profile::pki::get_cert(
  'WMF_testing_CA',
  $facts['fqdn'],
  {'outdir' => '/etc/foobar', 'owner' => 'foobar'}
) >> {
  'cert' => '/etc/foobar/WMF_testing_CA_$fqdn.pem',    # owned by foobar
  'key' => '/etc/foobar/WMF_testing_CA_$fqdn-key.pem', # owned by foobar
  'ca' => '/etc/foobar/WMF_testing_CA_chain.pem',      # owned by foobar
}

Hiera

We can also add certificates by adding the configuration to hiera via the profile::pki::client::certs. Entries here are passed directly to cfssl::cert and support all parameters for that resource. In the following example we add provide_chain: true which should probably be the default but at the time of writing there are some improvements which could be made to bundle handling and for puppet managed hosts it may make more senses to push all intermediate cert to all puppet agents.

profile::pki::client::certs:
  '%{facts.networking.fqdn}':
    # label is escaped with regsubst('[^\w\-]', '_', 'G')
    label: WMF_testing_CA
    provide_chain: true
  foobar.example.com:
    label: WMF_test_intermediate_ca
    provide_chain: true
    profile: server

As mentioned above certificates are set with a short TTL and are set to auto expiry as such it is likely that you may need to auto restart some servie when the certificate is renewed for this you can use the notify_service parameter.

profile::pki::client::certs:
  foobar.example.com:
    label: WMF_test_intermediate_ca
    provide_chain: true
    profile: server
    notify_service: foobar

in the above example Service['foobar'] will be notified when the certificate is renewed.


Puppet

Some times it is required to have more complex relations ships or keep resources in specific manifests to ease debugging and readability as such it is also possible to use the cfssl::cert resource directly.

class profile::foobar () {
  cfssl::cert{ $facts['networking']['fqdn']:
    label         => WMF_testing_CA,
    provide_chain => true,
  }
  cfssl::cert{ 'foobar.example.com':
    label         => WMF_testing_CA,
    provide_chain => true,
    profile       => 'server',
  }
}

Defining the resource in puppet allows for more complex relations as indicated in the following example

class profile::foobar () {
  exec {'foobar_exec':
    command     => 'run a command every time the client cert is renewed'
    refreshonly => true,
  }
  service {'foobar':
    ensure => 'running',
  }
  cfssl::cert{ $facts['networking']['fqdn']:
    label         => WMF_testing_CA,
    provide_chain => true,
    notify        => Exec['foobar_exec'],
  }
  cfssl::cert{ 'foobar.example.com':
    label          => WMF_testing_CA,
    provide_chain  => true,
    profile        => 'server',
    notify_service => 'foobar',
    # We could also use the following however notify_service is a bit more optimal
    # notify        => Service['foobar'],
  }
  service {'another_foobar':
    ensure => 'running',
    subscribe => Cfssl::Cert['foobar.example.com'],
  }
}

Proxy Signing

The pki client supports the ability to start a cfssl api service and proxy the signing request to the main multicaroot instance. The proxy server still needs to have access to a valid puppet agent certificate to use for outbound connections to the multiroot ca server. however it is able to offer a http endpoint locally which does not have this restrictions. This allows for an infrastructure where production puppet host can provide PKI services to none-puppet hosts. The main driver for this was to allow k8s hosts to provide PKI services to there containers over localhost. To enable the proxy services simply add the following configuration to hiera

profile::pki::client::enable_proxy: true
# By default the proxy service listens on http://127.0.0.1:8888/
# This can be changed with the following settings however one should 
# be very cautious in considering having this listen on anything other then localhost 
profile::pki::client::listen_addr: '127.0.0.1'
profile::pki::client::listen_port: 8888

CLI Interactions

Users are also able interact with CFSSL using the cfssl command line tool. The instructions below assume you are running commands from a puppet enabled host in the production network. the pki server is needs is protected via TLS mutal auth using the puppet CA, further there is an additional secret token held in /etc/cfssl/client-cfssl.conf.

First need to create a certificate signing request in json format. e.g. crs.json

{
  "hosts": [
    "cumin1001.eqiad.wmnet"
  ],
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "organisation": "SnakeOil", 
      "country": "uk"
    }
  ]
}

Once this is in place you can request send it to the pki server and as for a signature with the following command:

$ /usr/bin/cfssl gencert \
  -config /etc/cfssl/client-cfssl.conf \
  -tls-remote-ca $(facter -p puppet_config.localcacert) \
  -mutual-tls-client-cert $(facter -p puppet_config.hostcert) \
  -mutual-tls-client-key $(facter -p puppet_config.hostprivkey) \
  -label ${ca_label} -profile server \  
  csr.json

${ca_label} is the name of the intermediate, see of PKI/CA Operations#Current intermediates.

This command will produce a json response with the certificate, key and csr. To automatically parse out the json you can pip the command to cfssljson -bare $dir/$basename e.g.

$ /usr/bin/cfssl gencert \
  -config /etc/cfssl/client-cfssl.conf \
  -tls-remote-ca $(facter -p puppet_config.localcacert) \
  -mutual-tls-client-cert $(facter -p puppet_config.hostcert) \
  -mutual-tls-client-key $(facter -p puppet_config.hostprivkey) \
  -label ${ca_label} -profile server \  
  csr.json | /usr/bin/cfssljson -bare /path/to/certs/basename
$ ls -la /path/to/certs/
drwxr-xr-x  2 root root 4096 Jun 28 12:11 .
drwx------ 11 root root 4096 Jun 28 12:11 ..
-rw-r--r--  1 root root  395 Jun 28 12:11 basename.csr
-rw-------  1 root root  227 Jun 28 12:11 basename-key.pem
-rw-r--r--  1 root root 1054 Jun 28 12:11 basename.pem

Users can also resign the certificate by using pretty much the same command however change gencert to sign and use the current pem certificate instead of the json csr

$ /usr/bin/cfssl sign \
  -config /etc/cfssl/client-cfssl.conf \
  -tls-remote-ca $(facter -p puppet_config.localcacert) \
  -mutual-tls-client-cert $(facter -p puppet_config.hostcert) \
  -mutual-tls-client-key $(facter -p puppet_config.hostprivkey) \
  -label ${ca_label} -profile server \  
  /path/to/certs/basename.pem| /usr/bin/cfssljson -bare /path/to/certs/basename