PKI/CA Operations

The root CA is an offline signing CA. The root key is only stored on the root ca server and in (TODO: some offline location). When creating new intermediate certificates one will need to manully generate the on the root ca server and then copy the certificate content into puppet.

Intermediate Certificates

Creating an intermediate certificate is mostly a manual process however we do make use of puppet to ensure the certificate is created with the correct parameters

Current intermediates

• debmonitor: Used for connections between debmonitor clients and the internal debmonitor client auth projected daemon
• discovery: Used for connections between ATS and back-end origin servers

Adding a new intermediate

To add a new certificate we first neeed to add an entry to the profile::pki::root_ca::intermediates: array in hiera for the root CA server.

profile::pki::root_ca::intermediates:
 - WMF_test_intermediate_ca

Once you have done this you will need to run puppet and the puppet ca server to create the public and private key pair. They should be created in /etc/cfssl/ssl/$intermediate_name and you should also see the puppet output.

$ sudo puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for pki-root.pki.eqiad1.wikimedia.cloud
Info: Applying configuration version '(e739fdec88) Jbond - P:pki::root_ca: use correct title for intermediate certs'
Notice: The LDAP client stack for this host is: sssd/sudo
Notice: /Stage[main]/Profile::Ldap::Client::Labs/Notify[LDAP client stack]/message: defined 'message' as 'The LDAP client stack for this host is: sssd/sudo'
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca]/ensure: created (corrective)
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/Exec[Generate cert WMF_test_intermediate_ca]/returns: executed successfully (corrective)
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca/WMF_test_intermediate_ca.pem]/mode: mode changed '0644' to '0440' (corrective)
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca/WMF_test_intermediate_ca-key.pem]/mode: mode changed '0600' to '0440' (corrective)
Notice: /Stage[main]/Profile::Pki::Root_ca/Cfssl::Cert[WMF_test_intermediate_ca]/File[/etc/cfssl/ssl/WMF_test_intermediate_ca/WMF_test_intermediate_ca.csr]/mode: mode changed '0644' to '0440' (corrective)
Notice: Applied catalog in 4.94 seconds

Once you have generated the new certificate you will need to copy the certificate file to the public puppet repo (modules/profile/files/pki/intermediates/$CN.pem) and the private key to the secret store (currently puppet private repo modules/secret/secrets/pki/intermediates/$CN.pem). You will also need to add an entry under the profile::pki::multirooca::intermediates: key. Note the location of the certificate and key must go in the specified paths for puppet to find them e.g.

profile::pki::multirooca::intermediates:
  WMF_test_intermediate_ca:
    ocsp_port: 10001
    # The following parameters are optional and override the defaults
    nets:
      -192.0.2.0/24
    auth_keys:
      default_auth:
        key: $custom_key
        type: standard
    profile:
      database:
        expiry: 8760h
        usages:
          - 'digital signature'
          - 'key encipherment'
          - 'server auth'
          - 'client auth'

Adding to defaulkt trust store

You may want to add the new intermidiates to ca-certificates so it is automaticly trused to do this add it the new ca to profile::base::certificates::wmf_intermidiates in hieradata/common/profile/base/certificates.yaml

Renewing a new intermediate

The root pki root server will automaticity renew intermediated certificates it manages and resign the public key on disk on the root CA server however it will be required to upload the certificate to the puppet repo e.g. profile/pki/intermediates/$CA.pem. Further at the time of writing certificate bundles are only ever fetched once by puppet. As such you will need to run a cumin command to delete the current intermediate certificates and have puppet re-download the new one.

$ cumin 'R:cfssl::cert%label = $CA' 'rm /etc/cfssl/$CA_chain.pem'

This is not ideal and is an area we shuold improve. some possible solutions

• use puppet to fetch the resource via http directly. This would rely on the `last-modified` header so would automaticity update the bundles but would also cause a lot of unnecessary head requests to check this value on every puppet run
• make the re-querement to store the public intermidate cert in profile/pki/intermediates/ a MUST. This would allow other modules to pull the intermediate directly from puppet but ties the puppet code to WMF specifics
• Look to add bundle support to the genkey and sign commands https://github.com/cloudflare/cfssl/issues/779. Upstream doesn't seem to have moved on this so it may mean maintaining a patch
• reimplement the genkey and sign commands using the api directly, which does support the bundle option. Requires maintaining our own puppet provider/type

Revoking certificates

If you are required to revoke a certificate you can use the cfssl-certs revoke command which expects the public certificate file as input

$ cfssl-certs -vvvv revoke -R cessationOfOperation /etc/cfssl/signers/debmonitor_discovery_wmnet/ca/debmonitor_discovery_wmnet.pem
DEBUG:root:running: /usr/bin/cfssl revoke -db-config /etc/cfssl/db.conf -serial 436540461805550888668031556404920484612797891140 -aki 3bada271e634bd1bfc80bf35718391d0ef691336 -reason cessationOfOperation
DEBUG:root:b''

OCSP

OCSP responses are generated on the multicaroot server however the OCSP signing certificate is created on the Root CA server in a similar way to the the intermediates i.e. it is generated using the cfssl::cert resource and files are maintained on the root ca server. however you will need to copy the certificate to the puppet repo (modules/profile/files/pki/ROOT/$CN.pem) and secret store (pki/ROOT/$CN.pem). This is only an issue when boot strapping or if you need to extend the life of the ocsp certificate

Boot strapping

When installing a new root CA or in the unfortunate event of having to revoke and replace the current root CA one must follow a number of boot strapping steps to create the initial root ca key. for this we will first need to create a csr.json file like the following (the one below is used in the cloud pki project). Please save this file into /etc/cfssl/csr/$CN.json

{
    "CN": "WMF_TEST_CA",
    "CA": {
        "expiry": "87600h"
    },
    "key": {
        "algo": "ecdsa",
        "size": 521
    },
    "names": [
        {
            "C":  "US",
            "L":  "San Francisco",
            "O":  "Wikimedia Foundation, Inc",
            "OU": "Cloud Services",
            "ST": "California"
        }
    ]
}

With this we can create the initial root ca certificate and private key. One should ultimately store these files in /etc/cfssl/signers/$CN/ca/

$ cd /etc/cfssl/signers/WMF_TEST_CA/ca
$ cfssl genkey -initca /etc/cfssl/csr/WMF_TEST_CA.json | cfssljson -bare ca
2021/03/03 13:51:50 [INFO] generate received request
2021/03/03 13:51:50 [INFO] received CSR
2021/03/03 13:51:50 [INFO] generating key: ecdsa-521
2021/03/03 13:51:50 [INFO] encoded CSR
2021/03/03 13:51:50 [INFO] signed certificate with serial number 518380654990553637914228602044849101209208053364
$ ls -l
total 12
-rw-r--r-- 1 root root  708 Mar  3 13:52 ca.csr
-rw------- 1 root root  365 Mar  3 13:52 ca-key.pem
-rw-r--r-- 1 root root 1066 Mar  3 13:52 ca.pem
$ openssl x509 -in ca.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            5a:cc:fe:14:ca:d9:96:1b:4b:ed:cc:26:02:b6:f7:ae:4e:a3:5e:74
        Signature Algorithm: ecdsa-with-SHA512
        Issuer: C = US, ST = California, L = San Francisco, O = "Wikimedia Foundation, Inc", OU = Cloud Services, CN = WMF_TEST_CA
        Validity
            Not Before: Mar  3 13:54:00 2021 GMT
            Not After : Mar  1 13:54:00 2031 GMT
        Subject: C = US, ST = California, L = San Francisco, O = "Wikimedia Foundation, Inc", OU = Cloud Services, CN = WMF_TEST_CA
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (521 bit)
                pub:
                    04:01:5c:0a:69:97:9a:21:41:66:4f:c1:72:f7:af:
                    75:42:a1:58:5d:a7:91:f7:c8:4e:e0:98:cb:ed:f0:
                    80:af:62:16:f8:80:db:55:79:42:6d:8c:b4:5f:f6:
                    ca:e9:40:98:65:28:31:9b:97:74:ff:35:01:e6:3c:
                    e0:7c:1e:25:05:ad:20:01:8a:82:5b:d6:61:38:26:
                    fb:5a:f9:66:09:16:ba:3a:43:26:a2:79:84:69:d4:
                    99:da:0f:a0:3e:fe:8a:73:b7:7b:c1:15:56:24:d6:
                    92:d8:44:f9:10:9e:da:6f:1f:ca:6f:b6:fc:65:ca:
                    d5:83:ab:0a:cc:cb:50:b4:52:37:78:92:63
                ASN1 OID: secp521r1
                NIST CURVE: P-521
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier: 
                71:1E:84:97:83:0B:3D:AB:6C:08:F7:1F:6B:42:7D:12:56:CE:19:DF
    Signature Algorithm: ecdsa-with-SHA512
         30:81:87:02:41:2a:bd:f4:a8:8b:4f:8a:ed:03:08:bc:e7:c8:
         68:4b:02:08:e4:3e:89:90:21:ac:54:94:08:2d:e3:f6:6f:f7:
         f2:aa:23:77:06:a3:77:9b:b1:ca:1c:e6:b8:72:e0:e0:0b:7b:
         dc:30:43:6e:f9:a7:10:41:94:97:bb:90:97:a9:9e:37:02:42:
         00:f1:42:79:38:d5:ad:01:87:06:f4:80:70:d6:a5:d8:d7:1f:
         0b:44:43:e5:ae:52:8e:79:9c:f6:62:63:b1:db:0d:5a:37:2d:
         f0:8b:05:3d:d7:fc:64:23:71:88:86:5a:98:55:6c:1e:14:f4:
         c9:86:2d:a2:71:8c:e7:23:62:47:aa:75

Once you have the key you will need to upload the public part of the key to the puppet/operations repo recommended to place in modules/profile/files/pki/ROOT/$CN.pem