TLS certificates in Infra

TLS certificates in Infra¶

Common naming convention.¶

Ansible roles are using the following logic to distribute .key/.cert files

  * {{ public_name }}.key : TLS private key
  * {{ public_name }}.crt : signed TLS certificate
  * {{ public_name }}-CAchain.crt : Trusted chain from CA (usually a symlink in pkistore is enough as we have a very few)

Internal certificates¶

IPA/dogtag (central authentication)¶

While IPA enrolled nodes can directly request TLS certificates, for CentOS infra, due to the fact that almost all nodes can't be enrolled (no direct link with IPA infra), we have to delegate this on an enrolled node where we can then be granted "delegation" rights in IPA, so that a "fake" enrolled node (created for the real target server that will need a TLS cert) can be "managed by" the enrolled node.

From that enrolled node, we'll then be able to retrieve the TLS cert/keytab and also then export/import into pkistore , following the same naming convention as described above.

Depending on the Env (Prod vs STG), you can find which node[s] is/are enrolled (through applied ipa-client client role applied with also the ipa_client_tls_delegated_host boolean set to True (needed for the following script to be present in the ipa-client role).

Pre-requisites:

one node enrolled in the correct REALM we want to generate/retrieve TLS cert (and keytab) for
an IPA account that has enough privileges to add hosts/services and local sudo rights on the intermediate enrolled node
ipa-client role applied with correct script deployed

Note

The following steps are just for new certificates. As once you'll have requested this on the enrolled node, the certmonger process will automatically watch and request/renew new ones, so they'll land on the enrolled node automatically, from which you can then retrieve TLS files (from /etc/pki/tls/certs) and update pkistore (see above). To help with that see /koji/retrieve_from_ipa script

Once we have shell access on such enrolled node, we can proceed like this :

/usr/libexec/centos/generate_ipa_tls_krb5 

You need to call the script like this : /usr/libexec/centos/generate_ipa_tls_krb5 -arguments
 -n : node name / fqdn ([REQUIRED], example 'ppc64-01.cbs.centos.org')
 -d : Description for that host ([REQUIRED], example 'cbs koji builder')
 -s : service for principal ([REQUIRED], example 'compile' would create compile/ppc64-01.cbs.centos.org service in IPA)
 -h : display this help

You also need a valid kerberos ticket otherwise script will exit

Here is an output example for the command with arguments:

/usr/libexec/centos/generate_ipa_tls_krb5 -n mbs.mbox.stg.centos.org -d 'Koji mbox STG mbs' -s 'HTTP'
[+] Adding host in IPA and adding delegation to retrieve certs/keytab ...
------------------------------------
Added host "mbs.mbox.stg.centos.org"
------------------------------------
  Host name: mbs.mbox.stg.centos.org
  Description: Koji mbox STG mbs
  Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Password: False
  Keytab: False
  Managed by: mbs.mbox.stg.centos.org
  Host name: mbs.mbox.stg.centos.org
  Description: Koji mbox STG mbs
  Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org
-------------------------
Number of members added 1
-------------------------
------------------------------------------------------------------
Added service "HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG"
------------------------------------------------------------------
  Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Managed by: mbs.mbox.stg.centos.org
  Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org
-------------------------
Number of members added 1
-------------------------
  Host name: mbs.mbox.stg.centos.org
  Description: Koji mbox STG mbs
  Principal name: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Principal alias: host/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org
  Hosts allowed to retrieve keytab: <modified>.fedoraproject.org
-------------------------
Number of members added 1
-------------------------
  Principal name: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Principal alias: HTTP/mbs.mbox.stg.centos.org@STG.FEDORAPROJECT.ORG
  Managed by: mbs.mbox.stg.centos.org, <modified>.fedoraproject.org
  Hosts allowed to retrieve keytab: <modified>.fedoraproject.org
-------------------------
Number of members added 1
-------------------------
[+] Retrieving TLS and keytab files ...
New signing request "20210705130958" added.
Keytab successfully retrieved and stored in: /etc/pki/centos/certs/HTTP-mbs.mbox.stg.centos.org.keytab
[+] Validating TLS against IPA CA ...
/etc/pki/centos/certs/mbs.mbox.stg.centos.org.crt: OK

Tip

You need a valid kerberos ticket for the existing REALM, otherwise same script will exit 1. Worth knowing that if you have enabled OTP on your account (a must for admins) you can use the 2fa-kinit script, also deployed by ansible on each enrolled node by the ipa-client ansible role

You can now import into pkistore (and correct directory based on role) git-crypted repository the needed .crt and .key files from /etc/pki/centos/certs/ directory

TLS service account¶

While the mentioned above script is probably the one that we'll use the most for nodes, we can also have to create service account, just to retrieve TLS cert used to auth against other services. As we'll just do that on very limited use cases, we can just "manually" execute the following snippet, still with first a valid kerberos ticket to be able to add users in IPA (so also on an enrolled machine and ideally the same one we use for the node certificates) :

# Let's first define some variables
service_account="mbox_stg_kojira"
full_name="CentOS kojira mbox STG service user"
realm="STG.FEDORAPROJECT.ORG"

# Before next steps we *have* to have ipa rights to create ipa users and valid kerberos ticket
pushd /etc/pki/centos/certs >/dev/null
ipa user-show ${service_account} >/dev/null 2>&1 || ipa user-add --cn="${full_name}" --displayname="${full_name}" --password --first=${service_account} --last=${service_account} ${service_account}

# Now that we have created the account with strong and random password, we can create private key and csr and ask IPA CA to sign it
# Create a private key and csr first
test -e ${service_account}.key || openssl req -new -newkey rsa:2048 -days 3650 -nodes -keyout ${service_account}.key -out ${service_account}.csr -subj "/CN=${service_account}"
# kinit as user (will ask password and also twice to set new password which can be the same one we decided to use
kinit ${service_account}@${realm}
ipa cert-request ${service_account}.csr --principal=${service_account} --profile-id=userCerts --certificate-out=${service_account}.crt && rm ${service_account}.csr

You can now push both .key/.crt files into pkistore git-crypted repository and also record the service account password in the IPA-service-accounts file in that same git-crypted repository

Red Hat CA (internal setup only)¶

Public certificates¶

Letsencrypt¶

Warning

This is now the prefered way to retrieve and use TLS certs in the CentOS infra for all public services.

We use one dedicated node to obtain/renew certs for the acme http challenges, and also the same for dns challenges (for internal openshift setup). Actually that node is acme01.rdu2.centos.org.

How to obtain new cert (DNS challenge is the preferred way)¶

For dns challenge¶

We have automated the delegated dynamic zone needed for acme-challenge update with acme.sh We just have once to add in our zone (main one, so centos.org) a CNAME pointing to the delegated zone acme.centos.org Example (simple) for forums.centos.org :

_acme-challenge.forums IN  CNAME _acme-challenge.forums.acme.centos.org.

Now on certbot node, we can just ask acme.sh to dynamically update acme.centos.org with our ddns.key file (already present) that is permitted to update the acme.centos.org and instruct acme.sh that while asking for a record in centos.org, it has to update other TXT record for the 'acme-challenge' record

acme.sh --issue --keylength 2048 --dns dns_nsupdate -d forums.centos.org --challenge-alias forums.acme.centos.org

Let's see what this produces on our DNS node, basically updating TXT record:

Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: signer "ddns" approved
Dec 10 10:35:57 ns1 named[28134]: client @0x7fe6240e93a0 8.43.84.3#59340/key ddns: updating zone 'acme.centos.org/IN': adding an RR at '_acme-challenge.forums.acme.centos.org' TXT "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"

And back on the certbot node, where it just updates, then waits and finalizes the acme validation with letsencrypt servers:

[Tue 10 Dec 10:35:56 UTC 2019] Single domain='forums.centos.org'
[Tue 10 Dec 10:35:56 UTC 2019] Getting domain auth token for each domain
[Tue 10 Dec 10:35:57 UTC 2019] Getting webroot for domain='forums.centos.org'
[Tue 10 Dec 10:35:57 UTC 2019] Adding txt value: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain:  _acme-challenge.forums.acme.centos.org
[Tue 10 Dec 10:35:57 UTC 2019] adding _acme-challenge.forums.acme.centos.org. 60 in txt "mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ"
[Tue 10 Dec 10:35:57 UTC 2019] The txt record is added: Success.
[Tue 10 Dec 10:35:57 UTC 2019] Let's check each dns records now. Sleep 20 seconds first.
[Tue 10 Dec 10:36:19 UTC 2019] Checking forums.centos.org for _acme-challenge.forums.acme.centos.org
[Tue 10 Dec 10:36:19 UTC 2019] Domain forums.centos.org '_acme-challenge.forums.acme.centos.org' success.
[Tue 10 Dec 10:36:19 UTC 2019] All success, let's return
[Tue 10 Dec 10:36:19 UTC 2019] Verifying: forums.centos.org
[Tue 10 Dec 10:36:22 UTC 2019] Success
[Tue 10 Dec 10:36:22 UTC 2019] Removing DNS records.
[Tue 10 Dec 10:36:22 UTC 2019] Removing txt: mKq4hBQnYsbWEmTYEkZu6OjVsQJBGQsXWS0c8Zdu9hQ for domain: _acme-challenge.forums.acme.centos.org
[Tue 10 Dec 10:36:22 UTC 2019] removing _acme-challenge.forums.acme.centos.org. txt
[Tue 10 Dec 10:36:22 UTC 2019] Removed: Success
[Tue 10 Dec 10:36:22 UTC 2019] Verify finished, start to sign.
[Tue 10 Dec 10:36:22 UTC 2019] Lets finalize the order, Le_OrderFinalize: https://acme-v02.api.letsencrypt.org/acme/finalize/52322595/1718928579
[Tue 10 Dec 10:36:23 UTC 2019] Download cert, Le_LinkCert: https://acme-v02.api.letsencrypt.org/acme/cert/04cd1bdeeef194461a56d1323b11691aeccd
[Tue 10 Dec 10:36:24 UTC 2019] Cert success.

PS : for a wildcard, just add multiple -d and *.domain

acme.sh --issue --keylength 2048 --dns dns_nsupdate -d stg.centos.org -d '*.stg.centos.org' --challenge-alias stg.acme.centos.org

All certs/keys obtained through acme are under /root/.acme.sh/{hostname}/ so you'll then have to import those into this pkistore dir

Note

worth knowing that for AWS/Route53 hosted zones (as we now have some for CI infra and openshift), one can use --dns dns_aws [option](https://github.com/acmesh-official/acme.sh/wiki/dnsapi#10-use-amazon-route53-domain-api to request new TLS certs. And it will be automatic for renewal so nothing to worry aboutt

For http challenge¶

Normally we prefer DNS challenge, but there are corner cases like delegated records for which that would be problematic. That's the case for {buildlogs,cloud,vault}.centos.org nodes (delegated records to pdns/geoip)

You can add multiple SANs in the same certs. Here is one example with buildlogs.centos.org and SAN cloud.centos.org :

acme.sh --issue --keylength 2048 -d buildlogs.centos.org -d cloud.centos.org -w /var/www/html/

All files (certs/keys) are then available under /root/.acme.sh/{hostname} (you'll have to import those into this pkistore dir)

How to renew existing certs¶

For DNS challenges (existing records)¶

Each pkistore git repository (based on the env) will have a ./tools/letsencrypt-renew-import wrapper tool that will :

inspect .crt TLS files in the pkistore git repo
verify if that's signed by Let's Encrypt CA
access the central acme machine through ssh (from which you initially create new cert) and renew with acme.sh --renew -d ${domain} --force
retrieve the .crt and CA chain, and also corresponding .key
reencrypt (if needed, based on env) with ansible-vault

Once all done and validated, you can just git commit && git push back as usual

Example :

./tools/letsencrypt-renew-import
[+] Analyzing TLS cert accounts.centos.org.crt ...
 Renewing [accounts.centos.org.crt] on [acme01.rdu2.centos.org]
 TLS cert accounts.centos.org.crt remotely renewed so importing key/crt/cachain : SUCCESS 
 [accounts.centos.org.crt] validated against [/etc/pki/tls/certs/ca-bundle.crt accounts.centos.org-CAChain.crt] : SUCCESS 

[+] Analyzing TLS cert accounts.dev.centos.org.crt ...
 TLS [accounts.dev.centos.org.crt] file is a symlink so ignoring ... SKIPPED  

[+] Analyzing TLS cert accounts.stg.centos.org.crt ...
 Renewing [accounts.stg.centos.org.crt] on [acme01.rdu2.centos.org]
 TLS cert accounts.stg.centos.org.crt remotely renewed so importing key/crt/cachain : SUCCESS 
 [accounts.stg.centos.org.crt] validated against [/etc/pki/tls/certs/ca-bundle.crt accounts.stg.centos.org-CAChain.crt] : SUCCESS 
<...>

Note

if you'll have an error on a specific cert, just ssh into delegate machine for acme.sh and manually kick acme.sh --renew -d <domain> --force to see the output and fix the underlying issue (if any)

For HTTP challenges¶

Same as for dns challenges as we consolidated all under acme.sh (and no certbot anymore)

Deploying through ansible¶

Don't forget to have pushed the new/renewed certs/keys into this pkistore directory first. Important too : Please validate that the -CAChain.crt is linked to correct CA chain. As LetsEncrypt is rotating also their CA, please validate that your .crt is correctly being validated with correct CAchain with simple provided tool in this repository :

./validate_public_chain koji.mbox.centos.org.crt 
Validating cert [koji.mbox.centos.org.crt] with CAChain [koji.mbox.centos.org-CAChain.crt] and default trusted ca-bundle
koji.mbox.centos.org.crt: OK

Let's consider now three infrastructures and how to push renewed certs :

CentOS public infra (including .dev. , .stg. and ci infra)¶

Once it's committed/pushed to pkistore git repo, the ansible bot will deploy the renewed TLS certs automatically. You can still "force" the playbook execution if you want, from ansible bot host but should be done automatically and you can see reports through ARA.

CentOS Stream infra¶

Same as for other parts of infra, except that you have to encrypt with ansible-vault before git commit/git push operations (important). Once done :

ansible-playbook-stream playbooks/role-haproxy.yml --tags "tls,pki"

Note

Fedora Infra team is responsible for mirrors.centos.org one, as it's hosted on their infra. In the past we were generating/renewing cert but it seems now automated