SSH Host & User Certificates

SSH

Published

March 21, 2021

Modified

October 16, 2024

Allows one SSH key to sign another SSH key, resulting in an “SSH certificate”

OpenSSH certificates are an extension build using public keys
- Introduced in OpenSSH 5.4 (released 2010-03-08)
- does not use PEM, X.509 ASN.1 (as in SSL)
certificate authority (CA) is a special trusted party holding own public-private key-pairs
- alleviates the need to distribute SSH public keys
- an OpenSSH CA is a regular private key
- not used for authentication, only to sign SSH certificates
- signs user keys with capabilities and expiration date
- signs host keys for a domain
certificates include:
- nonce - unique ID to prevent signature collision attacks
- public key - associated with a private key
- yype - identifies user or host certificates
- key ID - identifies the user or host in loge messages
- (valid) principles - list of user or host names
- validity interval - start time, and expiration date
- critical options - supported client requests
- extensions - optional SSH extensions
- signature Key - CA public key, used to sign certificate (with private key)
- signature - CA issued signature of all preceding fields
the validity interval is specified like <start_time>:<end_time>
- start time always (no specified start time)
- end time forever (never expire)
- date format is YYYYMMDD, time format YYYYMMDDHHMM[SS]
- relative time starting with + or -, suffix w (week) d (day) m (minute)

+1w1d           # valid from node to 1 week and 1 day
-4w:+4w         # valid from four weeks ago to four weeks from now
-1d:20201231    # valid from yesterday to midnight December 31st 2020
-1m:forever     # valid from one minute ago and never expiring

Authorized Principles

AuthorizedPrincipalsFile specifies a file that lists principal names that are accepted for certificate authentication. When using certificates signed by a key listed in TrustedUserCAKeys, this file lists names, one of which must appear in the certificate for it to be accepted for authentication.

cf. AUTHORIZED_KEYS FILE FORMAT in the sshd_config.5 manual page
configuration in /etc/ssh/sshd_config:

AuthorizedPrincipalsFile /etc/ssh/%u_principals

accepts following tokens:
- %% a literal %
- %h home directory of the user
- %u user name
names are listed one per line including key options like

# principle     # options
devops          command=/path/to/command,no-agent-forwarding,no-port-forwarding,no-pty

Host Certificates

…alternative to host public key authentication
hosts send signed SSH certificates to clients in order to enable the verification of the host’s identity
- host certificate is signed by a trusted certificate authority (CA)
- host certificate includes host name (principle) and expiration date
- clients check if a trusted CA signed the host certificate
- trusted CA public keys are listed in known_hosts
- warning on signature check failure or if a CA is not trusted
- clients don’t store public keys for every host connection
generate a CA public key-pair to sign host keys with ssh-keygen:
- option -f defines the name of the (output) private key file
- option -C provides a comment to identify the CA key-pair

ssh-keygen -f devops-host_ca-$(mktemp -u XXXXXX) -C "Host signing key for DevOps"
# the public key gets .pub appended

best practice is to have multiple sets of CA keys with different expiration dates
- allows to revoke a key if required, while maintaining access to the infrastructure
- generally it is useful to follow a naming convention like <organisation>-<key_identifier>-<unique_id>

# the example above, would create a public and private key-pair like
devops-host_ca-Gb3t8s
devops-host_ca-Gb3t8s.pub

Signing

# SSH client connection configuration
cat > ssh_config <<EOF
StrictHostKeyChecking=no
UserKnownHostsFile=/dev/null
EOF
# download the public host key from a node
scp -F ssh_config root@lxdev01:/etc/ssh/ssh_host_rsa_key.pub .

Use the CA key-pair to sign the host public key using the ssh-keygen command:

Option -h creates a host certificate instead of a user certificate
Option -s specifies a path to a CA private key file
Option -V specifies a validity interval when signing a certificate
Option -n specifies one or more principals (host names)
Option -I specifies an identification string used in log output

# sign the host key with the CA signing key
host_ca_private_key=$(ls | egrep -o 'devops-host_ca-[A-Za-z0-9-]{6}' | uniq)
ssh-keygen -h -s $host_ca_private_key \
           -V -1d:+52w \
           -n lxdev01,lxdev01.devops.test \
           -I 'lxdev01.devops.test host certificate' \
    ssh_host_rsa_key.pub
# upload the host certificate to the node
scp -F ssh_config ssh_host_rsa_key-cert.pub root@lxdev01:/etc/ssh

The example above is just for illustration purpose, and not the recommended way of distributing host certificates

Inspect the host certificate:

» ssh-keygen -L -f ssh_host_rsa_key-cert.pub
ssh_host_rsa_key-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com host certificate
        Public key: RSA-CERT SHA256:iVBchuhVcTKvUA4XZb5ldnP2FMgiDKcqaIsWCq9ChIQ
        Signing CA: RSA SHA256:zTEUXG8CJ0j9l7s8wt1couYyHD+u8gFjpawbsNmxoFk
        Key ID: "lxdev01.devops.test host certificate"
        Serial: 0
        Valid: from 2020-01-23T11:26:51 to 2021-01-22T11:26:51
        Principals:
                lxdev01.devops.test
        Critical Options: (none)
        Extensions: (none)

Host Configuration

Enable a host certificate in the sshd server configuration:

ssh -F ssh_config root@lxdev01 <<EOF
    echo HostCertificate=/etc/ssh/ssh_host_rsa_key-cert.pub >> /etc/ssh/sshd_config 
    echo LogLevel=DEBUG3 >> /etc/ssh/sshd_config
    sudo systemctl restart sshd
EOF
# follow the log information
ssh -F ssh_config root@lxdev01 -C journalctl -fu sshd

Cf. sshd manual page:

HostCertificate specifies a file containing a public host certificate. The certificate’s public key must match a private host key already specified by HostKey.

Client Known Hosts

Clients use a host certificate to verify the node integrity with the CA public key. Therefore add the CA public key to known_hosts file:

echo "@cert-authority * $(cat $host_ca_private_key.pub)" > ssh_known_hosts

The resulting file uses a wildcard to match all hostnames with the CA:

@cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnGv...

Cf. sshd manual page:

Each line in these files contains the following fields: markers (optional), hostnames, bits, exponent, modulus, comment… The fields are separated by spaces… The marker is optional, but if it is present then it must be one of “@cert-authority”, to indicate that the line contains a certification authority (CA) key… Hostnames is a comma-separated list of patterns (* and ? act as wildcards); each pattern in turn is matched against the canonical host name In order to test the configurations enable host key checking, and use the SSH known host file create above:

cat > ssh_config <<EOF
StrictHostKeyChecking=yes
UserKnownHostsFile=ssh_known_hosts
EOF
# connect using the known hosts file
ssh -vvv -F ssh_config root@lxdev01

A successful host key verification will print following log information:

debug1: Server host certificate: ... "lxdev01.devops.test host certificate" CA...
debug2: Server host certificate hostname: lxdev01
debug2: Server host certificate hostname: lxdev01.devops.test
debug3: hostkeys_foreach: reading file "ssh_known_hosts"
debug3: record_hostkey: found ca key type RSA in file ssh_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from lxdev01
debug1: Host 'lxdev01' is known and matches the RSA-CERT host certificate.
debug1: Found CA key in ssh_known_hosts:1

User Certificates

Generate a CA certificate used as signing keys:

ssh-keygen -f user_ca
# writes user_ca & user_ca.pub

Make sure to keep these keys save, since to enable access to all nodes trusting the CA.

Server Configuration

Configure sshd to accept user certificates signed by an CA certificate:

Copy the public certificate to the servers /etc/ssh directory
Configure TrustedUserCAKeys in /etc/ssh/sshd_config

# copy the public certificate to the servers
scp user_ca.pub root@lxdev01:/etc/ssh
# make sshd trust the public CA certificate, and restart in debugging mode
ssh root@lxdev01 -C '
    echo TrustedUserCAKeys /etc/ssh/user_ca.pub >> /etc/ssh/sshd_config
    grep ^TrustedUserCAKeys /etc/ssh/sshd_config
    systemctl stop sshd
    $(which sshd) -d
'

Sign User Keys

Generate user certificates to grand access priviliges:

## create a user key (no password for testing)
ssh-keygen -q -t rsa -b 2048 -N '' -f id_rsa
# create id_rsa & id_rsa.pub
## sign the user public key
ssh-keygen -U -s user_ca -V +2w -n devops -I 'SSH key for user devops' id_rsa.pub
# creates id_rsa-cert.pub
## inspect the user certificate
ssh-keygen -L -f id_rsa-cert.pub

Available options:

-U enables ssh-agent host support
-s $sign_key CA user signing key
-V $interval validation interval (aka certificate life-time)
-I $key_id “key identifier” that is logged by the server when the certificate is used for authentication

Principles are defined with option -n:

By default, generated certificates are valid for all users or hosts.
Generate a certificate for a specified set of principals

# allows user and root login
ssk-keygen ... -n root,devops ...

Additional limitations on the validity and use of user certificates may be specified through certificate options with -O:

Disable features of the SSH session
Limit user to a particular source addresses
Force the use of a specific command

# Restrict the source addresses from which the certificate is considered valid.
ssh-keygen ... -O source-address=10.1.1.1/24 ...   # CIDR format
# forces execution of command instead of any shell or command specified
ssh-keygen ... -O force-command=/usr/bin/journalctl ...

Using User-Certificates

Connect with an private key identity, ssh will also try to load certificate information from the filename obtained by appending -cert.pub to identity filenames:

# for debugging
cat > ssh_config <<EOF
PasswordAuthentication=no
StrictHostKeyChecking=no
UserKnownHostsFile=/dev/null
EOF
# will implicitly load id_rsa-cert.pub if present
ssh -F ssh_config -v -i id_rsa devops@lxdev01

Debugging will show if the certificate is loaded:

Will attempt key: id_rsa RSA SHA256... explicit                    
Will attempt key: id_rsa RSA-CERT SHA256... explicit

And if the server accepted the key:

Offering public key: id_rsa RSA-CERT SHA256.... explicit
Server accepts key: id_rsa RSA-CERT SHA256... explicit
Authentication succeeded (publickey).

In case of a configuration problem following error message is emitted by sshd:

key_cert_check_authority: invalid
Certificate invalid: name is not a listed principal

Otherwise a successful login emits:

Accepted certificate ID "..." (serial 0) signed by RSA CA SHA256... via /etc/ssh/user_ca.pub
do_pam_account: called Accepted publickey for ... ssh2: RSA-CERT ID SSH key for user ... CA RSA SHA256...

--- title: SSH Host & User Certificates categories: - SSH date: 2021/03/21 date-modified: 2024/10/16 toc-expand: 3 --- Allows one SSH key to sign another SSH key, resulting in an “SSH certificate” * OpenSSH certificates are an extension build using public keys - Introduced in OpenSSH 5.4 (released 2010-03-08) - does not use PEM, X.509 ASN.1 (as in SSL) * **certificate authority** (CA) is a special trusted party holding own public-private key-pairs - alleviates the need to distribute SSH public keys - an OpenSSH CA is a regular private key - not used for authentication, only to sign SSH certificates - signs user keys with capabilities and expiration date - signs host keys for a domain * certificates include: - nonce - unique ID to prevent signature collision attacks - public key - associated with a private key - yype - identifies user or host certificates - key ID - identifies the user or host in loge messages - (valid) principles - list of user or host names - validity interval - start time, and expiration date - critical options - supported client requests - extensions - optional SSH extensions - signature Key - CA public key, used to sign certificate (with private key) - signature - CA issued signature of all preceding fields * the **validity interval** is specified like `<start_time>:<end_time>` - start time `always` (no specified start time) - end time `forever` (never expire) - date format is `YYYYMMDD`, time format `YYYYMMDDHHMM[SS]` - relative time starting with `+` or `-`, suffix `w` (week) `d` (day) `m` (minute) ```bash +1w1d # valid from node to 1 week and 1 day -4w:+4w # valid from four weeks ago to four weeks from now -1d:20201231 # valid from yesterday to midnight December 31st 2020 -1m:forever # valid from one minute ago and never expiring ``` # Authorized Principles > `AuthorizedPrincipalsFile` specifies a file that lists principal names that > are accepted for certificate authentication. When using certificates signed > by a key listed in `TrustedUserCAKeys`, this file lists names, one of which > must appear in the certificate for it to be accepted for authentication. * cf. `AUTHORIZED_KEYS FILE FORMAT` in the `sshd_config.5` manual page * configuration in `/etc/ssh/sshd_config`: ``` AuthorizedPrincipalsFile /etc/ssh/%u_principals ``` * accepts following **tokens**: - `%%` a literal % - `%h` home directory of the user - `%u` user name * names are listed one per line including key options like ``` # principle # options devops command=/path/to/command,no-agent-forwarding,no-port-forwarding,no-pty ``` # Host Certificates * ...alternative to host public key authentication * hosts send signed SSH certificates to clients in order to enable the verification of the host's identity - host certificate is signed by a trusted certificate authority (CA) - host certificate includes host name (principle) and expiration date - clients check if a trusted CA signed the host certificate - trusted CA public keys are listed in `known_hosts` - warning on signature check failure or if a CA is not trusted - clients don't store public keys for every host connection * **generate a CA public key-pair** to sign host keys with `ssh-keygen`: - option `-f` defines the name of the (output) private key file - option `-C` provides a comment to identify the CA key-pair ```bash ssh-keygen -f devops-host_ca-$(mktemp -u XXXXXX) -C "Host signing key for DevOps" # the public key gets .pub appended ``` * best practice is to have multiple sets of CA keys with different expiration dates - allows to revoke a key if required, while maintaining access to the infrastructure - generally it is useful to follow a naming convention like `<organisation>-<key_identifier>-<unique_id>` ```bash # the example above, would create a public and private key-pair like devops-host_ca-Gb3t8s devops-host_ca-Gb3t8s.pub ``` ## Signing ```bash # SSH client connection configuration cat > ssh_config <<EOF StrictHostKeyChecking=no UserKnownHostsFile=/dev/null EOF # download the public host key from a node scp -F ssh_config root@lxdev01:/etc/ssh/ssh_host_rsa_key.pub . ``` Use the CA key-pair to **sign the host public key** using the `ssh-keygen` command: * Option `-h` creates a host certificate instead of a user certificate * Option `-s` specifies a path to a **CA private key file** * Option `-V` specifies a **validity interval** when signing a certificate * Option `-n` specifies one or more **principals** (host names) * Option `-I` specifies an identification string used in log output ```bash # sign the host key with the CA signing key host_ca_private_key=$(ls | egrep -o 'devops-host_ca-[A-Za-z0-9-]{6}' | uniq) ssh-keygen -h -s $host_ca_private_key \ -V -1d:+52w \ -n lxdev01,lxdev01.devops.test \ -I 'lxdev01.devops.test host certificate' \ ssh_host_rsa_key.pub # upload the host certificate to the node scp -F ssh_config ssh_host_rsa_key-cert.pub root@lxdev01:/etc/ssh ``` _The example above is just for illustration purpose, and not the recommended way of distributing host certificates_ Inspect the host certificate: ```bash » ssh-keygen -L -f ssh_host_rsa_key-cert.pub ssh_host_rsa_key-cert.pub: Type: ssh-rsa-cert-v01@openssh.com host certificate Public key: RSA-CERT SHA256:iVBchuhVcTKvUA4XZb5ldnP2FMgiDKcqaIsWCq9ChIQ Signing CA: RSA SHA256:zTEUXG8CJ0j9l7s8wt1couYyHD+u8gFjpawbsNmxoFk Key ID: "lxdev01.devops.test host certificate" Serial: 0 Valid: from 2020-01-23T11:26:51 to 2021-01-22T11:26:51 Principals: lxdev01.devops.test Critical Options: (none) Extensions: (none) ``` ## Host Configuration **Enable a host certificate** in the `sshd` server configuration: ```bash ssh -F ssh_config root@lxdev01 <<EOF echo HostCertificate=/etc/ssh/ssh_host_rsa_key-cert.pub >> /etc/ssh/sshd_config echo LogLevel=DEBUG3 >> /etc/ssh/sshd_config sudo systemctl restart sshd EOF # follow the log information ssh -F ssh_config root@lxdev01 -C journalctl -fu sshd ``` Cf. `sshd` manual page: > `HostCertificate` specifies a file containing a public host certificate. The certificate's public key must match a private host key already specified by `HostKey`. ## Client Known Hosts Clients use a host certificate to verify the node integrity with the CA public key. Therefore **add the CA public key to `known_hosts` file**: ```bash echo "@cert-authority * $(cat $host_ca_private_key.pub)" > ssh_known_hosts ``` The resulting file uses a wildcard to **match all hostnames** with the CA: ``` @cert-authority * ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnGv... ``` Cf. `sshd` manual page: > Each line in these files contains the following fields: markers (optional), hostnames, bits, exponent, modulus, comment... The fields are separated by spaces... The **marker** is optional, but if it is present then it must be one of “`@cert-authority`”, to indicate that the line contains a certification authority (CA) key... > Hostnames is a comma-separated list of patterns (`*` and `?` act as wildcards); each pattern in turn is matched against the canonical host name In order to test the configurations enable host key checking, and use the SSH known host file create above: ```bash cat > ssh_config <<EOF StrictHostKeyChecking=yes UserKnownHostsFile=ssh_known_hosts EOF # connect using the known hosts file ssh -vvv -F ssh_config root@lxdev01 ``` A successful host key verification will print following log information: ```bash debug1: Server host certificate: ... "lxdev01.devops.test host certificate" CA... debug2: Server host certificate hostname: lxdev01 debug2: Server host certificate hostname: lxdev01.devops.test debug3: hostkeys_foreach: reading file "ssh_known_hosts" debug3: record_hostkey: found ca key type RSA in file ssh_known_hosts:1 debug3: load_hostkeys: loaded 1 keys from lxdev01 debug1: Host 'lxdev01' is known and matches the RSA-CERT host certificate. debug1: Found CA key in ssh_known_hosts:1 ``` # User Certificates Generate a **CA certificate** used as signing keys: ```bash ssh-keygen -f user_ca # writes user_ca & user_ca.pub ``` _Make sure to keep these keys save, since to enable access to all nodes trusting the CA._ ## Server Configuration Configure `sshd` to accept user certificates signed by an CA certificate: * Copy the public certificate to the servers `/etc/ssh` directory * Configure `TrustedUserCAKeys` in `/etc/ssh/sshd_config` ```bash # copy the public certificate to the servers scp user_ca.pub root@lxdev01:/etc/ssh # make sshd trust the public CA certificate, and restart in debugging mode ssh root@lxdev01 -C ' echo TrustedUserCAKeys /etc/ssh/user_ca.pub >> /etc/ssh/sshd_config grep ^TrustedUserCAKeys /etc/ssh/sshd_config systemctl stop sshd $(which sshd) -d ' ``` ## Sign User Keys Generate **user certificates** to grand access priviliges: ```bash ## create a user key (no password for testing) ssh-keygen -q -t rsa -b 2048 -N '' -f id_rsa # create id_rsa & id_rsa.pub ## sign the user public key ssh-keygen -U -s user_ca -V +2w -n devops -I 'SSH key for user devops' id_rsa.pub # creates id_rsa-cert.pub ## inspect the user certificate ssh-keygen -L -f id_rsa-cert.pub ``` Available options: * `-U` enables `ssh-agent` host support * `-s $sign_key` CA user signing key * `-V $interval` validation interval (aka certificate life-time) * `-I $key_id` "key identifier" that is logged by the server when the certificate is used for authentication **Principles** are defined with option `-n`: * By default, generated certificates are valid for all users or hosts. * Generate a certificate for a specified set of principals ```bash # allows user and root login ssk-keygen ... -n root,devops ... ``` Additional limitations on the validity and use of user certificates may be specified through **certificate options** with `-O`: * Disable features of the SSH session * Limit user to a particular source addresses * Force the use of a specific command ```bash # Restrict the source addresses from which the certificate is considered valid. ssh-keygen ... -O source-address=10.1.1.1/24 ... # CIDR format # forces execution of command instead of any shell or command specified ssh-keygen ... -O force-command=/usr/bin/journalctl ... ``` ## Using User-Certificates Connect with an private key identity, ssh will also try to load certificate information from the filename obtained by appending `-cert.pub` to identity filenames: ```bash # for debugging cat > ssh_config <<EOF PasswordAuthentication=no StrictHostKeyChecking=no UserKnownHostsFile=/dev/null EOF # will implicitly load id_rsa-cert.pub if present ssh -F ssh_config -v -i id_rsa devops@lxdev01 ``` Debugging will show if the certificate is loaded: ``` Will attempt key: id_rsa RSA SHA256... explicit Will attempt key: id_rsa RSA-CERT SHA256... explicit ``` And if the server accepted the key: ``` Offering public key: id_rsa RSA-CERT SHA256.... explicit Server accepts key: id_rsa RSA-CERT SHA256... explicit Authentication succeeded (publickey). ``` In case of a configuration problem following error message is emitted by `sshd`: ``` key_cert_check_authority: invalid Certificate invalid: name is not a listed principal ``` Otherwise a successful login emits: ``` Accepted certificate ID "..." (serial 0) signed by RSA CA SHA256... via /etc/ssh/user_ca.pub do_pam_account: called Accepted publickey for ... ssh2: RSA-CERT ID SSH key for user ... CA RSA SHA256... ```