We describe the procedure for emergency access to SSH hosts with hardware keys

In this post, we will develop a procedure for emergency access to SSH hosts using offline security keys. This is just one of the approaches, and you can adapt it for yourself. We will store the SSH certification authority for our hosts on a hardware security key. This scheme will work on almost any OpenSSH, including SSH with a single input.

Why all this? Well, this is an emergency option. This is a backdoor that will allow you to access your server when for some reason nothing else helps.

Why use certificates instead of public / private keys for emergency access?

Unlike public keys, certificate validity can be very short. You can generate a certificate valid for 1 minute or even 5 seconds. After this period, the certificate will become unusable for new connections. This is ideal for emergency access.
You can create a certificate for any account on your hosts and, if necessary, send such “one-time” certificates to colleagues.

What do you need

Hardware security keys that support resident keys.
Resident keys are cryptographic keys that are completely stored inside the security key. Sometimes they are protected by an alphanumeric PIN code. The public part of the resident key can be exported from the security key, if necessary, together with the private key descriptor. Support for resident keys are, for example, USB-keys of the Yubikey 5 series. It is desirable that they are intended only for emergency access to the host. For this post I will use only one key, but you must have an additional one for backup.
A safe place to store these keys.
OpenSSH version 8.2 or higher on your local computer and on the servers to which you want to get emergency access. Ubuntu 20.04 ships with OpenSSH 8.2.
(optional, but desirable) CLI certificate verification tool.

Training

First you need to create a certification authority, which will be located on the hardware security key. Insert the key and run:

$ ssh-keygen -t ecdsa-sk -f sk-user-ca -O resident -C [security key ID]

As a comment (-C), I indicated yubikey-9-512-742@smallstep.com so as not to forget which security key this certification authority belongs to.

In addition to adding the key to Yubikey, two files will be generated locally:

sk-user-ca, a key descriptor that refers to the private key stored in the security key,
sk-user-ca.pub, which will be the public key for your certification authority.

But don’t worry, Yubikey has one more private key that cannot be retrieved. Therefore, everything is reliable here.

On hosts as root, add (if you have not added it yet) to the configuration of your SSHD (/ etc / ssh / sshd_config) the following:

TrustedUserCAKeys /etc/ssh/ca.pub

Then on the host add the public key (sk-user-ca.pub) to /etc/ssh/ca.pub
Reboot the server:

systemctl sshd restart

Now we can try to access the host. But first, we need a certificate. Create a key pair that will be associated with the certificate:

$ ssh-keygen -t ecdsa -f emergency

Certificates and SSH pairs
Sometimes it is tempting to use a certificate as a replacement for a public / private key pair. But for user authentication, a single certificate is not enough. Each certificate also has a private key associated with it. That’s why we need to generate this pair of “emergency” keys before we issue a certificate to ourselves. It is important that we show the signed certificate to the server, indicating the key pair for which we have a private key.
Thus, the public key exchange is still alive and well. It rolls even with certificates. Certificates simply save the server from having to store public keys.

Next, create the certificate itself. I need ubuntu user authorization in a 10 minute interval. You can do it your own way.

$ ssh-keygen -s sk-user-ca -I test-key -n ubuntu -V -5m:+5m emergency

You will be asked to sign the certificate with your fingerprint. You can add additional usernames, separated by commas, for example, -n ubuntu, carl, ec2-user

That’s it, now you have a certificate! Next, you need to specify the correct permissions:

$ chmod 600 emergency-cert.pub

After that, you can familiarize yourself with the contents of your certificate:

$ step ssh inspect emergency-cert.pub

Here’s what mine looks like:

emergency-cert.pub
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
        Public key: ECDSA-CERT SHA256:EJSfzfQv1UK44/LOKhBbuh5oRMqxXGBSr+UAzA7cork
        Signing CA: SK-ECDSA SHA256:kLJ7xfTTPQN0G/IF2cq5TB3EitaV4k3XczcBZcLPQ0E
        Key ID: "test-key"
        Serial: 0
        Valid: from 2020-06-24T16:53:03 to 2020-06-24T17:03:03
        Principals:
                ubuntu
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Here, the public key is the emergency key we created, and sk-user-ca is associated with the certification authority.

Finally, we are ready to run the SSH command:


$ ssh -i emergency ubuntu@my-hostname
ubuntu@my-hostname:~$

Now you can create certificates for any user on a host who trusts your certification authority.
You can remove emergency. You can save sk-user-ca, but you do not need it, since it is also on the security key. You might also want to remove the original PEM public key from your hosts (for example, in ~ / .ssh / authorized_keys for the ubuntu user) if you used it for emergency access.

Emergency Access: Action Plan

Insert the security key and run the command:

$ ssh-add -K

Thus, you add the public key of the certification authority and the key descriptor to the SSH agent.
Now export the public key to make the certificate:

$ ssh-add -L | tail -1 > sk-user-ca.pub

Create a certificate with an expiration date, for example, no more than an hour:

$ ssh-keygen -t ecdsa -f emergency
$ ssh-keygen -Us sk-user-ca.pub -I test-key -n [username] -V -5m:+60m emergency
$ chmod 600 emergency-cert.pub

And now again SSH:

$ ssh -i emergency username@host

If your .ssh / config file causes some kind of connection problems, you can run ssh with the -F none option to get by without it. If you need to send a certificate to a colleague, the easiest and safest option is Magic wormhole. To do this, you will need only two files – in our case it is emergency and emergency-cert.pub.

What I like about this approach is the hardware support. You can put the security keys in a safe and they will not go anywhere.

As an advertisement

Epic servers – this is cheap vps with powerful processors from AMD, CPU core frequency up to 3.4 GHz. The maximum configuration allows you to solve almost any task – 128 CPU cores, 512 GB RAM, 4000 GB NVMe. Join now!