Sunday 1 February 2015

Key rotation in OpenSSH 6.8+

Update: Two things that I neglected to mention in the above: 1) host keys that are not offered as part of the server->client notification are automatically removed from known_hosts, and 2) the updating of known_hosts can be disabled using a new UpdateHostkeys option in ssh_config and ~/.ssh/config.

Update 2: Jann Horn (after Philipp Kern) points out that a malicious server (say, "host-a") could advertise the public key of another server (say, "host-b"). Then, when the client subsequently connects back to host-a, instead of answering the connection as usual itself, host-a could proxy the connection to host-b. This would cause the user to connect to host-b when they think they are connecting to host-a, which is a violation of the authentication the host key is supposed to provide. The solution for this is for the server to prove to the client that server is in possession of the private key as well as the public. I'll add this before release (or delay the feature), but it's going to make the key rotation more onerous.

Update 3: (last one, promise). See the follow-up post.

Something that's bugged me about the SSH protocol is its lack of key continuity - key algorithm changes and key rotations are basically unsupported, as there is no in-protocol way for a client to learn updated host keys for hosts that the user already trusts. About the best one can do is cat /etc/ssh/*.pub once logged in to manually learn the host's other keys, but this only works if you have shell access and is a kludge anyway...

This makes it difficult for users to switch away from weak public key algorithms like ssh-dss to stronger ones and makes it practically impossible for a host to gracefully rotate its hostkeys. (I'm ignoring host certificates here, which do solve the problem but are mostly useful within an organisation).

These problems have become more urgent as the DSA supported in the SSH protocol has not aged gracefully, being within the range of a motivated attacker now. OpenSSH has preferred other host key algorithms for over 14 years, but it is the only "REQUIRED" public key algorithm in the SSH transport protocol (RSA was only "RECOMMENDED" because its patent hadn't expired back when they were written, and ECDSA was still future-tech), so people who have maintained systems for long periods of time could still be using this crappy algorithm today.

I've wanted the SSH protocol to provide a way to get users onto better host key algorithms for a while and finally got around to implementing it a couple of weeks ago: OpenSSH 6.8 will ship with a protocol extension that allows a server to inform a client of all of its host keys, and support in the client to update known_hosts when such a message is received. So, when an OpenSSH ≥6.8 client connects to a OpenSSH ≥6.8 server (or any other client/server that adopts the extension) where the user already trusts or explicitly accepts the host key, the user's known_hosts file will be updated with all the server's host keys, not just the one that authenticated the host during key exchange.

This fixes both the shortcomings I mentioned above: first, the client learns all the server's host key types, and can select the best possible host key algorithm (ed25519 is our current favourite) on subsequent connections. Secondly, it allows a server to gracefully rotate keys by publishing additional keys for a period to allow clients to learn them, before removing the deprecated key(s) and letting the new ones become the primary ones.

To practically rotate host keys, the operator of a sshd server should add additional HostKey statements to their sshd_config for the new keys, while keeping the existing keys in place. I.e.

# Old keys
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_dsa_key
# New keys
HostKey /etc/ssh/ssh_host_ed25519_key_new
HostKey /etc/ssh/ssh_host_rsa_key_new
HostKey /etc/ssh/ssh_host_ecdsa_key_new
Once these lines are added, sshd can be restarted to begin offering the new keys to clients who connect. The old keys, appearing first in the configuration, will be the ones actually used to authenticate the host to the client. Once the operator is satisfied that enough users have had a chance to pick up the new keys, they can rename replace the original keys with the new keys and remove the extra HostKey lines from sshd_config.

This mechanism isn't perfect: first, it only works for users who actually connect to the server - if they don't happen to connect during the grace period when both old and new keys are offered then they will have to learn the new key manually afterwards. As such, it doesn't cope well with sudden key rotations (e.g. when the operator has reason to believe a key has been compromised).

Fortunately both these cases can be addressed with a bit of forethought: when setting up a server, generate some reserve keys. Keep their private halves offline (e.g in a safe), but list them in sshd_config as described above. OpenSSH ≥6.8 will notify clients of the reserve public keys on every connection, so if you ever need to rotate keys in a hurry then they are already ready to go. You could generate multiple sets of reserve keys if you like - you aren't limited to a single set.

The host key rotation support is in OpenBSD -current and the git HEAD version of portable OpenSSH. Please try it out!

If you implement SSH yourself and want to add the extension to your own software, then setails of the protocol extension are in the PROTOCOL file in the source distribution. It's trivial though: a global request, sent once after authentication completes containing one or more public key blobs encoded as strings. In my biased opinion, its a small, easy to implement tweak to the SSH protocol that provides a significant improvement to the protocol.


  1. This doesn't assert that the server has the private half of the keys it sent, though. Correct? (How does that work with regular host key exchange? I would assume that the handshake is then encrypted to the presented key and hence key control is asserted?)

    1. No, the server doesn't have to have the private key at hand. This is desirable, as it lets the server admin advertise keys that are offline.

      Yes, the message with the additional hostkeys is sent after key exchange and regular host authentication has completed. The channel is secure by this point, and the message is only acted upon by the client if the client either already trust the host key used to authenticate the host during key exchange (i.e. it was already in known_hosts) -or- they learned it for the first time by explicitly typing "yes" at the new host key prompt.

      Hostkey update messages are ignored from a server that was not authenticated (i.e. user ignored a hostkey warning), loosely authenticated (key was in known hosts, but the associated port number was wrong) or unauthenticated (StrictHostKeyChecking=no).

    2. [I think sending the comment failed the first time, retrying...]
      Isn't this a security problem anyway because it allows a server to forward connections to another server? In scripted interactions with servers, that might be a problem IMO. Imagine code like this running on a management server with access to a bunch of other servers:
      scp $host:/etc/ssh/sshd_config .
      sed -i 's|^AcceptEnv\s.*|#\0|g'
      scp sshd_config $host:/etc/ssh/sshd_config
      ssh $host /etc/init.d/ssh reload
      If the first connection goes to the correct server (imagine an unimportant webserver on which an attacker now has root access) but the second and third connection are forwarded to another server (the holy LDAP server or so), the attacker now probably has code execution on the other server.

      Can't you require some kind of offline signing for hostkeys where the new key must sign the old one?

    3. I don't follow - the first server can't replace the keys for other servers (they are stored under separate hostnames). A MITM can't impersonate the other servers without the keys for those servers.

      A hostile server could advertise the public key for another server, but that wouldn't work either unless it had the private half to match.

    4. But a hostile server can replace the host key for himself, correct? If an automated script or so, running with a highly privileged private key, now wants to perform an operation on the hostile server, the hostile server could forward that connection to a victim server, correct? The script would check that the host key of the victim server matches the one specified by the hostile server, then perform the command intended for the hostile server on the victim server instead.

    5. Ok, that makes sense. I'll disable the extension in HEAD and add a way for the client to ask the server to prove ownership of the private keys.

    6. Thanks Philipp and Jann for pointing this out - I've updated the blog post.

  2. While im not at liberty to discuss i will say there has been a lot of chatter on the wire in the darknets and APT attacks coinciding with the ISIS migration from Syria into the baltics and red sea which would benefit from this type of incremental security from reaching US soil. Bravo!

  3. Will this work even if server key is authenticated by other means, like secure SSHFP records or SSH certificates?

    1. ATM yes, but I might adjust this before release.

    2. Anything new on that?

      I think other authentication means (especially SSHFP) might be considered less secure than the typical directly peer authenticated keys of server/client (when done properly) with ssh keys.

      So it might be bad if a ssh host key can added via one of the other methods.

    3. I don't plan on changing this before release (we're down to bugfixes only now), and probably won't after release either - I don't think it is necessary to automatically learn other hostkeys when the client already has some trusted ways to discover them (i.e. a trusted CA or trusting DNSSEC).

    4. Mhh I think that's a bad choice, especially since when using something DNSSEC/CA based, there's no real need to do this flooding of the clients with keys... the CAs respectively DNSSEC itself already provides the means for automatic propagation of new keys.

    5. Uhm then I've misunderstood something, I guess.

      The original poster asked, whether this also works by other means.
      You: ATM, yes.
      I: Does it still work
      You: No plans to chance this.

      So I've assumed, new keys would still be retrieved when auth is done via a non-ssh-key method.

  4. Will this work the other way around as well, from the client to the server?

    Automatically updating the "authorized_keys" file would save me a couple of hours every time i change one of my clients keys (i do a lot of non-interactive stuff too, like mercurial over ssh).

    1. There were proposals for a service to programmatically add/remove public keys from authorized_keys, but nothing ever became of them.

    2. If something like that should ever be implemented, special care has to be taken as well how it is done.

      Imagine things like gitolite, where the one connecting isn't granted shell access and cannot edit ~git/.ssh/authorized_keys.
      If such key was blindly added automatically, just because the connector's key is part of ~git/.ssh/authorized_keys (which may however contain restrictions like command="..." as in the case of gitolite),... then he could potentially evade these.

      Even if something like "a user can just exchange the key material of the line of the key he is using to connect to" might be done (instead of adding new lines to authorized_keys... it could be a problem,... think again of cases like the above, where the user cannot write to authorized_keys and where the one who does (e.g. gitolite server admin) wouldn't want to allow "weak" keys like DSA.

      So I think such functionality would be even more tricky than the server advertising more keys... might offer higher level ways to exploit.

  5. Why not mentioning also the sshfp (fingerprints of the ssh keys in the dns secured with dnssec) ?

    1. 1) that feature has been there for many years and 2) approximately nobody uses DNSSEC

    2. It's beginning to be used.

      Same problem with ssl certs... Solution ? Dane.

  6. I'm principally sceptically about such things, i.e. giving people stronger keys using connections made (i.e authenticated) with potentially weaker ones...

    Generally, I'd want a configuration option to disable this feature on both the server and the client.
    Especially on the client though, adding anything automatically to my known_hosts or ssh_known_hosts is for me personally a no go.
    So can you make that configurable?

    Maybe you could give three choices on the client side like
    yes = i.e. take over new keys
    no = ignore any keys the server advertises
    comment = add them to known_hosts but comment them out... i.e. just collect but don't use them.


    1. There's UpdateHostKeys=ask that gives you a chance to review and refuse the changes.

  7. This comment has been removed by a blog administrator.

  8. Hello Damien!

    The idea of the rolling host key updates is really cool.
    How it is, that the host key verification function (check_hostkeys_by_key_or_type) eventually had not been modified, to support multiple keys with the same type?(it breaks from the loop, after the first keytype match)


  9. The protocol only sends a single key and signature at the end of key exchange, so there is no possibility of checking other types there.

  10. Ok, thank you for your reply!
    In this case I hope, that the protocol will evolve with time as - in my opinion - it is a severe security risk, that it is not possible to do fast hostkey rolling update in case of eg. compromised keys or vulnerable algos/key lengths.
    If there are only a few clients, then it is not a big deal to change the keys, but with lots of clients - separate organisations involved - it takes some time. :-)
    Thanks again!