hashicorp/vault

Vault SSH CA engine issuing certs with deprecated type ssh-rsa-cert-v01@openssh.com user certificate

Opened this issue · 4 comments

prbinu commented

Describe the bug
After upgrading OpenSSH to 9.0 on my Mac, I'm having trouble with my Vault SSH certificates. It appears that recent OpenSSH version dropped ssh-rsa-cert-v01@openssh.com from their PubkeyAcceptedAlgorithms default set [1]

The default for PubkeyAcceptedAlgorithms option is:

ssh-ed25519-cert-v01@openssh.com,
ecdsa-sha2-nistp256-cert-v01@openssh.com,
ecdsa-sha2-nistp384-cert-v01@openssh.com,
ecdsa-sha2-nistp521-cert-v01@openssh.com,
sk-ssh-ed25519-cert-v01@openssh.com,
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,
rsa-sha2-512-cert-v01@openssh.com,
rsa-sha2-256-cert-v01@openssh.com,
ssh-ed25519,
ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,
sk-ssh-ed25519@openssh.com,
sk-ecdsa-sha2-nistp256@openssh.com,
rsa-sha2-512,rsa-sha2-256

Here is one example SSH certificate issued by Vault (configured for rsa-sha2-256):

Type: **ssh-rsa-cert-v01@openssh.com** user certificate
        Public key: RSA-CERT SHA256:qRUOr61jF8hdGNdkno77grNgFgVoNhdfmMYd3X+LrIo
        Signing CA: RSA SHA256:mrRnq6txarrSCWaPhZ8235ZqrQIYgf9dN42WGcoTUxQ (using rsa-sha2-256)
        Key ID: "group=devops pubkey_fp=a9150eafad6317c85d18d7649e8efb82b36016056836175f98c61ddd7f8bac8a"
        Serial: 11912340821462643665
        Valid: from 2022-05-17T20:39:00 to 2022-05-18T20:39:30
        Principals:
                devops
        Critical Options: (none)
        Extensions:
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty

[1] https://man.openbsd.org/ssh_config#PubkeyAcceptedAlgorithms

A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Install latest OpenSSH version:
$ ssh -V
OpenSSH_9.0p1, OpenSSL 1.1.1n  15 Mar 2022

$ ssh -Q PubkeyAcceptedAlgorithms
ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
sk-ssh-ed25519@openssh.com
sk-ssh-ed25519-cert-v01@openssh.com
ssh-rsa
rsa-sha2-256
rsa-sha2-512
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
sk-ecdsa-sha2-nistp256@openssh.com
webauthn-sk-ecdsa-sha2-nistp256@openssh.com
ssh-rsa-cert-v01@openssh.com
rsa-sha2-256-cert-v01@openssh.com
rsa-sha2-512-cert-v01@openssh.com
ssh-dss-cert-v01@openssh.com
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521-cert-v01@openssh.com
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com
  1. Follow the steps as described here: https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-certificates:
  • To setup SSH Secret engine
  • To setup a target host with SSH CA
  • Fetch certificate and parse using ssh-add -L | ssh-keygen -L -f -. Check the Type field (first line). Even though we are using rsa-sha2-256, the Type is still showing ssh-rsa-cert-v01@openssh.com (instead of rsa-sha2-256-cert-v01@openssh.com type)

Expected behavior
I believe the SSH certificate based on rsa-sha2-256 shall have Type rsa-sha2-256-cert-v01@openssh.com

Environment:
OS: Mac Monterey
Client: OpenSSH 9.0

cipherboy commented

@prbinu Hmm, this is interesting. Note the key you pasted:

        Public key: RSA-CERT SHA256:qRUOr61jF8hdGNdkno77grNgFgVoNhdfmMYd3X+LrIo
        Signing CA: RSA SHA256:mrRnq6txarrSCWaPhZ8235ZqrQIYgf9dN42WGcoTUxQ (using rsa-sha2-256)

So we are effectively generating a rsa-sha2-256-cert-v01@openssh.com key type, and placing it in a ssh-rsa-cert-v01@openssh.com typed object (which is fine, I believe, from an OpenSSH standard -- should find a definitive reference though as to the delta between the two container formats).

Notably, in the x/crypto/ssh (library we use for this generation), I see:

	// CertAlgoRSASHA256v01 and CertAlgoRSASHA512v01 can't appear as a
	// Certificate.Type (or PublicKey.Type), but only in
	// ClientConfig.HostKeyAlgorithms.
	CertAlgoRSASHA256v01 = "rsa-sha2-256-cert-v01@openssh.com"
	CertAlgoRSASHA512v01 = "rsa-sha2-512-cert-v01@openssh.com"

This to me says that we can't just update the constant, we'd need to fix the upstream x/crypto/ssh package first. I've not yet looked to see how much work this would be; it looks like it might be tracked here: golang/go#49952 -- but it doesn't look like updating certificate creation formats is on the list? I might've missed something though.

The SSH engine was recently updated to support alternative key types, see e.g., #14006 or #14008. Perhaps switching to an ECDSA key is sufficient until x/crypto/ssh can be updated?

If its a simple fix and it already works, a PR would be appreciated :-)

prbinu commented

thanks @cipherboy

A workaround switching to ed25519 key algorithm is working:

$ ssh-keygen -t ed25519

$ vault write -field=signed_key ssh/sign/test public_key=@$HOME/.ssh/id_ed25519.pub valid_principals=jdoe | ssh-keygen -L -f -
(stdin):1:
Type: **ssh-ed25519-cert-v01@openssh.com** user certificate
Public key: ED25519-CERT SHA256:XhZL3wGE6JiWczuSsX5M2jwaYrlG8NEhuCPITuhUp5Y
Signing CA: RSA SHA256:sPKzOUJwvkR3aCFf2oCyHnc+JoMtFcow2UxcEz+cXo4 (using rsa-sha2-256)
Key ID: "group=test  pubkey_fp=5e164bdf0184e89896733b92b17e4cda3c1a62b946f0d121b823c84ee854a796
Serial: 11871586717846079993
Valid: from 2022-05-20T13:46:21 to 2022-05-21T01:46:51
Principals:
    jdoe
Critical Options: (none)
Extensions:
    permit-port-forwarding
    permit-pty

But with new OpenSSH changes, this issue is likely to affect many users.

vmhuy99 commented

thanks @cipherboy

A workaround switching to ed25519 key algorithm is working:

$ ssh-keygen -t ed25519

$ vault write -field=signed_key ssh/sign/test public_key=@$HOME/.ssh/id_ed25519.pub valid_principals=jdoe | ssh-keygen -L -f -
(stdin):1:
Type: **ssh-ed25519-cert-v01@openssh.com** user certificate
Public key: ED25519-CERT SHA256:XhZL3wGE6JiWczuSsX5M2jwaYrlG8NEhuCPITuhUp5Y
Signing CA: RSA SHA256:sPKzOUJwvkR3aCFf2oCyHnc+JoMtFcow2UxcEz+cXo4 (using rsa-sha2-256)
Key ID: "group=test  pubkey_fp=5e164bdf0184e89896733b92b17e4cda3c1a62b946f0d121b823c84ee854a796
Serial: 11871586717846079993
Valid: from 2022-05-20T13:46:21 to 2022-05-21T01:46:51
Principals:
    jdoe
Critical Options: (none)
Extensions:
    permit-port-forwarding
    permit-pty

But with new OpenSSH changes, this issue is likely to affect many users.
I used this method but the key is still sha256
ssh-ed25519-cert-v01@openssh.com AAAAIHNzaC1lZDI1NTE5LWNlcnQtdjAxQG9wZW5zca3guY29tAAAAIElLwQsBUe2lGY7oItw+407O3wEwdTNW9qCJaf52fhXQAAAAIBdtT/hQhs2kJPEjhKg4I/cIha5DaPKrnDG3Ub1io2vyhsIVpT3RizoAAAABAAAAS3ZhdWx0LXJvb3QtY2NjZDU5NzhhNjc1MTgzZjRhaNDNkMTBhODI5M2I3ZjI4ZWUwZDFjMzFiNGRhNTMwMmM0YWY5OTY3N2I3YjYwYwAAAAAAAAAAZmHT4wAAAABmjAQBAAAAAAAAAAAAAAAAAAAAMwAAAAtzc2gtZWQayNTUxOQAAACDkkIzmIcxGVjZiuDZKV2IXFr+7hKvM9W+Kuapx0MyJKgAAAFMAAaAALc3NoLWVkMjU1MTkAAABA0RjWWdh9mXs9U+tNlYaF4GF918dKR1oShYyT8fAl1tXKYEQKBJ4xCBoV3ngNB64iPzu3bMdYMuFrScXFiMT1Dw==

jlrcontegix commented

We are experiencing the same issue. After migrating to RHEL 8 machines with FIPS enabled, we are unable to use our Vault issued certs unless we relax the crypto policy for OpenSSH to allow "ssh-rsa-cert-v01@openssh.com" in PubkeyAcceptedKeyTypes and "ssh-rsa" in CASignatureAlgorithms.

All other issues about this I can find seem to indicate the problem is resolved when you change the Signing Algorithm, which we have, but we see the same as the OP. The certificate type is always generated as ssh-rsa-cert-v01@openssh.com.

# ssh-keygen -Lf tmpk6pb843_
tmpk6pb843_:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:hKyRsWm/49TFPj5mISV1jLYErnhacnCLghqTlwFSGZI
        Signing CA: RSA SHA256:Gw+BOYMg6ZGZWWCOkz28P5qB/IUvmL7to9krSR+dKBs (using rsa-sha2-256)
        Key ID: "vault-ldap-user-84ac9662125758c1b169bfe3d4c53e3eb604ae785a72708b821a939701521992"
        Serial: 6124806212823624310
        Valid: from 2024-06-27T11:21:27 to 2024-06-27T11:22:57
        Principals:
                user
        Critical Options: (none)
        Extensions:
                permit-port-forwarding
                permit-pty

With FIPS enabled, we cannot use these certs at the client level, much less to connect to a server:

# cat /usr/share/crypto-policies/FIPS/openssh.txt
Ciphers aes256-gcm@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
MACs hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha2-512
GSSAPIKeyExchange no
KexAlgorithms ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
PubkeyAcceptedKeyTypes ecdsa-sha2-nistp256,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-256,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-512-cert-v01@openssh.com
CASignatureAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512