SSH with libykcs11.so does not work with 4096 bit RSA keys

Question

SSH with libykcs11.so does not work with 4096 bit RSA keys

Closed this issue 24 days ago · 13 comments

Hello.

I generate RSA key pair and import them with this snippet of commands:
openssl genrsa -out private_key.pem 4096
openssl rsa -in private_key.pem -outform PEM -pubout -out public_key.pem
yubico-piv-tool -s 9a -a import-key -i private_key.pem
yubico-piv-tool -a verify-pin -a selfsign-certificate -s 9a -S "/CN=SSH key/" -i public_key.pem -o cert.pem
ssh-keygen -i -m pkcs8 -f public_key.pem > ssh.pem.pub
ssh-copy-id -f -i ssh.pem.pub <user>@<ip>

Afterwards I import the certificate into yubikey with yubico authenticator (tried with yubikey manager and got same result).
Then I try to connect with ssh -I /usr/lib/libykcs11.so <user>@<ip> of which output is:
Enter PIN for 'YubiKey PIV #<redacted>':
cannot find private key
pkcs11_get_key failed
sign_and_send_pubkey: signing failed for RSA "/CN=SSH key": error in libcrypto

This only happens with 4096 bit keys. 2048 and 3072 bit keys work fine.
Is the problem on my side or is this expected behaviour?

yubikey: YubiKey 5 NFC
yubico-piv-tool: 2.5.2
ssh: OpenSSH_9.8p1, OpenSSL 3.3.1 4 Jun 2024
openssl: OpenSSL 3.3.1 4 Jun 2024 (Library: OpenSSL 3.3.1 4 Jun 2024)
OS: ArchLinux
Kernel: 6.9.7-arch1-1

Answer 1 · 2024-08-10T01:34:06.000Z

Interesting... If the key is generated on yubikey it works even with 4096 bits.
I generated it with yubico authenticator, exported certificate, extracted public key with openssl x509 -pubkey -in cert.crt -noout > public_key.pem and converted it to ssh format.

This doesn't work for me as I want to have the same private key on 2 yubikeys in case of losing or breaking one, but posting just in case it can give you some clues.

Answer 2 · 2024-08-13T13:26:09.000Z

Hi,
I am using Yubikey 5c NFC. Imported RSA 4096 and ED25519 keys do not work (RSA 3072 works). My usecase is also to have a spare indentical Yubikey in case of if one breaks.
Would be nice if the issue gets fixed as soon as possible.

BR

Answer 3 · 2024-08-14T12:05:24.000Z

Does the certificate import return any error? If you run yubico-piv-tool -a status after certificate import, does it list the content of slot 9a?

Out of curiosity, is there a reason for importing the certificate with Yubico Authenticator or YubiKey Manager rather than yubico-piv-tool? Not that it should make a difference

Answer 4 · 2024-08-14T12:56:43.000Z

the output of yubico-piv-tool -a status is:
Version: 5.7.1
Serial Number: <redacted>
CHUID: <redacted>
CCC: No data available
Slot 9a:
Algorithm: RSA4096
Subject DN: CN=SSH key
Issuer DN: CN=SSH key
Fingerprint: 02f2e11c78a53d53bfa24a9f7df68df4988c984d330bbca2456da16d2ba61784
Not Before: Aug 14 12:43:22 2024 GMT
Not After: Aug 14 12:43:22 2025 GMT
PIN tries left: 3

Output is identical when using yubico-piv-tool -a import-certificate -s 9a -i cert.pem to import certificate and not GUI.
I haven't noticed any errors when importing either private key or certificate.

As for the reason to use GUI, there's none, just a bit of misunderstanding. I've been trying to do this for a few days before I created an issue. For the first time, I did this purely with CLI and got the same error so I assumed that the act of importing certificate has overwritten private key and I needed to use PKCS12 to import them at the same time. so I switched to importing through GUI (especially authenticator because it shows whether private key is present) and just made a habit of it. Kind of stupid assumption but it was the first time I ever held any Yubikey.

Answer 5 · 2024-08-14T13:44:58.000Z

It was not a stupid assumption, we can try to make the documentation clearer :)

Can you set the environment variable YKCS11_DBG to any value higher than 1 then run the ssh -I /usr/lib/libykcs11.so command? Please post the output here (removing any sensitive data if there are any).

Answer 6 · 2024-08-14T15:56:39.000Z

It was not a stupid assumption, we can try to make the documentation clearer :)

Thank you :). The documentation is fine, it was mostly the error that it cannot find private key that led me to my assumption. But what I'd appreciate is some way to check if private key is present like Yubico Autheticator is capable of doing, but that's topic for another time.

And now for the debug log.
I've set YKCS11_DBG to 2 and tried to connect
Currently only slot 9a contains any private key or certificate
The log is huge so I used pastebin, if you prefer something else just let me know.
https://pastebin.com/HA90cJpm

Answer 7 · 2024-08-16T12:59:19.000Z

As far as I can tell from the debug output and some tests, it seems to be looking for a certificate but it can't find it for some reason. Just to eliminate some scenarios, the imported certificate is an X509 certificate, correct?

Answer 8 · 2024-08-16T13:09:54.000Z

Never mind. I remembered that it's the selfsigned cert produced by the YubiKey so it's definitely an X509.

Running some more tests

Answer 9 · 2024-08-16T13:15:18.000Z

Instead of converting the public key produced by OpenSSL, can you download it from the YubiKey instead and use that with ssh just in case it's a format issue? Use the following command to download the public keys from the YubiKey

ssh-keygen -D /usr/lib/libykcs11.so

Answer 10 · 2024-08-16T18:55:05.000Z

running ssh-keygen -D /usr/lib/libykcs11.so gave me 2 ssh keys. First one is Public key for PIV Attestation and second one is identical to the one obtained by ssh-keygen -i -m pkcs8 -f public_key.pem > ssh.pem.pub except it has /CN=SSH key at the end.

I tried copying only the one with /CN=SSH key at the end which gave exactly same output as before.

Then I tried copying both of them and connecting which produced the output of

Enter PIN for 'YubiKey PIV #<redacted>': 
C_Sign failed: 7
sign_and_send_pubkey: signing failed for RSA "Public key for PIV Attestation": error in libcrypto
cannot find private key
pkcs11_get_key failed
sign_and_send_pubkey: signing failed for RSA "/CN=SSH key": error in libcrypto

I deleted ~/.ssh/authorized_keys each time to start with clean slate.

Answer 11 · 2024-08-16T19:34:55.000Z

I am too getting successful login with RSA 2048 and RSA 3072, but RSA 4096 stops with the message "error in libcrypto". I have tried with openSC and RSA 4096 doesn't work there either.
The RSA 4096 key on the disk successfully connects to the remote server. So I would say PKCS11 seems to be the culprit here.

Answer 12 · 2024-08-19T07:39:09.000Z

I also having trouble using yubikey 5.7.1 with libykcs11.so pkcs11 library to work with RSA 4096.
I have:

generated self-signed RSA4096 with openssl and imported key and certificate into yubikey's 9a slot
cloned yubico-piv-tool repository and build it with -DYKCS11_DBG=9

run p11tool --provider /usr/local/lib/libykcs11.so --list-privkeys --login --set-pin 123456

here is the debug output: https://pastebin.com/raw/6qw2rT2Y

this is the output of the p11tool without debug output (it doesn't list the rsa 4096 at 9a slot, but lists the rsa 2048 attest key)

Object 0:
    URL: pkcs11:model=YubiKey%20YK5;manufacturer=Yubico%20%28www.yubico.com%29;serial=29507445;token=YubiKey%20PIV%20%2329507445;id=%19;object=Private%20key%20for%20PIV%20Attestation;type=private
    Type: Private key (RSA-2048)
    Label: Private key for PIV Attestation
    Flags: CKA_PRIVATE; CKA_NEVER_EXTRACTABLE; CKA_SENSITIVE; 
    ID: 19

the environment I am testing on is: ubuntu 24.04, openssl 3.0.13, p11tool (gnutls-bin package 3.8.3-1.1.ubuntu3.1)

the ykman piv info output is:

PIV version:              5.7.1
PIN tries remaining:      7/7
PUK tries remaining:      5/5
Management key algorithm: AES256
WARNING: Using default PIN!
WARNING: Using default PUK!
Management key is stored on the YubiKey, protected by PIN.
CHUID: <redacted>
CCC:   No data available
Slot 9A (AUTHENTICATION):
  Private key type: RSA4096
  Public key type:  RSA4096
  Subject DN:       CN=TEST4096
  Issuer DN:        CN=TEST4096
  Serial:           60:2a:ab:cd:90:79:d1:57:a5:50:da:64:37:f4:c4:73:28:07:ce:32
  Fingerprint:      238dbdf3d9964a5989b346fd9007250310c84550afd86313d3dd597752e026b5
  Not before:       2024-08-18T12:14:41+00:00
  Not after:        2044-08-13T12:14:41+00:00

I have tried to set 4096 here at ykpiv.h#L713 with no success
I have tried opensc pkcs11 successfully (therefore using rsa4096 with pkcs11 is practically possible).
But the opensc has to be patched in two places to support rsa4096 over pkcs11: card-piv.c#L4585 and card-piv.c#L5741.

Answer 13 · 2024-08-20T14:01:05.000Z

It looks like this was a bug that got pass our tests. The next release scheduled for tomorrow should contain a fix. Please let us know if the problem still persist.