Yubico/yubico-piv-tool

sign_and_send_pubkey: signing failed: agent refused operation (after some inactivity)

alexeyantropov opened this issue Β· 41 comments

Remote ssh-server can't verify my private key from YubiKey after thirty ~ fourty five minutes ssh-agent inactivity.

I use YubiKey 5C Nano under MacOS 11.5.2 (Apple M1) with lib from yubico-piv-tool-2.2.0-mac-arm64.pkg package. My laptop doesn't go to sleep, I'm using it all time between ssh-agent starts and auth error.

Example, start ssh-agent:

Console one:

user@host1 ~ $ ssh-agent -d -a /Users/user/.ssh/ssh_auth_sock -P /usr/local/lib/libykcs11.dylib
SSH_AUTH_SOCK=/Users/user/.ssh/ssh_auth_sock; export SSH_AUTH_SOCK;
echo Agent pid 59899;
debug2: fd 3 setting O_NONBLOCK

Console two:

user@host1~ $ SSH_AUTH_SOCK=$HOME/.ssh/ssh_auth_sock ssh-add -s /usr/local/lib/libykcs11.dylib
Enter passphrase for PKCS#11:
Card added: /usr/local/lib/libykcs11.dylib

Console one:

...
debug3: fd 4 is O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 20
debug1: process_add_smartcard_key: add /usr/local/lib/libykcs11.dylib
debug1: pkcs11_start_helper: starting /usr/libexec/ssh-pkcs11-helper -vvv
debug1: process_add
debug1: provider /usr/local/lib/libykcs11.dylib: manufacturerID <Yubico (www.yubico.com)> cryptokiVersion 2.40 libraryDescription <PKCS#11 PIV Library (SP-800-73)> libraryVersion 2.20
debug1: provider /usr/local/lib/libykcs11.dylib slot 0: label <YubiKey PIV #10114264> manufacturerID <Yubico (www.yubico.com)> model <YubiKey YK5> serial <10114264> flags 0x40d
debug1: have 1 keys
debug1: have 2 keys
debug1: pkcs11_k11_free: parent 0x145b1a730 ptr 0x145b1a620 idx 0
debug1: pkcs11_provider_unref: 0x145804350 refcount 3
debug1: pkcs11_k11_free: parent 0x145b19ea0 ptr 0x145b19d90 idx 0
debug1: pkcs11_provider_unref: 0x145804350 refcount 3
debug1: pkcs11_k11_free: parent 0x145b1a930 ptr 0x145b198f0 idx 0
debug1: pkcs11_provider_unref: 0x145804350 refcount 3

# All right (mark one)
debug3: fd 4 is O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug1: process_message: socket 1 (fd=4) type 13
debug1: process_sign
debug1: check 0x145b19590 /usr/local/lib/libykcs11.dylib
debug1: pkcs11_check_obj_bool_attrib: provider 0x145804350 slot 0 object 86: attrib 514 = 0
debug1: pkcs11_k11_free: parent 0x144711900 ptr 0x0 idx 0

# All right too (mark two)
debug3: fd 4 is O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug1: process_message: socket 1 (fd=4) type 13
debug1: process_sign
debug1: check 0x145b19590 /usr/local/lib/libykcs11.dylib
debug1: pkcs11_check_obj_bool_attrib: provider 0x145804350 slot 0 object 86: attrib 514 = 0
debug1: pkcs11_k11_free: parent 0x144711900 ptr 0x0 idx 0

# Error after inactivity (mark three)
debug3: fd 4 is O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug1: process_message: socket 1 (fd=4) type 13
debug1: process_sign
debug1: check 0x145b19590 /usr/local/lib/libykcs11.dylib
debug1: pkcs11_check_obj_bool_attrib: provider 0x145804350 slot 0 object 86: attrib 514 = 0
C_Sign failed: 48
debug1: pkcs11_k11_free: parent 0x145805450 ptr 0x0 idx 0
process_sign_request2: sshkey_sign: error in libcrypto

# Same error again (mark four)
debug3: fd 4 is O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11
debug1: process_message: socket 1 (fd=4) type 13
debug1: process_sign
debug1: check 0x145b19590 /usr/local/lib/libykcs11.dylib
debug1: pkcs11_check_obj_bool_attrib: provider 0x145804350 slot 0 object 86: attrib 514 = 0
C_Sign failed: 48
debug1: pkcs11_k11_free: parent 0x144711900 ptr 0x0 idx 0
process_sign_request2: sshkey_sign: error in libcrypto

Console three after ssh-agent (re)start:

user@host1 ~ $ ssh -v host3
...
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available: Disk quota exceeded

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available: Disk quota exceeded

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /usr/local/lib/libykcs11.dylib
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to host3 ([10.1.1.1]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
...
-bash-4.2$ hostname
host 3

Console three after some time (between MARK TWO and MARK THREE), I'm on the remote host and usging agent forwarding:

user@host2 ~ $ ssh -v host3
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /home/user/.ssh/config
debug1: /home/user/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 1: Applying options for *
debug1: Connecting to host3 [10.1.1.1] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to host3:22 as 'user'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:cyhMOsis5CrYnZ/U102a1pksG6DTJaprzVgySp182GE
debug1: Host 'host3' is known and matches the ECDSA host key.
debug1: Found key in /home/user/.ssh/known_hosts:2540
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available: Disk quota exceeded

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available: Disk quota exceeded

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /usr/local/lib/libykcs11.dylib
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
sign_and_send_pubkey: signing failed: agent refused operation
debug1: Offering RSA public key: /usr/local/lib/libykcs11.dylib
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Trying private key: /home/user/.ssh/id_rsa
debug1: Trying private key: /home/user/.ssh/id_dsa
debug1: Trying private key: /home/user/.ssh/id_ecdsa
debug1: Trying private key: /home/user/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Command "ssh-add -l" always gives same results (during normal work and after failure)

user@host1 ~ $ ssh-add -l
2048 SHA256:dvIR9e14pNm5VUxVMzNjCMnBnD0I8wCj+PKvEbjymhA /usr/local/lib/libykcs11.dylib (RSA)
2048 SHA256:Xk7dUop5/qjwucmsUduwTtG9hgBmOE/jD3UJh+wqlVY /usr/local/lib/libykcs11.dylib (RSA)
skevy commented

Very possible that this is related to #330.

If you get a chance @alexeyantropov, can you run your same test but with export YKCS11_DBG=1?

Very possible that this is related to #330.

If you get a chance @alexeyantropov, can you run your same test but with export YKCS11_DBG=1?

I collected log, there is more one thousand strings. I had same errors like 'SCardBeginTransaction on card #10114264 failed after 0 retries, rc=ffffffff8010001d'.

Paste all log here?

$ cat /tmp/ssh-agent.log|grep -C3 'failed after'
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/ykcs11.c:1786 (C_GetAttributeValue): Out
debug1: pkcs11_check_obj_bool_attrib: provider 0x155f04390 slot 0 object 86: attrib 514 = 0
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/ykcs11.c:2834 (C_Sign): In
SCardBeginTransaction on card #10114264 failed after 0 retries, rc=ffffffff8010001d
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/mechanisms.c:299 (sign_mechanism_final): ykpiv_sign_data with key 9a failed: Error in PCSC call
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/ykcs11.c:2894 (C_Sign): sign_mechanism_final failed
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/ykcs11.c:2907 (C_Sign): Out
--
--
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/ykcs11.c:1786 (C_GetAttributeValue): Out
debug1: pkcs11_check_obj_bool_attrib: provider 0x155f04390 slot 0 object 86: attrib 514 = 0
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/ykcs11.c:2834 (C_Sign): In
SCardBeginTransaction on card #10114264 failed after 0 retries, rc=ffffffff8010001d
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/mechanisms.c:299 (sign_mechanism_final): ykpiv_sign_data with key 9a failed: Error in PCSC call
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/ykcs11.c:2894 (C_Sign): sign_mechanism_final failed
debug: /Users/jokkon/code/yubico-piv-tool/ykcs11/ykcs11.c:2907 (C_Sign): Out

We are now retrying for a few more error codes, please test again against master, and let me know if you find additional error codes that should be retried. See ShouldReconnect(). Of particular interest is if retrying on the error code SCARD_E_NO_SERVICE helps.

OK, retrying on SCARD_E_NO_SERVICE doesn't help. Will have to look into this furter.

Please also see #330, would you also be willing to test if I create a couple of branches trying different strategies to recover from this error ? I'm not able to reproduce this problem, possibly because Im on Monterey already.

Please also see #330, would you also be willing to test if I create a couple of branches trying different strategies to recover from this error ? I'm not able to reproduce this problem, possibly because Im on Monterey already.

Of course! No problem! But one little question, could you build a lib? :) I will try, but I can't promise successful build.

Here is some code that tests an alternative approach, please let me know if this makes any difference. Not that the code is just a draft to test if this approach has any merit. #332

Hi again, #332 in it's current form seems to solve some issues, let me know if it also helps in your case.

Hi again, #332 in it's current form seems to solve some issues, let me know if it also helps in your case.

Yes, I'm here! I can try https://github.com/Yubico/yubico-piv-tool/actions/runs/1439971471 (it's last now) build ?

Yes, it would be excellent to get your feedback, thx !

Yes, it would be excellent to get your feedback, thx !

There is only x86 binary release, I can't run it :(, sorry.

I'll see if I can arrange an arm build

I am getting this problem consistently. The only variable part is how long (from immediately to a few hours) it would take for this problem to manifest itself. And once it does - the only solution is to kill ssh-agent. Current master does not remedy this problem.

I'm experiencing this problem with Apple ssh-agent coming with the OS (the following is on Big Sur), and with Macports-installed OpenSSH that's built from sources on my machine.

$ ssh-add -s /usr/local/lib/libykcs11.dylib 
Enter passphrase for PKCS#11: 
Card added: /usr/local/lib/libykcs11.dylib
$ ssh-add -L
ssh-rsa AAA. .  .  .  aL /usr/local/lib/libykcs11.2.2.1.dylib
ssh-rsa AAA. .  .  .  7r /usr/local/lib/libykcs11.2.2.1.dylib
ssh-rsa AAA. .  .  .  Gj /usr/local/lib/libykcs11.2.2.1.dylib
ssh-rsa AAA. .  .  .  Dj /usr/local/lib/libykcs11.2.2.1.dylib
ssh-rsa AAA. .  .  .  oCz /usr/local/lib/libykcs11.2.2.1.dylib
$ ssh -Y myhostname
sign_and_send_pubkey: signing failed: agent refused operation
Password:

Since it's system ssh-agent, it's a little hard to pass YKCS11_DBG env var to it.

I am getting this problem consistently. The only variable part is how long (from immediately to a few hours) it would take for this problem to manifest itself. And once it does - the only solution is to kill ssh-agent. Current master does not remedy this problem....

How much memory do you have? 8 Gb, right? This problem is around the memory management in MacOS. MacOS unloads the PKCS library from runtime (like the OOM) when memory (and swap) limit reached and loads its again, but ssh agent's library can't restore a Yubikey context.

All we are still waiting for a new release witch fix it.

How much memory do you have? 8 Gb, right?

Well, it's 64 GB and 10 physical CPU cores. Wouldn't you say it's sufficient?

Well, it's 64 GB and 10 physical CPU cores. Wouldn't you say it's sufficient?

Wow! I couldn’t reproduce the problem on same systems. Only on Macbooks with 8-16Gb memory.

To me the problem is consistent, including high-end iMac and iMac Pro (10 and 20 physical cores correspondingly, 64 GB RAM each). Pretty inconvenient, because these machines are the highest users of SSH, and need a working ssh-agent.

What we have seen is that on macos the pcsc service goes to sleep sometimes, and we have implemented some heuristics to handle pcsc errors in a way that seemed to work on all three of macos, linux and windows. Someone was able to produce logs on what happened, do you think you could do the same ? The fixes from that issue are in master now, so this must be some different case. Or we have a bug.. But the issue looked to be solved, hence I'd appreciate som logs.

Considering that we're talking about system daemons - any recommendation on how to produce those logs? I'd be happy to do it.

I want to try a new version and check, but I need packages for MacOS :(

Maybe this thread #330 can help, or someone here can tell how they debugged this. I guess you could try killing the ssh-agent and then restart it with debugging on for ykcs11, ot recompile it with debugging always on. But I'm not familiar with where logging ends up in the normal case.

Regarding packages Im sorry we haven't made a new release yet. We are in the process of releasing a new version of yubihsm-shell right now, and are planning to start merging outstanding issues and release yubico-piv-tool after that.
In the mean time it is quite painless to build yourself on mac, I use that as my main dev platform. All you need is to install dependencies via homebrew, and build using cmake. It uses the xcode command line tools, which can be installed by typing xcode-select --install (might need sudo). When building you need to specify where homebrew installed openssl. You can find where that is by typing brew info openssl. For me on an Intel mac it looks like this:
(after creating an empty directory i usually call build inside the top level directory where you cloned the git repo)
PKG_CONFIG_PATH="/usr/local/opt/openssl@1.1/lib/pkgconfig" cmake ..
make
make install

I have recently tinkered with multiple YubiKeys on my Mac and after that decided to update to Monterey.
After attempt to use main YubiKey 5Ci with resident SSH keys in git, I started getting in situations where if ssh-add -l is not showing any identities (right after ssh-agent is killed), the card behaves fine and prompts me for:

  • key passphrase (not using Mac Keychain)
  • PIV PIN
  • confirm presence on the pins (physicall)

Each attempt to use SSH resident keys for any git op. that needs auth., immediately after that 1st attempt, would fail with error described in this issue's title:
sign_and_send_pubkey: signing failed: agent refused operation (after some inactivity)

For me the problem initially looked like a change in openssh:8.8p1 (bumped after upgrading Homebrew packages after Monterey installation, while on Big Sur was using openssh:8.6p1).
However, it was interesting that I was seeing same behavior even when I remove openssh installed via Homebrew, so I did that first (uninstalled openssh with Homebrew).

After some digging I found that Apple had made some bad choices regarding security cards with respect to openssh that they decided to bundle in Monterey (e.g. see Yubico/libfido2#464)

Considering that I was thinkering with other Yubico sec. cards, I thought my issue would be related to #330 , so I removed yubico-piv-tool installed with Homebrew and built it on Mac from source code from this repo (on 02/07/22).
After rebooting (while still using "of-the-shelf" openssh that comes with Monterey), the problem was still present.
Then I installed openssh:8.8p1 again via Homebrew and after rebooting, problem was still present.

I kept digging and eventually .... I found this: https://apple.stackexchange.com/questions/430363/monterey-ssh-with-hardware-key-only-works-once
According to the blog post in https://aditsachde.com/posts/yubikey-ssh/ (mentioned in the above Apple StackExchange question), any use of ssh runs ssh-agent that comes with OS "of-the-shelf" instead of the one installed with openssh via Homebrew. So after disabling OS default ssh-agent and following through the blog, my issue is gone and consecutive attempts to use SSH resident keys on Yubikey work as before ( I always get prompted to enter PIN, confirm presence, etc.)

Now, what I am missing here is whether the "of-the-shelf" openssh that comes with Monterey did some additional bad decisions in regards the security cards, or there is still opportunity that needs to be addressed with yubico-piv-tool.

@alexeyantropov , from your logs in the very first post on this issue you are using very old openssh

OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017

Please try upgrading openssh via homebrew and follow my post above if you can? I would be curious to see if this also solves the issue for you. I had to use min openssh:8.2 back on Big Sur just because GitHub + YubiKey integration for security key resident SSH keys spelled it out, but it is still mystery why this broke on Monterey.

Are you talking about using ssh with U2F / FIDO2 ? If so it has nothing to do with yubico-piv-tool (or libykcs11).

try running gpg-connect-agent updatestartuptty /bye. I experienced the same error but I dont know if it's the same cause. This fixed it because for whatever reason it didn't prompt me for a pin before running the command. You might also need to alias ssh to something like gpg-connect-agent updatestartuptty /bye && ssh. For me, it works across restarts and everything now.

try running gpg-connect-agent updatestartuptty /bye. I experienced the same error but I dont know if it's the same cause. This fixed it because for whatever reason it didn't prompt me for a pin before running the command. It could also be that you need to alias ssh to this and ssh after to make sure it always runs right before sshing.

I had a similar issue like OP and this fixed it for me, thank you @VixieTSQ

@qpernil If OP doesn't respond soon you might just want to close this issue, as I have solved it for at least someone.

try running gpg-connect-agent updatestartuptty /bye. I experienced the same error but I dont know if it's the same cause. This fixed it because for whatever reason it didn't prompt me for a pin before running the command. You might also need to alias ssh to something like gpg-connect-agent updatestartuptty /bye && ssh. For me, it works across restarts and everything now.

Thank you so much! I've been running into this all day today and this fixed it!!!

@alexeyantropov , from your logs in the very first post on this issue you are using very old openssh

OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017

Please try upgrading openssh via homebrew and follow my post above if you can? I would be curious to see if this also solves the issue for you. I had to use min openssh:8.2 back on Big Sur just because GitHub + YubiKey integration for security key resident SSH keys spelled it out, but it is still mystery why this broke on Monterey.

I missed your answer, sorry! I would like to use native ssh-client from Apple. Using a third-party build is strange way. IMHO!

I saw a message about the new build in #330. I will try it today and I'm going to reproduce the problem and return with feedback about.

I tested the new version yubico-piv-tool-2.3.0-mac-universal.pkg! It works fine!

I couldn't reproduce problem after update. E.g. put my system in swap or kill com.apple.ctkpcscd. I think 2.3.0 release solved this issue!

Closing this issue now as it seems to be mostly solved, please open a new issue if you still have problems.

Have same issue (i guess, plz sorry if it's off topic):
After some time of inactivity, ssh connection fails with

$ ssh user@host
sign_and_send_pubkey: signing failed for RSA "PIV AUTH pubkey" from agent: agent refused operation
user@host: Permission denied (publickey).

i tried to debug this, but don't get the point of log output:

debug1: Offering public key: PIV AUTH pubkey RSA SHA256:+%sometoken% token agent
debug1: Server accepts key: PIV AUTH pubkey RSA SHA256:+%sometoken% token agent
sign_and_send_pubkey: signing failed for RSA "PIV AUTH pubkey" from agent: agent refused operation
debug1: pkcs11_k11_free: parent 0x6000003883c0 ptr 0x0 idx 0

Usually, i just run alias ssh-add -e /usr/local/lib/opensc-pkcs11.so; ansible-vault view ~/.ssh/.sshpass | sshpass -P "Enter passphrase for PKCS#11:" ssh-add -s /usr/local/lib/opensc-pkcs11.so but it's kinda annoying πŸ˜„

Have same issue (i guess, plz sorry if it's off topic): After some time of inactivity, ssh connection fails with

$ ssh user@host
sign_and_send_pubkey: signing failed for RSA "PIV AUTH pubkey" from agent: agent refused operation
user@host: Permission denied (publickey).

i tried to debug this, but don't get the point of log output:

debug1: Offering public key: PIV AUTH pubkey RSA SHA256:+%sometoken% token agent
debug1: Server accepts key: PIV AUTH pubkey RSA SHA256:+%sometoken% token agent
sign_and_send_pubkey: signing failed for RSA "PIV AUTH pubkey" from agent: agent refused operation
debug1: pkcs11_k11_free: parent 0x6000003883c0 ptr 0x0 idx 0

Usually, i just run alias ssh-add -e /usr/local/lib/opensc-pkcs11.so; ansible-vault view ~/.ssh/.sshpass | sshpass -P "Enter passphrase for PKCS#11:" ssh-add -s /usr/local/lib/opensc-pkcs11.so but it's kinda annoying πŸ˜„

You aren’t using library from a Yubico package. You have to update (or install) the Yubico pkg and use a yubico lib.

Link to the pkg https://developers.yubico.com/yubico-piv-tool/Release_Notes.html , look for the libykcs11.dylib inside and add it instead the OpenCS lib.

Finally figured out with libykcs11.dylib and i didn't understand some things:
if libykcs11.dylib added into agent, like ssh-add -s libykcs11.dylib - ssh connection always fails with:

need pin entry
login failed for always-auth key
pkcs11_get_key failed

If remove this via ssh-add -D its ok, but - is there a way to use pin from keychain?

There might be an issue using always-auth keys with ssh, could you try using a different slot ? Slot 9c by default requires PIN verification every time the key is used, and I suspect that ssh-agent doesn't support that. Slot 9a by default only requires PIN once, and might work better.

Slot 9a by default only requires PIN once, and might work better.

I use it, not 9c and don't have the problem described above.

9d also requires PIN only once by default. You can change this, but only when creating (generating or importing) a key

try running gpg-connect-agent updatestartuptty /bye. I experienced the same error but I dont know if it's the same cause. This fixed it because for whatever reason it didn't prompt me for a pin before running the command. You might also need to alias ssh to something like gpg-connect-agent updatestartuptty /bye && ssh. For me, it works across restarts and everything now.

Thanks a lot! This fixed my issue.

Are you talking about using ssh with U2F / FIDO2 ? If so it has nothing to do with yubico-piv-tool (or libykcs11).

In the subject mentioned error is related to ykcs11 package which in my case on Ubuntu 22.04 was missing.
Discoverable Key Instructions didn't describe anything about it.