Error performing curl for OTP validation w/ Yubico Cloud
bturley2 opened this issue · 13 comments
I'm attempting to use the pam_yubico module to validate a user's OTP during SSH login. The request to the Yubico cloud to validate the user's OTP is failing. The tail end of the debug file shows:
debug: pam_yubico.c:1154 (pam_sm_authenticate): Token is associated to the user. Validating the OTP...
debug: pam_yubico.c:1156 (pam_sm_authenticate): ykclient return value (109): Error performing curl
debug: pam_yubico.c:1157 (pam_sm_authenticate): ykclient url used:
debug: pam_yubico.c:1220 (pam_sm_authenticate): done. [Authentication service cannot retrieve authentication info]
The contents of my /etc/pam.d/sshd file are:
auth sufficient pam_yubico.so id=<removed> key=<removed> debug debug_file=/home/opc/log
I'm running this on a cloud-based instance of Oracle Linux 7.8.
It almost seems as though there is no url specified by the ykclient. How do I fix this issue? Any suggestions?
Wonder if you could manually specify the URL parameter?
pam_yubico.so id=<> key=<> urllist=https://api.yubico.com/wsapi/2.0/verify
You might also try running curl from the command line against Yubico's infrastructure?
curl "https://api.yubico.com/wsapi/2.0/verify?id=IDHERE&nonce=0123456789abcdef&otp=PASTEOTPHERE"
Had an issue the other week where curl was refusing to connect to self-hosted OTP validation infrastructure due to an expired intermediate CA certificate, which broke pam_yubico; running it from the command-line revealed the problem.
I updated the first line of my /etc/pam.d/sshd file to:
auth sufficient pam_yubico.so id=<removed> key=<removed> debug debug_file=/home/opc/log urllist=https://api.yubico.com/wsapi/2.0/verify
But I'm still having the same issue as before. The output to the debug log is exactly the same as before:
debug: pam_yubico.c:1154 (pam_sm_authenticate): Token is associated to the user. Validating the OTP...
debug: pam_yubico.c:1156 (pam_sm_authenticate): ykclient return value (109): Error performing curl
debug: pam_yubico.c:1157 (pam_sm_authenticate): ykclient url used:
debug: pam_yubico.c:1220 (pam_sm_authenticate): done. [Authentication service cannot retrieve authentication info]
Running the curl request seemed successful:
$ curl "https://api.yubico.com/wsapi/2.0/verify?id=<removed>&nonce=0123456789abcdef&otp=<OTP-removed>"
h=<removed>
t=2020-06-12T20:12:32Z0294
otp=<OTP-removed>
nonce=0123456789abcdef
sl=100
status=OK
That's bizarre; not sure what to try next. Maybe including the version of pam_yubico from the RPM repo you're using might help others diagnose the issue?
I've just tried this on a oracle linux 7.8 vagrant machine and it works for me. Do you have any other special configuration on the machine? Could you have something not allowing network egress from pam?
On this vagrant instance I got pam_yubico version 2.26 and ykclient version 2.15 does this match what you have?
Can you try to use the binary ykclient to verify the OTP? This uses the same library as pam_yubico to do the validation but should maybe print more in it's debug output.
I've been attempting to get this working on a fresh install of a free-tier instance of Oracle Linux 7.8 running from the Oracle Cloud. After booting up and downloading the yubico pam module, I have ykclient version 2.15with pam_yubico version 2.26.
As far as I'm aware, I'm don't have any special configurations on this server. This instance was created on a brand new free-tier Oracle Cloud account created just to test this issue.
After the weekend it seems as though I can't get anything to work right. It doesn't appear to be prompting the user for their OTP anymore on SSH login, and nothing is appearing in the debug_file when I specify one. I was able to get this fully functional on a free-tier instance running Ubuntu 16, but unfortunately I need to get this working on Oracle Linux. My goal is to require public/private keypairs + OTP for ssh login.
When I use the ykclient it provides a similar error as before:
$ ykclient --debug --url "https://localhost/wsapi/2.0/verify?id=%d&otp=%s" --apikey <secret_key_removed> <client_id_removed> vvccccntkbrrvjfiridlvnuftbbreibeuiggrrvvdekb
Input:
validation URL: https://localhost/wsapi/2.0/verify?id=%d&otp=%s
client id: <client_id_removed>
token: vvccccntkbrrvjfiridlvnuftbbreibeuiggrrvvdekb
api key: <secret_key_removed>
Response from:
Verification output (109): Error performing curl
otp: (null)
nonce: (null)
t: (null)
timestamp: (null)
sessioncounter: (null)
sessionuse: (null)
sl: (null)
status: (null)
The exact steps I've been following to install are as follows:
(These setup instructions assume you have already configured your Yubikey with an OTP in slot 2)
- (On server) Install the Yubico PAM module:
sudo yum install pam_yubico - (On server) Modify /etc/pam.d/sshd by adding the following line at the beginning of the file:
auth sufficient pam_yubico.so id=[Your API Client ID] debug
and comment out the following line:
# auth substack password-auth - (On Server) Modify /etc/ssh/sshd_config to ensure the following settings are present:
ChallengeResponseAuthentication yes
PasswordAuthentication no
AuthenticationMethods publickey,keyboard-interactive
By specifying AuthenticationMethods as shown above, we indicate to the sshd that it should require both the user's public/private keypair and utilize the Yubico PAM module for login.
4. (On server) Setup the authorization mapping files. Make a directory '.yubico' under the current user's home directory. In my case, go to /home/opc and run:
mkdir .yubico
5. (On server) Go to the newly created .yubico directory, create a file called authorized_yubikeys. Input the following line into the file:
<user name>:<YubiKey token ID>
The YubiKey token ID is the first 12 characters from any generated OTP. For example, if you generate a OTP by holding the touch capacitor on the Yubikey and it generates
vvccccntkbrrijulebeubigbkvcvrjhvvfthidbfjjke
Then the corresponding Yubikey token ID would be:
vvccccntkbrr
In my case, the file's content is:
opc:vvccccntkbrr
6. (On server) Restart the SSH daemon:
sudo systemctl restart sshd
7. (On Client) Do an SSH login to your server. The server should require a valid SSH key, and should prompt you for your OTP and password individually. Be sure to hold the gold button for ~3 seconds to generate a yubikey from slot 2 when prompted:
ssh <user name>@<server's IP address>
One thing at a time.
Since ykclient is showing the same problems lets go from that, it's going to be easier for us to debug. Is it possible to get an oracle cloud machine like that running on free tier?
Are you trying this with local validation now? Does straight curl work on that url?
The error you're getting is indicative of a network or certificate problem.
This morning I realized I had a couple errors in my setup (such as using and @ symbol instead of : in the authorized_yubikeys file). After fixing those errors I'm back at the original problem laid out in the original post.
Also, it looks like I forgot to include the nonce= section of the url yesterday when checking if the ykclient can validate a given OTP. The following command appears to work just fine:
$ ykclient --debug <client_id> <OTP>
Input:
client id: <client_id>
token: <OTP>
Response from: https://api2.yubico.com/wsapi/2.0/verify?id=<client_id>&nonce=rnfwdxuisbmaxlcxhthokapqnexauton&otp=<OTP>×tamp=1
Verification output (0): Success
otp: <OTP>
nonce: rnfwdxuisbmaxlcxhthokapqnexauton
t: 2020-06-16T15:48:58Z0691
timestamp: 11000358
sessioncounter: 7
sessionuse: 14
sl: 100
status: OK
Curl also appears to be working just fine:
$ curl "https://api2.yubico.com/wsapi/2.0/verify?id=<client_id>&nonce=rnfwdxuisbmaxlcxhthokapqnexauton&otp=<OTP>"
h=9PC7UQy7vhZBSSD2lx1v9ZSvo6A=
t=2020-06-16T15:53:57Z0217
otp=<OTP>
nonce=rnfwdxuisbmaxlcxhthokapqnexauton
sl=100
status=OK
So now ykclient works for you? Are you running it with the same url you configured for yubico-pam (or no url)?
Can there be some form of policy that denies network connections from pam? Can you look in your syslog if there's anything about that?
I believe the ykclient always worked for me. I had specified a URL which was missing a parameter when I tried running it previously. So this command works:
$ ykclient --debug <client_id> vvccccntkbrrjtebhfurbfhrngkhkjtgttnfbdrilruj
Input:
client id: <client_id>
token: vvccccntkbrrjtebhfurbfhrngkhkjtgttnfbdrilruj
Response from: https://api.yubico.com/wsapi/2.0/verify?id=<client_id>&nonce=hszadxyiznjvhopwzwlafnnkfqllxese&otp=vvccccntkbrrjtebhfurbfhrngkhkjtgttnfbdrilruj×tamp=1
Verification output (0): Success
otp: vvccccntkbrrjtebhfurbfhrngkhkjtgttnfbdrilruj
nonce: hszadxyiznjvhopwzwlafnnkfqllxese
t: 2020-06-17T15:09:41Z0704
timestamp: 11969498
sessioncounter: 9
sessionuse: 0
sl: 100
status: OK
But this one does not:
$ ykclient --debug --url "https://localhost/wsapi/2.0/verify?id=%d&otp=%s" --apikey <secret_key> <client_id> vvccccntkbrrbeicfbtunjlrkjhrjtbedclnkuigvcub
Input:
validation URL: https://localhost/wsapi/2.0/verify?id=%d&otp=%s
client id: <client_id>
token: vvccccntkbrrbeicfbtunjlrkjhrjtbedclnkuigvcub
api key: <secret_key>
Response from:
Verification output (109): Error performing curl
otp: (null)
nonce: (null)
t: (null)
timestamp: (null)
sessioncounter: (null)
sessionuse: (null)
sl: (null)
status: (null)
I would imagine that not specifying a url for the yubico-pam would cause it to use whatever the defaults are for the ykclient, and as you can see from the above output, when I use the defaults on the ykclient the request goes through.
I couldn't find any policies in the syslog related to network connections or pam modules.
I've been trying to think of root causes for this issue:
- As you mentioned earlier, Oracle Linux may have additional restrictions set on the pam modules. For example, pam modules may not have the ability to access the internet.
- The pam module installed may not have full permissions or may be incorrectly accessing the ykclient program. Perhaps the ykclient is located in the wrong installation folder and the pam module can't find it.
The error output I've been receiving in the debug log showing:
debug: pam_yubico.c:1157 (pam_sm_authenticate): ykclient url used:
leads me to believe that the ykclient is attempting curl with a blank url, although when I manually specify a url in the /etc/pam.d/sshd file using the url= or urllist= parameters this debug line does not change.
So: are you trying to use yubicloud or your own validation server? when you specify localhost as the url I guess you have a validation server running there? If you want to run with yubicloud, please don't use the url/urllist options at all.
The url is only reported back if there was a valid answer, so that it's empty is normal in the case of network problems.
I think the fastest way to get past guessing about this is turning on debugging for curl, that requires modifying the libykclient library and re-compiling it to use that. Would you be comfortable with that?
Since this is cloud machines, could you give me access to a machine where this is happening? You can contact me on klas@yubico.com if this would be an option.
I'm attempting to get this to work with yubicloud.
I've tried the same setup with CentOS and it appears to have the exact same issue. The debug logs are exactly the same.
I am happy to give you access to the machine. Please check your email.
Thanks @bturley2 that helped with debugging this.
So what happened here is that selinux doesn't allow outgoing http connections by default, the command to let this work is: $ setsebool -P authlogin_yubikey 1.
Documentation around this is at https://developers.yubico.com/yubico-pam/YubiKey_and_SELinux.html