FreeRADIUS/pam_radius

pam_radius version 2 not woking with multi factor authentication/2FA

muzammel111 opened this issue · 32 comments

We have setup (MFA/2FA) on linux based servers using first pam_radius to authenticate with radius server then authenticate with OS password. our pam.d/sshd contains below for auth.

#%PAM-1.0
auth required pam_sepermit.so
auth required pam_radius_auth.so
auth include password-auth
etc etc

Same things working fine on pam_radius-1.4.0-15, and when we update pam_radius to version 2.0 , the MFA has been broken, however if we use only for radius auth it works only as a single auth. but if we want to MFA as mentioned then its not work. So kindly please advise on it, i think new version has some changes which not supporting MFA. Kindly please advise. Thanks.

Regards,

Any update on this please ?

I would suggest giving a little more information than "it doesn't work".

Also, it's open source. No one is getting paid to fix issues. We have no obligation to jump when you file a bug.

The source code is online. You can either submit a patch, or look through the commit history to track down which commit broke this functionality. That effort should take only a small amount of time, and will help us prioritize a fix.

@muzammel111 if you don't share any input, logs error/warning messages, and entire config files. it's totally impossible to figure out what is happening.

Dear @jpereira and @alandekok Thank you, for your prompt response, i have now enabled debug logs of pam_radius and after upgraded to version 2.0, the radius auth breaks and password prompt repeating to send requests to radius server, see below logs, as the raidus auth already successful but repeated password prompt sending OS password to radius also.

Oct 31 02:37:28 Oracle-Linux-9 sshd[3789900]: pam_radius_auth: Got user name infra.nabil
Oct 31 02:37:28 Oracle-Linux-9 sshd[3789900]: pam_radius_auth: ignore last_pass, force_prompt set
Oct 31 02:37:54 Oracle-Linux-9 sshd[3789900]: pam_radius_auth: Sending RADIUS request code 1
Oct 31 02:37:54 Oracle-Linux-9 sshd[3789900]: pam_radius_auth: DEBUG: get_ipaddr(10.50.1.1) returned 0.
Oct 31 02:37:54 Oracle-Linux-9 sshd[3789900]: pam_radius_auth: Got RADIUS response code 2
Oct 31 02:37:54 Oracle-Linux-9 sshd[3789900]: pam_radius_auth: authentication succeeded
Oct 31 02:37:54 Oracle-Linux-9 sshd[3789900]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.50.3.117 user=infra.nabil
Oct 31 02:37:56 Oracle-Linux-9 sshd[3789893]: error: PAM: Authentication failure for infra.nabil from 10.50.3.117
Oct 31 02:37:56 Oracle-Linux-9 sshd[3792074]: pam_radius_auth: Got user name infra.nabil
Oct 31 02:37:56 Oracle-Linux-9 sshd[3792074]: pam_radius_auth: ignore last_pass, force_prompt set
Oct 31 02:38:24 Oracle-Linux-9 sshd[3792074]: pam_radius_auth: Sending RADIUS request code 1
Oct 31 02:38:24 Oracle-Linux-9 sshd[3792074]: pam_radius_auth: DEBUG: get_ipaddr(10.50.1.1) returned 0.
Oct 31 02:38:25 Oracle-Linux-9 sshd[3792074]: pam_radius_auth: Got RADIUS response code 3
Oct 31 02:38:25 Oracle-Linux-9 sshd[3792074]: pam_radius_auth: authentication failed
Oct 31 02:38:27 Oracle-Linux-9 sshd[3789893]: error: PAM: Authentication failure for infra.nabil from 10.50.3.117

And once i downgrade to pam 1.4 its works,

Oct 31 02:30:33 Oracle-Linux-9 sshd[3756610]: pam_radius_auth: Got user name infra.nabil
Oct 31 02:30:33 Oracle-Linux-9 sshd[3756610]: pam_radius_auth: ignore last_pass, force_prompt set
Oct 31 02:30:50 Oracle-Linux-9 sshd[3756610]: pam_radius_auth: Sending RADIUS request code 1
Oct 31 02:30:50 Oracle-Linux-9 sshd[3756610]: pam_radius_auth: Got RADIUS response code 2
Oct 31 02:30:50 Oracle-Linux-9 sshd[3756610]: pam_radius_auth: authentication succeeded
Oct 31 02:30:52 Oracle-Linux-9 sshd[3756610]: pam_sepermit(sshd:account): Parsing config file: /etc/security/sepermit.conf
Oct 31 02:30:52 Oracle-Linux-9 sshd[3756610]: pam_sepermit(sshd:account): Access will not be allowed on match
Oct 31 02:30:52 Oracle-Linux-9 sshd[3756610]: pam_sepermit(sshd:account): sepermit_match returned: -1
Oct 31 02:30:52 Oracle-Linux-9 sshd[3756610]: pam_nologin(sshd:account): unknown option: debug
Oct 31 02:30:52 Oracle-Linux-9 sshd[3756608]: Accepted keyboard-interactive/pam for infra.nabil from 10.50.3.117 port 42595 ssh2
Oct 31 02:30:52 Oracle-Linux-9 sshd[3756608]: pam_unix(sshd:session): session opened for user infra.nabil(uid=1000) by (uid=0)

I found below changes between 1.4 and 2.0

https://src.fedoraproject.org/rpms/pam_radius/c/c2a3314689968d9a1ebb798224f8d585810a78a9?branch=epel8
https://bugzilla.redhat.com/show_bug.cgi?id=2103904

I am a systemadmin not programmer, kindly please guide me further if you need require more info. Thanks again for your great support.

Regards,
Muzammel

@muzammel111

  1. What is your OS?
  2. What is the source of packages? eg: did you build from source or install from an external repository?
  3. What is the exact version of packages? e.g:
If Debian, e.g: dpkg -l | grep pam_radius
or if Redhat-based, e.g: rpm -qa | grep pam_radius
  1. Share the entire content of /etc/security/sepermit.conf and /etc/pam_radius_auth.conf
  2. Based on the log messages
# the Radius replied Access-Accept (code 2)
Oct 31 02:37:54 Oracle-Linux-9 sshd[3789900]: pam_radius_auth: Got RADIUS response code 2
Oct 31 02:37:54 Oracle-Linux-9 sshd[3789900]: pam_radius_auth: authentication succeeded

# therefore, there is something not correct in your pam.conf or pam.conf.d 
Oct 31 02:37:54 Oracle-Linux-9 sshd[3789900]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.50.3.117 user=infra.nabil
Oct 31 02:37:56 Oracle-Linux-9 sshd[3789893]: error: PAM: Authentication failure for infra.nabil from 10.50.3.117
  1. Take a look at your sshd pam configuration. if you are using a Ubuntu-based system. Take a look at our CI script and the used PAM SSD example.

Dear @jpereira thanks for you prompt response, here below the answers.

  1. Oracle Linux 9 based on RHEL (Redhat Enterprise Linux 9) and RHEL8 also.

  2. installed from EPEL repository.

  3. pam_radius-2.0.0-3.el9.x86_64

  4. cat /etc/security/sepermit.conf (Note: we are not using this, as SELINUX already disabled)
    all lines are hashed

cat /etc/pam_radius.conf

10.50.1.1:1812 radiussecret 5

  1. No we are not tested on ubuntu, we tested on RHEL 8 and 9 based distros.

Please let me know freely for further information. Thanks.

Further, below is the pattern for ssh using 2FA (radius + OS local auth)

In pam_radius version 2.0.0-3

ssh testuser@x.x.x.x
password (enter radius password) --> auth Ok successful.
password (enter OS password) --> failed , as this password request again going to radius server, instead to local auth, again popup password prompt
password (enter OS password) --> failed again request goes to radius server.

Worked fine in pam_radius 1.4.15

ssh testuser@x.x.x.x
password (enter radius password) --> radius auth Ok successful.
password (enter OS password) --> local auth ok successful.
user login to ssh successfully.

@muzammel111 By default, the pam_radius 2.x expects to use the /etc/pam_radius_auth.conf instead of /etc/pam_radius.conf. or you could set the config file in /etc/pam.d/sshd like:

[jpereira@sugarloaf-lan:pam_radius.git]$ cat /etc/pam.d/sshd
# Example of SSHD+PAM for CentOS
#%PAM-1.0
auth     required   pam_radius_auth.so conf=/etc/pam_radius_auth.conf debug retry=123 client_id=666
account  required   pam_nologin.so
account  include    password-auth
password include    password-auth
# pam_selinux.so close should be the first session rule
session  required   pam_selinux.so close
session  required   pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user
#context
session  required   pam_selinux.so open env_params
session  required   pam_namespace.so
session  optional   pam_keyinit.so force revoke
session  include    password-auth
[jpereira@sugarloaf-lan:pam_radius.git]$

It has already been tested and validated against CentOS. As we don't have any Redhat server to reproduce it. please, take a look carefully at each line and see what is missing in your setup.

Hi, the pam_radius pacakge use below files

rpm -ql pam_radius (list files in pam_radius package)
/etc/pam_radius.conf (radius config file)
/lib64/security/pam_radius_auth.so
etc etc

As pam_radius_auth.so file call in pam.d/sshd file as below on first top mentioned in bold

auth required pam_radius_auth.so
auth substack password-auth
auth include postlogin
account required pam_sepermit.so
account required pam_nologin.so
account include password-auth
password include password-auth
session required pam_selinux.so close
session required pam_loginuid.so
session required pam_selinux.so open env_params
session required pam_namespace.so
session optional pam_keyinit.so force revoke
session optional pam_motd.so
session include password-auth
session include postlogin

@jpereira same config files working fine on version 1.4, but when we upgrade to version 2 and restart sshd service, it failed on 2FA, as the config files are working fine as we got successful auth hit on radius server log, but password prompt appearing again and again as mentioned above.

@muzammel111 As I said: the pam_auth 2.x expects the /etc/pam_radius_auth.conf. so, please edit the pam.d/sshd adding the extra parameters in auth required pam_radius_auth.so:

e.g:

auth required pam_radius_auth.so conf=/etc/pam_radius_auth.conf debug retry=123 client_id=666

@jpereira sir, the /etc/pam_radius_auth.conf file is not present, i think its come in ubuntu pacakge, as i am using RHEL distro, the pam_radius has total below files in their rpm

[root@Linux-9 ~]# rpm -ql pam_radius
/etc/pam_radius.conf
/lib64/security/pam_radius_auth.so
/usr/lib/.build-id
/usr/lib/.build-id/76
/usr/lib/.build-id/76/19f3377e577cc580abef670886263afb3c4bac
/usr/share/doc/pam_radius
/usr/share/doc/pam_radius/Changelog
/usr/share/doc/pam_radius/INSTALL
/usr/share/doc/pam_radius/LICENSE
/usr/share/doc/pam_radius/README.rst
/usr/share/doc/pam_radius/USAGE

Any way i will try your method like below and let you know.
auth required pam_radius_auth.so debug retry=123 client_id=666

@muzammel111 If not work, please do the below steps:

  1. Remove any pam_radius-2x packages previously installed from Redhat repository and install directly from the official repository https://github.com/FreeRADIUS/pam_radius (all dependencies and steps can be seen in the CI/RPM script https://github.com/FreeRADIUS/pam_radius/blob/master/.github/workflows/ci-rpm.yml)
  2. do the change against /etc/pam.d/sshd.
  3. then, perform the tests again.

Thanks @jpereira i will test and update you.

@muzammel111 We've fixed the CI tasks and now the module has been built and tested successfully against CentOS8, Rocky8, and Rocky9 (same as Redhat) as can be seen here #81

Please let us know any news.

Dear @jpereira thanks for the update and prompt response, now you have fixed some thing ? as now i am currently testing on Oracle Linux 8 and 9 which is based on Redhat Enterprise, so upstream will need to build rpm for new changes ? then i will test. Please confirm.

@jpereira , today i compiled the pam_radius master branch from git, as you mentioned and compiled it and update pam.d/sshd file first line as below
auth required pam_radius_auth.so conf=/etc/pam_radius_auth.conf debug retry=123 client_id=666

But result its same as mentioned above in detail, first attempt of radius auth is successful, and password prompts continue appears. Below are the log messages. please guide.

ov 6 06:34:58 Oracle-Linux-9 sshd[874179]: pam_radius_auth: 2.0.1, built on Nov 6 2023 at 05:11:19
Nov 6 06:34:58 Oracle-Linux-9 sshd[874179]: pam_radius_auth: DEBUG: conf='/etc/pam_radius_auth.conf' use_first_pass=no try_first_pass=no skip_passwd=no retry=123 localifdown=no client_id='666' accounting_bug=no ruser=no prompt='Password: ' force_prompt=no prompt_attribute=no max_challenge=0 privilege_level=no
Nov 6 06:34:58 Oracle-Linux-9 sshd[874179]: pam_radius_auth: Got user name: 'infra.nabil'
Nov 6 06:34:58 Oracle-Linux-9 sshd[874179]: pam_radius_auth: ignore last_pass, force_prompt set
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: Sending RADIUS request code 1 (Access-Request)
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: DEBUG: get_ipaddr(10.50.1.1) returned 0.
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: Got RADIUS response code 2 (Access-Accept)
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: authentication succeeded
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.50.3.117 user=infra.nabil
Nov 6 06:35:19 Oracle-Linux-9 sshd[874161]: error: PAM: Authentication failure for infra.nabil from 10.50.3.117
Nov 6 06:35:19 Oracle-Linux-9 sshd[875909]: pam_radius_auth: 2.0.1, built on Nov 6 2023 at 05:11:19
Nov 6 06:35:19 Oracle-Linux-9 sshd[875909]: pam_radius_auth: DEBUG: conf='/etc/pam_radius_auth.conf' use_first_pass=no try_first_pass=no skip_passwd=no retry=123 localifdown=no client_id='666' accounting_bug=no ruser=no prompt='Password: ' force_prompt=no prompt_attribute=no max_challenge=0 privilege_level=no
Nov 6 06:35:19 Oracle-Linux-9 sshd[875909]: pam_radius_auth: Got user name: 'infra.nabil'
Nov 6 06:35:19 Oracle-Linux-9 sshd[875909]: pam_radius_auth: ignore last_pass, force_prompt set
Nov 6 06:35:32 Oracle-Linux-9 sshd[875909]: pam_radius_auth: Sending RADIUS request code 1 (Access-Request)
Nov 6 06:35:32 Oracle-Linux-9 sshd[875909]: pam_radius_auth: DEBUG: get_ipaddr(10.50.1.1) returned 0.
Nov 6 06:35:33 Oracle-Linux-9 sshd[875909]: pam_radius_auth: Got RADIUS response code 3 (Access-Reject)
Nov 6 06:35:33 Oracle-Linux-9 sshd[875909]: pam_radius_auth: authentication failed
Nov 6 06:35:35 Oracle-Linux-9 sshd[874161]: error: PAM: Authentication failure for infra.nabil from 10.50.3.117
Nov 6 06:35:35 Oracle-Linux-9 sshd[877275]: pam_radius_auth: 2.0.1, built on Nov 6 2023 at 05:11:19
Nov 6 06:35:35 Oracle-Linux-9 sshd[877275]: pam_radius_auth: DEBUG: conf='/etc/pam_radius_auth.conf' use_first_pass=no try_first_pass=no skip_passwd=no retry=123 localifdown=no client_id='666' accounting_bug=no ruser=no prompt='Password: ' force_prompt=no prompt_attribute=no max_challenge=0 privilege_level=no
Nov 6 06:35:35 Oracle-Linux-9 sshd[877275]: pam_radius_auth: Got user name: 'infra.nabil'
Nov 6 06:35:35 Oracle-Linux-9 sshd[877275]: pam_radius_auth: ignore last_pass, force_prompt set
Nov 6 06:35:35 Oracle-Linux-9 sshd[874161]: Postponed keyboard-interactive for infra.nabil from 10.50.3.117 port 29238 ssh2 [preauth]
Nov 6 06:35:39 Oracle-Linux-9 sshd[874161]: Connection closed by authenticating user infra.nabil 10.50.3.117 port 29238 [preauth]

@muzammel111 we can see below that the pam_radius_auth is handling and accepting as well.

Nov 6 06:34:58 Oracle-Linux-9 sshd[874179]: pam_radius_auth: Got user name: 'infra.nabil'
Nov 6 06:34:58 Oracle-Linux-9 sshd[874179]: pam_radius_auth: ignore last_pass, force_prompt set
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: Sending RADIUS request code 1 (Access-Request)
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: DEBUG: get_ipaddr(10.50.1.1) returned 0.
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: Got RADIUS response code 2 (Access-Accept)
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: authentication succeeded

Therefore, Probably you're missing something in your pam_unix(sshd:auth) setup.

Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.50.3.117 user=infra.nabil

Please, do the test using force_prompt=yes and take a look around the logs if the /var/log/secure, /var/log/auth and the others could have some clue.

@muzammel111 please, share the output of the command: id infra.nabil

@muzammel111 please, share the output of the command: id infra.nabil

[infra.nabil@Oracle-Linux-9 ~]$ id infra.nabil
uid=1000(infra.nabil) gid=1000(infra.nabil) groups=1000(infra.nabil)

@muzammel111 we can see below that the pam_radius_auth is handling and accepting as well.

Yes but only one time, its successful but, password prompt repeating again and again and as auth request goes again to radius server instead to perform local auth.

Nov 6 06:34:58 Oracle-Linux-9 sshd[874179]: pam_radius_auth: Got user name: 'infra.nabil'
Nov 6 06:34:58 Oracle-Linux-9 sshd[874179]: pam_radius_auth: ignore last_pass, force_prompt set
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: Sending RADIUS request code 1 (Access-Request)
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: DEBUG: get_ipaddr(10.50.1.1) returned 0.
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: Got RADIUS response code 2 (Access-Accept)
Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_radius_auth: authentication succeeded

Therefore, Probably you're missing something in your pam_unix(sshd:auth) setup.

Nov 6 06:35:17 Oracle-Linux-9 sshd[874179]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.50.3.117 user=infra.nabil

Please, do the test using force_prompt=yes and take a look around the logs if the /var/log/secure, /var/log/auth and the others could have some clue.

Ok, but surprising thing is that, same settings including (pam.d/sshd) file works on version 1.4, but when we upgraded it stop working, can you please guide how to set force_prompt=yes ?

force_prompt=yes

@jpereira i have tried to set force_prompt=yes but its gives (unrecognized option 'force_prompt=yes').

auth required pam_radius_auth.so conf=/etc/pam_radius_auth.conf debug retry=123 client_id=666 force_prompt=yes

Nov 8 02:57:25 Oracle-Linux-9 sshd[1139817]: pam_radius_auth: 2.0.1, built on Nov 6 2023 at 05:11:19
Nov 8 02:57:25 Oracle-Linux-9 sshd[1139817]: pam_radius_auth: unrecognized option 'force_prompt=yes'
Nov 8 02:57:25 Oracle-Linux-9 sshd[1139817]: pam_radius_auth: DEBUG: conf='/etc/pam_radius_auth.conf' use_first_pass=no try_first_pass=no skip_passwd=no retry=0 localifdown=no client_id='' accounting_bug=no ruser=no prompt='Password: ' force_prompt=no prompt_attribute=no max_challenge=0 privilege_level=no
Nov 8 02:57:25 Oracle-Linux-9 sshd[1139817]: pam_radius_auth: Got user name: 'infra.nabil'
Nov 8 02:57:25 Oracle-Linux-9 sshd[1139817]: pam_radius_auth: ignore last_pass, force_prompt set
Nov 8 02:57:45 Oracle-Linux-9 sshd[1139817]: pam_radius_auth: Sending RADIUS request code 1 (Access-Request)
Nov 8 02:57:45 Oracle-Linux-9 sshd[1139817]: pam_radius_auth: DEBUG: get_ipaddr(10.50.1.1) returned 0.
Nov 8 02:57:45 Oracle-Linux-9 sshd[1139817]: pam_radius_auth: Got RADIUS response code 2 (Access-Accept)
Nov 8 02:57:45 Oracle-Linux-9 sshd[1139817]: pam_radius_auth: authentication succeeded
Nov 8 02:57:45 Oracle-Linux-9 sshd[1139817]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.50.3.117 user=infra.nabil
Nov 8 02:57:47 Oracle-Linux-9 sshd[1139805]: error: PAM: Authentication failure for infra.nabil from 10.50.3.117

Hi @muzammel111

Well, I was a bit intrigued by that report. then, as we don't have any Redhat9 license to perform the tests. I've decided to perform the tests against the Rocky9 (clone of Redhat9)

I added all my tests and steps in such Dockerfile then below you can see the steps to reproduce the exact tests:

  1. Get my 'docker1' branch.
$ git clone --branch docker1 https://github.com/jpereira/pam_radius
$ cd pam_radius/scripts/docker/rocky9/
$ make
$ docker run --name docker-rocky9 -h docker-rocky9 -ti networkradius/pam_radius:rocky9
[root@docker-rocky9 pam_radius.git]#
  1. As we're inside the container instance.

... the user already exists.

[root@docker-rocky9 pam_radius.git]# id tapioca
uid=1000(tapioca) gid=1000(tapioca) groups=1000(tapioca)
[root@docker-rocky9 pam_radius.git]# 

... the FreeRADIUS instance is authenticating the user properly.

[root@docker-rocky9 pam_radius.git]# radtest -x tapioca queijo localhost 0 testing123
Sent Access-Request Id 73 from 0.0.0.0:51854 to 127.0.0.1:1812 length 77
	User-Name = "tapioca"
	User-Password = "queijo"
	NAS-IP-Address = 172.17.0.2
	NAS-Port = 0
	Message-Authenticator = 0x00
	Cleartext-Password = "queijo"
Received Access-Accept Id 73 from 127.0.0.1:1812 to 127.0.0.1:51854 length 36
	Reply-Message = "Hello, tapioca"
[root@docker-rocky9 pam_radius.git]#

... Also the ssh is working as well.

[root@docker-rocky9 ~]# ssh tapioca@localhost
The authenticity of host 'localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:Es0TU3O/LlOmSxBZTwYUJXXXCMD5jbb5ANBfohWzKKA.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
tapioca@localhost's password:
[tapioca@docker-rocky9 ~]$

Therefore, my conclusion is you have something broken in your setup.

@jpereira you are good to go with rocklinux 9, actually the issue comes in 2FA (Multi Factor Authentication) (radius auth + pam_unix) like you ssh using 2FA first auth with radius then with OS password. As i already told that if we bypass 2FA only radius auth successful but the issue came when you setup SSH with 2FA.

Therefore i am requesting you to please test SSH with 2FA using radius + OS auth as i mentioned in detail above.

@jpereira are you doing testing or any update please ?