SELinux prevents deployment of ceph-mgr after host package updates, EL8.7
Venator-Fox opened this issue · 2 comments
Bug Report
What happened: After performing a dnf update -y to RHEL 8.7 and associated packages; ceph no longer successfully deploys due to an avc denial with keyrings for ceph-mgr. I don't think? this is an 8.7 issue but rather some other system packages that were updated which causes this. Setting selinux to permissive allows a deploy. An exact deployment on a host on 8.6 (with enforcing) last fully patched ~1 month ago is successful; updating to 8.7+ a month worth of updates to all other packages appears to cause this issue. No changes in the working repository from the deployment admin node.
What you expected to happen: Successful deployment with SELinux enforcing
How to reproduce it (minimal and precise):
Attempt to run a deployment on fully updated RHEL 8.7 and associated packages
The deployment will fail. Investigating a failing node it was noted the ceph manager container was not running. Attempting to start ceph-mgr@service will cause the podman container to immediately crash due to permission denial on keyrings. A restorecon on /etc/ceph and /var/lib/ceph with systemctl restart ceph-mgr@service did not seem to resolve the issue. The following denials are present in audit. Setting to permissive allows a deploy and will generate these messages:
type=AVC msg=audit(1673733722.343:7745): avc: denied { read } for pid=55613 comm="ceph-mgr" name="keyring" dev="dm-0" ino=68051744 scontext=system_u:system_r:container_t:s0:c373,c524 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1673733722.343:7745): avc: denied { open } for pid=55613 comm="ceph-mgr" path="/var/lib/ceph/mgr/ceph-lab-csp-sc1-301/keyring" dev="dm-0" ino=68051744 scontext=system_u:system_r:container_t:s0:c373,c524 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
When enforcing, this causes the container to exit on startup
-- Unit ceph-mgr@cs-csp-io2-5a01.service has finished starting up.
--
-- The start-up result is done.
Jan 14 02:44:11 cs-csp-io2-5a01 ceph-mgr-csp-io2-5a01[50295]: find: '/var/lib/ceph/mgr/ceph-csp-io2-5a01/keyring': Permission denied
Jan 14 02:44:11 cs-csp-io2-5a01 ceph-mgr-csp-io2-5a01[50295]: chown: cannot access '/var/lib/ceph/mgr/ceph-csp-io2-5a01/keyring': Permission denied
Raw Audit Messages
type=AVC msg=audit(1673733722.343:7745): avc: denied { read } for pid=55613 comm="ceph-mgr" name="keyring" dev="dm-0" ino=68051744 scontext=system_u:system_r:container_t:s0:c373,c524 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=AVC msg=audit(1673733722.343:7745): avc: denied { open } for pid=55613 comm="ceph-mgr" path="/var/lib/ceph/mgr/ceph-lab-csp-sc1-301/keyring" dev="dm-0" ino=68051744 scontext=system_u:system_r:container_t:s0:c373,c524 tcontext=system_u:object_r:var_lib_t:s0 tclass=file permissive=1
type=SYSCALL msg=audit(1673733722.343:7745): arch=x86_64 syscall=openat success=yes exit=EIO a0=ffffff9c a1=5573c93aaf90 a2=80000 a3=0 items=0 ppid=55553 pid=55613 auid=4294967295 uid=167 gid=167 euid=167 suid=167 fsuid=167 egid=167 sgid=167 fsgid=167 tty=(none) ses=4294967295 comm=ceph-mgr exe=/usr/bin/ceph-mgr subj=system_u:system_r:container_t:s0:c373,c524 key=(null)ARCH=x86_64 SYSCALL=openat AUID=unset UID=unknown(167) GID=unknown(167) EUID=unknown(167) SUID=unknown(167) FSUID=unknown(167) EGID=unknown(167) SGID=unknown(167) FSGID=unknown(167)
Hash: ceph-mgr,container_t,var_lib_t,file,read
This will in turn cause deployment to fail when it expects ceph-mgr
TASK [ceph-mgr : wait for all mgr to be up] *******************************************************************************************************************************************************************
Saturday 14 January 2023 14:52:11 -0700 (0:00:00.131) 0:04:07.554 ******
FAILED - RETRYING: wait for all mgr to be up (30 retries left).
FAILED - RETRYING: wait for all mgr to be up (29 retries left).
<...>
FAILED - RETRYING: wait for all mgr to be up (1 retries left).
fatal: [dev-csp-sc1-303 -> dev-csp-sc1-301]: FAILED! => changed=false
attempts: 30
Setting all nodes to permissive or adding a catchall local policy module appears to allow deployment but both cases are less than ideal.
Environment:
- OS (e.g. from /etc/os-release): PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)"
- Kernel (e.g.
uname -a): 4.18.0-425.10.1.el8_7.x86_64 SMP Wed Dec 14 16:00:01 EST 2022 x86_64 x86_64 x86_64 GNU/Linux - Docker version if applicable (e.g.
docker version): podman 4.2.0 - Ansible version (e.g.
ansible-playbook --version): ansible-playbook 2.10.17 - ceph-ansible version (e.g.
git head or tag or stable branch): tag v6.0.28 d7bf53a - Ceph version (e.g.
ceph -v): 16.2.10 via quay.io/ceph/daemon:v6.0.9-stable-6.0-pacific-centos-stream8-x86_64
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.