awslabs/mountpoint-s3-csi-driver

SELinux support for S3 CSI Driver for EKS Addon

GiamPy5 opened this issue · 6 comments

GiamPy5 commented

/feature

Is your feature request related to a problem? Please describe.
Our enterprise desires all of their instances to be security-hardened with SELinux enabled (we're also installing the CIS buildkit on the AMIs, starting from the EKS-optimized Amazon Linux 2 AMI). However SELinux does not make the s3-plugin container (part of the S3 CSI driver pod) start as it fails to perform a mount operation. AWS Support has advised us to submit a feature request about this issue.

Describe the solution you'd like in detail
Enhance the support of S3 CSI driver for SELinux so that it can work without any issues.

Describe alternatives you've considered
The only alternatives would be to:

  • use audit2allow to generate SELinux custom policies (however it could be tedious to maintain long-term)
  • disable SELinux (the least favourite option)

Additional context
Pod logs:

failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/proc/395888/mounts" to rootfs at "/host/proc/mounts": change mount propagation through procfd: mount /host/proc/mounts (via /proc/self/fd/6), flags: 0x44000: permission denied: unknown

Audit.log logs:

avc: denied { mounton } for pid=40998 comm="runc:[2:INIT]" path="/run/containerd/io.containerd.runtime.v2.task/k8s.io/c79bb808487e15e9d58a01ad593c8d446fd4bb20643c9ef154437596283ee42b/rootfs/host/proc/mounts" dev="proc" ino=34311 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0 tclass=file permissive=0

Internal AWS support reference (case ID): 171041866401170

jjkr commented

Thank you for the request. Are you using the default SELinux policies for Amazon Linux 2?

The driver does have some basic SELinux settings as of 1.4.0 where you can customize the seLinuxOptions on the driver containers (https://github.com/awslabs/mountpoint-s3-csi-driver/blob/main/charts/aws-mountpoint-s3-csi-driver/values.yaml#L16). Depending on how SELinux is configured, this may be enough for some applications, but more investigation is needed to see if that's the case on AL2.

GiamPy5 commented

I think we are indeed using the default SELinux policies (if there are any? I haven't found any documentation about the SEL policies included with AL2) as the CIS buildkit does not create new SELinux policies, it only enables the enforcement.

As far as I know, AWS Support has reproduced this issue on the AL2 image provided by AWS even without installing the CIS buildkit.

We are relying on EKS addons to install the S3 CSI driver so we don't have control on what's being installed behind the scenes.

GiamPy5 commented

Any news on this one by any chance?

unexge commented

Hey @GiamPy5 sorry for the late response!

I tried to run Mountpoint CSI driver on a SELinux enabled host (AL2023) in “permissive” mode and didn't see any problems other than mounting /proc/mounts as you mentioned.

Currently, we mount /proc/mounts to understand if a given path is a mountpoint or not, in:

  • NodePublishVolume – to understand if a path is already mounted to skip remount
  • NodeUnpublishVolume – to understand if a path is already mounted to skip unmount

We talked within the team, and seems like using Mounter.IsMountPoint from mount-utils would allow us to perform the same check without relying on /proc/mounts.

Mounter.IsMountPoint does the following checks:

  1. Makes a openat2 syscall at target path with RESOLVE_NO_XDEV and checks if it fails with EXDEV
    • This basically says Linux kernel to open a file without traversing mountpoints and kernel returns EXDEV error code if it encounters a mountpoint during the traversal
    • This is supported on Linux kernels 5.6+
  2. Makes a stat syscall at target path, and it's parent, and checks if their device differs
    • This seems like a reliable way for non-bind mountpoints, which should be fine in our case
  3. Parses /proc/mounts
    • This is the same logic as what Mountpoint CSI driver uses today

In order to make sure we don't break non-SELinux users, we'll preserve 3. check by default and allow SELinux users to opt-out from mounting of /proc/mounts.

We're hoping that to allow using Mountpoint CSI driver in SELinux enforced hosts.

marekhotshot commented

Any update please?

unexge commented

Hey @marekhotshot, we'll make the changes described in #169 (comment), but we don't have a target date yet unfortunately.