genuinetools/img

nsenter: Permission denied on K8s containerd w/ Gitlab Runners

ChuckNoxis opened this issue · 2 comments

ChuckNoxis commented

Hello,

I'm actually trying to run img on a Gitlab Runners chart (0.34.0) deployed with Helm on a Kubernetes 1.22.3 cluster w/ Containerd.

img:
 version     : v0.5.11
 git hash    : 3667c6a0
 go version  : go1.13.15
 go compiler : gc
 platform    : linux/amd64
runc:
 version     : 1.0.0-rc10+dev
 commit      : 56aca5aa50d07548d5db8fd33e9dc562f70f3208
 spec        : 1.0.2

When I do an img pull, I got the following error :
nsenter: cannot change root filesystem propagation: Permission denied

I saw issues #115 #266 #284 and also the Running with Kubernetes part in the Readme.md.

In the Gitlab Runner Helm Chart, I have set the following settings as suggested :

# Run all containers with the privileged flag enabled
# This will allow the docker:dind image to run if you need to run Docker 
# commands. Please read the docs before turning this on:  
# ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind 
runners:
  privileged: true

  ## Specify pod labels for CI job pods
  ## Even if deprecated, it's still working
  ## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
  podLabels:
    container.apparmor.security.beta.kubernetes.io/img: unconfined
    container.seccomp.security.alpha.kubernetes.io/img: unconfined

## Configure securitycontext 
## ref: http://kubernetes.io/docs/user-guide/security-context/
##       
securityContext:
  runAsUser: 100
  # runAsGroup: 65533
  fsGroup: 65533
  # supplementalGroups: [65533]
  privileged: true
  procMount: Unmasked

But I still got this Permission denied error when I'm trying to pull a Docker image.

towolf commented

I have this and it works with 1.21 on GKE with COS_CONTAINERD.

Could be, that you need pod annotations and not labels?

runners:

  # template configuration, which is merged into the config.toml
  config: |
    [[runners]]
      [runners.kubernetes]

        service_account = "default"
        service_account_overwrite_allowed = ".*"

        [runners.kubernetes.pod_annotations]
          # Annotations needed for https://github.com/genuinetools/img#running-with-kubernetes
          "container.apparmor.security.beta.kubernetes.io/build" = "unconfined"
          "container.seccomp.security.alpha.kubernetes.io/build" = "unconfined"
  • 👀2
towolf commented

BTW, procMount: Unmasked is set on the Runner pod, not on the spawned job pods.

  • 👍1