CrashLoopBackopff in Collector's Deamon Set on OpenShift 4.9

Question

CrashLoopBackopff in Collector's Deamon Set on OpenShift 4.9

Balaji-MP opened this issue 2 years ago · 25 comments

Hello Team, received the following error while deploying the collector in openshift 4.9. Initially thought this is a permission issue and added the required SCC to collector's service account, but still the issue persists.

terminate called after throwing an instance of 'scap_open_exception'
  what():  can't create map: Permission denied
collector[0x448f7d]
/lib64/libc.so.6(+0x4eb80)[0x7f726981fb80]
/lib64/libc.so.6(gsignal+0x10f)[0x7f726981faff]
/lib64/libc.so.6(abort+0x127)[0x7f72697f2ea5]
/lib64/libstdc++.so.6(+0x9009b)[0x7f726a1c109b]
/lib64/libstdc++.so.6(+0x9653c)[0x7f726a1c753c]
/lib64/libstdc++.so.6(+0x96597)[0x7f726a1c7597]
/lib64/libstdc++.so.6(+0x967f8)[0x7f726a1c77f8]
/usr/local/lib/libsinsp-wrapper.so(+0x240ef5)[0x7f726c82cef5]
/usr/local/lib/libsinsp-wrapper.so(_ZN5sinsp4openERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x36)[0x7f726c866c16]
collector[0x4d2b34]
collector[0x46631c]
collector[0x442bec]
/lib64/libc.so.6(__libc_start_main+0xe5)[0x7f726980bd85]
collector[0x448e2e]
Caught signal 6 (SIGABRT): Aborted
/bootstrap.sh: line 94:    10 Aborted                 eval exec "$@"

Answer 1 · 2023-02-16T13:04:40.000Z

@Balaji-MP It definitely looks like lack of permissions to load eBPF probe. Just in case, could you share the definition of DaemonSet and the SecurityContext you've got in the end?

Answer 2 · 2023-02-16T13:54:46.000Z

@erthalion here is the definition and security context within in it

`apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
deprecated.daemonset.template.generation: "4"
email: support@stackrox.com
meta.helm.sh/release-name: stackrox-secured-cluster-services
meta.helm.sh/release-namespace: rhacs-operator
owner: stackrox
creationTimestamp: "2023-02-16T08:25:19Z"
generation: 4
labels:
app: collector
app.kubernetes.io/component: collector
app.kubernetes.io/instance: stackrox-secured-cluster-services
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: stackrox
app.kubernetes.io/part-of: stackrox-secured-cluster-services
app.kubernetes.io/version: 3.73.2
auto-upgrade.stackrox.io/component: sensor
helm.sh/chart: stackrox-secured-cluster-services-73.2.0
service: collector
name: collector
namespace: rhacs-operator
ownerReferences:

apiVersion: platform.stackrox.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: SecuredCluster
name: stackrox-secured-cluster-services
uid: 5b40f3be-1e30-4ded-8480-67fb0a8b03b8
resourceVersion: "1074444903"
uid: b4786759-f8f7-4bb8-bdef-ee975923e740
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
service: collector
template:
metadata:
annotations:
email: support@stackrox.com
meta.helm.sh/release-name: stackrox-secured-cluster-services
meta.helm.sh/release-namespace: rhacs-operator
owner: stackrox
creationTimestamp: null
labels:
app: collector
app.kubernetes.io/component: collector
app.kubernetes.io/instance: stackrox-secured-cluster-services
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: stackrox
app.kubernetes.io/part-of: stackrox-secured-cluster-services
app.kubernetes.io/version: 3.73.2
helm.sh/chart: stackrox-secured-cluster-services-73.2.0
service: collector
namespace: rhacs-operator
spec:
containers:
- env:
  - name: COLLECTOR_CONFIG
    value: '{"tlsConfig":{"caCertPath":"/var/run/secrets/stackrox.io/certs/ca.pem","clientCertPath":"/var/run/secrets/stackrox.io/certs/cert.pem","clientKeyPath":"/var/run/secrets/stackrox.io/certs/key.pem"}}'
  - name: COLLECTION_METHOD
    value: EBPF
  - name: GRPC_SERVER
    value: sensor.rhacs-operator.svc:443
  - name: SNI_HOSTNAME
    value: sensor.stackrox.svc
    image: registry.redhat.io/advanced-cluster-security/rhacs-collector-rhel8@sha256:c15a9d534e6b0bd73bee22aa8c67503e53266b47f9dd9ef11f9f05f6d007ae02
    imagePullPolicy: Always
    name: collector
    resources:
    limits:
    cpu: 750m
    memory: 1Gi
    requests:
    cpu: 50m
    memory: 320Mi
    securityContext:
    capabilities:
    drop:
    - NET_RAW
      privileged: true
      readOnlyRootFilesystem: true
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
  - mountPath: /host/var/run/docker.sock
    name: var-run-docker-sock
    readOnly: true
  - mountPath: /host/proc
    name: proc-ro
    readOnly: true
  - mountPath: /module
    name: tmpfs-module
  - mountPath: /host/etc
    name: etc-ro
    readOnly: true
  - mountPath: /host/usr/lib
    name: usr-lib-ro
    readOnly: true
  - mountPath: /host/sys
    name: sys-ro
    readOnly: true
  - mountPath: /host/dev
    name: dev-ro
    readOnly: true
  - mountPath: /run/secrets/stackrox.io/certs/
    name: certs
    readOnly: true
- command:
  - stackrox/compliance
    env:
  - name: ROX_NODE_NAME
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: spec.nodeName
  - name: ROX_ADVERTISED_ENDPOINT
    value: sensor.rhacs-operator.svc:443
    image: registry.redhat.io/advanced-cluster-security/rhacs-main-rhel8@sha256:727e14f925b7f6bbde4ed291a6b9c4c0e068519364b6fea5ef86126775a0cc9e
    imagePullPolicy: IfNotPresent
    name: compliance
    resources:
    limits:
    cpu: "1"
    memory: 2Gi
    requests:
    cpu: 10m
    memory: 10Mi
    securityContext:
    readOnlyRootFilesystem: true
    runAsUser: 0
    seLinuxOptions:
    type: container_runtime_t
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
  - mountPath: /etc/ssl/
    name: etc-ssl
  - mountPath: /etc/pki/ca-trust/
    name: etc-pki-volume
  - mountPath: /host
    name: host-root-ro
    readOnly: true
  - mountPath: /run/secrets/stackrox.io/certs/
    name: certs
    readOnly: true
    dnsPolicy: ClusterFirst
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext:
    fsGroup: 2000
    runAsGroup: 3000
    runAsUser: 1000
    serviceAccount: collector
    serviceAccountName: collector
    terminationGracePeriodSeconds: 30
    tolerations:
- operator: Exists
  volumes:
- hostPath:
  path: /var/run/docker.sock
  type: ""
  name: var-run-docker-sock
- hostPath:
  path: /proc
  type: ""
  name: proc-ro
- emptyDir:
  medium: Memory
  name: tmpfs-module
- hostPath:
  path: /etc
  type: ""
  name: etc-ro
- hostPath:
  path: /usr/lib
  type: ""
  name: usr-lib-ro
- hostPath:
  path: /sys/
  type: ""
  name: sys-ro
- hostPath:
  path: /dev
  type: ""
  name: dev-ro
- name: certs
  secret:
  defaultMode: 420
  items:
  - key: collector-cert.pem
    path: cert.pem
  - key: collector-key.pem
    path: key.pem
  - key: ca.pem
    path: ca.pem
    secretName: collector-tls
- hostPath:
  path: /
  type: ""
  name: host-root-ro
- emptyDir: {}
  name: etc-ssl
- emptyDir: {}
  name: etc-pki-volume
  updateStrategy:
  rollingUpdate:
  maxSurge: 0
  maxUnavailable: 1
  type: RollingUpdate
  status:
  currentNumberScheduled: 3
  desiredNumberScheduled: 3
  numberMisscheduled: 0
  numberReady: 0
  numberUnavailable: 3
  observedGeneration: 4
  updatedNumberScheduled: 3`

Answer 3 · 2023-02-16T14:24:42.000Z

@Balaji-MP any chance to do kubectl describe ds collector to get the events as well?

Answer 4 · 2023-02-16T14:38:55.000Z

@erthalion here is the events, current state of the pod is CrashLoopBackOff

`Events:
Type Reason Age From Message

Normal SuccessfulCreate 10s daemonset-controller Created pod: collector-jst65
Normal SuccessfulCreate 3s daemonset-controller Created pod: collector-x86fj`

Answer 5 · 2023-02-16T16:03:22.000Z

@erthalion I guess, the permission issue is caused because of the eval in line 94. I might be wrong, any thoughts on this ?

Answer 6 · 2023-02-16T16:14:32.000Z

bootstrap.sh (including the eval part) is only responsible for starting Collector. The issue you observe is happening when Collector tries to load eBPF probes.

Answer 7 · 2023-02-20T07:46:37.000Z

@erthalion any thoughts on this one ?

Answer 8 · 2023-02-20T09:10:31.000Z

What happens if you remove this part from the security context?

seLinuxOptions:
  type: container_runtime_t

Answer 9 · 2023-02-20T09:35:12.000Z

same error and nothing changed.

Answer 10 · 2023-02-21T08:40:22.000Z

@Balaji-MP what about the SCC, you haven't posted it yet, can you show scc/stackrox-collector?

Answer 11 · 2023-02-21T09:17:48.000Z

@erthalion here is the security context in stackrox-collector

securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 containers:

Answer 12 · 2023-02-21T09:25:46.000Z

@erthalion here is the security context in stackrox-collector

securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 containers:

There is also a SecurityContextConstraints (SCC), which should have more information, e.g. if a privileged containers are allowed and similar. Having said that, can you describe more your Openshift setup, is there anything special?

Answer 13 · 2023-02-21T09:58:55.000Z

@erthalion here is the SCC applied for this collector

`runAsUser:
type: RunAsAny
seLinuxContext:
type: RunAsAny
seccompProfiles:

'*'
supplementalGroups:
type: RunAsAny`

My cluster is standard and no additional restriction are in place.

Answer 14 · 2023-02-21T10:19:59.000Z

@erthalion can you please share the directory location where the collector will create the map ??

Answer 15 · 2023-02-22T13:02:21.000Z

@erthalion can you please share the directory location where the collector will create the map ??

It's a BPF map, so it's not located on the filesystem. The problem here is your Openshift setup somehow prevent Collector from executing the bpf syscall, we need to find out why is that.

here is the SCC applied for this collector

runAsUser:
type: RunAsAny
seLinuxContext:
type: RunAsAny
seccompProfiles:

'*'
supplementalGroups:
type: RunAsAny

This doesn't look complete, isn't there anything saying something like below?

allowPrivilegeEscalation: true
allowPrivilegedContainer: true

Answer 16 · 2023-02-27T10:34:46.000Z

@erthalion no I don't see anything related to allowPriviledged escalation / container.

Answer 17 · 2023-03-03T15:57:01.000Z

no I don't see anything related to allowPriviledged escalation / container.

That sounds strange to me. So the output of oc get scc/stackrox-collector -o yaml doesn't show anything else except what you've posted?

Answer 18 · 2023-03-03T17:05:33.000Z

Yes, that's correct

Answer 19 · 2023-05-25T06:40:19.000Z

@stackrox/collector-team any updates on this issue?

Answer 20 · 2023-05-25T16:07:44.000Z

Unfortunately no, nobody had a capacity to look further into it.

Answer 21 · 2023-05-25T16:28:28.000Z

@Balaji-MP TBH Openshift 4.9 is quite dated... might even be out of support? Would it be feasible for you to upgrade to a more recent version?

Answer 22 · 2023-05-25T16:36:54.000Z

@porridge let me update to the latest version and can check. In the mean time, do you have a recommended version or above ?

Answer 23 · 2023-05-26T05:12:25.000Z

4.12 would be my first choice

Answer 24 · 2023-06-01T06:41:43.000Z

@porridge You are correct, after upgrading to version 4.12 it fixed the issue.

Answer 25 · 2023-06-01T07:43:57.000Z

Awesome! Let us know if you need anything else.