netdata/helmchart

Cannot create directory '/var/lib/netdata/registry' - Maybe a permission problem?

fzyzcjy opened this issue · 7 comments

fzyzcjy commented

When using this chart, the netdata-parent-69f84c876b-9pdst pod fails.
P.S. Using minikube.

Temporary (dirty) solution

Look at the PersistentVolume that this pod (deployment) uses. Then do sudo chmod -R 777 /path/to/that/volume. Then everything works. However, obviously we should never chmod in production env!

Logs

Netdata entrypoint script starting
2020-08-05 08:18:14: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
2020-08-05 08:18:14: netdata ERROR : MAIN : Ignoring host prefix '/host': path '/host' failed to stat() (errno 2, No such file or directory)
2020-08-05 08:18:14: netdata INFO  : MAIN : SIGNAL: Enabling reaper
2020-08-05 08:18:14: netdata INFO  : MAIN : process tracking enabled.
2020-08-05 08:18:14: netdata ERROR : MAIN : LISTENER: Invalid listen port 0 given. Defaulting to 19999. (errno 22, Invalid argument)
2020-08-05 08:18:14: netdata INFO  : MAIN : resources control: allowed file descriptors: soft = 1048576, max = 1048576
2020-08-05 08:18:14: netdata INFO  : MAIN : Out-Of-Memory (OOM) score is already set to the wanted value 1000
2020-08-05 08:18:14: netdata ERROR : MAIN : Cannot adjust netdata scheduling policy to idle (5), with priority 0. Falling back to nice. (errno 38, Function not implemented)
2020-08-05 08:18:14: netdata ERROR : MAIN : Cannot get my current process scheduling policy. (errno 38, Function not implemented)
2020-08-05 08:18:14: netdata INFO  : MAIN : netdata started on pid 1.
2020-08-05 08:18:14: netdata FATAL : MAIN : Cannot create directory '/var/lib/netdata/registry'. # : Invalid argument
2020-08-05 08:18:14: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 1...
2020-08-05 08:18:14: netdata INFO  : MAIN : EXIT: cleaning up the database...
2020-08-05 08:18:14: netdata INFO  : MAIN : Cleaning up database [0 hosts(s)]...
2020-08-05 08:18:14: netdata INFO  : MAIN : EXIT: all done - netdata is now exiting - bye bye...

kubectl describe

Name:         netdata-parent-69f84c876b-9pdst
Namespace:    default
Priority:     0
Node:         minikube/172.17.0.2
Start Time:   Wed, 05 Aug 2020 16:10:10 +0800
Labels:       app=netdata
              pod-template-hash=69f84c876b
              release=netdata
              role=parent
Annotations:  checksum/config: 094d5c7af68d3e6e34ac86b75f3769089726a09e3bc30cb40755a88f4ebcaa22
Status:       Running
IP:           172.18.0.25
IPs:
  IP:           172.18.0.25
Controlled By:  ReplicaSet/netdata-parent-69f84c876b
Containers:
  netdata:
    Container ID:   docker://8d64835fe907ba9be34ff07eefe74190dfb94c91940c62f278d14e1eacdb37a6
    Image:          netdata/netdata:v1.23.2
    Image ID:       docker-pullable://netdata/netdata@sha256:fc9fb61fb58f5428d8b9dee6ad5639bdfbb7639aec83e7c47b45d1b03dc36786
    Port:           19999/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 05 Aug 2020 16:18:14 +0800
      Finished:     Wed, 05 Aug 2020 16:18:14 +0800
    Ready:          False
    Restart Count:  5
    Liveness:       http-get http://:http/api/v1/info delay=0s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:http/api/v1/info delay=0s timeout=1s period=30s #success=1 #failure=3
    Environment:
      MY_POD_NAME:       netdata-parent-69f84c876b-9pdst (v1:metadata.name)
      MY_POD_NAMESPACE:  default (v1:metadata.namespace)
    Mounts:
      /etc/netdata/health_alarm_notify.conf from config (rw,path="health")
      /etc/netdata/netdata.conf from config (rw,path="netdata")
      /etc/netdata/stream.conf from config (rw,path="stream")
      /var/cache/netdata from database (rw)
      /var/lib/netdata from alarms (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from netdata-token-wjdbg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      netdata-conf-parent
    Optional:  false
  database:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  netdata-parent-database
    ReadOnly:   false
  alarms:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  netdata-parent-alarms
    ReadOnly:   false
  netdata-token-wjdbg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  netdata-token-wjdbg
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                     From               Message
  ----     ------            ----                    ----               -------
  Warning  FailedScheduling  <unknown>               default-scheduler  running "VolumeBinding" filter plugin for pod "netdata-parent-69f84c876b-9pdst": pod has unbound immediate PersistentVolumeClaims
  Warning  FailedScheduling  <unknown>               default-scheduler  running "VolumeBinding" filter plugin for pod "netdata-parent-69f84c876b-9pdst": pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled         <unknown>               default-scheduler  Successfully assigned default/netdata-parent-69f84c876b-9pdst to minikube
  Normal   Pulling           3m17s (x4 over 8m39s)   kubelet, minikube  Pulling image "netdata/netdata:v1.23.2"
  Normal   Pulled            3m12s (x4 over 4m29s)   kubelet, minikube  Successfully pulled image "netdata/netdata:v1.23.2"
  Normal   Created           3m12s (x4 over 4m28s)   kubelet, minikube  Created container netdata
  Normal   Started           3m11s (x4 over 4m28s)   kubelet, minikube  Started container netdata
  Warning  BackOff           2m34s (x10 over 4m12s)  kubelet, minikube  Back-off restarting failed container
ncmans commented

@fzyzcjy can you please provide the helm install command you used and/or values you passed to the chart so we can try to reproduce it here. Thanks.

ilyam8 commented

2020-08-05 08:18:14: netdata FATAL : MAIN : Cannot create directory '/var/lib/netdata/registry'. # : Invalid argument

same error as in #90 (comment)

cakrit commented

Partial duplicate of #122 . We have done something that caused this...

fzyzcjy commented

I use the default value for helm install.

Since minikube has a bug on provisioning volumes, I manually allocate several PVs, e.g.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Gi
  hostPath:
    path: /data/pv0001/
ilyam8 commented

i created persistend volumes (create PVC, use PVC) in minikube, i had no problems

[ilyam@ilyam-pc ~]$ minikube version
minikube version: v1.8.2
commit: eb13446e786c9ef70cb0a9f85a633194e62396a1
fzyzcjy commented

Hmmmm maybe that is related to the broken storage-provisioner? This issue -> kubernetes/minikube#7218

I will try to see whether this problem exists after the provisioner bug is fixed.

Thanks for the replies!

fzyzcjy commented

After I updated to 1.12.2 and the storage-provisioner bug solved, netdata works happily :) Thanks!

  • 🎉2