filebeat readinessProbe always fails when load elasticsearch password from env and env load that password from secret

Question

filebeat readinessProbe always fails when load elasticsearch password from env and env load that password from secret

PetrusZ opened this issue 2 years ago · 1 comments

Chart version:
7.17.3

Kubernetes version:
1.23.8

Kubernetes provider: E.g. GKE (Google Kubernetes Engine)
bare metal

Helm Version:
version.BuildInfo{Version:"v3.10.0", GitCommit:"ce66412a723e4d89555dc67217607c6579ffcb21", GitTreeState:"clean", GoVersion:"go1.19.1"}

helm get release output

e.g. helm get elasticsearch (replace elasticsearch with the name of your helm release)

Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting.

If you find some secrets in plain text in helm get release output you should use Kubernetes Secrets to managed them is a secure way (see Security Example).

Output of helm get release

NAME: filebeat
LAST DEPLOYED: Mon Nov 14 18:11:29 2022
NAMESPACE: monitoring
STATUS: deployed
REVISION: 3
TEST SUITE: None
USER-SUPPLIED VALUES:
daemonset:
  envFrom:
  - secretRef:
      name: elastic-security
  extraEnvs:
  - name: ELASTICSEARCH_HOST
    value: https://elastic.codeplayer.org:443
  filebeatConfig:
    filebeat.yml: |-
      setup.template:
        # name: kubernetes
        # pattern: kubernetes-*
        enabled: true
        #overwrite: true

      setup.ilm.enabled: true
      setup.ilm.overwrite: true

      setup.kibana.host: "https://kibana.codeplayer.org:443"
      setup.kibana.ssl.enabled: true
      setup.dashboards.enabled: true
      # setup.dashboards.index: kubernetes-*

      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        exclude_files: /var/log/containers/.*filebeat.*\.log
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/log/containers/"
      - drop_fields:
          fields:
          # - host
          - ecs
          - log
          - agent
          - input
          # - stream
          - container
          - kubernetes.node
          # - kubernetes
          ignore_missing: true

      output.elasticsearch:
        hosts: "${ELASTICSEARCH_HOST}"
        username: "elastic"
        password: "${elasticsearch-password}"
        # index: "kubernetes-%{+yyyy.MM.dd}"

COMPUTED VALUES:
affinity: {}
clusterRoleRules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - nodes
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
daemonset:
  affinity: {}
  annotations: {}
  enabled: true
  envFrom:
  - secretRef:
      name: elastic-security
  extraEnvs:
  - name: ELASTICSEARCH_HOST
    value: https://elastic.codeplayer.org:443
  extraVolumeMounts: []
  extraVolumes: []
  filebeatConfig:
    filebeat.yml: |-
      setup.template:
        # name: kubernetes
        # pattern: kubernetes-*
        enabled: true
        #overwrite: true

      setup.ilm.enabled: true
      setup.ilm.overwrite: true

      setup.kibana.host: "https://kibana.codeplayer.org:443"
      setup.kibana.ssl.enabled: true
      setup.dashboards.enabled: true
      # setup.dashboards.index: kubernetes-*

      filebeat.inputs:
      - type: container
        paths:
          - /var/log/containers/*.log
        exclude_files: /var/log/containers/.*filebeat.*\.log
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/log/containers/"
      - drop_fields:
          fields:
          # - host
          - ecs
          - log
          - agent
          - input
          # - stream
          - container
          - kubernetes.node
          # - kubernetes
          ignore_missing: true

      output.elasticsearch:
        hosts: "${ELASTICSEARCH_HOST}"
        username: "elastic"
        password: "${elasticsearch-password}"
        # index: "kubernetes-%{+yyyy.MM.dd}"
  hostNetworking: false
  labels: {}
  maxUnavailable: 1
  nodeSelector: {}
  resources:
    limits:
      cpu: 1000m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 100Mi
  secretMounts: []
  securityContext:
    privileged: false
    runAsUser: 0
  tolerations: []
deployment:
  affinity: {}
  annotations: {}
  enabled: false
  envFrom: []
  extraEnvs: []
  extraVolumeMounts: []
  extraVolumes: []
  filebeatConfig:
    filebeat.yml: |
      filebeat.inputs:
      - type: tcp
        max_message_size: 10MiB
        host: "localhost:9000"

      output.elasticsearch:
        host: '${NODE_NAME}'
        hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
  labels: {}
  nodeSelector: {}
  resources:
    limits:
      cpu: 1000m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 100Mi
  secretMounts: []
  securityContext:
    privileged: false
    runAsUser: 0
  tolerations: []
dnsConfig: {}
envFrom: []
extraContainers: ""
extraEnvs: []
extraInitContainers: []
extraVolumeMounts: []
extraVolumes: []
filebeatConfig: {}
fullnameOverride: ""
hostAliases: []
hostPathRoot: /var/lib
image: docker.elastic.co/beats/filebeat
imagePullPolicy: IfNotPresent
imagePullSecrets: []
imageTag: 7.17.3
labels: {}
livenessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      curl --fail 127.0.0.1:5066
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
managedServiceAccount: true
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext: {}
priorityClassName: ""
readinessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      filebeat test output
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
replicas: 1
resources: {}
secretMounts: []
serviceAccount: ""
serviceAccountAnnotations: {}
terminationGracePeriod: 30
tolerations: []
updateStrategy: RollingUpdate

HOOKS:
MANIFEST:
---
# Source: filebeat/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat-filebeat
  annotations:
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.17.3"
    heritage: "Helm"
    release: "filebeat"
---
# Source: filebeat/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-filebeat-daemonset-config
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.17.3"
    heritage: "Helm"
    release: "filebeat"
data:
  filebeat.yml: |
    setup.template:
      # name: kubernetes
      # pattern: kubernetes-*
      enabled: true
      #overwrite: true

    setup.ilm.enabled: true
    setup.ilm.overwrite: true

    setup.kibana.host: "https://kibana.codeplayer.org:443"
    setup.kibana.ssl.enabled: true
    setup.dashboards.enabled: true
    # setup.dashboards.index: kubernetes-*

    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      exclude_files: /var/log/containers/.*filebeat.*\.log
    processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
    - drop_fields:
        fields:
        # - host
        - ecs
        - log
        - agent
        - input
        # - stream
        - container
        - kubernetes.node
        # - kubernetes
        ignore_missing: true

    output.elasticsearch:
      hosts: "${ELASTICSEARCH_HOST}"
      username: "elastic"
      password: "${elasticsearch-password}"
      # index: "kubernetes-%{+yyyy.MM.dd}"
---
# Source: filebeat/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat-filebeat-cluster-role
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.17.3"
    heritage: "Helm"
    release: "filebeat"
rules:
  - apiGroups:
    - ""
    resources:
    - namespaces
    - nodes
    - pods
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - apps
    resources:
    - replicasets
    verbs:
    - get
    - list
    - watch
---
# Source: filebeat/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat-filebeat-cluster-role-binding
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.17.3"
    heritage: "Helm"
    release: "filebeat"
roleRef:
  kind: ClusterRole
  name: filebeat-filebeat-cluster-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: filebeat-filebeat
  namespace: monitoring
---
# Source: filebeat/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: filebeat-filebeat-role
  labels:
    app: "filebeat-filebeat"
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs: ["get", "create", "update"]
---
# Source: filebeat/templates/rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: filebeat-filebeat-role-binding
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.17.3"
    heritage: "Helm"
    release: "filebeat"
roleRef:
  kind: Role
  name: filebeat-filebeat-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: filebeat-filebeat
  namespace: monitoring
---
# Source: filebeat/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat-filebeat
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.17.3"
    heritage: "Helm"
    release: "filebeat"
spec:
  selector:
    matchLabels:
      app: "filebeat-filebeat"
      release: "filebeat"
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:

        configChecksum: ee7bc2b3e133bc96a044ce8f17df96b66dc36a1f1c0d9f6d961a7c8e5b19e76
      name: "filebeat-filebeat"
      labels:
        app: "filebeat-filebeat"
        chart: "filebeat-7.17.3"
        heritage: "Helm"
        release: "filebeat"
    spec:
      tolerations:
        []
      nodeSelector:
        {}
      affinity:
        {}
      serviceAccountName: filebeat-filebeat
      terminationGracePeriodSeconds: 30
      volumes:
      - name: filebeat-config
        configMap:
          defaultMode: 0600
          name: filebeat-filebeat-daemonset-config
      - name: data
        hostPath:
          path: /var/lib/filebeat-filebeat-monitoring-data
          type: DirectoryOrCreate
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      - name: varrundockersock
        hostPath:
          path: /var/run/docker.sock
      containers:
      - name: "filebeat"
        image: "docker.elastic.co/beats/filebeat:7.17.3"
        imagePullPolicy: "IfNotPresent"
        args:
        - "-e"
        - "-E"
        - "http.enabled=true"
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              curl --fail 127.0.0.1:5066
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              filebeat test output
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 1000m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: ELASTICSEARCH_HOST
          value: https://elastic.codeplayer.org:443
        envFrom:
          - secretRef:
              name: elastic-security
        securityContext:
          privileged: false
          runAsUser: 0
        volumeMounts:
        - name: filebeat-config
          mountPath: /usr/share/filebeat/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        # Necessary when using autodiscovery; avoid mounting it otherwise
        # See: https://www.elastic.co/guide/en/beats/filebeat/7.17/configuration-autodiscover.html
        - name: varrundockersock
          mountPath: /var/run/docker.sock
          readOnly: true
---
# Source: filebeat/templates/deployment.yaml
# Deploy singleton instance in the whole cluster for some unique data sources, like aws input

Describe the bug:
filebeat readinessProbe always failed when load elasticsearch password from env and env load that password from secret

Steps to reproduce:

load the elasticsearch password from secret to env
load the elasticsearch password from env to config
install helm charts and readniess probe alway fails

Expected behavior:

readniess probe pass.

Provide logs and/or server output (if relevant):

I try to manully run filebeat test output in pod, it return Error initializing output: missing field accessing 'output.elasticsearch.password' (source:'filebeat.yml').

Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting

Any additional context:

Answer 1 · 2023-05-13T11:23:53.000Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.