saferwall/saferwall

[logging] Elasticsearch don't have data from filebeat

Closed this issue · 2 comments

johnsaker97 commented

I enabled elasticsearch + filebeat + kibana in values.yaml. Those service still running OK, I access kibana:5601 then can't find out Log from filebeat to create index pattern: filebeat-*.

I read file values.yaml, I don't know value ${NODE_NAME} and ${ELASTICSEARCH_HOSTS:elasticsearch-master:9200} in module filebeat what is that?

Please guide to me how to setup ELK to use it.

LordNoteworthy commented

Hello @johnsaker97

That is right, after I changed the name of the deployments, I did not update the filebeat config to match them:

####### FILEBEAT #########
filebeat:
  enabled: false
  filebeatConfig:
    filebeat.yml: |-
      filebeat.autodiscover:
        providers:
          - type: kubernetes
            templates:
              - condition:
                  or:
                    - equals:
                        kubernetes.container.name: backend
                    - equals:
                        kubernetes.container.name: consumer
                config:
                  - type: container
                    paths:
                      - "/var/log/containers/*-${data.kubernetes.container.id}.log"
                    json.add_error_key: true
      processors:
        - add_cloud_metadata: ~
        - add_docker_metadata: ~
      output.elasticsearch:
        host: '${NODE_NAME}'
        hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'

Those conditions needs to change for it to work.

Besides that, I am considering using Loki from Grafana in the upcoming release, it is on my second item on the list.

I believe it is gonna take me 1 week to finish the item in the top of the list then maybe another week to fiddle with Loki.

If you just wanna use EFK stack, just match the conditions of the pods/deployment you want to consume their logs.

Cheers.

johnsaker97 commented

Yep tks. i have changed value in condition then get successful log

  • 🎉1