grafana/k8s-monitoring-helm

Tailing K8s pod logs

gabriel-yahav opened this issue · 2 comments

gabriel-yahav commented

Hey,
I've noticed the current chart's schema doesn't allow to tail the pod logs, but only from the beginning of the
file. Is there a workaround to still make the K8s pod logs scrapping work in tailing mode without chaning
manually the ConfigMap after installing the chart ?

https://grafana.com/docs/alloy/latest/reference/components/loki.source.file/

Thanks !

bentonam commented

@gabriel-yahav the chart supports gathering logs from:

  1. pod_logs
    a. by have a volume mounted /var/log that is to the worker node and directly tails the pod logs, logs.pod_logs.gatherMethod: volumes (default) or
    b. by using the kubernetes api for logs logs.pod_logs.gatherMethod: api
  2. PodLog Objects
  3. cluster_events by scraping the K8s API
  4. Journal logs from the worker.

When using pod_logs and gatherMethod: volumes which by default both are enabled, local.file_match is used as well as loki.source.file this can be seen in the rendered example logs.alloy config file in the default-values example

bentonam commented

@gabriel-yahav I mis-read your initial comment. Are you asking for the chart to support the tail_from_end argument in loki.source.file? I guess this could be added to the values file. It would have to be explicitly turned on, as when tail_from_end: true only new files are captured and tailed, any existing files would be ignored.

  • 👍2