Container name as label in logs
tr3mor opened this issue · 4 comments
Hello,
We are using scalyr Kubernetes agent with kubernetes_monitor and a pretty basic configuration.
Scalyr adds a lot of useful data from k8s labels and k8s dependencies to logs.
One thing we are missing and would like to suggest as a feature request is adding the name of the container (as it is named in k8s manifest) as a log label to be able to filter logs of sidecars, for example.
Now Scalyr adds only the contrainer_id label, which is just a hash and cant really be used to do filtering of specific container in pod.
Thanks for the suggestion. I've opened a feature request for the product to review. We'll notify you if we have any updates for you. Thanks
The Scalyr agent supports including the kubernetes container_name as an event attribute. It is disabled by default, and a configuration change must be made to enable it.
Enable container_name for all pods:
This gist provides step-by-step instructions for making the necessary ConfigMap changes to enable container_name for all pods.
Enable container_name for only selected pods:
The Scalyr agent Release Notes version 2.1.19 explains how to use labels to enable container_name for only specific pods.
Hello,
- Thank you for sharing, let's say this way, random gist is not a discoverable configuration documentation for the product.
- Steps provided in gist work supper inconsistently
i.e I can see the following in logs
2024-05-29 09:18:17.982Z Level 7 [monitor:kubernetes_monitor()] [scalyr_agent.monitor_utils.k8s:2975] Applying k8s_log config item 0. Matched pod_name ('cilium-w5msn', '*'), pod_namespace ('kube-system', '*'), k8s_container_name ('mount-bpf-fs', '*')
2024-05-29 09:18:17.982Z Level 7 [monitor:kubernetes_monitor()] [scalyr_agent.monitor_utils.k8s:2975] Applying k8s_log config item 0. Matched pod_name ('cilium-w5msn', '*'), pod_namespace ('kube-system', '*'), k8s_container_name ('clean-cilium-state', '*')
2024-05-29 09:18:18.073Z Level 7 [monitor:kubernetes_monitor()] [scalyr_agent.monitor_utils.k8s:2975] Applying k8s_log config item 0. Matched pod_name ('cilium-w5msn', '*'), pod_namespace ('kube-system', '*'), k8s_container_name ('install-cni-binaries', '*')
2024-05-29 09:18:18.074Z Level 7 [monitor:kubernetes_monitor()] [scalyr_agent.monitor_utils.k8s:2975] Applying k8s_log config item 0. Matched pod_name ('cilium-w5msn', '*'), pod_namespace ('kube-system', '*'), k8s_container_name ('cilium-agent', '*')
2024-05-29 09:18:18.173Z Level 7 [monitor:kubernetes_monitor()] [scalyr_agent.monitor_utils.k8s:2975] Applying k8s_log config item 0. Matched pod_name ('prometheus-grafana-56f586b7c-x25gw', '*'), pod_namespace ('monitoring', '*'), k8s_container_name ('grafana', '*')
2024-05-29 09:18:18.175Z Level 7 [monitor:kubernetes_monitor()] [scalyr_agent.monitor_utils.k8s:2975] Applying k8s_log config item 0. Matched pod_name ('prometheus-grafana-56f586b7c-x25gw', '*'), pod_namespace ('monitoring', '*'), k8s_container_name ('grafana-sc-dashboard', '*')
2024-05-29 09:18:18.176Z Level 7 [monitor:kubernetes_monitor()] [scalyr_agent.monitor_utils.k8s:2975] Applying k8s_log config item 0. Matched pod_name ('prometheus-grafana-56f586b7c-x25gw', '*'), pod_namespace ('monitoring', '*'), k8s_container_name ('grafana-sc-datasources', '*')
In the end, all logs from cilium have the required label, and all logs for Grafana do not have it.
Restart of agent/grafana deployment doesn't change the situation.
3) Annotation works better but requires every deployment to be annotated to include things by default.
So, if it is an officially supported feature, it doesn't work consistently. I can go to support, but I would love to understand how this feature should work in the first place (since there is no official documentation except release notes)
Okay, looking deeper into agent code, the reason for point 2 is that attributes from the annotations will replace attributes from k8s_logs.
so, if you provide ANY attribute via annotation, you cant use k8s_logs for this pod, only annotations.