Dropping logs
Closed this issue · 4 comments
Hello fellas,
I am trying to reduce the amount of logs I ingest, but I am unsure what is happening. I tried different ways, but none seemed to work. For instance, trying to drop logs from default and kube-system namespaces, I tried like this:
externalServices:
logs:
pod_logs:
extraRelabelingRules: |-
rule {
action = "labeldrop"
regex = "default|kube-system"
}
and
externalServices:
logs:
pod_logs:
extraRelabelingRules: |
rule {
action = "drop"
source_labels = ["__meta_kubernetes_namespace", "namespace"]
regex = "default|kube-system"
}
Am I doing something wrong?
You're very close!
The drop
action is what you need, but source_labels = ["__meta_kubernetes_namespace", "namespace"]
will result in both labels being combined into something like: default;default
This rule should work:
rule {
action = "drop"
source_labels = ["namespace"]
regex = "default|kube-system"
}
This works because we already set namespace
to the same as the meta label __meta_kubernetes_namespace
:
https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/templates/alloy_config/_pod_logs_discovery.alloy.txt#L15-L19
You're very close! The
drop
action is what you need, butsource_labels = ["__meta_kubernetes_namespace", "namespace"]
will result in both labels being combined into something like:default;default
This rule should work:
rule { action = "drop" source_labels = ["namespace"] regex = "default|kube-system" }
This works because we already set
namespace
to the same as the meta label__meta_kubernetes_namespace
: https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/templates/alloy_config/_pod_logs_discovery.alloy.txt#L15-L19
Thanks @petewall, but that didn't work :(
I still see the logs coming
I just tested this and it's working for my deployment. Sometimes, updated the Helm deployment will update the ConfigMap, but it takes a while for the Alloy pod to get the latest. Restarting your alloy logs instances would pick it up faster.