Can't override "level" attribute when parsing structured logs
Opened this issue · 0 comments
Describe the bug
We're using detected_level: true
in the limits config, but some logs are structured and include their own log level.
However when parsing the logs (for example using | json
) the level attribute is not overwritten, nor extracted.
When setting detected_level: false
the level attribute gets correctly extracted from the json parser and everything works as expected.
To Reproduce
Steps to reproduce the behavior:
- Start Loki (loki:3.2.1)
- Start otel collector to send logs to loki (see example logs below)
- Query:
{k8s_namespace_name="iss", k8s_container_name="iss-app-core"} | json
Expected behavior
The level
attribute from the json parsing result should be added as level_extracted
attribute (as the level
attribute itself already exists, this should be the expected behavior according to the documentation)
Environment:
- Infrastructure: Kubernetes
- Deployment tool: helm
Screenshots, Promtail config, or terminal output
Example log input (single line, formatted for better readability)
{
"@timestamp": "2024-10-31T09:12:25.491016763Z",
"@version": "1",
"message": "Scanner job status has changed, updating root image job",
"logger_name": "com.company.abc",
"thread_name": "pool-2-thread-1",
"level": "INFO",
"level_value": 20000
}
Example query output (broken behavior):
{
"_timestamp": "2024-10-31T09:12:25.491016763Z",
"_version": "1",
"k8s_container_name": "iss-app-core",
"k8s_container_restart_count": "0",
"k8s_namespace_name": "iss",
"k8s_pod_name": "iss-app-core-5f77c6d988-xsglm",
"k8s_pod_uid": "dc6eb4c2-e3cd-4297-8984-7b9b04998c22",
"level": "debug",
"level_value": "20000",
"log_iostream": "stdout",
"logger_name": "com.company.abc",
"logtag": "F",
"message": "Scanner job status has changed, updating root image job",
"observed_timestamp": "1730366344538077487",
"service_name": "unknown_service",
"thread_name": "pool-2-thread-1"
},
Example query output using detected_level: false
{
"_timestamp": "2024-10-31T09:12:25.491016763Z",
"_version": "1",
"k8s_container_name": "iss-app-core",
"k8s_container_restart_count": "0",
"k8s_namespace_name": "iss",
"k8s_pod_name": "iss-app-core-5f77c6d988-xsglm",
"k8s_pod_uid": "dc6eb4c2-e3cd-4297-8984-7b9b04998c22",
"level": "INFO",
"level_value": "20000",
"log_iostream": "stdout",
"logger_name": "com.company.abc",
"logtag": "F",
"message": "Scanner job status has changed, updating root image job",
"observed_timestamp": "1730366344538077487",
"service_name": "unknown_service",
"thread_name": "pool-2-thread-1"
},