grafana/loki

Can't override "level" attribute when parsing structured logs

Opened this issue · 0 comments

Describe the bug
We're using detected_level: true in the limits config, but some logs are structured and include their own log level.
However when parsing the logs (for example using | json) the level attribute is not overwritten, nor extracted.

When setting detected_level: false the level attribute gets correctly extracted from the json parser and everything works as expected.

To Reproduce
Steps to reproduce the behavior:

  1. Start Loki (loki:3.2.1)
  2. Start otel collector to send logs to loki (see example logs below)
  3. Query: {k8s_namespace_name="iss", k8s_container_name="iss-app-core"} | json

Expected behavior
The level attribute from the json parsing result should be added as level_extracted attribute (as the level attribute itself already exists, this should be the expected behavior according to the documentation)

Environment:

  • Infrastructure: Kubernetes
  • Deployment tool: helm

Screenshots, Promtail config, or terminal output

Example log input (single line, formatted for better readability)

{
  "@timestamp": "2024-10-31T09:12:25.491016763Z",
  "@version": "1",
  "message": "Scanner job status has changed, updating root image job",
  "logger_name": "com.company.abc",
  "thread_name": "pool-2-thread-1",
  "level": "INFO",
  "level_value": 20000
}

Example query output (broken behavior):

{
    "_timestamp": "2024-10-31T09:12:25.491016763Z",
    "_version": "1",
    "k8s_container_name": "iss-app-core",
    "k8s_container_restart_count": "0",
    "k8s_namespace_name": "iss",
    "k8s_pod_name": "iss-app-core-5f77c6d988-xsglm",
    "k8s_pod_uid": "dc6eb4c2-e3cd-4297-8984-7b9b04998c22",
    "level": "debug",
    "level_value": "20000",
    "log_iostream": "stdout",
    "logger_name": "com.company.abc",
    "logtag": "F",
    "message": "Scanner job status has changed, updating root image job",
    "observed_timestamp": "1730366344538077487",
    "service_name": "unknown_service",
    "thread_name": "pool-2-thread-1"
  },

Example query output using detected_level: false

 {
    "_timestamp": "2024-10-31T09:12:25.491016763Z",
    "_version": "1",
    "k8s_container_name": "iss-app-core",
    "k8s_container_restart_count": "0",
    "k8s_namespace_name": "iss",
    "k8s_pod_name": "iss-app-core-5f77c6d988-xsglm",
    "k8s_pod_uid": "dc6eb4c2-e3cd-4297-8984-7b9b04998c22",
    "level": "INFO",
    "level_value": "20000",
    "log_iostream": "stdout",
    "logger_name": "com.company.abc",
    "logtag": "F",
    "message": "Scanner job status has changed, updating root image job",
    "observed_timestamp": "1730366344538077487",
    "service_name": "unknown_service",
    "thread_name": "pool-2-thread-1"
  },