grafana/loki

IPv6 YAML Configuration Parsing Errors in Dual-Stack EKS Environment

Opened this issue · 0 comments

Describe the bug
When running Loki in EKS with IPv6 enabled (dual-stack), pods fail due to YAML parsing issues with IPv6 addresses. The issue occurs when using the downward API to inject the pod's IP address into Loki's configuration.

To Reproduce

  1. Running Loki fails due to IPv6's colon notation:
# Problematic configuration (/etc/loki/config/config.yaml)
frontend:
  address: ${POD_IP}  # When POD_IP is IPv6 like 2a05:d014:1c15:14:9808::
extraEnv:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  1. Error message received:
failed parsing config: /etc/loki/config/config.yaml: yaml: line 51: mapping values are not allowed in this context

Trying to add double quotes and escape them results in single quotes:

(Before deployment)
frontend:
  address: "\"${POD_IP}\""

(After deployment)
frontend:
  address: '"${POD_IP}"'

Setting frontend address to "::" results in non-working installation (trying to query logs in Grafana UI takes forever and fails when timeout is reached. In the logs of loki-read.* pods, I see that the query is being executed, but I think frontend doesn't know where to send back results to the gateway). However, when I scale down read replicas to 1, everything works again.

Current Workarounds

  1. Manually edit the ConfigMap to add quotes around ${POD_IP}
  2. Set frontend.address to "::" and scale read replicas down to 1.

Environment:

  • Infrastructure: Amazon EKS
  • Deployment mode: Simple Scalable
  • Deployment tool: helm
  • Loki version: 3.2.0
  • Configuration:
loki:
  extraMemberlistConfig:
    bind_addr:
      - "::"

  frontend:
    address: "$POD_IP"

  common:
    ring:
      instance_enable_ipv6: true

  compactor:
    compactor_ring:
      instance_enable_ipv6: true

  distributor:
    ring:
      instance_enable_ipv6: true

  memberlist:
    bind_addr:
      - "::"
    bind_port: 7946
    cluster_label: local.namespace
    randomize_node_name: false

  query_scheduler:
    scheduler_ring:
      instance_enable_ipv6: true

  ruler:
    ring:
      instance_enable_ipv6: true

  index_gateway:
    ring:
      instance_enable_ipv6: true

  commonConfig:
    instance_interface_names:
      - "eth0"
    replication_factor: 3
    ring:
      kvstore:
        store: "memberlist"
      instance_enable_ipv6: true

Avoiding setting the frontend.address is impossible: ipv6 addresses are still not implemented in GetFirstAddressOf and filterIPs functions in https://github.com/grafana/loki/blob/main/pkg/util/net.go. So I'm forced to set frontend.address, or Loki pods will not start, because they can't get any address for the network interface.