IPv6 YAML Configuration Parsing Errors in Dual-Stack EKS Environment
Opened this issue · 0 comments
Describe the bug
When running Loki in EKS with IPv6 enabled (dual-stack), pods fail due to YAML parsing issues with IPv6 addresses. The issue occurs when using the downward API to inject the pod's IP address into Loki's configuration.
To Reproduce
- Running Loki fails due to IPv6's colon notation:
# Problematic configuration (/etc/loki/config/config.yaml)
frontend:
address: ${POD_IP} # When POD_IP is IPv6 like 2a05:d014:1c15:14:9808::
extraEnv:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- Error message received:
failed parsing config: /etc/loki/config/config.yaml: yaml: line 51: mapping values are not allowed in this context
Trying to add double quotes and escape them results in single quotes:
(Before deployment)
frontend:
address: "\"${POD_IP}\""
(After deployment)
frontend:
address: '"${POD_IP}"'
Setting frontend address to "::" results in non-working installation (trying to query logs in Grafana UI takes forever and fails when timeout is reached. In the logs of loki-read.* pods, I see that the query is being executed, but I think frontend doesn't know where to send back results to the gateway). However, when I scale down read replicas to 1, everything works again.
Current Workarounds
- Manually edit the ConfigMap to add quotes around ${POD_IP}
- Set frontend.address to "::" and scale read replicas down to 1.
Environment:
- Infrastructure: Amazon EKS
- Deployment mode: Simple Scalable
- Deployment tool: helm
- Loki version: 3.2.0
- Configuration:
loki:
extraMemberlistConfig:
bind_addr:
- "::"
frontend:
address: "$POD_IP"
common:
ring:
instance_enable_ipv6: true
compactor:
compactor_ring:
instance_enable_ipv6: true
distributor:
ring:
instance_enable_ipv6: true
memberlist:
bind_addr:
- "::"
bind_port: 7946
cluster_label: local.namespace
randomize_node_name: false
query_scheduler:
scheduler_ring:
instance_enable_ipv6: true
ruler:
ring:
instance_enable_ipv6: true
index_gateway:
ring:
instance_enable_ipv6: true
commonConfig:
instance_interface_names:
- "eth0"
replication_factor: 3
ring:
kvstore:
store: "memberlist"
instance_enable_ipv6: true
Avoiding setting the frontend.address is impossible: ipv6 addresses are still not implemented in GetFirstAddressOf
and filterIPs
functions in https://github.com/grafana/loki/blob/main/pkg/util/net.go. So I'm forced to set frontend.address, or Loki pods will not start, because they can't get any address for the network interface.