Deployment in headless mode does not resolve public ip address of client
Closed this issue · 4 comments
viktor-sedov-gcx commented
Description
When deploying stunner in headless mode (following the example provided) stun resolves the client ip address to some internal kubernetes IP address instead of it's public ip address. Therefore STUN/TURN won't work.
Steps to Reproduce
Expected behavior: The clients IP address would be resolved correctly
Actual behavior: Internal kubernetes IP is being returned
Versions
Helm chart 0.18.0
Info
The issue seems to be the LoadBalancers externalTrafficPolicy. However when switching to "local", it works fine.
Is there any way of configuring the externalTrafficPolicy for the helm chart installation?
Gateway API status
apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1","kind":"Gateway","metadata":{"annotations":{"stunner.l7mp.io/enable-mixed-protocol-lb":"true"},"name":"stunner-netbird-mixed","namespace":"platform-stunner"},"spec":{"gatewayClassName":"stunner-netbird","listeners":[{"name":"udp-gateway","port":3478,"protocol":"TURN-UDP"},{"name":"tcp-gateway","port":3478,"protocol":"TURN-TCP"}]}}
stunner.l7mp.io/enable-mixed-protocol-lb: "true"
creationTimestamp: "2024-05-16T08:04:39Z"
generation: 3
name: stunner-netbird-mixed
namespace: platform-stunner
resourceVersion: "15060695"
uid: 7b39273a-1185-4d02-8c97-6b28ab83a98c
spec:
gatewayClassName: stunner-netbird
listeners:
- allowedRoutes:
namespaces:
from: Same
name: udp-gateway
port: 3478
protocol: TURN-UDP
- allowedRoutes:
namespaces:
from: Same
name: tcp-gateway
port: 3478
protocol: TURN-TCP
status:
addresses:
- type: IPAddress
value: 4.175.131.225
conditions:
- lastTransitionTime: "2024-05-16T08:04:42Z"
message: gateway accepted by controller stunner.l7mp.io/gateway-operator
observedGeneration: 3
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-05-17T12:22:07Z"
message: dataplane configuration successfully rendered
observedGeneration: 3
reason: Programmed
status: "True"
type: Programmed
listeners:
- attachedRoutes: 1
conditions:
- lastTransitionTime: "2024-05-17T12:22:07Z"
message: listener accepted
observedGeneration: 3
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-05-17T12:22:07Z"
message: listener protocol-port available
observedGeneration: 3
reason: NoConflicts
status: "False"
type: Conflicted
- lastTransitionTime: "2024-05-17T12:22:07Z"
message: listener object references sucessfully resolved
observedGeneration: 3
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
name: udp-gateway
supportedKinds:
- group: gateway.networking.k8s.io
kind: UDPRoute
- group: stunner.l7mp.io
kind: UDPRoute
- attachedRoutes: 1
conditions:
- lastTransitionTime: "2024-05-17T12:22:07Z"
message: listener accepted
observedGeneration: 3
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-05-17T12:22:07Z"
message: listener protocol-port available
observedGeneration: 3
reason: NoConflicts
status: "False"
type: Conflicted
- lastTransitionTime: "2024-05-17T12:22:07Z"
message: listener object references sucessfully resolved
observedGeneration: 3
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
name: tcp-gateway
supportedKinds:
- group: gateway.networking.k8s.io
kind: UDPRoute
- group: stunner.l7mp.io
kind: UDPRoute
- apiVersion: stunner.l7mp.io/v1
kind: GatewayConfig
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"stunner.l7mp.io/v1","kind":"GatewayConfig","metadata":{"annotations":{},"name":"stunner-netbird","namespace":"platform-stunner"},"spec":{"authRef":{"name":"stunner-basic-auth","namespace":"platform-stunner"},"dataplane":"default","logLevel":"all:INFO","realm":"stunner.l7mp.io"}}
creationTimestamp: "2024-05-16T08:04:40Z"
generation: 1
name: stunner-netbird
namespace: platform-stunner
resourceVersion: "14484047"
uid: f91bfbfc-7e7c-4082-a788-3815307a1749
spec:
authRef:
group: ""
kind: Secret
name: stunner-basic-auth
namespace: platform-stunner
authType: plaintext
dataplane: default
logLevel: all:INFO
realm: stunner.l7mp.io
- apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.networking.k8s.io/v1","kind":"GatewayClass","metadata":{"annotations":{},"name":"stunner-netbird"},"spec":{"controllerName":"stunner.l7mp.io/gateway-operator","description":"Default GatewayClass for Stunner used for netbird poc deployment","parametersRef":{"group":"stunner.l7mp.io","kind":"GatewayConfig","name":"stunner-netbird","namespace":"platform-stunner"}}}
creationTimestamp: "2024-05-15T12:25:39Z"
generation: 2
name: stunner-netbird
resourceVersion: "14484077"
uid: aeafe850-ba2a-4adf-abef-a26eab1e5a71
spec:
controllerName: stunner.l7mp.io/gateway-operator
description: Default GatewayClass for Stunner used for netbird poc deployment
parametersRef:
group: stunner.l7mp.io
kind: GatewayConfig
name: stunner-netbird
namespace: platform-stunner
status:
conditions:
- lastTransitionTime: "2024-05-16T08:04:42Z"
message: GatewayClass is now managed by controller "stunner.l7mp.io/gateway-operator"
observedGeneration: 2
reason: Accepted
status: "True"
type: Accepted
- apiVersion: stunner.l7mp.io/v1
kind: UDPRoute
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"stunner.l7mp.io/v1","kind":"UDPRoute","metadata":{"annotations":{},"name":"stunner-headless","namespace":"platform-stunner"},"spec":{"parentRefs":[{"name":"stunner-netbird-mixed"}],"rules":[{"backendRefs":[{"name":"stunner-netbird-mixed","namespace":"platform-stunner"}]}]}}
creationTimestamp: "2024-05-16T08:04:40Z"
generation: 1
name: stunner-headless
namespace: platform-stunner
resourceVersion: "15060696"
uid: dd4b9d3e-880f-4fc8-b818-5e4e6bf53124
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: stunner-netbird-mixed
rules:
- backendRefs:
- group: ""
kind: Service
name: stunner-netbird-mixed
namespace: platform-stunner
status:
parents:
- conditions:
- lastTransitionTime: "2024-05-17T12:22:07Z"
message: parent accepts the route
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2024-05-17T12:22:07Z"
message: all backend references successfully resolved
observedGeneration: 1
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: stunner.l7mp.io/gateway-operator
parentRef:
group: gateway.networking.k8s.io
kind: Gateway
name: stunner-netbird-mixed
kind: List
metadata:
resourceVersion: ""Operator logs
<details>
<summary>Logs</summary>
2024-05-17T12:24:47.436819973Z INFO renderer STUNner dataplane Deployment ready {"generation": 2261, "deployment": "{\"metadata\":{\"name\":\"stunner-netbird-mixed\",\"namespace\":\"platform-stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/enable-mixed-protocol-lb\":\"true\",\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"stunner-netbird-mixed\",\"uid\":\"7b39273a-1185-4d02-8c97-6b28ab83a98c\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"stunner-netbird-mixed\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"platform-stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"docker.io/l7mp/stunnerd:0.18.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"stunner-netbird-mixed\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"platform-stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://172.16.24.31:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-05-17T12:24:47.436863801Z INFO renderer STUNner dataplane configuration ready {"generation": 2261, "config": "{version=\"v1\",admin:{name=\"platform-stunner/stunner-netbird-mixed\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},static-auth:{realm=\"stunner.l7mp.io\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"platform-stunner/stunner-netbird-mixed/udp-gateway\":{turn://0.0.0.0:3478,public=4.175.131.225:3478,cert/key=-/-,routes=[platform-stunner/stunner-headless]},\"platform-stunner/stunner-netbird-mixed/tcp-gateway\":{turn://0.0.0.0:3478,public=4.175.131.225:3478,cert/key=-/-,routes=[platform-stunner/stunner-headless]}],clusters=[\"platform-stunner/stunner-headless\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.0.161.193,172.16.24.34]}]}"}
2024-05-17T12:24:47.436915163Z INFO cds-server processing config update event {"generation": 2261, "update": "update (gen: 2261): upsert-queue: gway-cls: 1, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-05-17T12:24:47.436994391Z INFO updater processing update event {"generation": 2261, "update": "update (gen: 2261): upsert-queue: gway-cls: 1, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-05-17T12:26:35.021495533Z INFO node-controller reconciling {"node": "/aks-poolopxi-37140328-vmss000001"}
2024-05-17T12:26:35.021660096Z INFO node-controller failed to find node with valid external address {"reason": "end of node list reached after searching through 4 node(s)"}
2024-05-17T12:26:35.272075164Z INFO renderer rendering configuration {"generation": 2262, "event": "render"}
2024-05-17T12:26:35.272103863Z INFO renderer commencing dataplane render {"mode": "managed"}
2024-05-17T12:26:35.272121792Z INFO renderer rendering configuration {"gateway-class": "/stunner-netbird"}
2024-05-17T12:26:35.272434234Z INFO renderer creating public service for gateway {"service": "platform-stunner/stunner-netbird-mixed", "gateway": "platform-stunner/stunner-netbird-mixed", "service": "{\"kind\":\"Service\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"stunner-netbird-mixed\",\"namespace\":\"platform-stunner\",\"uid\":\"3dfad60a-9815-4474-ac39-5107054cb7fa\",\"resourceVersion\":\"14991483\",\"creationTimestamp\":\"2024-05-16T08:04:42Z\",\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/enable-mixed-protocol-lb\":\"true\",\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"stunner-netbird-mixed\",\"uid\":\"7b39273a-1185-4d02-8c97-6b28ab83a98c\"}],\"finalizers\":[\"service.kubernetes.io/load-balancer-cleanup\"]},\"spec\":{\"ports\":[{\"name\":\"udp-gateway\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":31024},{\"name\":\"tcp-gateway\",\"protocol\":\"TCP\",\"port\":3478,\"targetPort\":3478,\"nodePort\":31024}],\"selector\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"clusterIP\":\"10.0.161.193\",\"clusterIPs\":[\"10.0.161.193\"],\"type\":\"LoadBalancer\",\"sessionAffinity\":\"None\",\"loadBalancerIP\":\"4.175.131.225\",\"externalTrafficPolicy\":\"Local\",\"healthCheckNodePort\":31809,\"ipFamilies\":[\"IPv4\"],\"ipFamilyPolicy\":\"SingleStack\",\"allocateLoadBalancerNodePorts\":true,\"internalTrafficPolicy\":\"Cluster\"},\"status\":{\"loadBalancer\":{\"ingress\":[{\"ip\":\"4.175.131.225\"}]}}}"}
2024-05-17T12:26:35.272591935Z INFO renderer update (gen: 2262): upsert-queue: gway-cls: 0, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 0
2024-05-17T12:26:35.27293623Z INFO renderer STUNner dataplane Deployment ready {"generation": 2262, "deployment": "{\"metadata\":{\"name\":\"stunner-netbird-mixed\",\"namespace\":\"platform-stunner\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/enable-mixed-protocol-lb\":\"true\",\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1\",\"kind\":\"Gateway\",\"name\":\"stunner-netbird-mixed\",\"uid\":\"7b39273a-1185-4d02-8c97-6b28ab83a98c\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchExpressions\":[{\"key\":\"app\",\"operator\":\"In\",\"values\":[\"stunner\"]},{\"key\":\"stunner.l7mp.io/related-gateway-name\",\"operator\":\"In\",\"values\":[\"stunner-netbird-mixed\"]},{\"key\":\"stunner.l7mp.io/related-gateway-namespace\",\"operator\":\"In\",\"values\":[\"platform-stunner\"]}]},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"app\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-netbird-mixed\",\"stunner.l7mp.io/related-gateway-namespace\":\"platform-stunner\"},\"annotations\":{\"stunner.l7mp.io/related-gateway-name\":\"platform-stunner/stunner-netbird-mixed\"}},\"spec\":{\"containers\":[{\"name\":\"stunner-daemon\",\"image\":\"docker.io/l7mp/stunnerd:0.18.0\",\"command\":[\"stunnerd\"],\"args\":[\"-w\",\"--udp-thread-num=16\"],\"env\":[{\"name\":\"STUNNER_ADDR\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"STUNNER_NAME\",\"value\":\"stunner-netbird-mixed\"},{\"name\":\"STUNNER_NAMESPACE\",\"value\":\"platform-stunner\"},{\"name\":\"STUNNER_CONFIG_ORIGIN\",\"value\":\"http://172.16.24.31:13478\"}],\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"500m\",\"memory\":\"128Mi\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/live\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/ready\",\"port\":8086,\"scheme\":\"HTTP\"},\"timeoutSeconds\":1,\"periodSeconds\":15,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"Always\"}],\"terminationGracePeriodSeconds\":3600}},\"strategy\":{}},\"status\":{}}"}
2024-05-17T12:26:35.273039314Z INFO renderer STUNner dataplane configuration ready {"generation": 2262, "config": "{version=\"v1\",admin:{name=\"platform-stunner/stunner-netbird-mixed\",logLevel=\"all:INFO\",health-check=\"http://:8086\"},static-auth:{realm=\"stunner.l7mp.io\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"platform-stunner/stunner-netbird-mixed/udp-gateway\":{turn://0.0.0.0:3478,public=4.175.131.225:3478,cert/key=-/-,routes=[platform-stunner/stunner-headless]},\"platform-stunner/stunner-netbird-mixed/tcp-gateway\":{turn://0.0.0.0:3478,public=4.175.131.225:3478,cert/key=-/-,routes=[platform-stunner/stunner-headless]}],clusters=[\"platform-stunner/stunner-headless\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[10.0.161.193,172.16.24.34]}]}"}
2024-05-17T12:26:35.273121369Z INFO cds-server processing config update event {"generation": 2262, "update": "update (gen: 2262): upsert-queue: gway-cls: 1, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
2024-05-17T12:26:35.273162868Z INFO updater processing update event {"generation": 2262, "update": "update (gen: 2262): upsert-queue: gway-cls: 1, gway: 1, route: 1, routeV1A2: 0, svc: 1, confmap: 0, dp: 1 / delete-queue: gway-cls: 0, gway: 0, route: 0, routeV1A2: 0, svc: 0, confmap: 0, dp: 0 / config-queue: 1"}
</details>
rg0now commented
This is intended. As long as the STUNner dataplane is deployed into ordinary Kubernetes pods and runs on private IPs (which is The Way For Deploying STUNner), it will return a fairly random private IP in STUN responses (unless you use some ugly Kubernetes hacks).
This is exactly why we do not encourage people to use STUNner as a STUN service. If you really insist on using STUN though, deploy the STUNner dataplane into the host-network namespace. But you'd better not do that, as it fails in many cloud providers' service.
The good news is that it does not matter: STUNner is intended to be used as a TURN Gateway and luckily TURN does not care about the client IP at all.
viktor-sedov-gcx commented
thank you very much for your quick response and already implementing this feature!
we're using stunner as the backbone for a peer-to-peer networking solution and the docs implied that the headless deployment did exactly what I stated above ( at least for me)
do you have a schedule for when this feature will be available in a release?
rg0now commented
We plan to release an RC-3 this or early next month, but there are still some outstanding issues we need to close first. Until then, you can test the latest goodies from the dev release channel.