On GKE, how to specify an IP for udp-gateway
nhha1602 opened this issue · 11 comments
Dear All,
I tried to deploy stunner on GKE. As your docs, when apply file stunner-helm/livekit-call-stunner.yaml, it will auto create an udp-gateway service then it will auto get an IP. But I want to specify an IP for it as below that I did, but it still get a different IP.
apiVersion: stunner.l7mp.io/v1alpha1
kind: GatewayConfig
metadata:
name: stunner-gatewayconfig
namespace: stunner-dev
spec:
realm: stunner.l7mp.io
authType: plaintext
userName: "user-1"
password: "pass-1"
loadBalancerServiceAnnotations:
networking.gke.io/load-balancer-type: Internal
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: udp-gateway
namespace: stunner-dev
spec:
gatewayClassName: stunner-gatewayclass
listeners:
- name: udp-listener
port: 3478
protocol: TURN-UDP
addresses:
- type: NamedAddress
value: my_reserved_ip4
The "my_reserved_ip4" created by command:
gcloud compute addresses create IP_ADDRESS_NAME \
--purpose=SHARED_LOADBALANCER_VIP \
--region=COMPUTE_REGION \
--subnet=SUBNET \
--project=PROJECT_ID
Could you, please help.
Thanks and regards.
Using a fix external IP is, theoretically, supported by STUNner, but this is something that is not quite standardized across cloud providers and so we haven't tested this feature extensively yet.
The problem is that the address type you're using in the Gateway spec (NamedAddress) is deprecated and so we currently do not accept NamedAddress as the address type.
I have two suggestions.
First, try to replace the NamedAddress type with IPAddress as in the below, hoping that GKE would accept it:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: udp-gateway
namespace: stunner-dev
spec:
gatewayClassName: stunner-gatewayclass
listeners:
- name: udp-listener
port: 3478
protocol: TURN-UDP
addresses:
- type: IPAddress
value: my_reserved_ip4
The other idea would be to create the LB Service manually (just copy-paste the YAML of the udp-gateway Service) and set the fields spec.externalIPs and spec.loadBalancerIP to the IP address you got from GKE. Would this work? Can you reach STUNner now over that IP? If you do then it is a quick patch to change the code to accept NamedAddress, that would at least fix GKE.
Also, can you please copy-paste the YAML of the service STUNner has created (kubectl -n stunner-dev get svc udp-gateway -o yaml)?
It just occurred to me that you could query the IP address assigned by gcloud compute addresses create ..., let the result be A.B.C.D. Then, set this IP with type IPAddress in the Gateway spec.addresses:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: udp-gateway
namespace: stunner-dev
spec:
gatewayClassName: stunner-gatewayclass
listeners:
- name: udp-listener
port: 3478
protocol: TURN-UDP
addresses:
- type: IPAddress
value: A.B.C.D
It seems the NamedAddress type is not supported outside Google's own in-house load-balancers and, regrettably, we're not Google. Would this solution work for you?
Thank you for your helps,
I tried as your comments but I got below error on stunner operator:
2023-10-27T01:35:25.676233438Z INFO renderer creating public service for gateway {"name": "stunner-dev/udp-gateway", "gateway": "udp-gateway", "service": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner-dev\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner-dev\"},\"annotations\":{\"networking.gke.io/load-balancer-type\":\"Internal\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-dev/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1beta1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"21560782-753a-4e93-9689-14de0157495c\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":0}],\"selector\":{\"app\":\"stunner\"},\"type\":\"LoadBalancer\",\"externalIPs\":[\"172.18.100.73\"],\"loadBalancerIP\":\"172.18.100.73\"},\"status\":{\"loadBalancer\":{}}}"}
2023-10-27T01:35:25.676331505Z INFO renderer STUNner dataplane configuration ready {"generation": 2762, "config": "{version=\"v1alpha1\",admin:{name=\"stunner-daemon\",logLevel=\"all:INFO\",health-check=\"http://0.0.0.0:8086\"},auth:{realm=\"stunner.l7mp.io\",type=\"plaintext\",username=\"<SECRET>\",password=\"<SECRET>\"},listeners=[\"stunner-dev/udp-gateway/udp-listener\":{turn://0.0.0.0:3478<32768-65535>,public=-:-,cert/key=-/-,routes=[stunner-dev/livekit-media-plane]}],clusters=[\"stunner-dev/livekit-media-plane\":{type=\"STATIC\",protocol=\"UDP\",endpoints=[172.18.112.32,172.31.2.26]}]}"}
2023-10-27T01:35:25.676385128Z INFO cds-server processing config update event {"generation": 2762, "update": "update (gen: 2762): upsert-queue: gway-cls: 1, gway: 1, route: 1, svc: 1, confmap: 1, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, svc: 0, confmap: 0, dp: 0"}
2023-10-27T01:35:25.67641283Z INFO updater processing update event {"generation": 2762, "update": "update (gen: 2762): upsert-queue: gway-cls: 1, gway: 1, route: 1, svc: 1, confmap: 1, dp: 0 / delete-queue: gway-cls: 0, gway: 0, route: 0, svc: 0, confmap: 0, dp: 0"}
2023-10-27T01:35:25.729584964Z ERROR updater cannot update service {"operation": "unchanged", "service": "{\"metadata\":{\"name\":\"udp-gateway\",\"namespace\":\"stunner-dev\",\"creationTimestamp\":null,\"labels\":{\"stunner.l7mp.io/owned-by\":\"stunner\",\"stunner.l7mp.io/related-gateway-name\":\"udp-gateway\",\"stunner.l7mp.io/related-gateway-namespace\":\"stunner-dev\"},\"annotations\":{\"networking.gke.io/load-balancer-type\":\"Internal\",\"stunner.l7mp.io/related-gateway-name\":\"stunner-dev/udp-gateway\"},\"ownerReferences\":[{\"apiVersion\":\"gateway.networking.k8s.io/v1beta1\",\"kind\":\"Gateway\",\"name\":\"udp-gateway\",\"uid\":\"21560782-753a-4e93-9689-14de0157495c\"}]},\"spec\":{\"ports\":[{\"name\":\"udp-listener\",\"protocol\":\"UDP\",\"port\":3478,\"targetPort\":0}],\"selector\":{\"app\":\"stunner\"},\"type\":\"LoadBalancer\",\"externalIPs\":[\"172.18.100.73\"],\"loadBalancerIP\":\"172.18.100.73\"},\"status\":{\"loadBalancer\":{}}}", "error": "cannot upsert service \"stunner-dev/udp-gateway\": services \"udp-gateway\" is forbidden: Use of external IPs is denied by admission control"}
github.com/l7mp/stunner-gateway-operator/internal/updater.(*Updater).ProcessUpdate
/workspace/internal/updater/updater.go:115
github.com/l7mp/stunner-gateway-operator/internal/updater.(*Updater).Start.func1
/workspace/internal/updater/updater.go:62
As I see, this error (is forbidden: Use of external IPs is denied by admission control) caused by duplicate by externalIP and loadbalacerIP ? One more thing, that I'm setting this LoadBalancer is Internal - \"networking.gke.io/load-balancer-type\":\"Internal\"
\"type\":\"LoadBalancer\",\"externalIPs\":[\"172.18.100.73\"],\"loadBalancerIP\":\"172.18.100.73\"}
Could you please advise this?
Hi,
I just found this link:
(l7mp/stunner-gateway-operator#32 (comment))
and stunner operator code:
// forward the first requested address to Kubernetes
if len(gw.Spec.Addresses) > 0 {
if gw.Spec.Addresses[0].Type == nil ||
(gw.Spec.Addresses[0].Type != nil &&
*gw.Spec.Addresses[0].Type == gwapiv1a2.IPAddressType) {
// only the first address can be used because
// stunner is limited to use a single public address
// https://github.com/l7mp/stunner-gateway-operator/issues/32#issuecomment-1648035135
svc.Spec.ExternalIPs = []string{gw.Spec.Addresses[0].Value}
svc.Spec.LoadBalancerIP = gw.Spec.Addresses[0].Value
}
}
So, may be this we set externalIP to LoadBalancerIP will raised above error. Because my network team dose not allow to use externalIP in GKE - they will do NAT the internal LoadBalancerIP to public.
Please advise this.
@nhha1602 Hi,
as you mentioned your team does not allow using the externapIP field, thus you would like to set the svc.Spec.LoadBalancerIP field perhaps. I don't know what the answer should be yet however the loadBalancerIP field is deprecated and ignored by most cloud providers. Do you have any proof that GKE supports it?
// Only applies to Service Type: LoadBalancer.
// This feature depends on whether the underlying cloud-provider supports specifying
// the loadBalancerIP when a load balancer is created.
// This field will be ignored if the cloud-provider does not support the feature.
// Deprecated: This field was under-specified and its meaning varies across implementations,
// and it cannot support dual-stack.
// As of Kubernetes v1.24, users are encouraged to use implementation-specific annotations when available.
// This field may be removed in a future API version.
// +optional
LoadBalancerIP string `json:"loadBalancerIP,omitempty" protobuf:"bytes,8,opt,name=loadBalancerIP"`
We need to figure out first what all these fields mean actually and which of them are supported, etc. I think we need more docs about this topic, if you got some feel free to share with us.
For a quick fix you can just remove the offending line below, build a new stunner image and deploy that:
svc.Spec.ExternalIPs = []string{gw.Spec.Addresses[0].Value}
Or you can remove the other problematic line and start anew:
svc.Spec.LoadBalancerIP = gw.Spec.Addresses[0].Value
Let us know which one works, if any.
Unfortunately support for static external IPs is so underspeficied in Kubernetes and the Gateway API that we don't know what's the most portable way to implement this. The official advice is to use cloud-provider specific Service annotations but we couldn't find any for GKE, let alone for EKS and the other one thousand cloud providers out there.
THank you for your helps,
I remove line: svc.Spec.ExternalIPs = []string{gw.Spec.Addresses[0].Value} and build new image for stunner operator and it worked.
Thank you.
Thanks! Just for summary: svc.Spec.ExternalIPs and svc.Spec.LoadBalancerIP together are superfluous. I'm leaning towards removing svc.Spec.ExternalIPs, @davidkornel wdyt? Do you remember why we added it in the first place?
I'm afraid both of them were added in order to support this feature on different cloud-providers' clusters (without realizing that they would collide and break). Other thing is that these fields are not that used and underspecified. Just reading their API documentation reveals that it is not advised to use these, since they are either not supported or the field itself is deprecated, etc. It seems that the svc.Spec.LoadBalancerIP works on GKE, however, it is most likely to not work on other platforms. TBH this quick 'removing that line' fix might have worked but in the long game, we should do some research on this and implement that annotation-based idea that I proposed in this issue, or something like that.
svc.Spec.LoadBalancerIP works on GKE. I tested it on my machine and is also recommended on Stack Overflow. And setting external IP breaks LoadBalancers in gke.
This has been addressed in l7mp/stunner-gateway-operator@d955a1b, at least for GKE. Feel free to reopen if bug regresses.