Provide a way to disable the load balancer handling
sixcorners opened this issue · 4 comments
I want to use a different load balancer controller and I don't really want to annotate every ingress with something to prevent a DO load balancer from being created.
Could there be something that disables the creation of digital ocean load balancers?
I think gke supports adding this flag to the cluster creation CLI command to do this --disable-addons HttpLoadBalancing.
Hey @sixcorners
to make sure I don't misunderstand: are you talking L3/L4 load-balancing or L7 Ingress? The annotations you're referring to is what most Ingress controllers support, and you'd set them on Ingress manifests. DO LBs, on the other hand, are hooked up by marking a Service as type LoadBalancer.
(To be fair, DO LBs also support forwarding HTTP* requests, but the initial wiring still happens through the Service object.)
I'm not really sure how everything is wired together. All I really know is if I point gitlab at kubernetes and I tell it to create an ingress then a load balancer and ip address get created in digital ocean. :
I read that ingress controllers scan periodically for ingresses and start doing things when they see one.
That annotations can be used if there is more than one.
But gitlab doesn't put annotations on the ingress and I don't really need for there to be multiple ingress controllers.
So I kind of want to remove the one that comes with the kubernetes cluster.
But it's not done using an ingress controller right? It's done using the cloud specific cloud controller manager right? I don't want to turn that off because then other things will stop working I think.
Is this making sense? Am I off the mark?
So the thing I want to do is configure the manager or something to stop caring about ingresses or load balancers so I can use external-dns to route traffic right to the host where the app lives or to route traffic to an nginx instance created by an nginx ingress controller.. Not all that sure.
@sixcorners I unfortunately do not know how Gitlab works exactly with regards to the Kubernetes integration. The issue you're describing though does not map directly to what CCM does:
But it's not done using an ingress controller right? It's done using the cloud specific cloud controller manager right? I don't want to turn that off because then other things will stop working I think.
CCM does not provide an (L7) Ingress controller, only L3/L4 load-balancers controlled through Service objects. These are usually cloud-specific, which means you normally cannot choose between different kinds (though it's theoretically possible, I have never seen it) and thus there's no need to deal with concurrent usage scenarios.
Ingress is a different beast because there's one Ingress manifest definition but potentially multiple Ingress controllers that could all watch for updates and configure "their" routers (Nginx, HAProxy, Traefik, Google's HTTP Ingress, etc.). That's why those annotations are available to disable specific controllers. But that's just for L7 Ingress controllers which DO's load-balancer isn't (and therefore CCM does not care about).
You mention that you want to use external-dns to route traffic directly to your nodes, so you wouldn't even want an L3/L4 load-balancer (I suppose for cost reasons?). To me it sounds like what you need to do then is teach Gitlab to not create Service objects of type LoadBalancer (which would spawn an L3/L4 DO LB) but use NodePort so that requests can reach your cluster's nodes directly as they are discovered through external-dns. The switch is already there to disable DO LBs (the type Service field), and tweaking that should allow you to turn things on or off as needed.
I'll be closing this issue as the discussion seems to have shown that there isn't work to do in CCM for the requested feature. Let me know though if you think otherwise or feel like we should continue chatting, I'm always happy to reopen tickets.