kubernetes/ingress-nginx

Use GCE load balancer controller with backend buckets

Closed this issue · 12 comments

omerzach commented

We're happily using the GCE load balancer controller in production to route traffic to a few different services. We'd like to have some paths point at backend buckets in Google Cloud Storage instead of backend services running in Kubernetes.

Right now if we manually create this backend bucket and then configure the load balancer to point certain paths at it the UrlMap is updates appropriately but almost immediately reverted to its previous setting, presumably because the controller sees it doesn't match the YAML we initially configured the Ingress with.

I have two questions:

  1. Is there any immediate workaround where we could continue using the Kubernetes controller but manually modify the UrlMap to work with a backend bucket?
  2. Would a pull request to add support for backend buckets in the load balancer controller be something the community is interested in? We're willing to invest engineering effort into this, though none of our team has experience with Go or the Kubernetes codebase so we might need a bit of guidance.

(For some context, we'd like to do something like this: https://cloud.google.com/compute/docs/load-balancing/http/using-http-lb-with-cloud-storage)

bprashanth commented

I don't think there's an immediate work around, as the controller will construct a url map based on your ingress and sync it continuously.

Something that says: serve static content for these paths from a content cache backed by [S3, GCS, memory etc] sounds like a good idea. We should allow GCE L7 x in-memory cache, but for the first cut we might get away with a simple boolean on the HTTPIngressPath (https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/types.go#L685). We'd have to assume GCS and allocates a private bucket if the Ingress is a GCE lb.

Maybe we should fork into another issue that tackles how we enable CDN on Ingress, and figure out the api part first? @nicksardo @thockin

  • 👍7
ConradIrwin commented

@nicksardo thanks for picking this up!

I'd also like to be able to proxy certain paths to CloudStorage — let me know if you want a sounding board for design decisions.

bbzg commented

This would be very useful for us. Has there been any progress since March?

  • 👍2
thockin commented
ConradIrwin commented
thockin commented
ConradIrwin commented
gcbirzan commented

As a workaround for this issue, wouldn't not touching rules that kubernetes cannot create (i.e. those not pointing to a backend service, but to a bucket) be okay?

As a side note, we had this working on 1.6.x, but after upgrading it started removing the extra rules in the url map...

  • 👍6
c3s4r commented

Any updates on this? Is it scheduled? Is there a timeline? ... Since I want the content served using https, right now the only workaround I can think on is to manually create another load balancer (not using ingress) just for the static content, which I don't like because it adds the cost of an additional load balancer :(

lostpebble commented

Just coming across this now after finishing setting up backend buckets for our system...

This is a major set back for us trying to set up static file routes alongside our server backends. I agree with @gcbirzan that perhaps the load balancer should be updated for the values that Kubernetes can control rather than replaced wholly (and in the process removing GCP-specific rules).

Right now things feel too flaky to rely on backend buckets for static file serving, if when we update the configuration we might lose those pathways and return bad requests.

Huge pity because the CDN and load bearing capabilities the backend buckets could afford us is a major asset to our system.

  • 👍9
jakobholmelund commented

Any news on this ?

bowei commented

This issue was moved to kubernetes/ingress-gce#33