Make workflow ingress compatible

Question

Make workflow ingress compatible

rothgar opened this issue 8 years ago · 4 comments

With an on-prem installation I don't have external load balancers and rely on ingress controllers for service routing. The problems with the existing router solution are

Not all of my services will be deployed/managed with deis router so I need to use an ingress controller for those services
To load balance my ingress controller I run it as a daemonset on all workers which means the deis router is unable to deploy pods due to port conflicts (ports 80 and 443)
If I'm not using the deis router then workflow deployed services won't automatically be routed because they rely on service annotations instead of ingress rules

It may be possible to deploy the router on alternative ports which might satisfy needs temporarily but it would be nice if all services were routed via ingress rules in the api server instead of needing to look multiple places for hostname/service routing.

I'm attempting to make some of the rules needed but running into problems. Mainly the ingress controller shows no backend service IPs for the below ingress rule (I figure figured out why yet)

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: deis-controller
  namespace: deis
spec:
  rules:
  - host: deis-controller
    http:
      paths:
      - backend:
          serviceName: deis-controller
          servicePort: http

As far as I could find I would need to make ingress rules for the controller (80, 443), builder (2222), and workflow-manager (80). There are others that are exposed with router.deis.io/routable="true" such as deis-monitor-grafana, deis-monitor-influxapi, and deis-monitor-influxui but these don't see as necessary to have externally exposed.

Answer 1 · 2016-11-28T16:40:56.000Z

To give some brief history on why the router isn't already a proper ingress controller... at the time Deis v1.x was re-platformed on Kubernetes and re-branded as Deis Workflow, it is something we had considered, however, ingress was very new at the time (and to be fair, it's still in beta). We were reluctant to build any component of Workflow around an API that was still subject to change because of the mess that could make for us in the future, assuming we wish to honor semantic versioning (which we do) and not be forced into a major release just because some k8s API that we knew was likely to change did. Labeling services as routable: "true" and annotating them with any other information necessary seems like a safer alternative.

Fast forward to the present day... turning the router into a proper ingress controller is possibly the most requested feature we've had. There's still a certain reluctance on my part to endorse that idea because, again, ingress isn't out of beta (still). That being said, I understand your motivations.

Aside: The router doesn't need to be bound to host ports 80 and 443. That's there for the convenience of anyone wanting to roll-their-own external load balancer who wants a reliable set of host ports which they can assume router to be listening on on any host where it does run. If that's not how you want to do things (for whatever reasons), just remove that bit from the chart.

Back to the ingress controller discussion...

To turn the router into a full fledged ingress controller would be a breaking change, and afaik at this time, we're not willing to entertain breaking changes since we honor semver and are avoiding a major release. That being said, such changes don't need to be considered breaking changes if they're hidden behind a feature flag. i.e. If you want this new behavior, you opt-in.

So, what does that look like?

In my honest opinion, the router does too much stuff. It treads a very fine line between attempting to be a one-size-fits-all web server (which is really hard to do) and a highly-configurable web server for everyone and their brother who have approached us in the last 3+ years and requested router support xyz. At this point, router's gone far beyond what its name suggests that it does-- which is routes. (Ostensibly, that is also the responsibility of an ingress controller-- to just route inbound requests.)

IMO, the best path forward for the router is that Workflow users be given the opportunity to opt out of the router and opt in to some other ingress controller. (I like traefik, personally.) The most major barriers to this:

The controller would need to be modified such that when an operator opts into this, the controller manages ingress resources instead of routable services.
Some developers might need to role web server functionality / responsibilities into their applications since the router (or, technically, its replacement) would not handle quite as much stuff as it did previously. This is easier said than done on account of Workflow not currently having any mechanism for deploying "side car" containers (e.g. an Nginx container in the same pod as your application container).

This has been formally proposed in deis/workflow#584

My best recommendation is that you comment that you upvote or comment on that issue.

Answer 2 · 2016-11-28T17:30:13.000Z

Thanks, I understand the history why the router was used (OpenShift did a similar thing before ingress controllers).
I'm not sure if I was clear but I prefer the option of using an existing ingress controller rather than extending the router to be an ingress controller. I had used traefik for a while but wanted features from nginx (TLS termination) so I moved to that. If the controller supported creating the kubernetes ingress objects that would be the best solution IMO.
I had planned on moving the router to different host ports but the only way I found to do it was by dumping the existing router kubectl get svc deis-router -n deis -o yaml changing the ports and then re-submitting the saved yaml. Is there a supported way to do that during install?

I subscribed to deis/workflow#584 so if you'd like to close this as a duplicate I'm fine with watching updates and providing feedback there.

Answer 3 · 2016-11-28T17:35:15.000Z

I had used traefik for a while but wanted features from nginx (TLS termination) so I moved to that.

Traefik can terminate TLS. That might be new since you last looked at it.

If the controller supported creating the kubernetes ingress objects that would be the best solution IMO.

Yes... that's the exact idea behind deis/workflow#584, with the caveat that it's opt-in.

I had planned on moving the router to different host ports

Depending on how you're doing ingress, as I said, you don't even need to move these. You can simply remove them.

Is there a supported way to do that during install?

You'd have to fork the chart.

Answer 4 · 2016-11-28T20:46:32.000Z

I believe deis/workflow#584 sufficiently captures the most relevant parts of this discussion and is a more appropriate place for that discussion to continue since the proposed changes affect Workflow as a whole, but do not affect this component (router) directly.