TCP support for Deis applications
deis-admin opened this issue · 22 comments
From @Xe on July 14, 2014 17:18
This issue obsoletes #1061 and #18.
Also see #535.
Can an admin close #1061 and #18 and inform conversations to be directed here?
Copied from original issue: deis/deis#1315
From @carmstrong on July 14, 2014 17:39
I don't think #18 is closed by this - that's not an issue of protocol support, but rather building in the functionality of uploading SSL certs and provisioning them on the router.
From @bacongobbler on July 14, 2014 17:57
the functionality of uploading SSL certs and provisioning them on the router.
This will also be (finally) resolved through #1184, since installing SSL certs on the load balancer is the recommended way to accomplish this.
From @carmstrong on July 14, 2014 18:0
since installing SSL certs on the load balancer is the recommended way to accomplish this.
It is for now, but that cert is only used for accessing Deis itself. The idea here is that a user wants SSL support for their particular app - this isn't handled currently. They should be able to upload their own SSL cert for a particular app, which is configured in nginx.
From @bacongobbler on July 15, 2014 20:28
I think the topic for this issue is a bit too broad, as by definition "arbitrary" means that we should support any network protocol, making this bug impossible to close. Could you please change the title to reflect TCP protocol support instead, and create another issue for UDP support? I don't think we should close other bugs and funnel them into a much larger "meta" bug unless they're all related to a single specific issue with the platform (such as #1331).
From @gust1n on July 29, 2014 6:34
Is this depending on the switch of routing? Our move to deis depends on this change so I'm curious if there is something I can do to help?
From @bacongobbler on July 29, 2014 8:38
Yes, this is ultimately depending on TCP support from the router. We have TCP support for the router (holy mother of line lengths, btw) and we use it for the builder, but it needs to be hooked up for applications that require TCP support. In other words, we're unsure how to flesh this feature out. I know that for Heroku they enable WebSocket support for apps with
$ heroku labs:enable websockets -a myapp
So maybe a deis apps:enable tcp -a myapp would be the right way?
...I think a spec is in order.
From @gust1n on July 29, 2014 9:34
Hmm yeah that seams allright... I guess that is a broader decision for you guys to make since it's not only about tcp but udp, websockets and so on. Our main issue is that we only need the services reported as running without answering 200 on a HTTP request to the port.
From @gabrtv on July 30, 2014 0:45
This is something we want to do, but it's post our stable release. Way too many open questions -- and too few TCP-capable routers ;) Leaving open for discussion.
From @bacongobbler on October 9, 2014 18:25
note to self: investigate speccing this out in #1042.
From @nekinie on January 2, 2015 11:3
Is there any news on tcp support in the router?
From @bacongobbler on January 2, 2015 16:29
We need to spec it out, make a proposal and hash out the details on this feature request. We also need to figure out the UX before moving onto development so we're all on the same page.
From @gust1n on January 12, 2015 16:51
Maybe it's a stupid question but doesn't nginx handle TCP? http://nginx.com/resources/admin-guide/tcp-load-balancing/
From @Xe on January 12, 2015 18:35
Another big problem is that a lot of applications that use IP addresses to figure out who is connecting and if they are banned or not use the remote TCP address to figure this out. With any sort of TCP proxy this is broken.
From @bacongobbler on January 12, 2015 20:11
TCP support for applications will most likely be written up as an addon through #231, so it won't break our current application/proxy stack. How it will be architected is a separate question which we'd probably need to take a look at #231 first, but I'd imagine that it'd be a separate proxy layer outside of the core component stack.
From @chendo on February 4, 2015 22:44
+1 for this feature.
There's a TCP module for nginx (https://github.com/yaoweibin/nginx_tcp_proxy_module) but I haven't used it yet, but we're planning to investigate it as a replacement for haproxy in our Marathon/Mesos cluster.
From @carmstrong on February 4, 2015 22:46
There's a TCP module for nginx
We're actually already using this - https://github.com/deis/deis/blob/ce484e5524b8570e511c041b327bdc38a70ff76a/router/build.sh#L66
From @chendo on February 4, 2015 22:49
Oh whoops, never mind me then.
Is the strategy to update the config, then reload nginx? I was planning on using nginx-mruby to have to dynamically provide routing.
From @carmstrong on February 4, 2015 22:52
Is the strategy to update the config, then reload nginx?
Yep, that's how it happens currently - confd will re-template the nginx configuration, and then sighup the process to reload.
From @chendo on February 4, 2015 22:59
How often does this happen? We have issues where our script currently reloads every 15s, but our deploys can happen faster than that so we end up with stale backends which cause 503s
From @carmstrong on February 4, 2015 23:2
How often does this happen?
confd re-templates every 5 seconds: https://github.com/deis/deis/blob/ce484e5524b8570e511c041b327bdc38a70ff76a/router/image/confd.toml#L3
I'm going to close this. This cannot reasonably be accomplished with an Nginx-based router. I also think it's well understood at this point that the workaround for this is making the application in question non-routable, then manually giving it a service of Type: LoadBalancer. This will bypass the router entirely and use standard k8s mechanisms to route raw TCP traffic to the application's pod(s).