Minikube Ingress TCP Services Not Forwarding Traffic

Question

Minikube Ingress TCP Services Not Forwarding Traffic

justinrcs opened this issue 6 years ago · 32 comments

justinrcs commented 6 years ago

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version):
minikube version: v0.27.0

OS (e.g. from /etc/os-release):
MacOS 10.12.6 Sierra
VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
virtualbox
ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):

minikube-v0.26.0.iso

Install tools:
brew

What happened:
Enabled ingress on minikube via addons manager
Created tcp-services.yaml in the kube-system namespace to forward traffic to a RabbitMQ service in the default namespace.
Verified RabbitMQ is accessible inside the minikube cluster via ClusterIP service
Verified that the nginx ingress has loaded the tcp-services.yaml ConfigMap and has a stanza in the nginx.conf file.
When attempting to telnet from host to minikube on the ingress port, get Connection Refused

What you expected to happen:
To be able to connect to a TCP-based service via the Ingress

How to reproduce it (as minimally and precisely as possible):
minikube addons enable ingress
create any tcp-based pod and appropriate service (RabbitMQ, Postgres, MySQL, etc)
add the following configmap to kube-system, substituting port, namespace and service-name

kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: kube-system data: port: "namespace/service-name:port"

Verify the TCP Services are configured with the Ingress Controller:
kubectl exec $(kubectl get pod -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n kube-system -l app=nginx-ingress-controller) -n kube-system cat /etc/nginx/nginx.conf

Attempt to connect to the TCP service: telnet $(minikube ip) <port>

woodcockjosh commented 5 years ago

/assign

Answer 1 · 2018-05-29T19:01:50.000Z

Hi,

Can you supply the following?

your tcp-services.yaml file
output from minikube status
output from minikube service list
output from minikube addons list
output from kubectl get ingress,services -n=kube-system
output from kubectl exec $(kubectl get pod -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n kube-system -l app=nginx-ingress-controller) -n kube-system cat /etc/nginx/nginx.conf

Regards

Answer 2 · 2018-05-30T04:49:07.000Z

Requested info below, thank you.

tcp-services.yaml file

kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: kube-system
data:
  5672: "default/rabbitmq:5672"

output from minikube status

minikube: Running
cluster: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

output from minikube service list

|-------------|----------------------|--------------------------------|
|  NAMESPACE  |         NAME         |              URL               |
|-------------|----------------------|--------------------------------|
| default     | kubernetes           | No node port                   |
| default     | rabbitmq             | No node port                   |
| kube-system | default-http-backend | http://192.168.99.100:30001    |
| kube-system | kube-dns             | No node port                   |
| kube-system | kubernetes-dashboard | http://192.168.99.100:30000    |
| kube-system | tiller-deploy        | No node port                   |
|-------------|----------------------|--------------------------------|

output from minikube addons list

- addon-manager: enabled
- coredns: disabled
- dashboard: enabled
- default-storageclass: enabled
- efk: disabled
- freshpod: disabled
- heapster: disabled
- ingress: enabled
- kube-dns: enabled
- metrics-server: disabled
- registry: disabled
- registry-creds: enabled
- storage-provisioner: enabled

output from kubectl get ingress,services -n=kube-system

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
svc/default-http-backend   NodePort    10.102.229.59    <none>        80:30001/TCP    6d
svc/kube-dns               ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   6d
svc/kubernetes-dashboard   NodePort    10.103.255.166   <none>        80:30000/TCP    6d
svc/tiller-deploy          ClusterIP   10.96.254.176    <none>        44134/TCP       6d

output from kubectl exec $(kubectl get pod -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' -n kube-system -l app=nginx-ingress-controller) -n kube-system cat /etc/nginx/nginx.conf

printing the relevant part for the rabbit ingress:

stream {
	log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
	
	access_log /var/log/nginx/access.log log_stream;
	
	error_log  /var/log/nginx/error.log;
	
	# TCP services
	
	upstream tcp-5672-default-rabbitmq-5672 {
		
		server                  172.17.0.10:5672;
		
	}
	server {
		
		listen                  5672;
		
		listen                  [::]:5672;
		
		proxy_timeout           600s;
		proxy_pass              tcp-5672-default-rabbitmq-5672;
		
	}
	
	# UDP services
	
}

Answer 3 · 2018-05-30T07:17:25.000Z

Hi,

You don't have an ingress defined :) The ingress controller is effectively your nginx reverse proxy. You need to define an ingress now to make use of it.

Example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: dashboard-ingress-path
  namespace: kube-system
  annotations:
    nginx.org/server-snippet: "proxy_ssl_verify off;"
    nginx.ingress.kubernetes.io/rewrite-target: "/"
spec:
  rules:
  - host: your.domain.name
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 80
        path: /dashboard

Then, add an entry in your DNS (or local hosts file, etc) for your.domain.name to point to your minikube ip. You should now be able to browse to http://your.domain.name/dashboard/ from your Mac. You could also define a host ingress if you prefer.

Regards.

Answer 4 · 2018-05-30T15:47:57.000Z

Appreciate the response, but that's not how the TCP/UDP services work with the Ingress Controller. The Ingress sample you have above is for http (or https)-based services.

Assuming Minikube is using the same Ingress Controller via the AddOn as Kubernetes does, here is the documentation for exposing TCP/UDP:
Exposing TCP and UDP services

We have a similar setup in our cloud environments.

Thank you

Answer 5 · 2018-05-30T22:58:06.000Z

Hi,

Struggling with it too.

On the minikube VM, after setting up the ingress controller, I can:

$ telnet 172.17.0.5 5672
GET / HTTP/1.1
AMQP	Connection closed by foreign host

Where 172.17.0.5 is the ingress controller and 172.17.0.7 is rabbitmq - so I can also:

$ telnet 172.17.0.7 5672
GET / HTTP/1.1
AMQP	Connection closed by foreign host

So the ingress controller works but I can't access rabbitmq outside the minikube VM. I think this is a limitation of the ingress in K8s. As the doc states, "Ingress does not support TCP or UDP services". The Nginx Ingress controller offers the work around based on ConfigMap but this has its limitations, I suspect. It can expose the port onto the Node but as this isn't a K8s Ingress, you cannot access the port outside the minikube VM - it isn't proxied.

The Helm package suggests these instructions (it didn't seem to work for me though):
Create a proxy server between localhost and the Kubernetes API Server:
kubectl proxy --port 8002

To Access the RabbitMQ AMQP port:

URL : amqp://127.0.0.1:8002/api/v1/proxy/namespaces/default/services/rabbitmq:5672/

To Access the RabbitMQ Management interface:

URL : http://127.0.0.1:8002/api/v1/proxy/namespaces/default/services/rabbitmq:15672/

Some other ideas (long shots):

You might be able to install nginx to proxy amqp between the minikube vm and the nginx ingress controller
Perhaps nc can do that too
Find another lightweight utility that can or write your own.
kubectl port-forward $POD_NAME 15672:15672

Please share here if you find a workaround / solution.

Best regards.

Answer 6 · 2018-07-15T01:59:43.000Z

Same issue here. If I connect to the nginx-ingress container I can confirm I can reach my container's tcp port through nginx, but it does not get exposed through minikube.

https://github.com/lrvick/weechat-helm/blob/master/templates/ingress.yaml

Answer 7 · 2018-07-27T18:40:43.000Z

@juan131 minikube only expose port 80,443,18080 in ingress pod, so you should create another ingress yourself, then forward 5672 to your rabbitmq.

Answer 8 · 2018-12-18T05:34:21.000Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Answer 9 · 2018-12-18T14:04:33.000Z

Are there any plans to change the current behaviour and let user easily expose additional ports via builtin nginx ingress controller?

And it would be really helpful to have a link to ome ressources describing neccessary steps to deploy a new ingress controller to minikube to archive this feature requested here.

Answer 10 · 2019-01-02T09:07:11.000Z

I compare the ingress controller between minikube and the k8s on server.

Find out minikube's ingress without the configuration as below:

spec.template.spec.hostNetwork = true

After I add it, port can be able to forwarded to the vm host.

Hope this can help.

Answer 11 · 2019-01-24T16:34:59.000Z

I don't know anything about Ingress yet, but would be happy to assist someone with getting this experience improved for all users. Help wanted!

Answer 12 · 2019-01-25T14:33:40.000Z

I am running into this same problem, where TCP services are not exposed on the IP the Minikube is running on. Ingress for HTTP works as expected, but not tcp-services defined as ConfigMap. My nginx config contains the relevant tcp mapping. I have also tried applying the 'hostNetwork = true' setting to the pod spec, but no success.

Answer 13 · 2019-02-27T14:07:01.000Z

Still not solved it seems. However, I have the same issue. Minikube's nginx ingress-controller does not do the job properly.

Please help.

Answer 14 · 2019-04-09T13:13:34.000Z

Update: The link to manifest file is fixed (was broken due to changes made in nginx repo)

Thanks to @MerlinPong I was able to make minikube forward TCP/UDP traffic. Here are the steps:

Disable default ingress addon
```
minikube addons disable ingress
```

Deploy standalone nginx ingress controller:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

Here's a link to the official documentation.

Tweak the deployment of nginx-ingress-controller.
Make sure it's running:

$ kubectl get deploy nginx-ingress-controller -n ingress-nginx

$ NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
  nginx-ingress-controller   1/1     1            1           8m39s

Save the output to a file:

kubectl get deploy nginx-ingress-controller -n ingress-nginx -o yaml>> nginx-ingress-controller.yaml

Edit the file by applying spec.template.spec.hostNetwork = true and apply the configuration.

Use the example provided in the docs -> https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

I also wrote more detailed explanation in the article (but steps are exactly the same as here) -> Exposing TCP and UDP Services via ingress on Minikube

Answer 15 · 2019-04-10T08:34:06.000Z

@skryvets @MerlinPong thank you, working as expected :)

@skryvets At the step Save the edited file and apply it by executing I have to use flag --force:

 kubectl apply -f nginx-ingress-controller.yaml  --force

Answer 16 · 2019-05-15T00:05:25.000Z

This worked for me:

kubectl patch deployment nginx-ingress-controller -n kube-system --patch '{"spec":{"template":{"spec":{"hostNetwork":true}}}}'

kubectl patch configmap tcp-services -n kube-system --patch '{"data":{"3306":"default/mysql-service:3306"}}'

The main issue is that Ingress controller doesn't support tcp services. I really hope to see this supported soon. I have no idea why they ports are limited for Ingress.

This is based on the nginx hackaround https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

Good luck!

Answer 17 · 2019-09-27T13:59:06.000Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Answer 18 · 2019-10-01T16:22:00.000Z

Relates to kubernetes/ingress-nginx#4624

Answer 19 · 2019-10-01T17:53:52.000Z

Relates to
kubernetes/ingress-nginx#4586
kubernetes/ingress-nginx#4585

Answer 20 · 2019-10-01T18:56:54.000Z

Relates to kubernetes/ingress-nginx#4626

Answer 21 · 2019-10-01T19:04:51.000Z

Resolved by #5517

Answer 22 · 2019-10-01T19:38:18.000Z

Relates to #5518

Answer 23 · 2019-10-01T20:19:44.000Z

The only other thing I think we could do here other than a tutorial is add a custom patching container that watches changes on the tcp and udp configmaps and updates the ingress deployment so that the pods listen on the appropriate ports. I will leave it up to the project maintainers to determine whether or not this issue can be closed without that capability.

Answer 24 · 2019-10-02T12:32:24.000Z

I don't know if could help or not.
After NGINX I tried Traefik and it does not work as well. But I don't know how to fix it...

Answer 25 · 2019-10-02T13:06:12.000Z

FYI https://minikube.sigs.k8s.io/docs/tutorials/nginx_tcp_udp_ingress/

Answer 26 · 2019-10-02T13:09:07.000Z

@woodcockjosh thanks a lot! NGINX ingress controller works perfectly right now.
I haven't found a fix for Traefik. Do you have an advice?

Answer 27 · 2019-10-02T13:15:55.000Z

@bygui86 https://github.com/woodcockjosh/traefik/blob/add-minikube-config-to-crd-acme-user-guide/docs/content/user-guides/crd-acme/index.md
Example files are in same directory as index.md.

Answer 28 · 2019-10-03T08:16:08.000Z

Thanks @woodcockjosh I will try asap and let you know!!

Answer 29 · 2019-10-14T16:12:40.000Z

@woodcockjosh mentioned that this should be closed now. Please /reopen if that is not the case. Thanks!

Answer 30 · 2020-06-29T19:27:50.000Z

Hi,

I am kind of stuck in one of the issues. I am learning and developing the node.js web app using microservices. I have one section where I have to route the request from one default namespace to ingress-nginx namespace but I can't find the ingress-nginx namespace in the Kubernetes cluster.
I am using minikube on windows 7 enterprise.
Please help me where I can find the ingress-nginx load balancer. Is there anything specific we need to don in minikube. I can see we have ingress-nginx in mac and docker windows app. but with minikube I can't find any services related to Loadbalancer ingress-nginx.
In fact, I found one in Kube-system, but it is cluster Ip not load balancer type and when I tried it, it didn't work.

Thanks in advance.

Answer 31 · 2020-06-29T20:45:59.000Z

@Aadil-Nabi I suggest you join the kubernetes slack channel for help on this. kubernetes.slack.com