pires/kubernetes-elasticsearch-cluster

can connect to elasticsearch through kubectl proxy, but not through kubectl port-forward elasticsearch-client 9200

Closed this issue · 11 comments

bw2 commented

I'm probably misunderstanding something, but I've deployed a cluster as follows:

2017-09-11 12:19:51,881 INFO     ==> kubectl get services
2017-09-11 12:19:52,219 INFO     NAME                      CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
2017-09-11 12:19:52,219 INFO     cockpit                   10.51.250.119   <none>        443/TCP          19d
2017-09-11 12:19:52,220 INFO     elasticsearch             10.51.253.89    <nodes>       9200:30001/TCP   19d
2017-09-11 12:19:52,220 INFO     elasticsearch-data        None            <none>        9300/TCP         19d
2017-09-11 12:19:52,220 INFO     elasticsearch-discovery   10.51.244.225   <none>        9300/TCP         19d
2017-09-11 12:19:52,220 INFO     kibana                    10.51.243.4     <nodes>       80:30002/TCP     19d
2017-09-11 12:19:52,220 INFO     kubernetes                10.51.240.1     <none>        443/TCP          19d
2017-09-11 12:19:52,222 INFO     ==> kubectl get pods
2017-09-11 12:19:52,676 INFO     NAME                         READY     STATUS    RESTARTS   AGE
2017-09-11 12:19:52,677 INFO     cockpit-hs4nv                1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-client-624501573-0zr2p    1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-client-624501573-k4k48    1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-data-0                    1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-data-1                    1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-data-2                    1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-data-3                    1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-data-4                    1/1       Running   1          19d
2017-09-11 12:19:52,677 INFO     es-data-5                    1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-data-6                    1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-data-7                    1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-master-2824812386-gl8b6   1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-master-2824812386-j6kg7   1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     es-master-2824812386-tj2gf   1/1       Running   0          19d
2017-09-11 12:19:52,677 INFO     kibana-2404200931-gkv7j      1/1       Running   0          19d

If I run kubectl proxy and go to
http://localhost:8001/api/v1/namespaces/default/services/elasticsearch:9200/proxy/
I get the standard json response.

but if I run kubectl port-forward es-client-624501573-0zr2p 9200 (or 30001) and go to
http://localhost:9200/ or http://localhost:30001/
I get:
image

pires commented

kubectl port-forward es-client-624501573-0zr2p 9200:9200.

timbrandin commented

@pires I can't get your command to work, it sets up the port-forward, but it spits out (when going to localhost:9200 in the browser):

$ kubectl port-forward elastic-elasticsearch-client-3854575734-3mnmt 9200:9200
Forwarding from 127.0.0.1:9200 -> 9200
Forwarding from [::1]:9200 -> 9200
Handling connection for 9200
E0929 11:51:43.049786   13035 portforward.go:331] an error occurred forwarding 9200 -> 9200: error forwarding port 9200 to pod b1e6c1ef521011672611b7a48bc8634898bd81f145ed4f7cbd0221b677090b48, uid : exit status 1: 2017/09/29 09:51:43 socat[14749] E connect(5, AF=2 127.0.0.1:9200, 16): Connection refused
  • 👍1
camechis commented

I am having the same issue. I am wondering if its something with the ES network binding where in the case of port forwarding it thinks its coming from localhost and ES isn't bound to that ? I noticed if I also exec into the container I cannot use localhost. I have to use the 10.x.x.x ip.

PanJ commented

@camechis @timbrandin I managed to solve the issue by setting NETWORK_HOST environment variable to _lo_,_eth0_.

Note that this setting may be varied due to your network interfaces (for me it's GKE).

  • 👍5
camechis commented
erkolson commented

I just ran into this today on AWS. If you do not specify a value, the container gets this by default NETWORK_HOST=_site_. Apparently the port-forward connects on the loopback interface, so this was required to get the pod to communicate with other pods (_site_) as well as permit port forwarding (_lo_):

- name: "NETWORK_HOST"
  value: "_site_,_lo_"
  • 👍1
bw2 commented

Thank you @PanJ and @erkolson !
Adding

env:
- name: "NETWORK_HOST"
  value: "_site_,_lo_"

to es-client.yaml fixed this for me too.

viet-wego commented

Thank @PanJ and @erkolson that works for me!

naarani commented

kubectl version >= 1.10 allow port-forwarding via kubectl port-forward svc/elasticsearch 9200:9200
where the elasticsearch name is the service name, so I don't point to the POD but to the service, but the problem is still replicated here: connection refused

  • 👍1
pawarmanoj commented

On GKE, this works for me -

$ kubectl port-forward service/kibana 5601 -n elastic-cluster
Forwarding from 127.0.0.1:5601 -> 5601
Forwarding from [::1]:5601 -> 5601
Handling connection for 5601
vinu commented

kubectl version >= 1.10 allow port-forwarding via kubectl port-forward svc/elasticsearch 9200:9200
where the elasticsearch name is the service name, so I don't point to the POD but to the service, but the problem is still replicated here: connection refused

@naarani just add
- name: NETWORK_HOST
value: "site,local"

into
es-data.yaml env