UI app does not load after exposing port after "yard expose"
c0debreaker opened this issue · 12 comments
c0debreaker commented
I'm at the very bottom of this tutorial - https://kubecon.demo.gs/01_index.html
I ran this command
yard expose --service-name svc/gateway-proxy-v2 --namespace gloo-system --port 9090:80
I ran netstat -na
# netstat -na|grep 'LISTEN ' |grep 9090
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN
I ran docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
15bc3ebeec58 nicholasjackson/consul-k8s-tools:0.5.5 "bash -c 'kubectl ge…" 24 minutes ago Up 24 minutes 0.0.0.0:9090->9090/tcp yard-shipyard-a7528967f35409d1c87f2bc8bc2e20e5
f4efab2ca620 nicholasjackson/consul-k8s-tools:0.5.5 "bash -c 'kubectl ge…" About an hour ago Up About an hour 0.0.0.0:16686->16686/tcp yard-shipyard-f519b20dd137a35cea099ab0dde33b47
dfc470cbb667 rancher/k3s:v0.9.1 "/bin/k3s server --h…" About an hour ago Up About an hour 0.0.0.0:64837->64837/tcp, 0.0.0.0:8500->30443/tcp, 0.0.0.0:8443->30444/tcp, 0.0.0.0:8200->30445/tcp k3d-shipyard-server
When I execute curl http://locahost:9090/ui, I get an empty response
curl: (52) Empty reply from server
I see the same results when I use my Chrome browser - https://i.imgur.com/vLSPZqs.png
Did I miss any step? I did this 3 times today, all results where identical. UI isn't reachable.
nicholasjackson commented
Hey @c0debreaker Just going to take a look now. From your output it looks like the port has been exposed in docker however. Maybe the Gloo API server has not come up.
I am just going to run through things myself but could you run a docker logs 15bc to see what the output from kubectl port_forward has been please.
c0debreaker commented
Here's the output of docker logs 15bc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 48m
consul-service NodePort 10.43.41.148 <none> 8500:30443/TCP 48m
vault-service NodePort 10.43.224.228 <none> 8200:30445/TCP 48m
consul-consul-server ClusterIP None <none> 8500/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP 48m
consul-consul-dns ClusterIP 10.43.195.79 <none> 53/TCP,53/UDP 48m
consul-consul-ui ClusterIP 10.43.70.197 <none> 80/TCP 48m
consul-consul-connect-injector-svc ClusterIP 10.43.51.234 <none> 443/TCP 48m
jaeger ClusterIP 10.43.194.239 <none> 5775/UDP,6831/UDP,6832/UDP,5778/TCP,16686/TCP,14268/TCP,9411/TCP 32m
web ClusterIP 10.43.253.195 <none> 9090/TCP 32m
NAME READY STATUS RESTARTS AGE
consul-consul-connect-injector-webhook-deployment-866c55c8m728v 1/1 Running 0 48m
consul-consul-server-0 1/1 Running 0 48m
consul-consul-h6ks5 1/1 Running 0 48m
currency-deployment-7c684cbfdb-7p8qd 2/2 Running 0 32m
web-deployment-6cd47964b9-ztwf5 2/2 Running 0 32m
api-deployment-v1-5c8d6d5f7c-wmb7x 2/2 Running 0 32m
jaeger-5694d7895d-xqshh 1/1 Running 0 32m
payment-deployment-blue-66cd4d7c76-ttgcm 2/2 Running 0 32m
Forwarding from 0.0.0.0:9090 -> 8080
Handling connection for 9090
E1128 06:47:59.546059 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:47:59 socat[22827] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:47:59.995502 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:47:59 socat[22828] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:48:04.289449 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:48:04 socat[22908] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:48:05.640821 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:48:05 socat[23076] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:48:28.151783 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:48:28 socat[24005] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:48:35.701533 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:48:35 socat[24388] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:48:40.776532 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:48:40 socat[24490] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:48:55.957462 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:48:55 socat[25172] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:48:57.588604 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:48:57 socat[25267] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:49:00.434292 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:49:00 socat[25341] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:49:04.756866 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:49:04 socat[25418] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:49:10.848322 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:49:10 socat[25736] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:49:12.449569 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:49:12 socat[25817] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:50:12.561991 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:50:12 socat[28322] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 06:54:53.798363 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 06:54:53 socat[7939] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:00:36.751210 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:00:36 socat[22592] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:00:38.626955 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:00:38 socat[22668] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:00:38.935376 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:00:38 socat[22669] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:00:39.081373 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:00:39 socat[22670] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:00:39.261363 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:00:39 socat[22672] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:00:39.402231 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:00:39 socat[22674] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:00:42.356724 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:00:42 socat[22820] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:01:40.949879 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:01:40 socat[25195] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:01:44.246473 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:01:44 socat[25374] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:05:42.511584 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:05:42 socat[2952] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:12:37.720266 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:12:37 socat[20752] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:12:41.237373 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:12:41 socat[20850] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:12:59.211793 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:12:59 socat[21616] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:13:03.711224 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:13:03 socat[21767] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:13:39.613717 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:13:39 socat[23331] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:13:46.102465 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:13:46 socat[23636] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
Handling connection for 9090
E1128 07:20:56.212489 21 portforward.go:400] an error occurred forwarding 9090 -> 8080: error forwarding port 8080 to pod 23dc619b9f8e6fad10c26656cb7b3fc8cd8b1a8a2c793c0b4e66d4677cc5be9d, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f634b769-da36-f944-08db-ada537755e6f": socat command returns error: exit status 1, stderr: "2019/11/28 07:20:56 socat[9281] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused\n"
c0debreaker commented
fyi, I re-executed glooctl add route --path-prefix / --dest-name default-web-9090 because I was thinking it failed last time. The output was
+-----------------+--------------+---------+------+----------+-----------------+--------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL | STATUS | LISTENERPLUGINS | ROUTES |
+-----------------+--------------+---------+------+----------+-----------------+--------------------------------+
| default | default | * | none | Accepted | | / -> |
| | | | | | | gloo-system.default-web-9090 |
| | | | | | | (upstream) |
| | | | | | | / -> |
| | | | | | | gloo-system.default-web-9090 |
| | | | | | | (upstream) |
+-----------------+--------------+---------+------+----------+-----------------+--------------------------------+
There's 2 rows now under ROUTES column. However, even if it was a single row before, it was already failing.
c0debreaker commented
Socat is trying to forward to a service running on TCP/8080. What service could this be?
nicholasjackson commented
Hey sorry was just running through those steps myself.
Everything looks ok, I added an additional route myself to check if that would cause a problem and it does not. I think something has gone wrong when with the kubectl port-forward which yard expose uses.
Could you try deleting the expose with:
yard expose --service-name svc/gateway-proxy-v2 --namespace gloo-system --port 9090:80 -d
Note the -d to delete exposed ports, the remainder of the command is the same.
And then re-add it:
yard expose --service-name svc/gateway-proxy-v2 --namespace gloo-system --port 9090:80
c0debreaker commented
It deleted the container then I reran the yard expose command. Output of docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3dda48dd3673 nicholasjackson/consul-k8s-tools:0.5.5 "bash -c 'kubectl ge…" About a minute ago Up About a minute 0.0.0.0:9090->9090/tcp yard-shipyard-a7528967f35409d1c87f2bc8bc2e20e5
f4efab2ca620 nicholasjackson/consul-k8s-tools:0.5.5 "bash -c 'kubectl ge…" 2 hours ago Up 2 hours 0.0.0.0:16686->16686/tcp yard-shipyard-f519b20dd137a35cea099ab0dde33b47
dfc470cbb667 rancher/k3s:v0.9.1 "/bin/k3s server --h…" 2 hours ago Up 2 hours 0.0.0.0:64837->64837/tcp, 0.0.0.0:8500->30443/tcp, 0.0.0.0:8443->30444/tcp, 0.0.0.0:8200->30445/tcp k3d-shipyard-server
Curl still failed
# curl http://localhost:9090/ui
curl: (52) Empty reply from server
nicholasjackson commented
Ok so it looks like it is not the yard expose command, I am thinking it is a setup issue with Gloo. We can test that the web service is directly accessible with the following command.
yard expose --service-name svc/web --port 9091:9090
and then:
curl localhost:9091
{
"name": "web",
"uri": "/",
"type": "HTTP",
"ip_addresses": [
"10.42.0.14"
],
"start_time": "2019-11-28T08:35:39.098497",
"end_time": "2019-11-28T08:35:39.163643",
"duration": "65.1463ms",
"body": "Hello World",
"upstream_calls": [
{
"name": "api-v1",
"uri": "http://localhost:9091",
"type": "HTTP",
"ip_addresses": [
"10.42.0.10"
],
"start_time": "2019-11-28T08:35:39.136774",
"end_time": "2019-11-28T08:35:39.161836",
"duration": "25.063ms",
"body": "Response from API v1",
"upstream_calls": [
{
"uri": "http://localhost:9091",
"code": 200
}
],
"code": 200
}
],
"code": 200
}
I have plans to improve the feedback loop in yard expose at the moment it just uses kubectl port-forward under the hood.
c0debreaker commented
Looks good on my side too
# curl localhost:9091
{
"name": "web",
"uri": "/",
"type": "HTTP",
"ip_addresses": [
"10.42.0.14"
],
"start_time": "2019-11-28T08:43:05.280294",
"end_time": "2019-11-28T08:43:05.359364",
"duration": "79.06959ms",
"body": "Hello World",
"upstream_calls": [
{
"name": "api-v1",
"uri": "http://localhost:9091",
"type": "HTTP",
"ip_addresses": [
"10.42.0.11"
],
"start_time": "2019-11-28T08:43:05.327085",
"end_time": "2019-11-28T08:43:05.346677",
"duration": "19.592371ms",
"body": "Response from API v1",
"upstream_calls": [
{
"uri": "http://localhost:9091",
"code": 200
}
],
"code": 200
}
],
"code": 200
}
Screenshot of it :) https://i.imgur.com/YiNEz70.png
c0debreaker commented
Thank you for staying this late. I'm going to bed since it's almost 3am here in Chicago. I'll catch you later. You should rest now too. :)
nicholasjackson commented
It looks like something has gone wrong with the Gloo installation. I am not 100% certain how to debug this. But could you try this please?
Delete Gloo
kubectl delete -f gloo-loop/gloo.yaml
Reinstall
kubectl apply -f gloo-loop/gloo.yaml
Wait for gloo to start
kubectl get pods --all-namespaces
Install the route
glooctl create vs default
+-----------------+--------------+---------+------+---------+-----------------+--------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL | STATUS | LISTENERPLUGINS | ROUTES |
+-----------------+--------------+---------+------+---------+-----------------+--------+
| default | default | * | none | Pending | | |
+-----------------+--------------+---------+------+---------+-----------------+--------+
glooctl add route --path-prefix / --dest-name default-web-9090
{"level":"info","ts":"2019-11-28T08:57:38.877Z","caller":"selectionutils/virtual_service.go:67","msg":"Selected virtual service with domain *","virtualService":"virtual_host:<domains:\"*\" > display_name:\"default\" status:<state:Accepted reported_by:\"gateway\" subresource_statuses:<key:\"*v1.Proxy.gloo-system.gateway-proxy-v2\" value:<state:Accepted reported_by:\"gloo\" > > > metadata:<name:\"default\" namespace:\"gloo-system\" resource_version:\"1338\" generation:2 > "}
+-----------------+--------------+---------+------+----------+-----------------+--------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL | STATUS | LISTENERPLUGINS | ROUTES |
+-----------------+--------------+---------+------+----------+-----------------+--------------------------------+
| default | default | * | none | Accepted | | / -> |
| | | | | | | gloo-system.default-web-9090 |
| | | | | | | (upstream) |
+-----------------+--------------+---------+------+----------+-----------------+--------------------------------+
Expose the service
yard expose --service-name svc/gateway-proxy-v2 --namespace gloo-system --port 9090:80
If it is easier tomorrow we can also jump on a Zoom call to try and debug what is going on. Thanks so much for you help.
c0debreaker commented
Ok, trying it one more time before heading to sleep :)
c0debreaker commented
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66f496764-jj7tv 1/1 Running 0 3h4m
kubernetes-dashboard dashboard-metrics-scraper-fb986f88d-cq5jn 1/1 Running 0 3h4m
local-path-storage local-path-provisioner-ccbdd96dc-h4mxj 1/1 Running 0 3h4m
default consul-consul-connect-injector-webhook-deployment-866c55c8m728v 1/1 Running 0 3h4m
kubernetes-dashboard kubernetes-dashboard-6d79989477-pgmlk 1/1 Running 0 3h4m
default consul-consul-server-0 1/1 Running 0 3h4m
default consul-consul-h6ks5 1/1 Running 0 3h4m
default currency-deployment-7c684cbfdb-7p8qd 2/2 Running 0 168m
default web-deployment-6cd47964b9-ztwf5 2/2 Running 0 168m
default api-deployment-v1-5c8d6d5f7c-wmb7x 2/2 Running 0 168m
default jaeger-5694d7895d-xqshh 1/1 Running 0 168m
default payment-deployment-blue-66cd4d7c76-ttgcm 2/2 Running 0 168m
gloo-system redis-5bbc7747dd-rkzmj 1/1 Running 0 16s
gloo-system svclb-gateway-proxy-v2-wzn2r 2/2 Running 0 17s
gloo-system gateway-proxy-v2-7bc7fcd6bb-pknk9 1/1 Running 0 16s
gloo-system extauth-86d884fc95-dk2mb 1/1 Running 0 16s
gloo-system discovery-5fc6c7dfbc-zmh5k 1/1 Running 0 16s
gloo-system rate-limit-cd84768fb-xtm6r 1/1 Running 1 16s
gloo-system gloo-7464b858c9-qj8s7 1/1 Running 0 16s
gloo-system gateway-certgen-pklxv 0/1 Completed 0 15s
gloo-system gateway-v2-b79ff6f74-dsjtq 1/1 Running 0 16s
gloo-system api-server-67d4686ff4-n699f 3/3 Running 0 15s
glooctl create vs default
+-----------------+--------------+---------+------+---------+-----------------+--------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL | STATUS | LISTENERPLUGINS | ROUTES |
+-----------------+--------------+---------+------+---------+-----------------+--------+
| default | default | * | none | Pending | | |
+-----------------+--------------+---------+------+---------+-----------------+--------+
glooctl add route --path-prefix / --dest-name default-web-9090
{"level":"info","ts":"2019-11-28T03:04:48.160-0600","caller":"selectionutils/virtual_service.go:67","msg":"Selected virtual service with domain *","virtualService":"virtual_host:<domains:\"*\" > display_name:\"default\" status:<state:Accepted reported_by:\"gateway\" subresource_statuses:<key:\"*v1.Proxy.gloo-system.gateway-proxy-v2\" value:<state:Accepted reported_by:\"gloo\" > > > metadata:<name:\"default\" namespace:\"gloo-system\" resource_version:\"11814\" generation:2 > "}
+-----------------+--------------+---------+------+----------+-----------------+--------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL | STATUS | LISTENERPLUGINS | ROUTES |
+-----------------+--------------+---------+------+----------+-----------------+--------------------------------+
| default | default | * | none | Accepted | | / -> |
| | | | | | | gloo-system.default-web-9090 |
| | | | | | | (upstream) |
+-----------------+--------------+---------+------+----------+-----------------+--------------------------------+
yard expose --service-name svc/gateway-proxy-v2 --namespace gloo-system --port 9090:80
_______. __ __ __ .______ ____ ____ ___ .______ _______
/ || | | | | | | _ \ \ \ / / / \ | _ \ | \
| (----`| |__| | | | | |_) | \ \/ / / ^ \ | |_) | | .--. |
\ \ | __ | | | | ___/ \_ _/ / /_\ \ | / | | | |
.----) | | | | | | | | | | | / _____ \ | |\ \----.| .--. |
|_______/ |__| |__| |__| | _| |__| /__/ \__\ | _| `._____||_______/
Version: 0.5.5
## Expose service: svc/gateway-proxy-v2 ports: 9090:80 using network null
netstat -tulpn |grep 9090
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 3713/docker-proxy
curl localhost:9090. It still failed
curl: (52) Empty reply from server
Yeah, Zoom call would be awesome tomorrow.