Cannot access the dashboard after a fresh install....

Question

Cannot access the dashboard after a fresh install....

natiki opened this issue 6 years ago · 7 comments

Hi,

I have a fresh Centos 7 install on which I have just run ./dind-cluster-v1.13.sh up. I did not set any other values and am using all the default values for networking.

All appears well:

[root@node01 dind-cluster]# kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
kube-master   Ready    master   23h   v1.13.0
kube-node-1   Ready    <none>   23h   v1.13.0
kube-node-2   Ready    <none>   23h   v1.13.0

[root@node01 dind-cluster]# kubectl config view
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: http://127.0.0.1:32769
  name: dind
contexts:
- context:
    cluster: dind
    user: ""
  name: dind
current-context: dind
kind: Config
preferences: {}
users: []
[root@node01 dind-cluster]# kubectl cluster-info
Kubernetes master is running at http://127.0.0.1:32769
KubeDNS is running at http://127.0.0.1:32769/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@node01 dind-cluster]#

and it appears healthy:

[root@node01 dind-cluster]# curl -w  '\n' http://127.0.0.1:32769/healthz
ok

I know the dashboard service is there:

[root@node01 dind-cluster]# kubectl get services kubernetes-dashboard -n kube-system
NAME                   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
kubernetes-dashboard   NodePort   10.102.82.8   <none>        80:31990/TCP   23h

however any attempt to access it is refused:

[root@node01 dind-cluster]# curl http://127.0.0.1:8080/api/v1/namespaces/kube-system/services/kubernetes-dashboard
curl: (7) Failed connect to 127.0.0.1:8080; Connection refused

[root@node01 dind-cluster]# curl http://127.0.0.1:8080/ui
curl: (7) Failed connect to 127.0.0.1:8080; Connection refused

I also see the following in the firewall log:

2019-02-05 19:45:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -p tcp -d 127.0.0.1 --dport 32769 -j DNAT --to-destination 10.192.0.2:8080 ! -i br-669b654fc9cd' failed: iptables: No chain/target/match by that name.

2019-02-05 19:45:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER ! -i br-669b654fc9cd -o br-669b654fc9cd -p tcp -d 10.192.0.2 --dport 8080 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).

2019-02-05 19:45:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -p tcp -s 10.192.0.2 -d 10.192.0.2 --dport 8080 -j MASQUERADE' failed: iptables: No chain/target/match by that name.

Any suggestions on how I actually access the dashboard?

Answer 1 · 2019-02-13T15:02:42.000Z

Please disable firewall on the host level.

Answer 2 · 2019-02-14T10:10:45.000Z

When I disable the firewall and attempt to run the install script it complains that it cannot create the necessary firewall rules. As you can see above some IPTables rules are created as part of the install.

Any other suggestions?

Answer 3 · 2019-03-16T15:30:52.000Z

I had same problem not being able to access dashboard at localhost:8080. My workaround is to use the server address of the dind cluster listed in kubectl config view . In your case, the dashboard should be accessible at http://127.0.0.1:32769/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/

Answer 4 · 2019-05-20T11:50:49.000Z

@jellonek #281 (comment)

Just tried again with no firewall and 1.14:

[root@node01 dind-cluster-1.14]# ./dind-cluster-v1.14.sh up
* Making sure DIND image is up to date
sha256:76da3588af69154942ed3590547aced9e42bc4e4d8486529305aaee702f41de0: Pulling from mirantis/kubeadm-dind-cluster
Digest: sha256:76da3588af69154942ed3590547aced9e42bc4e4d8486529305aaee702f41de0
Status: Image is up to date for mirantis/kubeadm-dind-cluster@sha256:76da3588af69154942ed3590547aced9e42bc4e4d8486529305aaee702f41de0
* Removing container: bbccc33beb79
bbccc33beb79
* Starting DIND container: kube-master
Error response from daemon: Failed to Setup IP tables: Unable to enable SKIP DNAT rule:  (iptables failed: iptables --wait -t nat -I DOCKER -i br-2fffb59cfc18 -j RETURN: iptables: No chain/target/match by that name.
 (exit status 1))
docker: Error response from daemon: network kubeadm-dind-net not found.
Error response from daemon: Container 359c94961afd717759069e0e6ca4ccf48d8ef7b3221c075620835d090864e7eb is not running
[root@node01 dind-cluster-1.14]#

Answer 5 · 2019-05-20T12:21:13.000Z

with firewall and 1.14 still produces heaps of errors:

2019-05-20 21:51:11 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:11 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION-STAGE-1 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:11 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION-STAGE-2 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:11 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:11 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i docker0 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:11 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:11 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:11 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i docker0 ! -o docker0 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER ! --dst 127.0.0.0/8' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -j DOCKER' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-ISOLATION-STAGE-1' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i docker0 -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -n -L DOCKER-USER' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-USER -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -j DOCKER-USER' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 10.192.0.0/24 ! -o br-0c83ead04470 -j MASQUERADE' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i br-0c83ead04470 -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i br-0c83ead04470 -o br-0c83ead04470 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i br-0c83ead04470 -o br-0c83ead04470 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i br-0c83ead04470 ! -o br-0c83ead04470 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o br-0c83ead04470 -j DOCKER' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:12 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o br-0c83ead04470 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -s 10.192.0.0/24 ! -o br-cdcd3904792f -j MASQUERADE' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -i br-cdcd3904792f -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -D FORWARD -i br-cdcd3904792f -o br-cdcd3904792f -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i br-cdcd3904792f -o br-cdcd3904792f -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -i br-cdcd3904792f ! -o br-cdcd3904792f -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o br-cdcd3904792f -j DOCKER' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C FORWARD -o br-cdcd3904792f -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION-STAGE-1 -i br-cdcd3904792f ! -o br-cdcd3904792f -j DOCKER-ISOLATION-STAGE-2' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:19 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION-STAGE-2 -o br-cdcd3904792f -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:22 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C DOCKER -p tcp -d 127.0.0.1 --dport 32769 -j DNAT --to-destination 10.192.0.2:8080 ! -i br-cdcd3904792f' failed: iptables: No chain/target/match by that name.
2019-05-20 21:51:22 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t filter -C DOCKER ! -i br-cdcd3904792f -o br-cdcd3904792f -p tcp -d 10.192.0.2 --dport 8080 -j ACCEPT' failed: iptables: Bad rule (does a matching rule exist in that chain?).
2019-05-20 21:51:22 WARNING: COMMAND_FAILED: '/usr/sbin/iptables -w2 -t nat -C POSTROUTING -p tcp -s 10.192.0.2 -d 10.192.0.2 --dport 8080 -j MASQUERADE' failed: iptables: No chain/target/match by that name.

So the IpTables things still appear to be broken? Following on from @redgetan suggestion #281 (comment) I used the following to attempt to connect (note the extra : before :/proxy not in @redgetan's original provided link:

* Access dashboard at: http://127.0.0.1:32769/api/v1/namespaces/kube-system/services/kubernetes-dashboard:/proxy
* Access dashboard at: http://127.0.0.1:32769/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy (if version>1.6 and HTTPS enabled)

So lynx http://127.0.0.1:32769/api/v1/namespaces/kube-system/services/kubernetes-dashboard:/proxy

however as the IpTables is broken I ultimately get a 503:

Answer 6 · 2019-07-28T12:00:29.000Z

Closing as this project is being deprecated and retired in favor of KinD.

Please see this issue for more details:
kubernetes/org#1016

/close

Answer 7 · 2019-07-28T12:00:31.000Z

@mrbobbytables: Closing this issue.

In response to this:

Closing as this project is being deprecated and retired in favor of KinD.

Please see this issue for more details:
kubernetes/org#1016

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.