kubeapi ssl issues
jchauncey opened this issue · 47 comments
jchauncey commented
stock kubesolo config has the secure port bound to 127.0.0.1 -
core@k8smaster-01 ~ $ netstat -alnp | grep LISTEN
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 172.17.15.101:7001 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:6443 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN -
tcp 0 0 172.17.15.101:2380 0.0.0.0:* LISTEN -
tcp6 0 0 :::3000 :::* LISTEN -
tcp6 0 0 :::4001 :::* LISTEN -
tcp6 0 0 :::2379 :::* LISTEN -
tcp6 0 0 :::5355 :::* LISTEN -
tcp6 0 0 :::8080 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
unix 2 [ ACC ] STREAM LISTENING 16895 - /var/run/early-docker.sock
unix 2 [ ACC ] STREAM LISTENING 11290 - /run/systemd/private
unix 2 [ ACC ] STREAM LISTENING 17451 - /var/run/fleet.sock
unix 2 [ ACC ] SEQPACKET LISTENING 11568 - /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 19222 - /var/lib/docker/network/files/953723e5ad7e585703752f2fb2f5df58e14f576447539cf794aedb7076542168.sock
unix 2 [ ACC ] STREAM LISTENING 7585 - /run/systemd/journal/stdout
unix 2 [ ACC ] STREAM LISTENING 13733 - /run/rkt/metadata-svc.sock
unix 2 [ ACC ] STREAM LISTENING 17648 - /var/lib/docker/network/files/ce8ae5075bf93b68725b63cc0dac44f221064b59e622b049b5082043a2395b22.sock
unix 2 [ ACC ] STREAM LISTENING 13742 - /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 13752 - /var/run/docker.sock
so I changed where the secure port is bound to -
core@k8smaster-01 ~ $ netstat -alnp | grep LISTEN
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 172.17.15.101:7001 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN -
tcp 0 0 172.17.15.101:2380 0.0.0.0:* LISTEN -
tcp6 0 0 :::3000 :::* LISTEN -
tcp6 0 0 :::4001 :::* LISTEN -
tcp6 0 0 :::6443 :::* LISTEN -
tcp6 0 0 :::2379 :::* LISTEN -
tcp6 0 0 :::5355 :::* LISTEN -
tcp6 0 0 :::8080 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
unix 2 [ ACC ] STREAM LISTENING 16895 - /var/run/early-docker.sock
unix 2 [ ACC ] STREAM LISTENING 11290 - /run/systemd/private
unix 2 [ ACC ] STREAM LISTENING 17451 - /var/run/fleet.sock
unix 2 [ ACC ] SEQPACKET LISTENING 11568 - /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 19222 - /var/lib/docker/network/files/953723e5ad7e585703752f2fb2f5df58e14f576447539cf794aedb7076542168.sock
unix 2 [ ACC ] STREAM LISTENING 7585 - /run/systemd/journal/stdout
unix 2 [ ACC ] STREAM LISTENING 13733 - /run/rkt/metadata-svc.sock
unix 2 [ ACC ] STREAM LISTENING 17648 - /var/lib/docker/network/files/ce8ae5075bf93b68725b63cc0dac44f221064b59e622b049b5082043a2395b22.sock
unix 2 [ ACC ] STREAM LISTENING 13742 - /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 13752 - /var/run/docker.sockcurling localhost:8080 seems to work.
core@k8smaster-01 ~ $ curl http://localhost:8080/api/v1/namespaces/deis/replicationcontrollers/deis-router
curling localhost:6443 does not
core@k8smaster-01 ~ $ curl -v http://localhost:6443/api/v1/namespaces/deis/replicationcontrollers/deis-router
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 6443 (#0)
> GET /api/v1/namespaces/deis/replicationcontrollers/deis-router HTTP/1.1
> Host: localhost:6443
> User-Agent: curl/7.43.0
> Accept: */*
>
* Connection #0 to host localhost left intact
curl -k -v https://127.0.0.1:6443/api/v1/namespaces/deis/replicationcontrollers/deis-router
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 6443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-SHA
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=kubernetes-master
* start date: 2015-12-22 19:37:04 GMT
* expire date: 2025-12-19 19:37:04 GMT
* issuer: CN=172.17.15.101@1450813023
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /api/v1/namespaces/deis/replicationcontrollers/deis-router HTTP/1.1
> Host: 127.0.0.1:6443
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: text/plain; charset=utf-8
< Date: Tue, 22 Dec 2015 20:26:16 GMT
< Content-Length: 13
<
Unauthorized
* Connection #0 to host 127.0.0.1 left intactjchauncey commented
Seems like we're missing token_auth_file
jchauncey commented
from within a container -
bash-4.3# curl http://172.17.15.101:8080/api
{
"kind": "APIVersions",
"versions": [
"v1"
]
}bash-4.3# curl 10.244.62.0:8080/api
{
"kind": "APIVersions",
"versions": [
"v1"
]
}bash-4.3# curl 10.244.62.0:8080/api
{
"kind": "APIVersions",
"versions": [
"v1"
]
bash-4.3# curl http://10.100.0.1:8080/api
^C
jchauncey commented
╰─○ kg svc --namespace=default
NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGE
kubernetes 10.100.0.1 <none> 443/TCP <none> 15m
so the kubernetes service lives at that ip but its not routable from within a container or on a host. but im not sure how that ip gets routable from within the container (im guessing flannel but the network interfaces dont line up)
core@k8smaster-01 ~ $ ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.62.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::42:eeff:fe3d:a4ba prefixlen 64 scopeid 0x20<link>
ether 02:42:ee:3d:a4:ba txqueuelen 0 (Ethernet)
RX packets 9 bytes 612 (612.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 18 bytes 1448 (1.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::a00:27ff:fe68:4567 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:68:45:67 txqueuelen 1000 (Ethernet)
RX packets 37295 bytes 47475341 (45.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 7040 bytes 596033 (582.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.15.101 netmask 255.255.255.0 broadcast 172.17.15.255
inet6 fe80::a00:27ff:fef4:844a prefixlen 64 scopeid 0x20<link>
ether 08:00:27:f4:84:4a txqueuelen 1000 (Ethernet)
RX packets 98752 bytes 19356531 (18.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 73392 bytes 74904617 (71.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.244.62.0 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::2484:95ff:fe68:16c9 prefixlen 64 scopeid 0x20<link>
ether 26:84:95:68:16:c9 txqueuelen 0 (Ethernet)
RX packets 12 bytes 806 (806.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 11 bytes 846 (846.0 B)
TX errors 0 dropped 18 overruns 0 carrier 0 collisions 0
the k8s api is listening on all interfaces
core@k8smaster-01 ~ $ sudo netstat -alnp | grep LISTEN
tcp 0 0 172.17.15.101:7001 0.0.0.0:* LISTEN 1089/etcd2
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN 1744/kube-scheduler
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 678/systemd-resolve
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 1727/kube-controlle
tcp 0 0 172.17.15.101:2380 0.0.0.0:* LISTEN 1089/etcd2
tcp6 0 0 :::3000 :::* LISTEN 1601/docker-proxy
tcp6 0 0 :::4001 :::* LISTEN 1089/etcd2
tcp6 0 0 :::6443 :::* LISTEN 1738/kube-apiserver
tcp6 0 0 :::2379 :::* LISTEN 1089/etcd2
tcp6 0 0 :::5355 :::* LISTEN 678/systemd-resolve
tcp6 0 0 :::8080 :::* LISTEN 1738/kube-apiserver
tcp6 0 0 :::22 :::* LISTEN 1/systemd
unix 2 [ ACC ] STREAM LISTENING 11302 1/systemd /run/systemd/private
unix 2 [ ACC ] SEQPACKET LISTENING 11320 1/systemd /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 19232 1272/docker /var/lib/docker/network/files/d14b6939fb337b80979eb98c97a8df0a158e1d1b14f7c1cc1bac8e733841c29e.sock
unix 2 [ ACC ] STREAM LISTENING 17645 1090/docker /var/lib/docker/network/files/691bf6b80670368573d8efe79f619b30cee2fb64e839e8355e8c451f5f45df11.sock
unix 2 [ ACC ] STREAM LISTENING 7595 1/systemd /run/systemd/journal/stdout
unix 2 [ ACC ] STREAM LISTENING 13744 1/systemd /var/run/dbus/system_bus_socket
unix 2 [ ACC ] STREAM LISTENING 13759 1/systemd /run/rkt/metadata-svc.sock
unix 2 [ ACC ] STREAM LISTENING 13768 1/systemd /var/run/docker.sock
unix 2 [ ACC ] STREAM LISTENING 16880 1/systemd /var/run/early-docker.sock
unix 2 [ ACC ] STREAM LISTENING 17404 1/systemd /var/run/fleet.sock
rimusz commented
will investigate it more
jchauncey commented
So the kubeapi via the service IP is completely unreachable from a host or a contianer/pod. However the kubeapi is reachable when using hte host ip its running on (either the public facing ip or the private).
If we can figure out why that is I think we can solve this issue
jchauncey commented
This issue also describes the error that I am seeing in the kube-proxy logs. - kubernetes/kubernetes#15676
rimusz commented
right, I see we need --configure-cbr0=true for kube-proxy
jchauncey commented
yeah i think thats the fix (at least it seems like it might work). Think you can get this pushed out?
rimusz commented
sure, I will in the next hour.
In the mean time you can test it adding it to kube-proxy fleet unit
jchauncey commented
wait why there?
seems like this should be added to the docker config
rimusz commented
sorry it is the kubelet fleet unit then as per kubernetes/kubernetes#15676 - the problem is kubelet is not started with --configure-cbr0=true
rimusz commented
hmm does not look it is working:
fleetctl status kube-kubelet.service
● kube-kubelet.service - Kubernetes Kubelet
Loaded: loaded (/run/fleet/units/kube-kubelet.service; linked-runtime; vendor preset: disabled)
Active: active (running) since Mon 2015-12-28 17:59:04 UTC; 2min 1s ago
Docs: https://github.com/GoogleCloudPlatform/kubernetes
Main PID: 23784 (kubelet)
Memory: 4.1M
CPU: 1.903s
CGroup: /system.slice/kube-kubelet.service
├─23784 /opt/bin/kubelet --address=0.0.0.0 --port=10250 --hostname_override=192.168.64.2 --register-node=true --container_runtime=docker --api_servers=http://127.0.0.1:8080 --allow_privileged=true --cluster_dns=10.100.0.10 --cluster_domain=cluster.local --configure-cbr0=true --logtostderr=true --cadvisor_port=4194 --healthz_bind_address=0.0.0.0 --healthz_port=10248
└─23835 journalctl -k -f
Dec 28 18:00:22 k8solo-01 kubelet[23784]: I1228 18:00:22.759654 23784 kubelet.go:1971] Skipping pod synchronization, network is not configured
Dec 28 18:00:27 k8solo-01 kubelet[23784]: I1228 18:00:27.765227 23784 kubelet.go:1971] Skipping pod synchronization, network is not configured
Dec 28 18:00:32 k8solo-01 kubelet[23784]: I1228 18:00:32.766949 23784 kubelet.go:1971] Skipping pod synchronization, network is not configured
Dec 28 18:00:36 k8solo-01 kubelet[23784]: W1228 18:00:36.927162 23784 kubelet.go:2320] ConfigureCBR0 requested, but PodCIDR not set. Will not configure CBR0 right now
Dec 28 18:00:37 k8solo-01 kubelet[23784]: I1228 18:00:37.767639 23784 kubelet.go:1971] Skipping pod synchronization, network is not configured
Dec 28 18:00:42 k8solo-01 kubelet[23784]: I1228 18:00:42.768202 23784 kubelet.go:1971] Skipping pod synchronization, network is not configured
jchauncey commented
i think we need this too Environment='DOCKER_OPTS=--bridge=cbr0 --iptables=false --ip-masq=false'
rimusz commented
yep, looks like it
rimusz commented
hmm docker does not like it - Environment='DOCKER_OPTS=--bridge=cbr0 --iptables=false --ip-masq=false'
Dec 28 18:35:00 k8solo-01 systemd[1]: Started Docker Application Container Engine.
Dec 28 18:35:00 k8solo-01 dockerd[1504]: time="2015-12-28T18:35:00.786339237Z" level=fatal msg="Error starting daemon: You specified -b & --bip, mutually exclusive options.
Dec 28 18:35:00 k8solo-01 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Dec 28 18:35:00 k8solo-01 systemd[1]: docker.service: Unit entered failed state.
Dec 28 18:35:00 k8solo-01 systemd[1]: docker.service: Failed with result 'exit-code'.
jchauncey commented
i got the following -
-- Logs begin at Mon 2015-12-28 18:22:16 UTC, end at Mon 2015-12-28 18:44:59 UTC. --
Dec 28 18:22:28 k8smaster-01 systemd[1]: [/etc/systemd/system/docker.service.d/50-insecure-registry.conf:5] Invalid environment assignment, ignoring: DOCKER_OPTS='--insecure-regis
Dec 28 18:22:28 k8smaster-01 systemd[1]: [/etc/systemd/system/docker.service.d/50-insecure-registry.conf:5] Invalid environment assignment, ignoring: DOCKER_OPTS='--insecure-regis
Dec 28 18:22:28 k8smaster-01 systemd[1]: [/etc/systemd/system/docker.service.d/50-insecure-registry.conf:5] Invalid environment assignment, ignoring: DOCKER_OPTS='--insecure-regis
Dec 28 18:22:47 k8smaster-01 systemd[1]: Started Docker Application Container Engine.
Dec 28 18:22:47 k8smaster-01 dockerd[1272]: time="2015-12-28T18:22:47.573875747Z" level=info msg="Firewalld running: false"
Dec 28 18:22:47 k8smaster-01 dockerd[1272]: time="2015-12-28T18:22:47.792736195Z" level=info msg="Loading containers: start."
Dec 28 18:22:47 k8smaster-01 dockerd[1272]: time="2015-12-28T18:22:47.793934430Z" level=info msg="Loading containers: done."
Dec 28 18:22:47 k8smaster-01 dockerd[1272]: time="2015-12-28T18:22:47.794189116Z" level=info msg="Daemon has completed initialization"
Dec 28 18:22:47 k8smaster-01 dockerd[1272]: time="2015-12-28T18:22:47.794417185Z" level=info msg="Docker daemon" commit=4419fdb-dirty execdriver=native-0.2 graphdriver=overlay ver
Dec 28 18:22:47 k8smaster-01 dockerd[1272]: time="2015-12-28T18:22:47.803280414Z" level=info msg="API listen on /var/run/docker.sock"
Dec 28 18:23:28 k8smaster-01 systemd[1]: [/etc/systemd/system/docker.service.d/50-insecure-registry.conf:5] Invalid environment assignment, ignoring: DOCKER_OPTS='--insecure-regis
Dec 28 18:23:28 k8smaster-01 systemd[1]: [/etc/systemd/system/docker.service.d/50-insecure-registry.conf:5] Invalid environment assignment, ignoring: DOCKER_OPTS='--insecure-regis
Dec 28 18:23:28 k8smaster-01 systemd[1]: [/etc/systemd/system/docker.service.d/50-insecure-registry.conf:5] Invalid environment assignment, ignoring: DOCKER_OPTS='--insecure-regis
Dec 28 18:23:28 k8smaster-01 dockerd[1272]: time="2015-12-28T18:23:28.829584688Z" level=info msg="POST /v1.21/images/create?fromImage=purpleworks%2Ffleet-ui%3Alatest"
Dec 28 18:23:40 k8smaster-01 dockerd[1272]: time="2015-12-28T18:23:40.493130402Z" level=info msg="DELETE /v1.21/containers/fleet_ui"
Dec 28 18:23:40 k8smaster-01 dockerd[1272]: time="2015-12-28T18:23:40.493337603Z" level=error msg="Handler for DELETE /v1.21/containers/fleet_ui returned error: no such id: fleet_
Dec 28 18:23:40 k8smaster-01 dockerd[1272]: time="2015-12-28T18:23:40.493353984Z" level=error msg="HTTP Error" err="no such id: fleet_ui" statusCode=404
Dec 28 18:23:40 k8smaster-01 dockerd[1272]: time="2015-12-28T18:23:40.527190285Z" level=info msg="POST /v1.21/containers/create?name=fleet_ui"
Dec 28 18:23:40 k8smaster-01 dockerd[1272]: time="2015-12-28T18:23:40.563287077Z" level=info msg="POST /v1.21/containers/f073e0736417863826456c8cf07ce153aa7ed57f22a1153a514e6b56c2
Dec 28 18:23:40 k8smaster-01 dockerd[1272]: time="2015-12-28T18:23:40.563839859Z" level=info msg="POST /v1.21/containers/f073e0736417863826456c8cf07ce153aa7ed57f22a1153a514e6b56c2
jchauncey commented
ok so that might only be for GCE - https://github.com/kubernetes/kubernetes/blob/master/docs/admin/networking.md#google-compute-engine-gce
rimusz commented
it looks it is for sure something is wrong with SSL/certs setup
rimusz commented
so this is related to runtime_config=extensions/v1beta1=true,extensions/v1beta1/daemonsets=true
rimusz commented
v0.6.1 got runtime_config=extensions/v1beta1=true,extensions/v1beta1/daemonsets=true disabled
Can you try it?
jchauncey commented
K I will later
On Dec 29, 2015 12:59 PM, "Rimas Mocevicius" notifications@github.com
wrote:
v0.6.1 got
runtime_config=extensions/v1beta1=true,extensions/v1beta1/daemonsets=true
disabled
Can you try it?—
Reply to this email directly or view it on GitHub
#33 (comment)
.
jchauncey commented
alright im still seeing issues even with the new version -
╭─jonathanchauncey at ENG000637 in ~/coreos-k8s-cluster using ‹2.2.2›
╰─○ curl http://172.17.15.101:8080/api
{
"kind": "APIVersions",
"versions": [
"v1"
]
}%
core@k8snode-02 ~ $ docker logs 752ad2dc250a
2015/12/31 17:09:40 INFO: Starting nginx...
2015/12/31 17:09:40 INFO: nginx started.
2015/12/31 17:09:40 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/31 17:09:50 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: read tcp 10.244.46.6:50936->10.100.0.1:443: read: connection reset by peer.
2015/12/31 17:10:00 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
2015/12/31 17:10:10 Error building model; not modifying configuration: Get https://10.100.0.1:443/api/v1/namespaces/deis/replicationcontrollers/deis-router: EOF.
rimusz commented
did you run the - Update OS X ... fleet units ?
jchauncey commented
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
EnvironmentFile=/etc/environment
ExecStart=/opt/bin/kube-apiserver \
--client-ca-file=/srv/kubernetes/ca.crt \
--tls-cert-file=/srv/kubernetes/server.cert \
--tls-private-key-file=/srv/kubernetes/server.key \
--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota \
--service_account_key_file=/srv/kubernetes/server.key \
--service_account_lookup=false \
--allow_privileged=true \
--insecure_bind_address=0.0.0.0 \
--insecure_port=8080 \
--kubelet_https=true \
--secure_port=6443 \
--service-cluster-ip-range=10.100.0.0/16 \
--etcd_servers=http://127.0.0.1:2379 \
--public_address_override=127.0.0.1 \
--logtostderr=true
# --runtime-config=extensions/v1beta1=true,extensions/v1beta1/daemonsets=true
Restart=always
RestartSec=10
[X-Fleet]
MachineMetadata=role=control
rimusz commented
weird, try on the fresh cluster
jchauncey commented
this is a fresh cluster
rimusz commented
right, will try to reproduce it
jchauncey commented
what do you see when you do
╰─○ k api-versions
extensions/v1beta1
v1
rimusz commented
the same, even it is not enabled
rimusz commented
can you try for me kube-solo app? it has the same setup but really works with v2 deis
jchauncey commented
rimusz commented
yep
and k api-versions does not mean it is installed, it only shows the available options
jchauncey commented
k going to try kube-solo
rimusz commented
crap, I see the same problem on this App, both share nearly identical fleet units and cloud-init files
jchauncey commented
does it normally take a while for kubernetes to come up?
rimusz commented
which app?
jchauncey commented
kube-solo
rimusz commented
no, usually quick
rimusz commented
I use kube-solo very intensively, have not noticed any problems
even k8s master env gets overwritten, but can you check that ?
jchauncey commented
trying again.
jchauncey commented
ok kube-solo seems to be working
rimusz commented
phew
need to find a time and finish porting this App to xhyve based one
jchauncey commented
so kube-solo works even with daemonsets enabled
jchauncey commented
╰─○ kgpo
NAME READY STATUS RESTARTS AGE
deis-builder-i02f4 1/1 Running 1 9m
deis-database-y8eeq 1/1 Running 0 9m
deis-etcd-1-3drw9 1/1 Running 0 9m
deis-etcd-1-h1b2s 1/1 Running 0 9m
deis-etcd-1-j9xrg 1/1 Running 0 9m
deis-etcd-discovery-4go4u 1/1 Running 0 9m
deis-logger-fluentd-cahba 0/1 Pending 0 53s
deis-logger-j3p5g 1/1 Running 0 53s
deis-minio-dhwdm 1/1 Running 0 9m
deis-registry-osbr9 1/1 Running 0 9m
deis-router-zoru6 1/1 Running 0 9m
deis-workflow-fgke9 1/1 Running 0 9m
rimusz commented
really? as it got disabled too for messing the router
jchauncey commented
yup so i think we should reenable it
rimusz commented
not the issue anymore