Unable to connect to the server: x509: certificate signed by unknown authority
agill17 opened this issue · 16 comments
Hi there, I am attempting to use this project along with dex + dex-k8s-authenticator on EKS. Whenever I run any kubectl commands I get the following error: Unable to connect to the server: x509: certificate signed by unknown authority
Here are values override for kube-oidc-proxy
oidc:
clientId: k8s-dex-authenticator
issuerUrl: my-dex-url
usernameClaim: email
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
hosts:
- host: oidc.endpoint.url
paths:
- /
kubeconfig file looks like this;
- cluster:
certificate-authority: certs/dev/k8s-ca.crt
server: https://oidc.endpoint.url
name: dev
users:
- name: admin-dev
user:
auth-provider:
config:
client-id: k8s-dex-authenticator
client-secret: super-secret..........
id-token: id-token-xxxx.......
idp-issuer-url: my-dex-url
refresh-token: refresh-token.....
name: oidc
contexts:
- context:
cluster: dev
user: admin-dev
name: admin-dev
oidc-proxy logs
I0704 22:39:40.211455 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0704 22:40:03.113429 1 probe.go:69] OIDC provider initialized, proxy ready
Note: I am terminating TLS on my load balancer
Any pointers?
Hi @agill17,
Since you are terminating TLS at your LB, you need to set the certificate authority of your kubeconfig to either the serving certificate of your LB, or ideally the CA that signed the certificate if not self signed. This will be set in the cluster config of your kubeconfig as either certificate-authority-data or certificate-authority. Hope that helps :)
Ah, just saw your update. Can you verify that certs/dev/k8s-ca.crt contains the serving certificate of your LB, or indeed the CA?
openssl s_client -showcerts -connect www.example.com:443 </dev/null
Actually the certs/dev/k8s-ca.crt is the EKS Certificate-Authority @JoshVanL . And I using AWS ACM certificate for ingress controller thats serving the LB
Can you make sure that this certificate is the CA or cert of the LB, and not the CA of the k8s api-server? This may be the issue.
@JoshVanL yes replaced that with cert of LB and now getting this
➜ kubectl get ns
E0704 19:06:03.269533 53612 request.go:879] Unexpected error when reading response body: unexpected EOF
E0704 19:06:03.308981 53612 request.go:879] Unexpected error when reading response body: unexpected EOF
E0704 19:06:03.342004 53612 request.go:879] Unexpected error when reading response body: unexpected EOF
E0704 19:06:03.372083 53612 request.go:879] Unexpected error when reading response body: unexpected EOF
E0704 19:06:03.406853 53612 request.go:879] Unexpected error when reading response body: unexpected EOF
error: unexpected error when reading response body. Please retry. Original error: unexpected EOF
Interesting error message... Anything on kube-oidc-proxy logs?
with verbose output
I0704 19:06:46.385345 53648 round_trippers.go:443] GET https://oidc.ingress.url/api?timeout=32s 400 Bad Request in 58 milliseconds
I0704 19:06:46.385370 53648 round_trippers.go:449] Response Headers:
I0704 19:06:46.385380 53648 round_trippers.go:452] Date: Sat, 04 Jul 2020 23:06:46 GMT
I0704 19:06:46.385387 53648 round_trippers.go:452] Server: nginx/1.15.6
I0704 19:06:46.385394 53648 round_trippers.go:452] Connection: keep-alive
E0704 19:06:46.385463 53648 request.go:879] Unexpected error when reading response body: unexpected EOF
I0704 19:06:46.385544 53648 cached_discovery.go:121] skipped caching discovery info due to unexpected error when reading response body. Please retry. Original error: unexpected EOF
nothing on kube-oidc-proxy logs
Seems to be a network issue then rather than a proxy one. I would suggest making sure that your ingress is wired up correctly. Your NGINX logs might have more info since that is where the connection is getting returned from.
I am seeing a lot of these on nginx ingress controller
2020/07/04 23:13:27 [error] 85#85: *10758 recv() failed (104: Connection reset by peer) while sending to client, client: 108.28.45.239, server: oidc.ingress.url, request: "GET /api?timeout=32s HTTP/1.1", upstream: "http://192.168.128.124:443/api?timeout=32s", host: "oidc.ingress.url"
108.28.45.239 - [108.28.45.239] - - [04/Jul/2020:23:13:27 +0000] "GET /api?timeout=32s HTTP/1.1" 400 54 "-" "kubectl/v1.16.9 (darwin/amd64) kubernetes/a17149e" 1138 0.000 [dex-oidc-proxy-kube-oidc-proxy-https] 192.168.128.124:443 48 0.000 400 5ae8b6b8987aa0dcf8d5ffcc9938f88c
2020/07/04 23:13:27 [error] 85#85: *10755 recv() failed (104: Connection reset by peer) while sending to client, client: 108.28.45.239, server: oidc.ingress.url, request: "GET /api?timeout=32s HTTP/1.1", upstream: "http://192.168.128.124:443/api?timeout=32s", host: "oidc.ingress.url"
108.28.45.239 - [108.28.45.239] - - [04/Jul/2020:23:13:27 +0000] "GET /api?timeout=32s HTTP/1.1" 400 54 "-" "kubectl/v1.16.9 (darwin/amd64) kubernetes/a17149e" 1138 0.000 [dex-oidc-proxy-kube-oidc-proxy-https] 192.168.128.124:443 48 0.000 400 e4d467371f7c389384da10b2a41478e6
2020/07/04 23:13:27 [error] 85#85: *10852 recv() failed (104: Connection reset by peer) while sending to client, client: 108.28.45.239, server: oidc.ingress.url, request: "GET /api?timeout=32s HTTP/1.1", upstream: "http://192.168.128.124:443/api?timeout=32s", host: "oidc.ingress.url"
108.28.45.239 - [108.28.45.239] - - [04/Jul/2020:23:13:27 +0000] "GET /api?timeout=32s HTTP/1.1" 400 54 "-" "kubectl/v1.16.9 (darwin/amd64) kubernetes/a17149e" 1138 0.000 [dex-oidc-proxy-kube-oidc-proxy-https] 192.168.128.124:443 48 0.000 400 5abda9ab28ed94ffe4e2c39b2068aede
2020/07/04 23:13:27 [error] 85#85: *10855 recv() failed (104: Connection reset by peer) while sending to client, client: 108.28.45.239, server: oidc.ingress.url, request: "GET /api?timeout=32s HTTP/1.1", upstream: "http://192.168.128.124:443/api?timeout=32s", host: "oidc.ingress.url"
108.28.45.239 - [108.28.45.239] - - [04/Jul/2020:23:13:27 +0000] "GET /api?timeout=32s HTTP/1.1" 400 54 "-" "kubectl/v1.16.9 (darwin/amd64) kubernetes/a17149e" 1138 0.000 [dex-oidc-proxy-kube-oidc-proxy-https] 192.168.128.124:443 48 0.000 400 724755997ca5058135787aa2d37a343d
2020/07/04 23:13:27 [error] 85#85: *10858 recv() failed (104: Connection reset by peer) while sending to client, client: 108.28.45.239, server: oidc.ingress.url, request: "GET /api?timeout=32s HTTP/1.1", upstream: "http://192.168.128.124:443/api?timeout=32s", host: "oidc.ingress.url"
108.28.45.239 - [108.28.45.239] - - [04/Jul/2020:23:13:27 +0000] "GET /api?timeout=32s HTTP/1.1" 400 54 "-" "kubectl/v1.16.9 (darwin/amd64) kubernetes/a17149e" 1138 0.001 [dex-oidc-proxy-kube-oidc-proxy-https] 192.168.128.124:443 48 0.000 400 a1ba8930cfb0749a2aa157d9743e29ee
oh does oidc proxy needs to be on Layer3 LB vs a Layer7?
Because currently I am using 1 ingress controller with LB https->http as its listener and I wonder if kube-oidc-proxy needs to be using SSL->TCP listener..
Which is the IP 192.168.128.124, the proxy of the NGINX? Also note that is going to a 443 when in theory it should be something like 80, though I'm not sure on the behavior for enforcing http(s) on ports with your setup.
oh does oidc proxy needs to be on Layer3 LB vs a Layer7?
In theory it shouldn't matter, although running kube-oidc-proxy unsecured isn't tested and is not supported really..
If you were to run kube-oidc-proxy at a higher log level you should be able to see more about the incoming connections. If you see nothing at all, then it is definitely a network/routing issue, rather than a proxy issue.
@JoshVanL that IP is actually the oidc-proxy pod ip
and I changed the oidc-proxy service port to 80, still seeing the same error.
This is all I see in oidc-proxy pod logs
I0704 23:27:36.738329 1 oidc.go:282] OIDC: No x509 certificates provided, will use host's root CA set
I0704 23:27:36.739115 1 secure_serving.go:178] Serving securely on [::]:443
I0704 23:27:36.739488 1 dynamic_serving_content.go:130] Starting serving-cert::/etc/oidc/tls/crt.pem::/etc/oidc/tls/key.pem
I0704 23:27:36.739592 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0704 23:34:02.492049 1 probe.go:69] OIDC provider initialized, proxy ready
I0704 23:34:02.492065 1 probe.go:70] OIDC provider initialized, readiness check returned error: oidc: verify token: oidc: expected audience "k8s-dex-authenticator" got []```
@JoshVanL Adding the following annotations to oidc-proxy ingress worked
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
kubectl get namespace
Error from server (Forbidden): namespaces is forbidden: User "user-from-ad" cannot list resource "namespaces" in API group "" at the cluster scope
Awesome 🎉 glad that is working for you now.
Feel free to reopen or open a new ticket if you have any other issues.
/close
@JoshVanL: Closing this issue.
In response to this:
Awesome 🎉 glad that is working for you now.
Feel free to reopen or open a new ticket if you have any other issues.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.