kubectl exec through kube-oidc-proxy fails
smurfralf opened this issue · 11 comments
I have a working kube-oidc-proxy instance but when I try to run kubectl exec -it
I get a failure. Is kubectl exec -it
supported by the proxy?
Here is what I ran and the result:
$ kubectl exec -it mypod -n mynamespace -- /bin/sh
error: error sending request: Post https://kube-oidc-proxy.mydomain.com/api/v1/namespaces/mynamespace/pods/mypod/exec?command=%2Fbin%2Fsh&container=maincontainer&stdin=true&stdout=true&tty=true: EOF
If make a request directly to the apiserver, it succeeds and I get a shell prompt inside the pod. But of course they are different users, since the direct call is using x509 authentication. However the kube-oidc-proxy user has a role with wildcards for every rule, I don't think permissions are the cause. There are no entries related to the request in the kube-oidc-proxy pod logs.
The image I'm using is quay.io/jetstack/kube-oidc-proxy:v0.2.0
Hi @smurfralf,
This indeed is a network issue rather than an auth one from what I can see. The latest release v0.3.0 included a change to enable a flush interval for long running connections however it looks like we are failing on a POST request here which is odd.
Would you be able to upgrade the image tag (there should't be any breaking changes), and increase the log level of the proxy (--v=10), and report back with the output? (don't forget it remove any secrets! 😬).
Hi @JoshVanL I'm experiencing the same issue however with a different error message.
hazmei.ar@Hazmeis-MBP charts % k exec dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc sh
Error from server (BadRequest): Upgrade request required
hazmei.ar@Hazmeis-MBP charts % k port-forward dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc 8080:80
error: error upgrading connection: Upgrade request required
I increased the log level of the proxy (--v=10) and this is the output for those requests
I0724 04:35:42.701980 1 handlers.go:51] authenticated request: 10.211.1.90:53230
I0724 04:35:42.702250 1 round_trippers.go:423] curl -k -v -XPOST -H "Content-Length: 0" -H "X-Forwarded-For: 172.31.0.195, 10.211.1.90" -H "Impersonate-User: user@example.com" -H "X-Amzn-Trace-Id: Root=xxx" -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "X-Forwarded-Proto: https" -H "User-Agent: kubectl/v0.0.0 (darwin/amd64) kubernetes/$Format" -H "X-Forwarded-Port: 443" -H "Impersonate-Group: team1@example.com" -H "Impersonate-Group: team2@example.com" -H "Impersonate-Group: system:authenticated" -H "Authorization: Bearer authorizationbearerhere" 'https://172.20.0.1:443/api/v1/namespaces/devops/pods/dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc/exec?command=sh&container=dex-k8s-authenticator&stderr=true&stdout=true'
I0724 04:35:42.710329 1 round_trippers.go:443] POST https://172.20.0.1:443/api/v1/namespaces/devops/pods/dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc/exec?command=sh&container=dex-k8s-authenticator&stderr=true&stdout=true 400 Bad Request in 8 milliseconds
I0724 04:35:42.710360 1 round_trippers.go:449] Response Headers:
I0724 04:35:42.710367 1 round_trippers.go:452] Audit-Id: a2f43851-ba92-4321-9f80-ed8913483f36
I0724 04:35:42.710373 1 round_trippers.go:452] Cache-Control: no-cache, private
I0724 04:35:42.710378 1 round_trippers.go:452] Content-Type: application/json
I0724 04:35:42.710383 1 round_trippers.go:452] Date: Fri, 24 Jul 2020 04:35:42 GMT
I0724 04:35:42.710388 1 round_trippers.go:452] Content-Length: 139
I0724 04:35:52.510581 1 handlers.go:51] authenticated request: 10.211.0.205:10340
I0724 04:35:52.510735 1 round_trippers.go:423] curl -k -v -XPOST -H "Impersonate-User: user@example.com" -H "X-Forwarded-Proto: https" -H "X-Forwarded-Port: 443" -H "Impersonate-Group: team1@example.com" -H "Impersonate-Group: team2@example.com" -H "Impersonate-Group: system:authenticated" -H "X-Forwarded-For: ip-address-1, ip-address-1" -H "X-Stream-Protocol-Version: portforward.k8s.io" -H "X-Amzn-Trace-Id: Root=xxx" -H "Authorization: Bearer authorizationbearerhere" -H "Content-Length: 0" -H "User-Agent: kubectl/v0.0.0 (darwin/amd64) kubernetes/$Format" 'https://172.20.0.1:443/api/v1/namespaces/devops/pods/dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc/portforward'
I0724 04:35:52.523721 1 round_trippers.go:443] POST https://172.20.0.1:443/api/v1/namespaces/devops/pods/dex-k8s-auth-dex-k8s-authenticator-6759b6c7-crthc/portforward 400 Bad Request in 12 milliseconds
I0724 04:35:52.523836 1 round_trippers.go:449] Response Headers:
I0724 04:35:52.523844 1 round_trippers.go:452] Content-Length: 139
I0724 04:35:52.523852 1 round_trippers.go:452] Audit-Id: d5bc1a29-b3b0-4cd3-9905-e8ee79ddb340
I0724 04:35:52.523856 1 round_trippers.go:452] Cache-Control: no-cache, private
I0724 04:35:52.523861 1 round_trippers.go:452] Content-Type: application/json
I0724 04:35:52.523913 1 round_trippers.go:452] Date: Fri, 24 Jul 2020 04:35:52 GMT
The proxy is running on EKS behind an ALB. Besides exec and port-forward, i'm able to run the other kubectl commands like kubectl get pods/deploy/secrets/...
, kubectl logs ...
.
Hi, we've got the same issue. Tried to expose the service on ELBs and ALBs and get the same output as the previous comment (except for hostnames). It works for everything except for kubectl exec
and kubectl port-forward
. Please let me know if you need any kind of logs. We are running 0.3.0.
Hi, we've got the same issue. Tried to expose the service on ELBs and ALBs and get the same output as the previous comment (except for hostnames). It works for everything except for
kubectl exec
andkubectl port-forward
. Please let me know if you need any kind of logs. We are running 0.3.0.
Hi @sebastienc . You can't run it behind ALB / ELB in AWS. kubectl exec
and kubectl port-forward
uses SPDY protocol which is not supported by those load balancers. I solve this by exposing the service on NLB.
@hazmei My kube-apiserver is exposed thru nginx ingress.
And the nginx ingress is behind HAProxy (corporate Loabalancer on-prem)
As per your comment, i need to make sure that SPDY protocol is configured in :
- nginx ingress
- HAProxy
My question is how to configure this protocol in these 2 layers?
Hi, I also have the same issue, but the log is different. I'm getting a 403 error. I'm using kube-oidc-proxy 0.3.0. I can't do port-forwarding. I'm using Istio -> Nginx Ingress -> kube-oidc-proxy.
here are the logs:
I0518 02:30:38.052646 88338 loader.go:373] Config loaded from file: ./kc.yaml
I0518 02:30:38.061036 88338 round_trippers.go:463] GET https://k8s/api/v1/namespaces/argocd/services/argocd-server
I0518 02:30:38.061052 88338 round_trippers.go:469] Request Headers:
I0518 02:30:38.061059 88338 round_trippers.go:473] Accept: application/json, /
I0518 02:30:38.061063 88338 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (darwin/arm64) kubernetes/4c94112
I0518 02:30:38.749894 88338 round_trippers.go:574] Response Status: 200 OK in 688 milliseconds
I0518 02:30:38.752714 88338 round_trippers.go:463] GET https://k8s/api/v1/namespaces/argocd/pods?labelSelector=app.kubernetes.io%2Fname%3Dargocd-server
I0518 02:30:38.752751 88338 round_trippers.go:469] Request Headers:
I0518 02:30:38.752769 88338 round_trippers.go:473] Accept: application/json, /
I0518 02:30:38.752781 88338 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (darwin/arm64) kubernetes/4c94112
I0518 02:30:38.780801 88338 round_trippers.go:574] Response Status: 200 OK in 27 milliseconds
I0518 02:30:38.790906 88338 round_trippers.go:463] GET https://k8s/api/v1/namespaces/argocd/pods/argocd-server-777945846-jrsld
I0518 02:30:38.790929 88338 round_trippers.go:469] Request Headers:
I0518 02:30:38.790937 88338 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (darwin/arm64) kubernetes/4c94112
I0518 02:30:38.790944 88338 round_trippers.go:473] Accept: application/json, /
I0518 02:30:38.806739 88338 round_trippers.go:574] Response Status: 200 OK in 15 milliseconds
I0518 02:30:38.807808 88338 round_trippers.go:463] POST https://k8s/api/v1/namespaces/argocd/pods/argocd-server-777945846-jrsld/portforward
I0518 02:30:38.807816 88338 round_trippers.go:469] Request Headers:
I0518 02:30:38.807823 88338 round_trippers.go:473] X-Stream-Protocol-Version: portforward.k8s.io
I0518 02:30:38.807829 88338 round_trippers.go:473] User-Agent: kubectl/v1.27.1 (darwin/arm64) kubernetes/4c94112
I0518 02:30:38.830910 88338 round_trippers.go:574] Response Status: 403 Forbidden in 23 milliseconds
I also get 403
when trying to use k port-forward
or when trying to use k exec
to connect to the proxy pod.
Any solution for this?
k port-forward -n xxx pod/redis-master-0 6379:6379 --v=6
I0921 17:31:52.061926 18020 loader.go:373] Config loaded from file: /Users/xxx/.kube/config
I0921 17:31:52.634821 18020 round_trippers.go:553] GET https://oidc-proxy.xxx/api/v1/namespaces/xxx/pods/redis-master-0 200 OK in 542 milliseconds
I0921 17:31:52.799925 18020 round_trippers.go:553] GET https://oidc-proxy.xxx/api/v1/namespaces/xxx/pods/redis-master-0 200 OK in 127 milliseconds
I0921 17:31:53.149233 18020 round_trippers.go:553] POST https://oidc-proxy.xxx/api/v1/namespaces/xxx/pods/redis-master-0/portforward 403 Forbidden in 345 milliseconds
error: error upgrading connection:
kubectl exec -it pod/kube-oidc-proxy-kube-oidc-proxy-8555dc5985-22b8k -n kube-oidc-proxy --v=6 -- /bin/bash
I0921 17:25:35.897751 17912 loader.go:373] Config loaded from file: /Users/xxx/.kube/config
I0921 17:25:36.394211 17912 round_trippers.go:553] GET https://oidc-proxy.xxx/api/v1/namespaces/kube-oidc-proxy/pods/kube-oidc-proxy-kube-oidc-proxy-8555dc5985-22b8k 200 OK in 486 milliseconds
I0921 17:25:36.397416 17912 podcmd.go:88] Defaulting container name to kube-oidc-proxy-kube-oidc-proxy
I0921 17:25:36.741460 17912 round_trippers.go:553] POST https://oidc-proxy.xxx/api/v1/namespaces/kube-oidc-proxy/pods/kube-oidc-proxy-kube-oidc-proxy-8555dc5985-22b8k/exec?command=%2Fbin%2Fbash&container=kube-oidc-proxy-kube-oidc-proxy&stdin=true&stdout=true&tty=true 403 Forbidden in 343 milliseconds
I0921 17:25:36.741959 17912 helpers.go:246] server response object: [{
"metadata": {}
}]
Error from server:
I'm also seeing this exact problem with NGINX... anybody have success yet?
In my case oidc-proxy is behind self-hosted NGINX balancer. And for kubectl-exec mode connection should be upgraded from regular HTTP/1.1 to WebSocket. So adding this to NGINX config fixed the problem:
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
for Istio the fix is to apply the following EnvoyFilter
:
configPatches:
- applyTo: NETWORK_FILTER
match:
context: GATEWAY
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
patch:
operation: MERGE
value:
typed_config:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
upgrade_configs:
- upgrade_type: spdy/3.1