jetstack/kube-oidc-proxy

Keep getting Unauthorized answer

Smana opened this issue · 10 comments

Smana commented

Hey folks,

I'd be glad to get some support please. Ideally I want to get the same authentication stack accross multiple cloud providers.
I already configured successfuly Dex with the newly released features OIDC for EKS and everything works like a charm.

Now I want to add the kube-oidc-proxy in front of that because GKE or AKS are not configurable in that way.

I deployed it using the helm chart but I'm struggling getting it working.
The first issue seems to be related to the certificates I provided. These certificates are valid and issued by a Trusted CA. (I already use them on my cloud loadbalancers). This is actually a wildcard *.domain.tld.

In order to avoid an issue with the cloud loadbalancers and the healthchecks, I ran a port-forward as follows

kubectl port-forward svc/oidc-proxy 4433:443&

First of all the server doesn't take care of my Host header

curl -H "Host: dummy.domain.tld" https://localhost:4433 -vvvv
*   Trying 127.0.0.1:4433...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 4433 (#0)
Handling connection for 4433
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
...
* Server certificate:
*  subject: CN=*.domain.tld
*  start date: Feb 10 00:00:00 2021 GMT
*  expire date: Mar  5 23:59:59 2022 GMT
*  subjectAltName does not match localhost
* SSL: no alternative certificate subject name matches target host name 'localhost'
* Closing connection 0
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (60) SSL: no alternative certificate subject name matches target host name 'localhost'
...

I had to add it to the /etc/hosts file. Anyway, that may be an issue on my side.
How should I configure the health checks on the loadbalancer side, because when I request the webserver, I always got an Unauthorized answer ?

Smana commented

In the meantime, I'll try to test it locally. I mean I'll run the proxy on my laptop in order to validate that everything works.

Smana commented

I'm probably missing something:
the kube-oidc-proxy seems to run properly

kubectl logs oidc-proxy-778cf76d79-zjsnx 
I0222 19:26:53.286424       1 oidc.go:282] OIDC: No x509 certificates provided, will use host's root CA set
I0222 19:26:53.286883       1 dynamic_serving_content.go:130] Starting serving-cert::/etc/oidc/tls/crt.pem::/etc/oidc/tls/key.pem
I0222 19:26:53.287070       1 secure_serving.go:178] Serving securely on [::]:443
I0222 19:26:53.287116       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0222 19:27:13.779390       1 probe.go:69] OIDC provider initialized, proxy ready

The issuer is reachable from the container

curl https://dex.domain.tld/dex/healthz
Health check passed

The container is running the command

kube-oidc-proxy --secure-port=443 --tls-cert-file=/etc/oidc/tls/crt.pem --tls-private-key-file=/etc/oidc/tls/key.pem --oidc-client-id=oidc-proxy --oidc-issuer-url=https://dex.domain.tld/dex --oidc-username-claim=email --oidc-signing-algs=RS256

But I'm not able to perform an authentication and all the calls return an Unauthorized answer

error: setup: authentication error: oidc error: oidc discovery error: 401 Unauthorized: Unauthorized
Smana commented

Well I'm using kubelogin plugin, maybe using Gangway will help me. But I'm not sure I'll be able to bypass the Unauthorized error and how to configure the proxy properly:

kubectl oidc-login setup --oidc-issuer-url https://proxy-xxx.domain.tld/dex --oidc-client-id eks-test-eu-2 --oidc-client-secret xxx --oidc-extra-scope=email --oidc-extra-scope=groups

error: setup: authentication error: oidc error: oidc discovery error: 401 Unauthorized: Unauthorized
Smana commented

Keep digging. Same unauthorized error using Gangway.

I increased the log level here are the options used by kube-oidc-proxy

      containers:       
      - args:                
        - --secure-port=443
        - --tls-cert-file=/etc/oidc/tls/crt.pem
        - --tls-private-key-file=/etc/oidc/tls/key.pem
        - --oidc-client-id=$(OIDC_CLIENT_ID)    
        - --oidc-issuer-url=$(OIDC_ISSUER_URL)      
        - --oidc-username-claim=$(OIDC_USERNAME_CLAIM)
        - --oidc-groups-claim=$(OIDC_GROUPS_CLAIM)
        - --oidc-signing-algs=$(OIDC_SIGNING_ALGS)
        - --token-passthrough
        - --v=10

I0223 12:52:39.339137       1 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/etc/oidc/tls/crt.pem::/etc/oidc/tls/key.pem"
I0223 12:52:39.339322       1 oidc.go:282] OIDC: No x509 certificates provided, will use host's root CA set
I0223 12:52:39.340842       1 dynamic_serving_content.go:130] Starting serving-cert::/etc/oidc/tls/crt.pem::/etc/oidc/tls/key.pem
I0223 12:52:39.341139       1 tlsconfig.go:200] loaded serving cert ["serving-cert::/etc/oidc/tls/crt.pem:....
I0223 12:52:39.341195       1 secure_serving.go:178] Serving securely on [::]:443
I0223 12:52:39.341251       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0223 12:52:54.929515       1 probe.go:69] OIDC provider initialized, proxy ready
I0223 12:52:54.929540       1 probe.go:70] OIDC provider initialized, readiness check returned error: oidc: verify token: oidc: expected audience "eks-test-eu-2" got []
I0223 12:55:54.138307       1 handlers.go:169] unauthenticated user request 192.168.1.1:42869
I0223 12:55:54.197655       1 handlers.go:169] unauthenticated user request 192.168.1.1:42869

I've an error regarding the audience verification, I might have missed something in the examples.

Smana commented

And I'd be glad to help writing a documentation with Helm charts instead of ksonnet which is not maintained anymore.

JoshVanL commented

Hi @Smana,

For the readiness probe, kube-oidc-proxy exposes an http readiness check on port 8080 /ready. Perhaps you can use that for your LB?

Your requests are being rejected here by the proxy as they are failing OIDC. Is oidc-proxy the same client-id in your token? Is the email claim also present?

Smana commented

Hi @JoshVanL,

Thanks for your answer,

For the readiness probe, kube-oidc-proxy exposes an http readiness check on port 8080 /ready. Perhaps you can use that for your LB?

Regarding the loadbalancing, I'll get back to it later on. I ended up using a TCP loadbalancer and that works.

For my main issue, well I surely misunderstood how to setup things so I'll detail the whole setup:

I'll use the following names as placeholders:
Dex URL (issuer): https://dex.domain.tld/dex
kube-oidc-proxy URL: https://proxy.domain.tld

  • kube-oidc-proxy helm values
oidc:
  clientId: "eks-test-eu-2"
  issuerUrl: "https://dex.domain.tld/dex"
  usernameClaim: "email"
  groupsClaim: "groups"
  • Dex is configured with Github as connector and I declared a cluster as staticClient
  connectors:
  - type: github
    id: github
    name: GitHub
    config:
      clientID: xxxx
      clientSecret: xxxx
      redirectURI: https://dex.domain.tld/dex/callback
      orgs:
      - name: mycompany
        teams:
          - sre
      loadAllGroups: true
      teamNameField: slug
      useLoginAsID: false
...
  staticClients:
    - id: eks-test-eu-2
      redirectURIs:
        - 'http://localhost:8000'
        - 'http://localhost:5555/callback'
      name: 'Kubernetes cluster eks-test-eu-2'
      secret: xxxx

Is there something missing on Dex side ? May be I should put another redirectURI address ?

Here I'm using kubelogin as a replacement of gangway with the following command

kubectl oidc-login setup --oidc-issuer-url https://proxy.domain.tld --oidc-client-id eks-test-eu-2 --oidc-client-secret xxxx --oidc-extra-scope=email --oidc-extra-scope=groups
authentication in progress...
error: setup: authentication error: oidc error: oidc discovery error: 401 Unauthorized: Unauthorized

Is oidc-proxy the same client-id in your token? Is the email claim also present?

I'm not sure to understand this question, I don't have any token yet because it fails with an Unauthorized error at the initial call. Do I have to provide some kind of credentials ?

Thanks again for your answers. The kube-oidc-proxy would be very useful in order to have the same authentication stack across all the cloud providers.
Again my Dex>Github setup works great with EKS and the OIDC configuration on AWS side.

JoshVanL commented

No problem! Let get this working 😎

Looks like you need to change the issuer URL for your oidc-login command --oidc-issuer-url https://proxy.domain.tld -> --oidc-issuer-url https://dex.domain.tld/dex

Smana commented

Ok doing so I got a valid kubeconfig (I can request the kubeapiserver with the proper RBAC configured), but there's no mention of the proxy anywhere.
Then I configured the kubeconfig with the proxy as apiserver and the proper CA.
Oooh I got it ❤️ . That was my missing part !

kubectl create clusterrolebinding --clusterrole=cluster-admin --group="company:sre" testsmana-githubauth

Then with my kubeconfig I've the perms.
You rock @JoshVanL

May I propose a documentation (demo) with 3 k3d clusters to mimic 3 different cloud providers ?

  • 👍1
  • 🎉1
JoshVanL commented

@Smana Awesome 🙂

Any demo contributions would be appreciated!