jetstack/kube-oidc-proxy

Unexpected error when reading response body: stream error: stream ID 1; INTERNAL_ERROR

Sandy1438 opened this issue · 2 comments

Sandy1438 commented

Hi,
First of all a big thanks to all the teams of Jetstack and Heptiolabs for this wonderful project!
I need your guidance/help with the manifest files to setup SSO for a cloud managed kubernetes cluster.
I'm using Digital Ocean as cloud provider and have spawned up a managed K8s cluster and want to enable SSO.
I have decided to use Dex as IDP and Github Org as external auth provider. Gangway as helper app and kube-oidc-proxy as proxy for K8s api server.

I have followed the deploy instruction for Dex and Gangway and was successful in configuring it how stuck with configuring kube-oidc-proxy and making changes to Gangway to integrate with kube-oidc-proxy.

I would be thankful if anyone has done this setup for managed kubernetes cluster and help me with the manifest for it.

Thanks in advance!

BR,
Sandeep.

Sandy1438 commented

I have managed to setup kube-oidc-proxy with dex and gangway but I get this error.
Am I doing something wrong with the configuration .

[root@k8-master gangway]# kubectl get nodes
error: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 9; INTERNAL_ERROR

Logs of kube-oidc-proxy:

[root@k8-master api-oidc-proxy]# kubectl logs kube-oidc-proxy-7b5fcb4bb6-gqzc4 -n oidc-proxy
I0604 08:26:45.493706 1 oidc.go:290] OIDC: No x509 certificates provided, will use host's root CA set
I0604 08:26:45.494158 1 proxy.go:50] waiting for oidc provider to become ready...
I0604 08:26:55.494384 1 secure_serving.go:116] Serving securely on [::]:443
I0604 08:26:55.494434 1 proxy.go:94] proxy ready

I'm using cert-manager for tls certificate and nginx ingress controller.

  1. dex.oidcdex.ga -- dex issuer
  2. login.oidcdex.ga -- kube-oidc.proxy

I have configured the server endpoints in kubeconfig file to point to login.oidcdex.ga and the dex issuer ca.crt file as certificate-authority provided by letsencrypt.

TLS is getting terminated at LB.

dex config:

issuer: https://dex.oidcdex.ga/
storage:
  type: kubernetes
  config:
    inCluster: true
web:
  http: 0.0.0.0:5556
oauth2:
  skipApprovalScreen: true
connectors:
- type: github
  id: github
  name: GitHub
  config:
    clientID: $GITHUB_CLIENT_ID
    clientSecret: $GITHUB_CLIENT_SECRET
    redirectURI: https://dex.oidcdex.ga/callback
    orgs:
    - name: demo4k8s
      teams:
      - devops
staticClients:
- id: gangway
  redirectURIs:
  - 'https://gangway.oidcdex.ga/callback'
  name: 'Gangway'
  secret: <>

proxy config:

spec:
  serviceAccountName: kube-oidc-proxy
  containers:
  - image: quay.io/jetstack/kube-oidc-proxy
    ports:
    - containerPort: 443
    - containerPort: 8080
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
    name: kube-oidc-proxy
    command: ["kube-oidc-proxy"]
    args:
      - "--secure-port=443"
      - "--tls-cert-file=/etc/oidc/tls/tls.crt"
      - "--tls-private-key-file=/etc/oidc/tls/tls.key"
      - "--oidc-client-id=gangway"
      - "--oidc-issuer-url=https://dex.oidcdex.ga/"
      - "--oidc-username-claim=email"

Any help please?

kingdonb commented

I got this error when my ingress-nginx was timing out, at 60s reliably, which was how I figured out that it was some timeout.

I setup my ingress like this and the issue was resolved, no more timeouts at 60s:

  values:
    oidc:
      ...
    ingress:
      enabled: true
      ingressClassName: internal
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-production
        nginx.ingress.kubernetes.io/backend-protocol: HTTPS
        #kubernetes.io/ingress.class: internal
        nginx.ingress.kubernetes.io/proxy-connect-timeout: "3600"
        nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
        nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
  • 👍1