Gatekeeper Policy Manager cannot retrieve data from GK controller manager
paunis opened this issue · 14 comments
We have GPM deployed in a cluster with access to multiple clusters. Everything is working fine except for 2 clusters. It was working until upgrade GK from 3.4.0 to 3.7.0 The error is :
<<
We had a problem while asking the API for Gatekeeper Constraint objects
Is Gatekeeper deployed in the cluster?
Environment:
Gatekeeper version: 3.7.0
Kubernetes version: (use kubectl version): Client Version: v1.21.0; Server Version: v1.23.16-eks-48e63af
Hi @paunis
I'm sorry Gatekeeper Policy Manager is not working as expected, could you please check the logs in gatekeeper-policy-manager pod for more details on the error?
GPM should work with Gatekeeper 3.7.0
These are all logs on GPM pod:
$ k logs gpm-gatekeeper-policy-manager-5cbb845bf6-2lt6z
[2023-03-28 08:39:28 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-03-28 08:39:28 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2023-03-28 08:39:28 +0000] [1] [INFO] Using worker: gthread
[2023-03-28 08:39:28 +0000] [7] [INFO] Booting worker with pid: 7
[2023-03-28 08:39:28 +0000] [8] [INFO] Booting worker with pid: 8
[2023-03-28 08:39:30 +0000] [7] [INFO] gunicorn log level is set to: INFO
[2023-03-28 08:39:30 +0000] [7] [INFO] application log level is set to: INFO
[2023-03-28 08:39:30 +0000] [7] [INFO] RUNNING WITH AUTHENTICATION DISABLED
[2023-03-28 08:39:30 +0000] [7] [INFO] Attempting init with KUBECONFIG from path ~/.kube/config
[2023-03-28 08:39:30 +0000] [8] [INFO] gunicorn log level is set to: INFO
[2023-03-28 08:39:30 +0000] [8] [INFO] application log level is set to: INFO
[2023-03-28 08:39:30 +0000] [8] [INFO] RUNNING WITH AUTHENTICATION DISABLED
[2023-03-28 08:39:30 +0000] [8] [INFO] Attempting init with KUBECONFIG from path ~/.kube/config
[2023-03-28 08:39:30 +0000] [7] [INFO] KUBECONFIG ~/.kube/config successfuly loaded.
[2023-03-28 08:39:30 +0000] [8] [INFO] KUBECONFIG ~/.kube/config successfuly loaded.
There doesn't seem to be any error on the backend, could you try to open your browser Developer Tools and check the frontend call to the API for more details on the error please?
Only for these 2 clusters we get Error 500:
<<
Request URL: https:///api/v1/constraints//
Request Method: GET
Status Code: 500
Remote Address: :443
Referrer Policy: strict-origin-when-cross-origin
mmm... what thas the response look like for those 500 errors?
are you sure there is nothing showing up in GPM's pod logs?
which version of GPM are you using? If you are using v1.0.3 maybe you can try downgrading GPM to v0.5.1 that was tested against Gatekeeper v3.7
API response is "{"action":"Is Gatekeeper deployed in the cluster?","description":"(401)\nReason: Unauthorized\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '42955441-02e5-4828-86c0-8ae90e0d550b', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 28 Mar 2023 10:50:21 GMT', 'Content-Length': '129'})\nHTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}\n\n","error":"We had a problem while asking the API for Gatekeeper Constraint objects"}
". Can this be something outdated in the kubeconfig that is passed to GPM?
Yes, it seems that the credentials included in the kubeconfig passed to GPM are not correct.
Hi @ralgozino , we did not updated the kubeconfig, still waiting for approval
Maybe you can try setting GPM logs to debug level and see if you can get some more details from the logs in the meantime.
You can do that by setting the GPM_LOG_LEVEL environment variable to DEBUG in GPM's pod.
Set log level to debug, still nothing weird in logs:
<<
[2023-04-03 09:53:24 +0000] [7] [DEBUG] GET /static/media/Poppins-Light.f41b63c0bb0963ace821.ttf
[2023-04-03 09:53:24 +0000] [7] [DEBUG] GET /static/media/gpm-logo.b802b8c92174e7e779bfd1b5f3f31127.svg
[2023-04-03 09:53:27 +0000] [7] [DEBUG] GET /constraints/
[2023-04-03 09:53:27 +0000] [7] [DEBUG] GET /static/css/main.0fb88184.css
[2023-04-03 09:53:27 +0000] [8] [DEBUG] GET /static/js/main.9d2388a4.js
[2023-04-03 09:53:27 +0000] [8] [DEBUG] GET /static/media/github-logo.2384f056f07cd6da5d2a11e846a50566.svg
[2023-04-03 09:53:27 +0000] [7] [DEBUG] GET /static/js/icon.heart.6a5439c3.chunk.js
[2023-04-03 09:53:27 +0000] [8] [DEBUG] GET /static/js/icon.popout.415e5814.chunk.js
[2023-04-03 09:53:27 +0000] [8] [DEBUG] GET /api/v1/constraints//
[2023-04-03 09:53:27 +0000] [8] [DEBUG] entering KUBECONFIG MODE and getting API objects
[2023-04-03 09:53:27 +0000] [7] [DEBUG] GET /api/v1/contexts/
[2023-04-03 09:53:27 +0000] [7] [DEBUG] GET /api/v1/auth/
[2023-04-03 09:53:27 +0000] [8] [DEBUG] GET /static/js/icon.arrow_down.64fbca8c.chunk.js
[2023-04-03 09:53:27 +0000] [8] [DEBUG] GET /static/media/Poppins-Bold.404e299be26d78e66794.ttf
[2023-04-03 09:53:27 +0000] [7] [DEBUG] GET /static/media/Poppins-Medium.9e1bb626874ed49aa343.ttf
[2023-04-03 09:53:27 +0000] [7] [DEBUG] GET /static/media/Poppins-Regular.8081832fc5cfbf634aa6.ttf
[2023-04-03 09:53:28 +0000] [7] [DEBUG] GET /manifest.json
[2023-04-03 09:53:28 +0000] [8] [DEBUG] GET /logo192.png
[2023-04-03 09:53:28 +0000] [7] [DEBUG] GET /static/js/icon.alert.16777ed2.chunk.js
[2023-04-03 09:53:28 +0000] [8] [DEBUG] GET /static/js/icon.arrow_right.b4dff9f3.chunk.js
[2023-04-03 09:53:33 +0000] [8] [DEBUG] GET /health
[2023-04-03 09:53:33 +0000] [7] [DEBUG] GET /health
[2023-04-03 09:53:33 +0000] [7] [DEBUG] Closing connection.
[2023-04-03 09:53:33 +0000] [8] [DEBUG] Closing connection.
@ralgozino Updating tokens solved the issue. Thank you.
great to hear that! thank you for letting us know