Okta crash
Flybro opened this issue · 4 comments
Flybro commented
Does anyone know if there is a specific reason why GPM crashes as soon as I enable Okta authentication? Below is a fragment of the log with a message that isn't entirely clear to me. Thanks for any advice.
[2023-06-13 06:39:52 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2023-06-13 06:39:52 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2023-06-13 06:39:52 +0000] [1] [INFO] Using worker: gthread
[2023-06-13 06:39:52 +0000] [7] [INFO] Booting worker with pid: 7
[2023-06-13 06:39:52 +0000] [8] [INFO] Booting worker with pid: 8
[2023-06-13 06:39:54 +0000] [7] [INFO] gunicorn log level is set to: DEBUG
[2023-06-13 06:39:54 +0000] [7] [INFO] application log level is set to: DEBUG
[2023-06-13 06:39:54 +0000] [7] [INFO] AUTHENTICATION ENABLED WITH OIDC
[2023-06-13 06:39:54 +0000] [8] [INFO] gunicorn log level is set to: DEBUG
[2023-06-13 06:39:54 +0000] [8] [INFO] application log level is set to: DEBUG
[2023-06-13 06:39:54 +0000] [8] [INFO] AUTHENTICATION ENABLED WITH OIDC
[2023-06-13 06:39:54 +0000] [7] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/gthread.py", line 92, in init_process
super().init_process()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
self.wsgi = self.app.wsgi()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
return self.load_wsgiapp()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
return util.import_app(self.app_uri)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/util.py", line 359, in import_app
mod = importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/app/app.py", line 111, in <module>
provider_config = ProviderConfiguration(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flask_pyoidc/provider_configuration.py", line 174, in __init__
self.client_settings = ClientSettings(timeout=self.DEFAULT_REQUEST_TIMEOUT,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/oic/utils/settings.py", line 96, in __init__
super().__init__(
File "/usr/local/lib/python3.11/site-packages/oic/utils/settings.py", line 51, in __init__
self.client_cert = client_cert
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/oic/utils/settings.py", line 73, in __setattr__
raise SettingsException(
oic.utils.settings.SettingsException: client_cert has a type of <class 'NoneType'>, expected any of (<class 'str'>, <class 'tuple'>).
[2023-06-13 06:39:54 +0000] [8] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/gthread.py", line 92, in init_process
super().init_process()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 134, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.11/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
self.wsgi = self.app.wsgi()
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
return self.load_wsgiapp()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
return util.import_app(self.app_uri)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/util.py", line 359, in import_app
mod = importlib.import_module(module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/app/app.py", line 111, in <module>
provider_config = ProviderConfiguration(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/flask_pyoidc/provider_configuration.py", line 174, in __init__
self.client_settings = ClientSettings(timeout=self.DEFAULT_REQUEST_TIMEOUT,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/oic/utils/settings.py", line 96, in __init__
super().__init__(
File "/usr/local/lib/python3.11/site-packages/oic/utils/settings.py", line 51, in __init__
self.client_cert = client_cert
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/oic/utils/settings.py", line 73, in __setattr__
raise SettingsException(
oic.utils.settings.SettingsException: client_cert has a type of <class 'NoneType'>, expected any of (<class 'str'>, <class 'tuple'>).
[2023-06-13 06:39:54 +0000] [7] [INFO] Worker exiting (pid: 7)
[2023-06-13 06:39:54 +0000] [8] [INFO] Worker exiting (pid: 8)
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 209, in run
self.sleep()
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 357, in sleep
ready = select.select([self.PIPE[0]], [], [], 1.0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
self.reap_workers()
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/gunicorn", line 8, in <module>
sys.exit(run())
^^^^^
File "/usr/local/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/usr/local/lib/python3.11/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/usr/local/lib/python3.11/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 229, in run
self.halt(reason=inst.reason, exit_status=inst.exit_status)
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 342, in halt
self.stop()
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 393, in stop
time.sleep(0.1)
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 242, in handle_chld
self.reap_workers()
File "/usr/local/lib/python3.11/site-packages/gunicorn/arbiter.py", line 525, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
ralgozino commented
Hi @Flybro
Have you tried setting all the environment variables that start with GPM_OIDC_*?
Autodiscovery of OIDC parameters (reading the .well-known endpoint) is not working for some reason that we need to investigate.
If you have, maybe there's some issue with Okta's certificate? see this line:
oic.utils.settings.SettingsException: client_cert has a type of <class 'NoneType'>, expected any of (<class 'str'>, <class 'tuple'>).
Flybro commented
Thanks @ralgozino for getting back to me.
This is what my environment variables look like when we use auth with "Anonymous":
gpm@gatekeeper-policy-manager-f9b97c5d-pqzf4:/app$ env | grep -i gpm
GPM_OIDC_CLIENT_SECRET=Y_fDNKhaWzQkmdg31eWl1mkaXgZ
GPM_OIDC_AUTHORIZATION_ENDPOINT=https://preprod.connect.hidden.net/oauth2/v1/authorize
GPM_OIDC_USERINFO_ENDPOINT=https://preprod.connect.hidden.net/oauth2/v1/userinfo
GPM_SECRET_KEY=TZGAydr2bmj2pug
GPM_OIDC_JWKS_URI=https://preprod.connect.hidden.net/oauth2/v1/keys
GPM_AUTH_ENABLED=Anonymous
GPM_PREFERRED_URL_SCHEME=HTTPS
GPM_OIDC_ISSUER=https://preprod.connect.hidden.net
GPM_OIDC_TOKEN_ENDPOINT=https://preprod.connect.hidden.net/oauth2/v1/token
GPM_OIDC_REDIRECT_DOMAIN=https://gpm.test.security.hidden.net/
GPM_OIDC_CLIENT_ID=a8htnlspunJRT
GPM_OIDC_INTROSPECTION_ENDPOINT=https://preprod.connect.hidden.net/oauth2/v1/introspect
GPM_OIDC_END_SESSION_ENDPOINT=https://preprod.connect.hidden.net/oauth2/v1/logout
GPM_LOG_LEVEL=DEBUG
I've intentionally not connected any certificates, so I'm not sure what the message you mentioned means.
ralgozino commented
Thanks for the deatils @Flybro
You are using v1.0.4, right? can I ask you to try with GPM v1.0.3 instead? It seems that something broke with oidc when we updated the dependencies in v1.0.4
Flybro commented
Hi @ralgozino
You're absolutely right. The problem disappeared with version 1.0.3 and all works fine.