oidc-client-relation-broken hook failed
Closed this issue · 2 comments
jardon commented
dex-auth charm goes into error state on an oidc-client-relation-broken hook failure. Will update the issue when I can determine exact steps to reproduce the issue.
2022-03-03 06:42:40 INFO juju-log Running legacy hooks/upgrade-charm.
2022-03-03 06:42:41 INFO juju.worker.caasoperator.uniter.dex-auth/5.operation runhook.go:152 ran "upgrade-charm" hook (via hook dispatching script: dispatch)
2022-03-03 06:42:41 INFO juju.worker.caasoperator.uniter.dex-auth/5 resolver.go:154 found queued "config-changed" hook
2022-03-03 06:42:43 ERROR juju-log oidc-client:2: Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 1284, in _run
result = run(args, **kwargs)
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-dex-auth-4/relation-get', '-r', '2', '-', '', '--app', '--format=json')' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 209, in <module>
main(Operator)
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/main.py", line 394, in main
charm = charm_class(framework)
File "./src/charm.py", line 44, in __init__
self.interfaces = get_interfaces(self)
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/serialized_data_interface/__init__.py", line 263, in get_interfaces
requires = {
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/serialized_data_interface/__init__.py", line 264, in <dictcomp>
name: SerializedDataInterface(
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/serialized_data_interface/__init__.py", line 110, in __init__
others = {
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/serialized_data_interface/__init__.py", line 111, in <dictcomp>
app.name: bag.get("_supported_versions")
File "/usr/lib/python3.8/_collections_abc.py", line 660, in get
return self[key]
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 400, in __getitem__
return self._data[key]
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 384, in _data
data = self._lazy_data = self._load()
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 748, in _load
return self._backend.relation_get(self.relation.id, self._entity.name, self._is_app)
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 1351, in relation_get
return self._run(*args, return_output=True, use_json=True)
File "/var/lib/juju/agents/unit-dex-auth-4/charm/venv/ops/model.py", line 1286, in _run
raise ModelError(e.stderr)
ops.model.ModelError: b'ERROR "" is not a valid unit or application\n'
2022-03-03 06:42:43 ERROR juju.worker.caasoperator.uniter.dex-auth/4.operation runhook.go:146 hook "oidc-client-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1
2022-03-03 06:42:43 INFO juju.worker.caasoperator.uniter.dex-auth/4 resolver.go:150 awaiting error resolution for "relation-broken" hook
dex-auth2.log
juju-crashdump-677b46e1-2d75-4ed2-887a-989e22e1d807.zip
kf-bundle.zip
shayancanonical commented
I ran into the same exception, below are a few observations on my deployment:
- The following is a portion of the output of
juju status:
App Version Status Scale Charm Store Channel Rev OS Address Message
admission-webhook res:oci-image@fc124ea active 1 admission-webhook charmhub stable 12 kubernetes 10.152.183.145
argo-controller res:oci-image@0eec3c1 active 1 argo-controller charmhub stable 55 kubernetes
dex-auth res:oci-image@a74f783 active 3/2 dex-auth charmhub 2.28/stable 78 kubernetes 10.152.183.192
...
Unit Workload Agent Address Ports Message
...
dex-auth/6* error idle 10.1.94.223 5556/TCP hook failed: "oidc-client-relation-broken"
dex-auth/7 waiting idle 10.1.94.219 5556/TCP Waiting for leadership
dex-auth/8 waiting idle 10.1.94.214 5556/TCP Waiting for leadership
- There are only two pods for dex-auth, both showing logs similar to:
time="2022-03-10T22:48:41Z" level=info msg="config using log level: debug"
time="2022-03-10T22:48:41Z" level=info msg="config issuer: http://10.64.140.43.nip.io/dex"
time="2022-03-10T22:48:41Z" level=info msg="kubernetes client apiVersion = dex.coreos.com/v1"
time="2022-03-10T22:48:41Z" level=info msg="creating custom Kubernetes resources"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource authcodes.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource authcodes.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource authrequests.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource authrequests.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource oauth2clients.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource oauth2clients.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource signingkeies.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource signingkeies.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource refreshtokens.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource refreshtokens.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource passwords.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource passwords.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource offlinesessionses.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource offlinesessionses.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource connectors.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource connectors.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource devicerequests.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource devicerequests.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="checking if custom resource devicetokens.dex.coreos.com has been created already..."
time="2022-03-10T22:48:41Z" level=info msg="The custom resource devicetokens.dex.coreos.com already available, skipping create"
time="2022-03-10T22:48:41Z" level=info msg="config storage: kubernetes"
time="2022-03-10T22:48:41Z" level=info msg="config static client: Ambassador Auth OIDC"
time="2022-03-10T22:48:41Z" level=info msg="config connector: local passwords enabled"
time="2022-03-10T22:48:41Z" level=info msg="config skipping approval screen"
time="2022-03-10T22:48:41Z" level=info msg="listening (http) on 0.0.0.0:5556"
- I am able to log into the kubeflow dashboard with the user/pass I set with
juju config
DomFleischmann commented
Closing this issue in favor of #53 which contains more information