Missing `secret-key` config value validation
Opened this issue · 4 comments
It is not clear from the charm docs that the secret-key needs to be at least 8 char long
It is also not very verbose from Juju POV what is actually happening on the charm
Ideally I would expect to have some config value validation on the charm to set it maybe on blocked state but avoid having the actual service down
Reproduce
juju config minio secret-key=minioLogs
I cannot access MinIO website
$ juju status | grep minio
minio res:oci-image@1755999 waiting 1 minio ckf-1.7/stable 186 10.152.183.165 no
mlflow-minio res:oci-image@1755999 active 1 minio ckf-1.7/edge 186 10.152.183.108 no
minio/0* error idle 10.1.149.251 9000/TCP,9001/TCP crash loop backoff: back-off 2m40s restarting failed container=minio pod=minio-0_kubeflow(1980c8fe-8cb3-4099-b9eb-2c6...
mlflow-minio/0* active idle 10.1.150.41 9000/TCP,9001/TCP$ microk8s.kubectl get pods -n kubeflow | grep minio
minio-operator-0 1/1 Running 0 40d
mlflow-minio-operator-0 1/1 Running 0 67m
mlflow-minio-0 1/1 Running 0 66m
minio-0 0/1 CrashLoopBackOff 5 (105s ago) 6m24s$ microk8s.kubectl logs -n kubeflow minio-0
Defaulted container "minio" out of: minio, juju-pod-init (init)
ERROR Unable to validate credentials inherited from the shell environment: Invalid credentials
> Please provide correct credentials
HINT:
Access key length should be at least 3, and secret key length at least 8 characters@orfeas-k I just spent a few hours having to debug this with trial-and-error, because juju debug-log doesn't help at all :) Could you add some check in the charm, and set it to Blocked with a nice message?
Thank you for reporting us your feedback!
The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6057.
This message was autogenerated
The bug still persists. I tried to configure the secret-key as minio and it changed to error state without any warning.
I was able to verify the reason of the error in the logs as it had a hint there.
Before configuring minio secret-key -
After configuring minio secret-key - juju config minio secret-key=minio
Destroying the model and redeploying the bundle seems to be the only way out. I tried changing the minio secret-key to minio1234 but it doesn't budge. I also tried to restart the node by deleting the minio-0 pod but it returns back to the error state.
These are some logs and metrics that might help:
kubectl logs -n kubeflow minio-0
juju config minio
kubectl get po -n kubeflow | grep minio
kubectl describe po -n kubeflow minio-0
shrishtikarkera@rag-demo-jh:~$ kubectl describe po -n kubeflow minio-0
Name: minio-0
Namespace: kubeflow
Priority: 0
Service Account: default
Node: juju-df992b-0/10.128.0.17
Start Time: Thu, 24 Oct 2024 17:26:26 +0000
Labels: app.kubernetes.io/name=minio
apps.kubernetes.io/pod-index=0
controller-revision-hash=minio-6c98d6979f
statefulset.kubernetes.io/pod-name=minio-0
Annotations: apparmor.security.beta.kubernetes.io/pod: runtime/default
charm.juju.is/modified-version: 0
cni.projectcalico.org/containerID: 0edba5c721d19cab0f453b3b50bcd1b4c558f1ea99da427c976854d3079777a5
cni.projectcalico.org/podIP: 10.1.209.188/32
cni.projectcalico.org/podIPs: 10.1.209.188/32
controller.juju.is/id: b5d69926-8e8a-4363-8b01-49608c270755
model.juju.is/id: e60163c9-3b29-4c02-8129-918cf0fbec30
seccomp.security.beta.kubernetes.io/pod: docker/default
unit.juju.is/id: minio/0
Status: Running
IP: 10.1.209.188
IPs:
IP: 10.1.209.188
Controlled By: StatefulSet/minio
Init Containers:
juju-pod-init:
Container ID: containerd://59b404417f50ebd8d4dfe780b3a77279a680f37a3776d5df50d53ba34e5356c1
Image: docker.io/jujusolutions/jujud-operator:3.5.4
Image ID: docker.io/jujusolutions/jujud-operator@sha256:c00558be1d56a960451686327a422aaa766b1024730126600b5d79ad9ea10b84
Port: <none>
Host Port: <none>
Command:
/bin/sh
Args:
-c
export JUJU_DATA_DIR=/var/lib/juju
export JUJU_TOOLS_DIR=$JUJU_DATA_DIR/tools
mkdir -p $JUJU_TOOLS_DIR
cp /opt/jujud $JUJU_TOOLS_DIR/jujud
initCmd=$($JUJU_TOOLS_DIR/jujud help commands | grep caas-unit-init)
if test -n "$initCmd"; then
exec $JUJU_TOOLS_DIR/jujud caas-unit-init --debug --wait;
else
exit 0
fi
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 24 Oct 2024 17:26:28 +0000
Finished: Thu, 24 Oct 2024 17:26:41 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/lib/juju from juju-data-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6glbp (ro)
Containers:
minio:
Container ID: containerd://1000e74102d4c18c555f77bbb5e2ce010a24891e68b1c24b5e7b889a998e1474
Image: registry.jujucharms.com/charm/81j63o4a2ldarn1umc22iyjz1q9l9g0sx5b8j/oci-image@sha256:220b31a68d3264f53a746a364207f28868887a7c62c61cc650fd52d8e557641a
Image ID: registry.jujucharms.com/charm/81j63o4a2ldarn1umc22iyjz1q9l9g0sx5b8j/oci-image@sha256:220b31a68d3264f53a746a364207f28868887a7c62c61cc650fd52d8e557641a
Ports: 9000/TCP, 9001/TCP
Host Ports: 0/TCP, 0/TCP
Args:
server
/data
--certs-dir
/minio/.minio/certs
--console-address
:9001
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 24 Oct 2024 17:37:33 +0000
Finished: Thu, 24 Oct 2024 17:37:33 +0000
Ready: False
Restart Count: 7
Environment Variables from:
minio-secret Secret Optional: false
Environment:
MINIO_PROMETHEUS_AUTH_TYPE: public
configmap-hash: 654cf2f1d31af8f2f86f275ea9f423a05743a81a2bfdfd055048c1cad270e388
Mounts:
/data from minio-data-75d42bd0 (rw)
/minio/.minio/certs/CAs from ssl-ca (rw)
/usr/bin/juju-exec from juju-data-dir (rw,path="tools/jujud")
/var/lib/juju from juju-data-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6glbp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
minio-data-75d42bd0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: minio-data-75d42bd0-minio-0
ReadOnly: false
juju-data-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
ssl-ca:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-6glbp:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/arch=amd64
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 4m36s (x46 over 14m) kubelet Back-off restarting failed container minio in pod minio-0_kubeflow(7808ee0c-01f4-42f8-8dae-adfb8354568e)
Hey @ShrishtiKarkera, I think this should be due to not having backported this fix. I 'll check with the team and get back to you.