minio revisions>57 cannot be deployed in charmed kubernetes
Closed this issue · 6 comments
Observed behaviour
minio ckf-1.6/beta hangs in a WaitingStatus for a long time and the storage that is attached to the unit remains in a pending status also. This causes minio to never be active.
juju status
minio/0* waiting idle waiting for container
Steps to reproduce
juju add-model minio-test
juju deploy minio --channel ckf-1.6/beta
juju status
Environment
- Charmed Kubernetes 1.22 on AWS
- RBAC and Metallb enabled
- Node constraints (kubernetes workers): kubernetes-worker cores=8 mem=32G root-disk=100G
Workaround
Remove the application and deploy an older version
juju remove-application minio
juju deploy minio --channel latest/stable
I am unable to reproduce on microk8s 1.22, ckf-1.6/beta goes to active for me and has an attached PVC.
I wonder if this is a charmed k8s thing. Do you have a default storage class? If you haven't already, try inspecting the pvcs and storageclass and see if there's something wrong there
Yeah this works for me as well in microk8s 1.23. This feels like something specific to charmed k8s. If it is a storage class thing though I have no idea why some minios would work and others would not, unless something has changed in juju?
I checked the storage class, nothing seems off.
Here are my findings
ubuntu@charm-dev:~$ juju status --storage
Model Controller Cloud/Region Version SLA Timestamp
minio-test juju-aws charmedk8s/default 2.9.33 unsupported 14:06:21-05:00
App Version Status Scale Charm Channel Rev Address Exposed Message
minio res:oci-image@1755999 waiting 1 minio ckf-1.6/beta 95 10.152.183.244 no waiting for container
Unit Workload Agent Address Ports Message
minio/0* waiting idle waiting for container
Storage Unit Storage ID Type Mountpoint Size Status Message
minio/0 minio-data/0 filesystem pending
ubuntu@charm-dev:~$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
cdk-ebs kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 25m
ubuntu@charm-dev:~$ kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
minio-test minio-data-7cc89ba9-minio-0 Pending cdk-ebs 39s
ubuntu@charm-dev:~$ kubectl describe pvc minio-data-7cc89ba9-minio-0 -nminio-test
Name: minio-data-7cc89ba9-minio-0
Namespace: minio-test
StorageClass: cdk-ebs
Status: Pending
Volume:
Labels: app.kubernetes.io/managed-by=juju
app.kubernetes.io/name=minio
storage.juju.is/name=minio-data
Annotations: controller.juju.is/id: da23ae37-5f6d-444d-8c5c-2fba9890bf22
juju-storage-owner: minio
model.juju.is/id: bb0e61b7-1b5f-4e4a-8f70-8490d70b1498
storage.juju.is/name: minio-data
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal WaitForFirstConsumer <invalid> (x5 over <invalid>) persistentvolume-controller waiting for first consumer to be created before binding
ubuntu@charm-dev:~$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx-kubernetes-worker default-http-backend-kubernetes-worker-cd9b77777-9btb7 1/1 Running 0 30m
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-9g2qp 1/1 Running 0 30m
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-kxsmj 1/1 Running 0 23m
ingress-nginx-kubernetes-worker nginx-ingress-controller-kubernetes-worker-qxh7j 1/1 Running 0 28m
kube-system calico-kube-controllers-5f6798768b-4kjg5 1/1 Running 0 30m
kube-system coredns-6f867cd986-6lvnr 1/1 Running 0 32m
kube-system kube-state-metrics-7799879d89-mlr8b 1/1 Running 0 32m
kube-system metrics-server-v0.5.0-6445c586d6-5kz6k 2/2 Running 1 (23m ago) 24m
kubernetes-dashboard dashboard-metrics-scraper-8458d7fdf6-rsm7f 1/1 Running 0 32m
kubernetes-dashboard kubernetes-dashboard-5784589f96-dsqcw 1/1 Running 0 32m
minio-test minio-operator-0 1/1 Running 0 8m34s
minio-test modeloperator-5dfb74dd8b-7v9lz 1/1 Running 0 8m59s
ubuntu@charm-dev:~$
juju debug-log
ubuntu@charm-dev:~$ juju debug-log --replay controller-0: 14:05:14 INFO juju.worker.apicaller [bb0e61] "machine-0" successfully connected to "localhost:17070" controller-0: 14:05:14 INFO juju.worker.logforwarder config change - log forwarding not enabled controller-0: 14:05:14 INFO juju.worker.logger logger worker started controller-0: 14:05:14 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for minio-test (bb0e61b7-1b5f-4e4a-8f70-8490d70b1498) controller-0: 14:05:14 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for minio-test (bb0e61b7-1b5f-4e4a-8f70-8490d70b1498) controller-0: 14:05:36 INFO juju.worker.caasapplicationprovisioner.runner start "minio" controller-0: 14:05:39 INFO juju.worker.caasprovisioner started operator for application "minio" application-minio: 14:05:41 INFO juju.cmd running jujud [2.9.33 e83d2a73f904080c5cdf4aaed2821abd4f58253a gc go1.18.5] application-minio: 14:05:41 DEBUG juju.cmd args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=minio", "--debug"} application-minio: 14:05:41 DEBUG juju.agent read agent config, format "2.0" application-minio: 14:05:41 INFO juju.worker.upgradesteps upgrade steps for 2.9.33 have already been run. application-minio: 14:05:41 INFO juju.cmd.jujud caas operator application-minio start (2.9.33 [gc]) application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2022-08-23 19:05:41.080298513 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "clock" manifold worker started at 2022-08-23 19:05:41.081384086 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2022-08-23 19:05:41.081609017 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-minio" application-minio: 14:05:41 DEBUG juju.worker.dependency "agent" manifold worker started at 2022-08-23 19:05:41.081759277 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker completed successfully application-minio: 14:05:41 DEBUG juju.worker.introspection stats worker now serving application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2022-08-23 19:05:41.090215139 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.apicaller connecting with old password application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2022-08-23 19:05:41.092658526 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2022-08-23 19:05:41.092976327 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2022-08-23 19:05:41.104384422 +0000 UTC application-minio: 14:05:41 DEBUG juju.api successfully dialed "wss://172.31.17.77:17070/model/bb0e61b7-1b5f-4e4a-8f70-8490d70b1498/api" application-minio: 14:05:41 INFO juju.api connection established to "wss://172.31.17.77:17070/model/bb0e61b7-1b5f-4e4a-8f70-8490d70b1498/api" application-minio: 14:05:41 INFO juju.worker.apicaller [bb0e61] "application-minio" successfully connected to "172.31.17.77:17070" application-minio: 14:05:41 DEBUG juju.api RPC connection died application-minio: 14:05:41 DEBUG juju.worker.dependency "api-caller" manifold worker completed successfully application-minio: 14:05:41 DEBUG juju.worker.apicaller connecting with old password application-minio: 14:05:41 DEBUG juju.api successfully dialed "wss://3.101.105.248:17070/model/bb0e61b7-1b5f-4e4a-8f70-8490d70b1498/api" application-minio: 14:05:41 INFO juju.api connection established to "wss://3.101.105.248:17070/model/bb0e61b7-1b5f-4e4a-8f70-8490d70b1498/api" application-minio: 14:05:41 INFO juju.worker.apicaller [bb0e61] "application-minio" successfully connected to "3.101.105.248:17070" application-minio: 14:05:41 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2022-08-23 19:05:41.147930424 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker completed successfully application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2022-08-23 19:05:41.157414467 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2022-08-23 19:05:41.158587465 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2022-08-23 19:05:41.158667493 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2022-08-23 19:05:41.158725745 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2022-08-23 19:05:41.158782836 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully application-minio: 14:05:41 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2022-08-23 19:05:41.160326745 +0000 UTC application-minio: 14:05:41 INFO juju.worker.caasupgrader abort check blocked until version event received application-minio: 14:05:41 DEBUG juju.worker.caasupgrader current agent binary version: 2.9.33 application-minio: 14:05:41 INFO juju.worker.caasupgrader unblocking abort check application-minio: 14:05:41 INFO juju.worker.migrationminion migration phase is now: NONE application-minio: 14:05:41 DEBUG juju.worker.logger initial log config: "=DEBUG" application-minio: 14:05:41 DEBUG juju.worker.dependency "logging-config-updater" manifold worker started at 2022-08-23 19:05:41.174569061 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "proxy-config-updater" manifold worker started at 2022-08-23 19:05:41.17469002 +0000 UTC application-minio: 14:05:41 INFO juju.worker.logger logger worker started application-minio: 14:05:41 DEBUG juju.worker.dependency "api-address-updater" manifold worker started at 2022-08-23 19:05:41.174742387 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.dependency "charm-dir" manifold worker started at 2022-08-23 19:05:41.174777148 +0000 UTC application-minio: 14:05:41 DEBUG juju.worker.logger reconfiguring logging from "=DEBUG" to "=INFO" application-minio: 14:05:41 DEBUG juju.worker.dependency "hook-retry-strategy" manifold worker started at 2022-08-23 19:05:41.196472077 +0000 UTC application-minio: 14:05:41 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" application-minio: 14:05:41 INFO juju.worker.caasoperator.charm downloading ch:amd64/focal/minio-95 from API server application-minio: 14:05:41 INFO juju.downloader downloading from ch:amd64/focal/minio-95 application-minio: 14:05:41 INFO juju.downloader download complete ("ch:amd64/focal/minio-95") application-minio: 14:05:41 INFO juju.downloader download verified ("ch:amd64/focal/minio-95") application-minio: 14:05:47 INFO juju.worker.caasoperator operator "minio" started application-minio: 14:05:47 INFO juju.worker.caasoperator.runner start "minio/0" application-minio: 14:05:47 INFO juju.worker.leadership minio/0 promoted to leadership of minio application-minio: 14:05:47 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-minio-0 application-minio: 14:05:47 INFO juju.worker.caasoperator.uniter.minio/0 unit "minio/0" started application-minio: 14:05:47 INFO juju.worker.caasoperator.uniter.minio/0 resuming charm install application-minio: 14:05:47 INFO juju.worker.caasoperator.uniter.minio/0.charm downloading ch:amd64/focal/minio-95 from API server application-minio: 14:05:47 INFO juju.downloader downloading from ch:amd64/focal/minio-95 application-minio: 14:05:47 INFO juju.downloader download complete ("ch:amd64/focal/minio-95") application-minio: 14:05:47 INFO juju.downloader download verified ("ch:amd64/focal/minio-95") application-minio: 14:05:54 INFO juju.worker.caasoperator.uniter.minio/0 hooks are retried true application-minio: 14:05:54 INFO juju.worker.caasoperator.uniter.minio/0 found queued "install" hook application-minio: 14:05:55 INFO unit.minio/0.juju-log Running legacy hooks/install. application-minio: 14:05:56 WARNING unit.minio/0.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion. application-minio: 14:05:58 INFO unit.minio/0.juju-log SSL: No secret specified in charm config. Proceeding without SSL. application-minio: 14:06:00 INFO juju.worker.caasoperator.uniter.minio/0.operation ran "install" hook (via hook dispatching script: dispatch) application-minio: 14:06:00 INFO juju.worker.caasoperator.uniter.minio/0 found queued "leader-elected" hook application-minio: 14:06:01 WARNING unit.minio/0.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion. application-minio: 14:06:03 INFO unit.minio/0.juju-log SSL: No secret specified in charm config. Proceeding without SSL. application-minio: 14:06:04 INFO juju.worker.caasoperator.uniter.minio/0.operation ran "leader-elected" hook (via hook dispatching script: dispatch) application-minio: 14:06:05 WARNING unit.minio/0.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion. application-minio: 14:06:07 INFO unit.minio/0.juju-log SSL: No secret specified in charm config. Proceeding without SSL. application-minio: 14:06:08 INFO juju.worker.caasoperator.uniter.minio/0.operation ran "config-changed" hook (via hook dispatching script: dispatch) application-minio: 14:06:08 INFO juju.worker.caasoperator.uniter.minio/0 found queued "start" hook application-minio: 14:06:09 INFO unit.minio/0.juju-log Running legacy hooks/start. application-minio: 14:06:09 WARNING unit.minio/0.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion. application-minio: 14:06:11 INFO juju.worker.caasoperator.uniter.minio/0.operation ran "start" hook (via hook dispatching script: dispatch) application-minio: 14:11:34 WARNING unit.minio/0.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion. application-minio: 14:11:36 INFO juju.worker.caasoperator.uniter.minio/0.operation ran "update-status" hook (via hook dispatching script: dispatch) application-minio: 14:16:05 WARNING unit.minio/0.juju-log 0 containers are present in metadata.yaml and refresh_event was not specified. Defaulting to update_status. Metrics IP may not be set in a timely fashion. application-minio: 14:16:07 INFO juju.worker.caasoperator.uniter.minio/0.operation ran "update-status" hook (via hook dispatching script: dispatch)
After closer inspection to minio (in particular the minio StatefulSet), the following appears to be the reason why the charm cannot be deployed.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 54m statefulset-controller create Claim minio-data-7cc89ba9-minio-0 Pod minio-0 in StatefulSet minio success
Warning FailedCreate 2m21s (x28 over 54m) statefulset-controller create Pod minio-0 in StatefulSet minio failed error: Pod "minio-0" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy
Turns out the kubernetes.securityContext.privileged setting, added by this commit is conflicting with my cluster. Turning that to false is a good workaround.
On microk8s the kube-apiserver is started with --allow-privileged=true by default which allows for that security context, while it's disabled by default in charmed k8s.
I wonder if it's required for the recently added SSL support. If so, allow-privileged config option should be set to true in kubernetes-master. But otherwise we should remove the security context as you explained in #70.
@jardon is there a reason for keeping it?
@natalian98 I have asked @jardon offline and he agreed there is no need for the privileged setting. We can close this issue with #70 . Thanks for checking!