Installs stalls during PG initialization

Question

Installs stalls during PG initialization

Closed this issue 2 years ago · 1 comments

$ helm version version.BuildInfo{Version:"v3.7.2", GitCommit:"663a896f4a815053445eec4153677ddc24a0a361", GitTreeState:"clean", GoVersion:"go1.16.10"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.20-eks-8c49e2", GitCommit:"8c49e2efc3cfbb7788a58025e679787daed22018", GitTreeState:"clean", BuildDate:"2021-10-17T05:13:46Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

[cloudshell-user galaxy]$ kubectl get pods
NAMESPACE NAME READY STATUS RESTARTS AGE
galaxy-prod galaxy-galaxy-galaxy-postgres-0 1/1 Running 0 18m
galaxy-prod galaxy-galaxy-postgres-5b5fcfff54-lznw6 1/1 Running 0 18m
galaxy-prod galaxy-init-db-1rntx-g4rph 0/1 Init:0/1 0 18m
galaxy-prod galaxy-init-mounts-kr7mv-db2r9 0/1 Completed 0 18m
galaxy-prod galaxy-job-0-c5779bcdf-hk2kc 0/1 Init:0/1 0 18m
galaxy-prod galaxy-nginx-8695bc94d5-t469x 1/1 Running 0 18m
galaxy-prod galaxy-web-69b6bb5fb5-bj75n 0/1 Init:0/1 0 18m
galaxy-prod galaxy-workflow-7f87d9787-nblkc 0/1 Init:0/1 0 18m

seems to be waiting on the DB, and looking into that container, i find this

$ kubectl logs galaxy-galaxy-galaxy-postgres-0 -n galaxy-prod
2022-02-16 00:38:13,642 - bootstrapping - INFO - Figuring out my environment (Google? AWS? Openstack? Local?)
2022-02-16 00:38:13,656 - bootstrapping - INFO - Looks like your running aws
2022-02-16 00:38:13,683 - bootstrapping - INFO - Configuring bootstrap
2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring pgqd
2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring log
2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring wal-e
2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring standby-cluster
2022-02-16 00:38:13,684 - bootstrapping - INFO - Configuring certificate
2022-02-16 00:38:13,685 - bootstrapping - INFO - Generating ssl certificate
2022-02-16 00:38:13,721 - bootstrapping - INFO - Configuring pgbouncer
2022-02-16 00:38:13,721 - bootstrapping - INFO - No PGBOUNCER_CONFIGURATION was specified, skipping
2022-02-16 00:38:13,721 - bootstrapping - INFO - Configuring crontab
2022-02-16 00:38:13,721 - bootstrapping - INFO - Skipping creation of renice cron job due to lack of SYS_NICE capability
2022-02-16 00:38:13,722 - bootstrapping - INFO - Configuring patroni
2022-02-16 00:38:13,730 - bootstrapping - INFO - Writing to file /run/postgres.yml
2022-02-16 00:38:13,731 - bootstrapping - INFO - Configuring pam-oauth2
2022-02-16 00:38:13,731 - bootstrapping - INFO - Writing to file /etc/pam.d/postgresql
2022-02-16 00:38:15,087 WARNING: Kubernetes RBAC doesn't allow GET access to the 'kubernetes' endpoint in the 'default' namespace. Disabling 'bypass_api_service'.
2022-02-16 00:38:15,109 INFO: No PostgreSQL configuration items changed, nothing to reload.
2022-02-16 00:38:15,112 INFO: Lock owner: galaxy-galaxy-galaxy-postgres-0; I am galaxy-galaxy-galaxy-postgres-0
2022-02-16 00:38:15,247 INFO: Leader key released
2022-02-16 00:38:15,305 INFO: released leader key voluntarily as data dir empty and currently leader
2022-02-16 00:38:15,306 INFO: Lock owner: None; I am galaxy-galaxy-galaxy-postgres-0
2022-02-16 00:38:15,306 INFO: waiting for leader to bootstrap

Not sure if that is why it is having issues with this CRD.

$ kubectl logs galaxy-galaxy-postgres-5b5fcfff54-lznw6 -n galaxy-prod
time="2022-02-16T00:37:36Z" level=info msg="Fully qualified configmap name: galaxy-prod/galaxy-galaxy-postgres"
time="2022-02-16T00:37:36Z" level=info msg="Spilo operator v1.6.3\n"
time="2022-02-16T00:37:36Z" level=info msg="Parse role bindings" pkg=controller
time="2022-02-16T00:37:36Z" level=info msg="successfully parsed" pkg=controller
time="2022-02-16T00:37:36Z" level=info msg="Listening to all namespaces" pkg=controller
time="2022-02-16T00:37:36Z" level=info msg="customResourceDefinition "postgresqls.acid.zalan.do" is already registered and will only be updated" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg="{" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ReadyWaitInterval": 3000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ReadyWaitTimeout": 30000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ResyncPeriod": 1800000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "RepairPeriod": 300000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableCRDValidation": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ResourceCheckInterval": 3000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ResourceCheckTimeout": 600000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodLabelWaitTimeout": 600000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodDeletionWaitTimeout": 600000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SpiloRunAsUser": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SpiloRunAsGroup": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SpiloFSGroup": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodPriorityClassName": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ClusterDomain": "cluster.local"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SpiloPrivileged": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SpiloAllowPrivilegeEscalation": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "AdditionalPodCapabilities": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ClusterLabels": {" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "application": "spilo"" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" }," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "InheritedLabels": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "InheritedAnnotations": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DownscalerAnnotations": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ClusterNameLabel": "cluster-name"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DeleteAnnotationDateKey": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DeleteAnnotationNameKey": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodRoleLabel": "spilo-role"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodToleration": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DefaultCPURequest": "100m"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DefaultMemoryRequest": "100Mi"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DefaultCPULimit": "1"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DefaultMemoryLimit": "500Mi"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "MinCPULimit": "250m"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "MinMemoryLimit": "250Mi"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodEnvironmentConfigMap": "/"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodEnvironmentSecret": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "NodeReadinessLabel": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "MaxInstances": -1," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "MinInstances": -1," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ShmVolume": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SecretNameTemplate": "{username}.{cluster}.credentials.{tprkind}.{tprgroup}"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PamRoleName": "zalandos"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PamConfiguration": "https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "TeamsAPIUrl": "https://teams.example.com/api/\"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "OAuthTokenSecretName": "galaxy-prod/postgresql-operator"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "InfrastructureRolesSecretName": "/"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "InfrastructureRoles": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "InfrastructureRolesDefs": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SuperUsername": "postgres"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ReplicationUsername": "standby"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ScalyrAPIKey": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ScalyrImage": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ScalyrServerURL": "https://upload.eu.scalyr.com\"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ScalyrCPURequest": "100m"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ScalyrMemoryRequest": "50Mi"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ScalyrCPULimit": "1"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ScalyrMemoryLimit": "500Mi"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupSchedule": "30 00 * * *"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupDockerImage": "registry.opensource.zalan.do/acid/logical-backup:v1.6.3"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupProvider": "s3"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupS3Bucket": "my-bucket-url"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupS3Region": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupS3Endpoint": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupS3AccessKeyID": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupS3SecretAccessKey": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupS3SSE": "AES256"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupGoogleApplicationCredentials": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogicalBackupJobPrefix": "logical-backup-"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "NumberOfInstances": 2," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "Schema": "pooler"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "User": "pooler"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "Image": "registry.opensource.zalan.do/acid/pgbouncer:master-16"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "Mode": "transaction"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "MaxDBConnections": 60," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ConnectionPoolerDefaultCPURequest": "500m"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ConnectionPoolerDefaultMemoryRequest": "100Mi"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ConnectionPoolerDefaultCPULimit": "1"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ConnectionPoolerDefaultMemoryLimit": "100Mi"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "WatchedNamespace": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "KubernetesUseConfigMaps": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EtcdHost": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DockerImage": "registry.opensource.zalan.do/acid/spilo-13:2.0-p7"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SidecarImages": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SidecarContainers": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodServiceAccountName": "postgres-pod"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodServiceAccountDefinition": "{\"apiVersion\":\"v1\",\"kind\":\"ServiceAccount\",\"metadata\":{\"name\":\"postgres-pod\"}}"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodServiceAccountRoleBindingDefinition": "{\"apiVersion\":\"rbac.authorization.k8s.io/v1\",\"kind\":\"RoleBinding\",\"metadata\":{\"name\":\"postgres-pod\"},\"roleRef\":{\"apiGroup\":\"rbac.authorization.k8s.io\",\"kind\":\"ClusterRole\",\"name\":\"postgres-pod\"},\"subjects\":[{\"kind\":\"ServiceAccount\",\"name\":\"postgres-pod\"}]}"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "MasterPodMoveTimeout": 1200000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DbHostedZone": "db.example.com"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "AWSRegion": "eu-central-1"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "WALES3Bucket": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "LogS3Bucket": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "KubeIAMRole": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "WALGSBucket": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "GCPCredentials": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "AdditionalSecretMount": ""," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "AdditionalSecretMountPath": "/meta/credentials"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableEBSGp3Migration": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableEBSGp3MigrationMaxSize": 1000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "DebugLogging": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableDBAccess": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableTeamsAPI": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableTeamSuperuser": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "TeamAdminRole": "admin"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "RoleDeletionSuffix": "_deleted"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableTeamMemberDeprecation": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableAdminRoleForUsers": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnablePostgresTeamCRD": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnablePostgresTeamCRDSuperusers": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableMasterLoadBalancer": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableReplicaLoadBalancer": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "CustomServiceAnnotations": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "CustomPodAnnotations": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnablePodAntiAffinity": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodAntiAffinityTopologyKey": "kubernetes.io/hostname"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "StorageResizeMode": "pvc"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableLoadBalancer": null," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ExternalTrafficPolicy": "Cluster"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "MasterDNSNameFormat": "{cluster}.{team}.{hostedzone}"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ReplicaDNSNameFormat": "{cluster}-repl.{team}.{hostedzone}"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PDBNameFormat": "postgres-{cluster}-pdb"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnablePodDisruptionBudget": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableInitContainers": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableSidecars": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "Workers": 8," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "APIPort": 8080," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "RingLogLines": 100," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ClusterHistoryEntries": 1000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "TeamAPIRoleConfiguration": {" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "log_statement": "all"" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" }," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodTerminateGracePeriod": 300000000000," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PodManagementPolicy": "ordered_ready"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "ProtectedRoles": [" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "admin"" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" ]," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "PostgresSuperuserTeams": [" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "postgres_superusers"" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" ]," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "SetMemoryRequestToLimit": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableLazySpiloUpgrade": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnablePgVersionEnvVar": true," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "EnableSpiloWalPathCompat": false," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "MajorVersionUpgradeMode": "off"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "MinimalMajorVersion": "9.5"," pkg=controller
time="2022-02-16T00:37:40Z" level=info msg=" "TargetMajorVersion": "13"" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg="}" pkg=controller
time="2022-02-16T00:37:40Z" level=debug msg="acquiring initial list of clusters" pkg=controller
time="2022-02-16T00:37:40Z" level=debug msg="added new cluster: "galaxy-prod/galaxy-galaxy-galaxy-postgres"" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg="SYNC event has been queued" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-02-16T00:37:40Z" level=info msg="there are 1 clusters running" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg="started working in background" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg="listening on :8080" pkg=apiserver
time="2022-02-16T00:37:40Z" level=info msg="ADD event has been queued" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-02-16T00:37:40Z" level=debug msg="new node has been added: /ip-10-0-2-74.us-west-2.compute.internal (aws:///us-west-2b/i-04feb62d6407c3e17)" pkg=controller
time="2022-02-16T00:37:40Z" level=info msg="syncing of the cluster started" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-02-16T00:37:40Z" level=debug msg="team API is disabled" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:40Z" level=debug msg="team API is disabled" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:40Z" level=info msg="syncing secrets" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:40Z" level=debug msg="secret galaxy-prod/galaxydbuser.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:40Z" level=debug msg="secret galaxy-prod/postgres.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:40Z" level=debug msg="secret galaxy-prod/standby.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do already exists, fetching its password" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:40Z" level=debug msg="syncing master service" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:40Z" level=debug msg="syncing replica service" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:41Z" level=debug msg="No load balancer created for the replica service" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:41Z" level=debug msg="syncing volumes using "pvc" storage resize mode" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:41Z" level=info msg="volume claims do not require changes" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:41Z" level=debug msg="syncing statefulsets" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:41Z" level=info msg="cluster's statefulset does not exist" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:41Z" level=debug msg="Generating Spilo container, environment variables" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:41Z" level=debug msg="[{SCOPE galaxy-galaxy-galaxy-postgres nil} {PGROOT /home/postgres/pgdata/pgroot nil} {POD_IP &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:status.podIP,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {POD_NAMESPACE &EnvVarSource{FieldRef:&ObjectFieldSelector{APIVersion:v1,FieldPath:metadata.namespace,},ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:nil,}} {PGUSER_SUPERUSER postgres nil} {KUBERNETES_SCOPE_LABEL cluster-name nil} {KUBERNETES_ROLE_LABEL spilo-role nil} {PGPASSWORD_SUPERUSER &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:postgres.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PGUSER_STANDBY standby nil} {PGPASSWORD_STANDBY &EnvVarSource{FieldRef:nil,ResourceFieldRef:nil,ConfigMapKeyRef:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:standby.galaxy-galaxy-galaxy-postgres.credentials.postgresql.acid.zalan.do,},Key:password,Optional:nil,},}} {PAM_OAUTH2 https://info.example.com/oauth2/tokeninfo?access_token= uid realm=/employees nil} {HUMAN_ROLE zalandos nil} {PGVERSION 13 nil} {KUBERNETES_LABELS {"application":"spilo"} nil} {SPILO_CONFIGURATION {"postgresql":{},"bootstrap":{"initdb":[{"auth-host":"md5"},{"auth-local":"trust"}],"users":{"zalandos":{"password":"","options":["CREATEDB","NOLOGIN"]}},"dcs":{}}} nil} {DCS_ENABLE_KUBERNETES_API true nil}]" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:41Z" level=debug msg="created new statefulset "galaxy-prod/galaxy-galaxy-galaxy-postgres", uid: "947d6dec-841e-409c-b349-0c6f6347cb5d"" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:37:44Z" level=debug msg="Waiting for 1 pods to become ready" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:47:41Z" level=warning msg="error while syncing cluster state: could not sync statefulsets: cluster is not ready: pod labels error: still failing after 200 retries" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=cluster
time="2022-02-16T00:47:41Z" level=error msg="could not sync cluster: could not sync statefulsets: cluster is not ready: pod labels error: still failing after 200 retries" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0
time="2022-02-16T00:47:41Z" level=info msg="recieved add event for already existing Postgres cluster" cluster-name=galaxy-prod/galaxy-galaxy-galaxy-postgres pkg=controller worker=0

Answer 1 · 2022-03-17T03:00:06.000Z

nevermind, will write my own if no one maintains this one.