I get an error CrashLoopBackOff and can't install 3scale
Closed this issue · 2 comments
pre-yein commented
Hello, I want to install 3scale in my OCP on-premise VM environment.
I tried s3 deployment.
I created threescale-registry-auth secret.
but CrashLoopBackOff error occored in pod system-mysql-1-nxzxt.
[admin@localhost ~]$ oc get pods
NAME READY STATUS RESTARTS AGE
apicast-production-1-7lkrx 0/1 Init:0/1 0 5m28s
apicast-production-1-deploy 1/1 Running 0 6m49s
apicast-staging-1-deploy 0/1 Completed 0 6m49s
apicast-staging-1-qv8tc 1/1 Running 0 5m28s
backend-cron-1-deploy 0/1 Completed 0 6m57s
backend-cron-1-fvp7z 1/1 Running 0 6m14s
backend-listener-1-4hqdk 1/1 Running 1 (4m17s ago) 6m17s
backend-listener-1-deploy 0/1 Completed 0 6m58s
backend-redis-1-deploy 0/1 Completed 0 6m56s
backend-redis-1-q5nbs 1/1 Running 0 6m22s
backend-worker-1-4q8zd 1/1 Running 0 6m22s
backend-worker-1-deploy 0/1 Completed 0 6m58s
system-app-1-deploy 1/1 Running 0 6m53s
system-app-1-hook-pre 1/1 Running 0 5m56s
system-memcache-1-deploy 0/1 Completed 0 6m58s
system-memcache-1-ng22s 1/1 Running 0 6m16s
system-mysql-1-deploy 1/1 Running 0 6m55s
system-mysql-1-nxzxt 0/1 CrashLoopBackOff 5 (39s ago) 6m3s
system-redis-1-49q5q 1/1 Running 0 6m10s
system-redis-1-deploy 0/1 Completed 0 6m55s
system-sidekiq-1-deploy 1/1 Running 0 6m54s
system-sidekiq-1-h9mjh 0/1 Init:0/1 0 5m58s
system-sphinx-1-deploy 1/1 Running 0 6m55s
system-sphinx-1-pp7f6 0/1 Init:0/1 0 5m57s
threescale-operator-controller-manager-v2-f7dd4fbc6-q4v7v 1/1 Running 1 (7m45s ago) 8m34s
zync-1-deploy 0/1 Completed 0 6m50s
zync-1-glng8 1/1 Running 0 5m32s
zync-database-1-24chm 1/1 Running 0 5m33s
zync-database-1-deploy 0/1 Completed 0 6m49s
zync-que-1-deploy 0/1 Completed 0 6m49s
zync-que-1-x5gm9 1/1 Running 5 (35s ago) 5m31s
[admin@localhost ~]$ oc describe pod system-mysql-1-nxzxt
Name: system-mysql-1-nxzxt
Namespace: precursor
Priority: 0
Node: crc-hsl9k-master-0/192.168.126.11
Start Time: Fri, 14 Jan 2022 01:59:55 -0500
Labels: app=3scale-api-management
com.redhat.component-name=system-mysql
com.redhat.component-type=application
com.redhat.component-version=
com.redhat.product-name=3scale
com.redhat.product-version=2.11
deployment=system-mysql-1
deploymentConfig=system-mysql
deploymentconfig=system-mysql
threescale_component=system
threescale_component_element=mysql
Annotations: k8s.v1.cni.cncf.io/network-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.217.0.93"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.217.0.93"
],
"default": true,
"dns": {}
}]
openshift.io/deployment-config.latest-version: 1
openshift.io/deployment-config.name: system-mysql
openshift.io/deployment.name: system-mysql-1
openshift.io/scc: restricted
Status: Running
IP: 10.217.0.93
IPs:
IP: 10.217.0.93
Controlled By: ReplicationController/system-mysql-1
Containers:
system-mysql:
Container ID: cri-o://1065c41df5d6667da4da2bd5880bd2ea67cba2083503e365753afb12e02046d6
Image: registry.redhat.io/rhscl/mysql-57-rhel7@sha256:9a781abe7581cc141e14a7e404ec34125b3e89c008b14f4e7b41e094fd3049fe
Image ID: registry.redhat.io/rhscl/mysql-57-rhel7@sha256:9a781abe7581cc141e14a7e404ec34125b3e89c008b14f4e7b41e094fd3049fe
Port: 3306/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 14 Jan 2022 02:05:08 -0500
Finished: Fri, 14 Jan 2022 02:05:16 -0500
Ready: False
Restart Count: 5
Liveness: tcp-socket :3306 delay=30s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/bin/sh -i -c MYSQL_PWD="$MYSQL_PASSWORD" mysql -h 127.0.0.1 -u $MYSQL_USER -D $MYSQL_DATABASE -e 'SELECT 1'] delay=10s timeout=5s period=30s #success=1 #failure=3
Environment:
MYSQL_USER: <set to the key 'DB_USER' in secret 'system-database'> Optional: false
MYSQL_PASSWORD: <set to the key 'DB_PASSWORD' in secret 'system-database'> Optional: false
MYSQL_DATABASE: system
MYSQL_ROOT_PASSWORD: 2imxzYfc
MYSQL_LOWER_CASE_TABLE_NAMES: 1
MYSQL_DEFAULTS_FILE: /etc/my-extra/my.cnf
Mounts:
/etc/my-extra from mysql-main-conf (rw)
/etc/my-extra.d from mysql-extra-conf (rw)
/var/lib/mysql/data from mysql-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-zv5ct (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
mysql-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: mysql-storage
ReadOnly: false
mysql-extra-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mysql-extra-conf
Optional: false
mysql-main-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: mysql-main-conf
Optional: false
kube-api-access-zv5ct:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional: <nil>
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m21s default-scheduler Successfully assigned precursor/system-mysql-1-nxzxt to crc-hsl9k-master-0
Normal AddedInterface 5m59s multus Add eth0 [10.217.0.93/23] from openshift-sdn
Warning Unhealthy 4m45s (x3 over 5m8s) kubelet Liveness probe failed: dial tcp 10.217.0.93:3306: connect: connection refused
Normal Killing 4m40s kubelet Container system-mysql failed liveness probe, will be restarted
Warning Unhealthy 4m15s (x6 over 5m33s) kubelet Readiness probe failed: sh: cannot set terminal process group (-1): Inappropriate ioctl for device
sh: no job control in this shell
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)
Warning Unhealthy 3m48s kubelet Readiness probe errored: rpc error: code = NotFound desc = container is not created or running: checking if PID of c35317e36b55dfba933959027b100caaa534f4139b8e7078f31720ae926e8dc6 is running failed: container process not found
Normal Pulled 2m54s (x4 over 5m57s) kubelet Container image "registry.redhat.io/rhscl/mysql-57-rhel7@sha256:9a781abe7581cc141e14a7e404ec34125b3e89c008b14f4e7b41e094fd3049fe" already present on machine
Normal Created 2m50s (x4 over 5m46s) kubelet Created container system-mysql
Normal Started 2m48s (x4 over 5m42s) kubelet Started container system-mysql
Warning BackOff 54s (x10 over 3m8s) kubelet Back-off restarting failed container
Can I know what was wrong? Any sugesstion?
Do I need to install MySQL on localhost?
Please help me.
Thank you.
eguzki commented
It seems to me that this can be a temporary error.
It could be that the password used is not correct. Make sure to delete 3scale secrets (from previous installations) before deploying APIManager object
kubectl delete secrets backend-internal-api backend-listener backend-redis system-app system-database system-master-apicast system-events-hook system-memcache system-recaptcha system-redis system-seed system-smtp zync
pre-yein commented
resolved. Thanks!