Back-off restarting failed container check-db
Opened this issue · 3 comments
stephen-lazarionok commented
Checks
- I have checked for existing issues.
- This report is about the
User-Community Airflow Helm Chart
.
Chart Version
8.9.0
Kubernetes Version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:47:25Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.10", GitCommit:"0fa26aea1d5c21516b0d96fea95a77d8d429912e", GitTreeState:"clean", BuildDate:"2024-01-17T13:38:41Z", GoVersion:"go1.20.13", Compiler:"gc", Platform:"linux/amd64"}
Helm Version
version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}
Description
I was trying to redeploy Airflow but it
- keep Airflow pods created keep restarting with "Back-off restarting failed container check-db "
- eventually I got "{db.py:1758} INFO - Connection successful."
Relevant Logs
airflow-db-migrations-6867d67f79-b5hl9 0/1 Init:0/1 7 (5m14s ago) 18m
airflow-scheduler-59fd578c45-xv7hl 0/2 Init:0/2 7 (5m26s ago) 18m
airflow-scheduler-65c666bcb9-75rwj 2/2 Running 0 145m
airflow-sync-connections-7b99456945-clh52 0/1 Init:0/2 7 (5m21s ago) 18m
airflow-sync-pools-8967f946d-mwt4h 1/1 Running 0 18m
airflow-sync-users-7cd5c575cb-5v9gg 0/1 Init:0/2 7 (5m30s ago) 18m
airflow-web-6f5fffb68d-gqss5 1/1 Running 1 (112m ago) 145m
airflow-web-cb655b569-77l4g 0/1 Init:0/2 7 (5m20s ago) 18m
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned airflow2/airflow-web-cb655b569-77l4g to zing-staging-k8spool-2c16-ram-ji8a2
Normal Pulling 12m kubelet Pulling image "z/z-airflow-dags:0.45.0-staging"
Normal Pulled 12m kubelet Successfully pulled image "z/z-airflow-dags:0.45.0-staging" in 1.69428358s (1.69430606s including waiting)
Normal Created 6m46s (x5 over 12m) kubelet Created container check-db
Normal Started 6m46s (x5 over 12m) kubelet Started container check-db
Normal Pulled 6m46s (x4 over 11m) kubelet Container image "z/z-airflow-dags:0.45.0-staging" already present on machine
Warning BackOff 2m15s (x19 over 10m) kubelet Back-off restarting failed container check-db in pod airflow-web-cb655b569-77l4g_airflow2(d2347d9f-3f7c-462d-bd7d-b18ae2c85fea)
Custom Helm Values
No response
thesuperzapper commented
@stephen-lazarionok almost certainly this is a connectivity or credentials issue between your cluster and the Postgres DB, and not an issue with the chart.
You should check the logs of the container itself (rather than just the events) to get more context as to why it's failing.
harsh0522 commented
thesuperzapper commented
@harsh0522 in your image, the postgres container is "Pending", this is probably due to some kind of PVC issue.
Check that your cluster has a default StorageClass (or use the values to set a specific one).
Also, you can confirm this by checking the events of the Postgres pod (with kubectl describe
).