error: a container name must be specified for pod airflow-scheduler-0, choose one of: [scheduler git-sync scheduler-log-groomer]...
giacomorognoni opened this issue · 3 comments
giacomorognoni commented
Checks
- I have checked for existing issues.
- This report is about the
User-Community Airflow Helm Chart.
Chart Version
1.9.0
Kubernetes Version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.10-gke.2300", GitCommit:"1d7ae0799b40b0cd95502e3a5e698db62572e341", GitTreeState:"clean", BuildDate:"2023-02-22T09:28:49Z", GoVersion:"go1.19.6 X:boringcrypto", Compiler:"gc", Platform:"linux/amd64"}Helm Version
version.BuildInfo{Version:"v3.11.3", GitCommit:"66a969e7cc08af2377d055f4e6283c33ee84be33", GitTreeState:"clean", GoVersion:"go1.20.3"}Description
I am trying to deploy a customised airflow instance in kubernetes and in doing so have updated the values.yml file to support git sync using ssh key. However in doing so I receive the following error:
error: a container name must be specified for pod airflow-scheduler-0, choose one of: [scheduler git-sync scheduler-log-groomer] or one of the init containers: [wait-for-airflow-migrations git-sync-init]
Any idea why that might be the case?
Relevant Logs
Name: airflow-scheduler-0
Namespace: airflow
Priority: 0
Node: <none>
Labels: component=scheduler
controller-revision-hash=airflow-scheduler-6786988ff9
release=airflow
statefulset.kubernetes.io/pod-name=airflow-scheduler-0
tier=airflow
Annotations: checksum/airflow-config: 9695dc3ae68929e3ade8c5ac0d6884375bd30c9277da6ba1c9e5dc9ab5881158
checksum/extra-configmaps: 2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
checksum/extra-secrets: bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
checksum/metadata-secret: a5c9ef957c6afd5660a6f087e3ebf8a97991086c2f3ce203ca0c541366ddf63a
checksum/pgbouncer-config-secret: da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
checksum/result-backend-secret: 74e3e99feee51248d44224665d60fab543dd6b25ba95f04e6fcb0e5758342056
cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status: Pending
IP:
IPs: <none>
Controlled By: StatefulSet/airflow-scheduler
Init Containers:
wait-for-airflow-migrations:
Image: apache/airflow:2.5.3
Port: <none>
Host Port: <none>
Args:
airflow
db
check-migrations
--migration-wait-timeout=60
Environment:
AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in secret 'airflow-fernet-key'> Optional: false
AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false
AIRFLOW_CONN_AIRFLOW_DB: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false
AIRFLOW__WEBSERVER__SECRET_KEY: <set to the key 'webserver-secret-key' in secret 'airflow-webserver-secret-key'> Optional: false
Mounts:
/opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
/opt/airflow/config/airflow_local_settings.py from config (ro,path="airflow_local_settings.py")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n8nh7 (ro)
git-sync-init:
Image: registry.k8s.io/git-sync/git-sync:v3.6.3
Port: <none>
Host Port: <none>
Environment:
GIT_SSH_KEY_FILE: /etc/git-secret/ssh
GIT_SYNC_SSH: true
GIT_KNOWN_HOSTS: false
GIT_SYNC_REV: HEAD
GIT_SYNC_BRANCH: master
GIT_SYNC_REPO: XXX
GIT_SYNC_DEPTH: 1
GIT_SYNC_ROOT: /git
GIT_SYNC_DEST: repo
GIT_SYNC_ADD_USER: true
GIT_SYNC_WAIT: 5
GIT_SYNC_MAX_SYNC_FAILURES: 0
GIT_SYNC_ONE_TIME: true
Mounts:
/etc/git-secret/ssh from git-sync-ssh-key (ro,path="gitSshKey")
/git from dags (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n8nh7 (ro)
Containers:
scheduler:
Image: apache/airflow:2.5.3
Port: 8793/TCP
Host Port: 0/TCP
Args:
bash
-c
exec airflow scheduler
Liveness: exec [sh -c CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
airflow jobs check --job-type SchedulerJob --local
] delay=10s timeout=20s period=60s #success=1 #failure=5
Environment:
AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in secret 'airflow-fernet-key'> Optional: false
AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false
AIRFLOW_CONN_AIRFLOW_DB: <set to the key 'connection' in secret 'airflow-airflow-metadata'> Optional: false
AIRFLOW__WEBSERVER__SECRET_KEY: <set to the key 'webserver-secret-key' in secret 'airflow-webserver-secret-key'> Optional: false
Mounts:
/opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
/opt/airflow/config/airflow_local_settings.py from config (ro,path="airflow_local_settings.py")
/opt/airflow/dags from dags (ro)
/opt/airflow/logs from logs (rw)
/opt/airflow/pod_templates/pod_template_file.yaml from config (ro,path="pod_template_file.yaml")
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n8nh7 (ro)
git-sync:
Image: registry.k8s.io/git-sync/git-sync:v3.6.3
Port: <none>
Host Port: <none>
Environment:
GIT_SSH_KEY_FILE: /etc/git-secret/ssh
GIT_SYNC_SSH: true
GIT_KNOWN_HOSTS: false
GIT_SYNC_REV: HEAD
GIT_SYNC_BRANCH: master
GIT_SYNC_REPO: XXX
GIT_SYNC_DEPTH: 1
GIT_SYNC_ROOT: /git
GIT_SYNC_DEST: repo
GIT_SYNC_ADD_USER: true
GIT_SYNC_WAIT: 5
GIT_SYNC_MAX_SYNC_FAILURES: 0
Mounts:
/etc/git-secret/ssh from git-sync-ssh-key (ro,path="gitSshKey")
/git from dags (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n8nh7 (ro)
scheduler-log-groomer:
Image: apache/airflow:2.5.3
Port: <none>
Host Port: <none>
Args:
bash
/clean-logs
Environment:
AIRFLOW__LOG_RETENTION_DAYS: 15
Mounts:
/opt/airflow/logs from logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n8nh7 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
logs:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: logs-airflow-scheduler-0
ReadOnly: false
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-airflow-config
Optional: false
dags:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
git-sync-ssh-key:
Type: Secret (a volume populated by a Secret)
SecretName: git-sync-ssh-key
Optional: false
kube-api-access-n8nh7:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 78s (x2 over 4m14s) default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the conditionCustom Helm Values
No response
giacomorognoni commented
Happy to share any further information I have forgotten and thank you for your help!
thesuperzapper commented
@giacomorognoni I think you might be using the official chart from the apache/airflow repo, as version 1.9.0 is not a version of the User-Community Airflow Chart.
So I think you should either:
- Raise your issue on the
apache/airflowrepo - If you want, try using the chart from this repo (
airflow-helm/charts) and see if you have any problems
giacomorognoni commented
Hello @thesuperzapper , thank you very much and sorry for the confusion, will give the correct charts a go.