Pods fail to initialize when using azure blob store for logs
anirao opened this issue · 1 comments
anirao commented
Checks
- I have checked for existing issues.
- This report is about the
User-Community Airflow Helm Chart
.
Chart Version
8.8..0
Kubernetes Version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.10
Helm Version
version.BuildInfo{Version:"v3.13.1", GitCommit:"3547a4b5bf5edb5478ce352e18858d8a552a4110", GitTreeState:"clean", GoVersion:"go1.20.8"}
Description
When I use an azure blob store pvc, check-db on all airflow pods fails and the pods do not run.
I can confirm that my pvc configuration is okay because I can use it for non-airflow images, mount volumes and write to it.
Relevant Logs
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/local/lib/python3.9/pathlib.py", line 1323, in mkdir
self._accessor.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2023-12-12'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/logging/config.py", line 564, in configure
handler = self.configure_handler(handlers[name])
File "/usr/local/lib/python3.9/logging/config.py", line 745, in configure_handler
result = factory(**kwargs)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/log/file_processor_handler.py", line 49, in __init__
Path(self._get_log_directory()).mkdir(parents=True, exist_ok=True)
File "/usr/local/lib/python3.9/pathlib.py", line 1332, in mkdir
if not exist_ok or not self.is_dir():
File "/usr/local/lib/python3.9/pathlib.py", line 1439, in is_dir
return S_ISDIR(self.stat().st_mode)
File "/usr/local/lib/python3.9/pathlib.py", line 1232, in stat
return self._accessor.stat(self)
PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/2023-12-12'
Custom Helm Values
USER-SUPPLIED VALUES:
airflow:
config:
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "True"
HTTP_PROXY: <proxy>
HTTPS_PROXY: <proxy>
connections: []
executor: KubernetesExecutor
extraEnv:
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
key: fernet-key
name: airflow-secrets
namespace: airflow-cluster
- name: AIRFLOW__WEBSERVER__SECRET_KEY
valueFrom:
secretKeyRef:
key: webserver-secret-key
name: airflow-secrets
namespace: airflow-cluster
extraPipPackages: []
extraVolumeMounts:
- mountPath: /opt/airflow/cfgdags
name: dags-volume
extraVolumes:
- configMap:
name: airflow-dags-config
optional: true
name: dags-volume
fernetKey: ""
image:
pullPolicy: Always
pullSecret: <>
repository: apache-airflow
tag: 2.6.1-python3.9
kubernetesPodTemplate:
extraPipPackages: []
extraVolumeMounts: []
extraVolumes: []
image:
pullPolicy: Always
repository: apache-airflow
tag: 2.6.1-python3.9
resources: {}
stringOverride: ""
legacyCommands: false
pools: []
resources: {}
users:
- email: admin@example.com
firstName: admin
lastName: admin
password: admin
role: Admin
username: admin
variables: []
webserverSecretKey: ""
dags:
gitSync:
enabled: false
path: /opt/airflow/cfgdags
persistence:
enabled: false
externalDatabase:
authType: scram-sha-256
database: airflow_db
host: ""
namespace: airflow-cluster
password: postgres-password
port: 5432
serverSSL:
mode: verify-ca
type: postgres
user: postgres-username
externalRedis:
host: localhost
extraManifests: []
flower:
enabled: false
ingress:
enabled: true
web:
host: <host>
ingressClassName: nginx
logs:
path: /opt/airflow/logs
persistence:
enabled: true
existingClaim: airflow-logs-pvc
pgbouncer:
enabled: true
image:
pullSecret: <>
repository: airflow-helm/pgbouncer
tag: 1.18.0-patch.1
resources: {}
postgresql:
enabled: true
image:
pullPolicy: IfNotPresent
pullSecrets:
- <>
registry: <>
repository: airflow-helm/postgresql-bitnami
tag: 11.16-patch.0
persistence:
enabled: true
size: 4Gi
storageClass: ""
redis:
enabled: false
scheduler:
image:
pullPolicy: Always
pullSecret: <>
repository: apache-airflow
tag: 2.6.1-python3.9
livenessProbe:
enabled: true
taskCreationCheck:
enabled: false
schedulerAgeBeforeCheck: 180
thresholdSeconds: 300
logCleanup:
enabled: false
retentionMinutes: 21600
replicas: 1
resources:
limits:
cpu: 1000m
memory: 1G
requests:
cpu: 500m
memory: 512Mi
serviceAccount:
annotations: {}
create: true
name: ""
triggerer:
capacity: 1000
enabled: true
image:
pullPolicy: Always
pullSecret: <>
repository: apache-airflow
tag: 2.6.1-python3.9
replicas: 1
resources:
limits:
cpu: 1000m
memory: 1G
requests:
cpu: 500m
memory: 512Mi
web:
image:
pullPolicy: Always
pullSecret: <>
repository: apache-airflow
tag: 2.6.1-python3.9
replicas: 1
resources:
limits:
cpu: 1000m
memory: 1G
requests:
cpu: 500m
memory: 512Mi
service:
externalPort: 8080
type: ClusterIP
webserverConfig:
existingSecret: ""
stringOverride: |
from airflow import configuration as conf
from flask_appbuilder.security.manager import AUTH_DB
# the SQLAlchemy connection string
SQLALCHEMY_DATABASE_URI = conf.get("core", "SQL_ALCHEMY_CONN")
# use embedded DB for auth
AUTH_TYPE = AUTH_DB
workers:
enabled: false
logCleanupEnabled: false