teamclairvoyant/airflow-maintenance-dags

Add support for Mysql Backed airflow.

amolsr opened this issue · 1 comments

[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:293} INFO - Retrieving max_execution_date from XCom
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:306} INFO - Configurations:
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:307} INFO - max_date: 2024-01-13 05:59:25.875810+00:00
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:308} INFO - enable_delete: True
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:309} INFO - session: <sqlalchemy.orm.session.Session object at 0x7f618406e910>
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:310} INFO - _db_model: <class '.models.taskinstance.TaskInstance'>
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:311} INFO - state: None
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:312} INFO - age_check_column: ColumnAssociationProxyInstance(AssociationProxy('dag_run', 'execution_date'))
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:313} INFO - keep_last: False
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:314} INFO - keep_last_filters: None
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:315} INFO - keep_last_group_by: None
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:317} INFO -
[2024-01-23, 11:30:29 IST] {airflow-db-cleanup.py:319} INFO - Running Cleanup Process...
[2024-01-23, 11:30:29 IST] {taskinstance.py:1775} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 175, in execute
return_value = self.execute_callable()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 192, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/opt/airflow/dags/maintainance/airflow-db-cleanup.py", line 324, in cleanup_function
logging.info("INITIAL QUERY : " + str(query))
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2946, in str
return str(statement.compile(bind))
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 503, in compile
return self._compiler(dialect, *kw)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 567, in _compiler
return dialect.statement_compiler(dialect, self, kw)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 809, in init
Compiled.init(self, dialect, statement, kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 464, in init
self.string = self.process(self.statement, compile_kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 499, in process
return obj._compiler_dispatch(self, kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 82, in _compiler_dispatch
return meth(self, kw)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/compiler.py", line 3406, in visit_select
select_stmt, self, kwargs
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/base.py", line 510, in create_for_statement
return klass.create_for_statement(statement, compiler, kw)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/context.py", line 736, in create_for_statement
opt.process_compile_state(self)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 267, in process_compile_state
and not compile_state.compile_options._for_refresh_state,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 924, in _process
raiseerr,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/strategy_options.py", line 1085, in _bind_loader
"mapper option expects " "string key or list of attributes"
sqlalchemy.exc.ArgumentError: mapper option expects string key or list of attributes
[2024-01-23, 11:30:29 IST] {taskinstance.py:1326} INFO - Marking task as FAILED. dag_id=
-db-cleanup, task_id=cleanup_TaskInstance, execution_date=20240122T000000, start_date=20240123T060029, end_date=20240123T060029
[2024-01-23, 11:30:29 IST] {logging_mixin.py:137} WARNING - /home/
/.local/lib/python3.7/site-packages/
/models/param.py:74 RemovedInAirflow3Warning: The use of non-json-serializable params is deprecated and will be removed in a future release
[2024-01-23, 11:30:29 IST] {logging_mixin.py:137} WARNING - /home/
/.local/lib/python3.7/site-packages/
/utils/email.py:152 RemovedInAirflow3Warning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
[2024-01-23, 11:30:29 IST] {email.py:268} INFO - Email alerting: attempt 1
[2024-01-23, 11:30:30 IST] {email.py:280} INFO - Sent an alert email to ['amol.saini@infoedge.com', 'sumit.k1@gmail.com']
[2024-01-23, 11:30:30 IST] {standard_task_runner.py:105} ERROR - Failed to execute job 393423 for task cleanup_TaskInstance (mapper option expects string key or list of attributes; 32510)
[2024-01-23, 11:30:30 IST] {local_task_job.py:212} INFO - Task exited with return code 1
[2024-01-23, 11:30:30 IST] {taskinstance.py:2585} INFO - 0 downstream tasks scheduled from follow-on schedule check