Debug failure reason for s3_to_databricks DAG

[vatsrahul1001]

Currently, our 's3_to_databricks' DAG is failing in the integration test. We should investigate this issue, as frequent failures may result in the leakage of a valid bug

Deployment link

Logs

[2023-12-18, 07:09:33 UTC] {base.py:83} INFO - Using connection ID 'aws_conn' for task execution.
[2023-12-18, 07:09:34 UTC] {adapter.py:122} WARNING - Failed to emit OpenLineage event of id ebf1b197-3ca4-3186-bc1e-1c5c56202b89
[2023-12-18, 07:09:34 UTC] {taskinstance.py:2699} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/airflow/models/taskinstance.py", line 433, in _execute_task
result = execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/astro/sql/operators/load_file.py", line 131, in execute
return self.load_data(input_file=self.input_file, context=context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/astro/sql/operators/load_file.py", line 136, in load_data
return self.load_data_to_table(input_file, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/astro/sql/operators/load_file.py", line 155, in load_data_to_table
database.load_file_to_table(
File "/usr/local/lib/python3.11/site-packages/astro/databases/databricks/delta.py", line 151, in load_file_to_table
load_file_to_delta(
File "/usr/local/lib/python3.11/site-packages/astro/databases/databricks/load_file/load_file_job.py", line 88, in load_file_to_delta
create_and_run_job(
File "/usr/local/lib/python3.11/site-packages/astro/databases/databricks/api_utils.py", line 174, in create_and_run_job
raise AirflowException(f"Databricks job failed. Job info {final_job_state}")
airflow.exceptions.AirflowException: Databricks job failed. Job info {'job_id': 735015794232116, 'run_id': 169955093810641, 'creator_user_name': 'phani.kumar@astronomer.io', 'number_in_job': 169955093810641, 'state': {'life_cycle_state': 'TERMINATED', 'result_state': 'FAILED', 'state_message': 'Workload failed, see run output for details', 'user_cancelled_or_timedout': False}, 'task': {'spark_python_task': {'python_file': 'dbfs:/mnt/pyscripts/load_file__tmp_jxlilhgkr15ca28f0oy733tmqkjdpnocildizl0m3vjf8hprrxxho2a2i.py'}}, 'cluster_spec': {'existing_cluster_id': '0403-094356-wab883hn'}, 'cluster_instance': {'cluster_id': '0403-094356-wab883hn', 'spark_context_id': '5137278826035226949'}, 'start_time': 1702883353507, 'setup_duration': 1000, 'execution_duration': 13000, 'cleanup_duration': 0, 'end_time': 1702883367582, 'run_name': 'Untitled', 'run_page_url': 'https://dbc-9c390870-65ef.cloud.databricks.com/?o=4256138892007661#job/735015794232116/run/169955093810641', 'run_type': 'SUBMIT_RUN', 'attempt_number': 0, 'format': 'SINGLE_TASK'}

[phanikumv]

Credentials got removed on the Databricks workspace, hence the need to change the DAG.