cal-itp/data-infra

Reconfigure Elavon Loader to Fetch Secrets at Execution

Closed this issue · 0 comments

User story / feature request

As a warehouse developer, I want to be able to utilize a local instance of Airflow for development even if I don't have access to all the creds used by each DAG. Because Airflow's DagBag import fails if even one incoming file experiences errors at import time, any issue that comes up during the DAG import process prevents interacting with all the other DAGs as normal.

One such common blocker is the sync_elavon DAG, which uses elavon_to_gcs_raw.py to complete its work. That Python file calls get_secret_by_name() at the top level when constructing constant, and the call fails if the user running the Airflow instance doesn't have access to the CALITP__ELAVON_SFTP_PASSWORD secret in Google Secret Manager.

Acceptance Criteria

The get_secret_by_name() function call should be moved somewhere that doesn't fail the DagBag import process, likely bypassing construction of a constant altogether and feeding it into the password variable construction here.