Run DBT jobs that are observed with Prefect and keep credential in your environemnt (don't a Prefect Cloud Credentials Block). Generalizes to other integrations.
Could use Docker, K8s, ECS, or whatever infrastructure type you like, just make sure your environment variable is injected.
- Install prefect-dbt
- Register prefect-dbt module blocks
- Get API key from DBT Cloud
- Create work pool
- Create flow code
- Create deployment code (specify work pool)
- Could use a SecretVolume