In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our cloud provider. This project illustrate the process of data ingestion to the lakehouse, data integration with ADF and data transformation with Databricks, and DBT.
Try running the following commands:
- dbt run # for running models
- dbt test # for tests
- dbt snapshot # for snapshotting and slowly changing dimensions
- dbt docs generate # for documentation
- dbt docs serve # for documentation preview
