/simple_dbt_project

Code for dbt tutorial

Primary LanguageDockerfile

This is the code repo for dbt tutorial at https://www.startdataengineering.com/post/dbt-data-build-tool-tutorial

Setup

Prerequisites

  1. Docker and Docker compose
  2. dbt
  3. pgcli
  4. git

Clone the git repo and start the warehouse & dbt docker containers, as shown below.

git clone https://github.com/josephmachado/simple_dbt_project.git
cd simple_dbt_project
make up
make sh

We use make sh to log into the dbt docker container, & run dbt commands.

Run dbt

Once you are inside the dbt docker container, run the following commands.

cd $WORKDIR # go to the directory where we have dbt code
dbt deps
dbt snapshot
dbt run --select sde_dbt_tutorial
dbt test
dbt docs generate
dbt docs serve

Go to http://localhost:8080 to see the dbt documentation (press ctrl+c).

[Optional] We can create a data observability report with Elemetary as shown below:

dbt run --select elementary
edr report

From your file system, open the html file at the path sde_dbt_tutorial/edr_target/elementary_report.html on your broswer to see the data observability report.

You can exit the dbt container using exit.

Let's do some testing, Insert some data into source customer table, to demonstrate dbt snapshots. From your terminal (after exiting dbt container) run the following command.

make warehouse

You will be logged into your warehouse, here use the below command:

COPY warehouse.customers(customer_id, zipcode, city, state_code, datetime_created, datetime_updated) FROM '/input_data/customer_new.csv' DELIMITER ',' CSV HEADER;
\q

Run snapshot and create models again.

make sh
cd $WORKDIR # go to the directory where we have dbt code
dbt snapshot --select sde_dbt_tutorial
dbt run --select sde_dbt_tutorial

You can exit the dbt container using exit. From your terminal (after exiting dbt container) run the following command.

make warehouse
select * from warehouse.customer_orders limit 3;
\q

Stop docker container

make down