dbt + Trino: TPC-H demo
Presented at Cinco de Trino!
Inspired by:
Initial setup
What you'll need:
tpch
catalog, giving you access to the standard TPC-H source datasetanalytics
catalog, in which to save your transformed datasets (dbt models)- Compute resource with access to both of those catalogs
I used Starburst Galaxy to get up & running quickly. I was able to configure the TPC-H "source" catalog, my analytical "target" catalog (S3 + Starburst Galaxy metastore), and use the free "sample" cluster with access to both:
- https://docs.starburst.io/starburst-galaxy/catalogs/tpch.html
- https://docs.starburst.io/starburst-galaxy/catalogs/s3.html
Relevant docs if using self-hosted Trino:
- https://trino.io/docs/current/connector/tpch.html
- https://trino.io/docs/current/connector/hive-s3.html
Getting started
-
Clone this GitHub repo to your local machine:
git clone https://github.com/dbt-labs/trino-dbt-tpch-demo.git
-
Install the
dbt-trino
adapter plugin, which allows you to use dbt together with Trino / Starburst Galaxy. You may want to do this inside a Python virtual environment.
pip install dbt-trino
- Copy
sample.profiles.yml
to the root of your machine,~/dbt/profiles.yml
. (Why? This file will contain yourpassword
for connecting to Trino/Starburst, so you don't want it checked intogit
.)
$ cp ./sample.profiles.yml ~/.dbt/profiles.yml
-
Open the file, and update the fields denoted by
<>
with your own user, password, cluster, etc. -
Verify that you can connect to Trino / Starburst Galaxy. (If your Galaxy cluster is stopped, it may take a few moments for it to resume.)
dbt debug
- Install dbt packages (
dbt_utils
) for use in the project:
dbt deps
- Try running dbt:
dbt run
dbt test
dbt build
- Generate and view documentation:
dbt docs generate
dbt docs serve
- You may find you want to do things like...
- Fix a failing test
- Reenable models with more complex transformations:
models/marts/aggregates
andall_months
- Write some models of your own!
More on dbt
- Read the introduction and viewpoint
- Be part of the conversation in the dbt Community Slack
More on dbt + Trino
Watch recordings from past Trino community broadcasts: