/dbt-spark-demo

Dagster + DBT + Spark + Iceberg

Primary LanguagePythonApache License 2.0Apache-2.0

dbt-spark-demo

Use GitHub Codespaces / VS Code Remote Dev Toolkit

Or

Local setup steps:

  1. Install requirements (Linux)
  2. Deploy spark server
  3. Run dagster web ui

Requirement

Debian

apt-get update
# For thrift
apt-get -y install gcc libsasl2-modules libsasl2-dev libsasl2-modules-gssapi-heimdal

Spark Server

services

  • minio
  • spark thrift server (STS)
docker-compose up -d

Dagster + DBT

cd my-dagster-project
pip install -e ".[dev]"
dagster dev

Open Web UI: http://127.0.0.1:3000

Misc

Disable auth

Disable auth by removing hive-site.xml mount of sts services.