/dagster

Dagster is an opinionated system and programming model for data pipelines.

Primary LanguagePython

https://user-images.githubusercontent.com/28738937/44878798-b6e17e00-ac5c-11e8-8d25-2e47e5a53418.png

Introduction

Dagster is an opinionated system and programming model for data pipelines. This process goes by many names -- ETL (extract-load-transform), ELT (extract-transform-load), model production, data integration, and so on -- but in essence they all describe the same activity: Performing a set of computations structured as a DAG (directed, acyclic graph) that end up producing data assets, whether those assets be tables, files, machine-learning models, etc.

There are a few tools in this repo:

  • Dagster: The core programming model and abstraction stack; a stateless single-node and -process execution engine; and a CLI tool for driving that engine.

Go to https://dagster.readthedocs.io/en/latest/ for documentation!