minidagster

This is a Dagster project scaffolded with dagster project scaffold.

Getting started

First, install your Dagster code location as a Python package. By using the --editable flag, pip will install your Python package in "editable mode" so that as you develop, local code changes will automatically apply.

pip install -e ".[dev]"

Then, start the Dagster UI web server:

dagster dev

Info

Linear with partitions

Linear graph of assets where every asset has the same partition definition.

flowchart LR
    Download --> Parse --> Index

What happens when I materialize partition X of the Download asset?
1. Partition X of Download will materialize
2. Partition X of Parse will materialize
3. Partition X of Index will materialize
What happens when I materialize partition X of the Parse asset? Or any other asset in the middle of the graph.
1. Partition X of Parse will materialize
2. Partition X of Index will materialize

Linear without partitions

Linear graph of assets where every asset doesn't have any partition definition.

flowchart LR
    Download --> Parse --> Index

What happens when I materialize the Download asset?
1. Download will materialize
2. Parse will materialize
3. Index will materialize
What happens when I materialize the Parse asset? Or any other asset in the middle of the graph.
1. Parse will materialize
2. Index will materialize

Parent with partitions

Linear graph of assets where the parent asset has partitions but the child asset does not.

flowchart LR
    Download --> Parse