/dagster-poc

Proof of concept user codebase with dagster.

Primary LanguagePython

dagster_poc

This is a Dagster project that can be used as a template for data engineering @ RKI

Getting started

This project is set up with make and poetry. To get started, run:

make install

Then, start the Dagster UI web server:

make dev

Open http://localhost:3000 with your browser to see the project.

You can start writing assets in dagster_poc/assets.py. The assets are automatically loaded into the Dagster code location as you define them.

Development

Adding new Python dependencies

You can add new Python dependencies in with poetry.

Unit testing

Tests are in the dagster_poc_tests directory and you can run tests using pytest:

make test

to run unit tests once or

make watch

to run unit tests on file changes.

Schedules and sensors

If you want to enable Dagster Schedules or Sensors for your jobs, the Dagster Daemon process must be running. This is done automatically when you run dagster dev.

Once your Dagster Daemon is running, you can start turning on schedules and sensors for your jobs.

Deploy on Dagster on premise

This project is set up with a Jenkinsfile that can be used to deploy to a Dagster on premise instance. To use it, set up a Jenkins pipeline on your Jenkins Instance.

Contact your Dagster Administrator to set up continous deployment of the image to our Dagster on premise deployment.