Please visit the offical awesome-dagster repo by dagster. I moved the relevant content over there.
A curated list of dagster code snippets for data engineers
List of content
- Data Warehouse
- Apache Druid Connector resource to send GET/POST REST-Requests
- Relational
- MySQL Instantiate a connection to a MySQL database and run queries
- Monitoring
- HealthcheckIO Send updates to Cron job monitoring HealthcheckIO
- Spark & Delta-Lake
- Spark and Delta Lake Solids to create Delta-Table
create_delta_table, load data from a Delta-Table into a Spark-dfcreate_delta_table
- Spark and Delta Lake Solids to create Delta-Table
- Scrapting
- Web-Scraping and Caching Web-scraping with BeautifulSoup
- Success and Failure Hook and Pipeline Example How to use Hooks on failure or on error
- JsonType A valid representation of a JSON, validated with json.loads()
- Everything but the Kitchen Sink Image Image to run Spark and a whole lot more
- Spark Image to run Spark and Delta-Lake
- GitHub Discussions News, ideas, Q&A and more about Dagster
- Slack Channel Questions and Discussions
- Data Engineering Podcast - Build Maintainable And Testable Data Applications With Dagster - Episode 104 October 28, 2019: Fist podcast about Dagster where Nick Schrock the founder explains the motivation and the story behind Dagster.
- Software Engineering Daily - Dagster with Nick Schrock November 15, 2019: Second podcast with different insight from the software engineering perspective
- Python Podcast - Digging Into Dagster: An Opinionated Open Source Framework For Data Orchestration - Episode 279 September 7, 2020: Update on Dagster after a year.
- Running Pipelines In Production Using Dagster to replace a homegrown Cron job
- Bestplaces and their Journey A journey from various heterogeneous Tools to integrated Dagster Environment
- Dagster Migration At Prezi From JSON based homegrown orchestrator to Dagster
Inspired by the Awesome list of Data Engineering by Igor Barinov.
To the extent possible under law, Simon Späti has waived all copyright and related or neighboring rights to this work.