This repository contains numerous work examples of code I use in my day to day work as a data engineer, all of which has been modified as minimum reproducible examples. My favourite tools are Snowflake, Python, and dbt, and I also have an interest in DevOps as it pertains to data engineering.
- Python Examples
- Snowpark example - backload data from SQL Server
- Snowpark example - backload data from API
- Automated SQL insert statements from a CSV file
- Extract data from SQL Server, transform, and load to Snowflake
- Batch load JSON files to Snowflake
- SQL Server data Pull - 100 Records from every view in a database
- SQL Examples
- Only grant permissions on tables with > 0 rows of data - Snowflake
- Auto Ingest Snowpipe from Azure Blob to Snowflake
- Shorten large union queries using Snowflake
- Basic Snowflake CDC Pipeline using Streams and Tasks
- Find missing dates in a date field - Snowflake
- Snowflake data pipeline from internal stage
- Snowflake CI/CD using Azure Pipelines - SQLFluff testing, build and deploy using SnowSQL
- SQLFluff and yamllint pipelines for a dbt project
# Clone the repository
$ git clone https://github.com/MartyC-137/Data-Engineering.git
# Connect to the repository
$ cd Data-Engineering