srinivasseema/made-template

Template repository for the Methods of Advanced Data Engineering course at FAU

Project Plan

Weathering the Storm: Forecasting Energy Consumption and Pricing Trends Amidst Changing Climate Conditions.

Correlation Between Energy and Weather to anticipate energy demand surges and dips.

Description

In a rapidly changing climate, accurate energy demand and pricing forecasts are crucial for ensuring grid stability and sustainable resource management. This project aims to analyse energy consumption and pricing trends.

The model will analyze historical energy consumption data, weather patterns, and climate projections to identify key relationships driving energy demand and pricing dynamics. This will enable utilities to:

• Proactively prepare for demand fluctuations caused by extreme weather events.

• Optimize energy pricing strategies to ensure financial sustainability while maintaining affordability for consumers.

Further, By harnessing the power of machine learning, this project will equip stakeholders with the knowledge and tools to navigate the evolving energy landscape and ensure energy security for all.

Datasources

Datasource1: Energy Data

• Metadata URL: https://transparency.entsoe.eu/
• Data URL: https://transparency.entsoe.eu/generation/r2/actualGenerationPerProductionType/show?name=&defaultValue=false&viewType=TABLE&areaType=BZN&atch=false&datepicker-day-offset-select-dv-date-from_input=D&dateTime.dateTime=08.11.2023+00:00|CET|DAYTIMERANGE&dateTime.endDateTime=08.11.2023+00:00|CET|DAYTIMERANGE&area.values=CTY|10Y1001A1001A83F!BZN|10Y1001A1001A82H&productionType.values=B01&productionType.values=B02&productionType.values=B03&productionType.values=B04&productionType.values=B05&productionType.values=B06&productionType.values=B07&productionType.values=B08&productionType.values=B09&productionType.values=B10&productionType.values=B11&productionType.values=B12&productionType.values=B13&productionType.values=B14&productionType.values=B20&productionType.values=B15&productionType.values=B16&productionType.values=B17&productionType.values=B18&productionType.values=B19&dateTime.timezone=CET_CEST&dateTime.timezone_input=CET+(UTC+1)+/+CEST+(UTC+2)#
• Data Type: CSV

This dataset contains yearly electrical consumption, generation data for european countries. Consumption and generation data was retrieved from ENTSOE a public portal for Transmission Service Operator (TSO) data.

Dataengineering framework

Prefect Dashboard

Prefect Deployment Flow:

Prefect Flow Runs:

Prefect Schedule:

CI/CD

• Prefect framework handling the automated deployment, schedule of the flows
• Has integration with Jupyter notebook to run notebook as task

Prefect Server Commands

To start server locally:

• prefect server start

To create deployment:

By invoking the flow with serve method and specifying name of the flow:

executePipeline.serve(name="energy-data-flow")

• We can run the deployment of the pipeline with below command: prefect deployment run 'executePipeline/my-first-deployment'

To access Prefect UI:

http://127.0.0.1:4200/deployments

Additional Documentation

Slides Project Video Report

Work Packages

1. Explore Datasources [#1]
2. Analyze data pipeline requirements