CarbonCast

CarbonCast: Multi-Day Forecasting of Grid Carbon Intensity (PDF). An extended version of our paper appeared in ACM SIGEnergy Energy Informatics Review, and can be found here.
CarbonCast provides average carbon intensity forecasts for up to 96 hours. This is an extension of DACF, which provides only day-ahead carbon intensity forecasts.

Version: 2.1
Authors: Diptyaroop Maji, Prashant Shenoy, Ramesh K Sitaraman
Affiliation: University of Massachusetts, Amherst

CarbonCast v3.0, which can provide forecasts in real-time, is now in the testing phase and can be found here. Please feel free to use and report any issues. We are working to release it in the main branch soon.

0. Disclaimer

We will be periodically updating this repo as we update/improve/extend CarbonCast. Even after cloning the repo, please check back in a while to see if anything is updated.
If something is not working, please check whether some recent update has already fixed that.
In case something is not working in the latest version, or if there are any doubts/questions/suggestions, please feel free to reach us at dmaji at cs dot umass dot edu.

0.1 Current status:

Code files: Up to date as of 03/11/2023.
Data files: Up to date as of 01/10/2023.
Latest stable commit:

1. Regions covered

US (US region data collected from EIA):
- California (CISO)
- Florida (FPL)
- New England (ISO-NE. We refer the region as ISNE.)
- New York (NYISO)
- Pennsylvania-Jersey-Maryland Interconnection (PJM)
- Texas (ERCOT. We refer the region as ERCO)
- Washington (BPAT)
Europe (European regions are monitored by ENTSOE):
- Germany (DE)
- Netherlands (NL)
- Spain (ES)
- Sweden (SE)
- Poland (PL)
Australia (Data for Australian regions is available at OpenNEM)
- Queensland (AUS-QLD)

2. Data Sources

US ISO electricity generation by source: EIA hourly grid monitor

European regions electricity generation by source: ENTSOE

Australian regions electricity generation by source: OpenNEM

Weather forecasts: GFS weather forecast archive

Day-ahead solar/wind Forecasts:

CISO: OASIS
European regions: ENTSOE
We currently do not have solar/wind forecasts for other regions, or for periods beyond 24 hours. Hence, we generate them using ANN models along with 96-hour forecasts for other sources.

3. Background on carbon intensity

3.1 Carbon emission factor (CEF):

CEF of a source is the amount of carbon emitted into the atmosphere per unit of electricity generated by that source. There can be two types of CEFs for a source:

Direct emission factors: These are the operational emissions when a source is converted into electricity.
Lifecycle emission factors: These include operational as well as infrastructural emissions up the supply chain.
(See paper for further details.)

3.2 Calculating average carbon intensity:

We use the following formula for calculating avg carbon intensity:
, where

CI_avg = Average carbon intensity (real-time or forecast) of a region
E_i = Electricity produced by source i.
CEF_i = Carbon emission factor (lifecycle/direct) of source i.

We have provided the file carbonIntensityCalculator.py to calculate both real-time/historical average CI values as well as carbon intensity forecasts from source prodution forecasts. Please refer to Section 4.4 for details.

4. Running CarbonCast with existing datasets and models

4.1 Installing dependencies:

CarbonCast requires Python3.
Other required packages:

Required python modules are listed in requirements.txt.
Run pip3 install -U -r requirements.txt for installing the dependecies.
wgrib2 (for weather data). Please refer here for compilation/installation details. If you are using MacOS and having trouble compiling wgrib2, please refer to this article. Once wgrib2 is installed, go to the grib2 directory and run the following command to access wgrib2 from anywhere in the terminal: cp -rfv wgrib2/wgrib2 /usr/local/bin/wgrib2

4.2 Running CarbonCast using saved models/Reproducing results from paper:

We have saved second-tier models for each region which you can use with existing & new datasets to get 96-hour CI forecasts. These models are trained with data from Jan-Dec 2020 and validated with data from Jan-Jun 2021, so that results similar to the paper can be obtained when tested over Jul-Dec 2021. Each region has 2 saved models --- one for lifecycle CEF & the other for direct CEF. If you are using new datasets, you may need to update the models with new training data or generate new models.
To run CarbonCast using the saved model for any region, run:
python3 secondTierForecasts.py <configFileName> <-l/-d> <-s>
Configuration file name: secondTierConfig.json
Regions: CISO, PJM, ERCO, ISNE, NYISO, FPL, BPAT, SE, DE, ES, NL, PL, AUS_QLD. You can specify the region(s) in the configuration file.
<-l/-d>: Lifecycle/Direct. Relevant saved model for the specified region(s) will be loaded.
<-s>: Use saved model. Parameter that tells CarbonCast to use saved models and not train a new model.

5 Running CarbonCast from scratch

To run CarbonCast from scratch (with new data/for new regions etc.), first install the dependencies mentioned in Section 4.1.

5.1 Getting Weather data:

The aggregated and cleaned weather forecasts that we have used for our regions are provided in data/. If you need weather forecasts for other regions, or even for the same regions (e.g., if you want to use a different aggregation method or if you want to forecast for a different time period), the procedure is mentioned below.

We fetch weather data from the GFS weather forecast archive. You will need to register before you can get weather data. Once you have registered, do the following:
GitHub repo of script to fetch weather data can be found here. You can follow the instructions there and download weather forecasts in grib2 format. The repo has a sample Jupyter Notebook with step-by-step instructions. Remember to modify the notebook as required (e.g., changing the dataset id (dsid)).
Otherwise, clone the above repo and add the following two files in the rda-apps-clients/src/python folder:
src/weather/getWeatherData.py, src/weather/ds084.1_control.ctl
getWeatherData.py uses ds084.1_control.ctl as a template file to download 96-hour weather forecasting data for a particular region. Change the template file for different regions and weather variables (weather variables include wind speed, temperature, dewpoint temperature, solar irradiance (dswrf), and precipitation). The template file has instructions on how to modify it for different regions and weather variables. After you have configured the template file, run: python3 getWeatherData.py
You may need to add your credentials in rda-apps-clients/src/python/rdams_client.py for API calls to work. To do that, add the following as the first line in get_authentication():
write_pw_file(<username>, <password>)
Once you have obtained the grib2 files, use the following files to aggregate and clean the data:
python3 dataCollectionScript.py -- this file uses code from here for aggregating weather forecasts over a specified region.
python3 cleanWeatherData.py -- this file cleans the data and generates hourly files for the above specified weather variables.
You will need to modify the relevant fields in the above two files to successfully parse & clean the weather data.
If you are using any other weather aggregating method, please feel free to modify the above files as required.

5.2 Getting source production forecasts:

You will need to obtain, clean, & format the datasets before you can get source production forecasts. You may also need to modify the configuration file as required.
For getting source production forecasts in the first-tier, run the following file:
python3 firstTierForecasts.py <configFileName>
Configuration file name: firstTierConfig.json
Regions: CISO, PJM, ERCO, ISNE, NYISO, FPL, BPAT, SE, DE, ES, NL, PL, AUS_QLD
Sources: coal, nat_gas, oil, solar, wind, hydro, unknown, geothermal, biomass, nuclear
You can get source production forecasts of multiple regions together. Just add the new regions in the "REGION" parameter.

5.3 Calculating carbon intensity (real-time/historical/from source production forecasts):

For calculating real-time/historical carbon intensity from source data, or carbon intensity forecasts from the source production forecast data using the formula, run the following file:
python3 carbonIntensityCalculator.py <region> <-l/-d> <-f/-r> <num_sources>
Regions: CISO, PJM, ERCO, ISNE, NYISO, FPL, BPAT, SE, DE, ES, NL, PL, AUS_QLD
<-l/-d>: Lifecycle/Direct
<-f/-r>: Forecast/Real-time (or, historical)
num_sources: No. of electricity producting sources in that region.

5.4 Getting carbon intensity forecasts using CarbonCast:

For getting 96-hour average carbon intensity forecasts, run the following file:
python3 secondTierForecasts.py <configFileName> <-l/-d>
Configuration file name: secondTierConfig.json
Regions: CISO, PJM, ERCO, ISNE, NYISO, FPL, BPAT, SE, DE, ES, NL, PL, AUS_QLD
<-l/-d>: Lifecycle/Direct
You can get carbon intensity forecasts of multiple regions together. Just add the new regions in the "REGION" parameter.

6. Developer mode

We welcome users to suggest modifications to improve CarbonCast and/or add new features or models to the existing codebase. Please feel free to contact us at dmaji at cs dot umass dot edu with suggestions (or even working patches!)

7. Citing CarbonCast

If you use CarbonCast, please consider citing our paper. The BibTex format is as follows:
@inproceedings{maji2022carboncast,
title={CarbonCast: multi-day forecasting of grid carbon intensity},
author={Maji, Diptyaroop and Shenoy, Prashant and Sitaraman, Ramesh K},
booktitle={Proceedings of the 9th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation},
pages={198--207},
year={2022}
}

8. Acknowledgements

This work is part of the CarbonFirst project, supported by NSF grants 2105494, 2021693, and 2020888, and a grant from VMware.

fershad/CarbonCast