openclimatefix/quartz-frontend

Front End repo for the Nowcasting project.

Quartz Energy

A set of forecasting products driven by the exciting modelling work in Open Climate Fix and the community

The code is as open source as we can possibly make it (safely) and is powered by various forecast APIs, which are also available as services under the same Quartz umbrella.

Head to quartz.solar to find out more or to get in touch about using our Production services.

Solar Electricity Nowcasting UI

The nowcasting-app is the repository for Open Climate Fix's solar electricity nowcasting project. See this great Wired article about OCF's solar electricity forecasting work for a good intro to solar electricity nowcasting.

The plan is to enable the community to build the world's best near-term forecasting system for solar electricity generation, and then let anyone use it! :) We'll do this by using state-of-the-art machine learning and 5-minutely satellite imagery to predict the movement of clouds over the next few hours, and then use this to predict solar electricity generation.

The term "nowcasting" just means "forecasting for the next few hours using statistical techniques".

Why is all this stuff open-source?

In OCF, we're curious to see if it's possible to rapidly mitigate climate change by:

1. Enabling thousands of people to help solve ML problems which, if solved, might help reduce CO2 emissions
2. Running small(ish) pilot projects to implement the best solution in industry
3. Enabling thousands of practitioners to use the code in their products.

What's the likely climate impact?

It's really, really, really hard to estimate climate impact of forecasting! But, as a super-rough back-of-the-envelope calculation, we estimate that better solar forecasts, if rolled out globally, could reduce CO2 emissions by about a billion tonnes between now and 2035.

Getting involved

• List of "good first issues": GitHub "issues" which describe changes we'd like to make to the code.
• OCF's coding style
• The main tools we use include: PyTorch, PyTorch Lighting, xarray, pandas, pvlib

Overview of OCF's nowcasting repositories

Downloading data & getting the data in the right shape for ML experiments

• nowcasting_dataset: Pre-prepares ML training batches. Loads satellite data, numerical weather predictions, solar PV power generation timeseries, and other datasets. Outputs pre-prepared ML training batches as NetCDF files (one batch per NetCDF file).
• Satip: Retrieve, transform and store EUMETSAT data.
• pvoutput: Python code for downloading PV data from PVOutput.org.

Older code (no longer maintained)

• satellite_image_processing
• eumetsat: Tools for downloading and processing satellite images from EUMETSAT

Machine Learning

Main repositories for our experiments:

• satflow: Satellite Optical Flow with machine learning models. Predicting the next few hours of satellite imagery from the recent history of satellite imagery (and other data sources).
• predict_pv_yield: Using optical flow (and the output of satflow) & machine learning to predict solar PV yield (i.e. to predict the power generated by solar electricity systems over the next few hours). An older set of experiments is in predict_pv_yield_OLD, which is no longer maintained..
• nowcasting_utils: Forecasting performance metrics, plotting functions, loss functions, etc.
• nowcasting_dataloader: PyTorch dataloader for taking pre-prepared batches from nowcasting-dataset and getting them into our models.

PyTorch implementations of ML models from the literature

• MetNet: PyTorch Implementation of Google Research's MetNet (Sønderby et al. 2020), inspired from Thomas Capelle's metnet_pytorch.
• skillful_nowcasting: Implementation of DeepMind's Skillful Nowcasting GAN (Ravuri et al. 2021) in PyTorch Lightning.
• perceiver-pytorch: Implementation of DeepMind's Perceiver (Jaegle et al. 2021) and Perceiver IO (Jaegle et al. 2021) in Pytorch. Forked from lucidrains/perceiver-pytorch.

Older code (no longer maintained)

• solar-power-mapping-data: Code to create rich harmonised geographic data for PV installations from OpenStreetMap and other sources. Mostly by Dan Stowell, The Turing Institute, and Sheffield Solar. The code behind the 2020 paper "A harmonised, high-coverage, open dataset of solar photovoltaic installations in the UK" by Stowell et al.
• predict_pv_yield_OLD
• predict_pv_yield_NWP: Build a baseline model for predicting PV yield using NWP (numerical weather predictions), as opposed to satellite imagery. This model is intentionally very simple, so we can get an end-to-end system up and running quickly to interate on.
• metoffice_ec2: Extract specific parts of the UK Met Office's UKV and MOGREPS-UK numerical weather predictions from AWS, compress, and save to S3 as Zarr. Intended to run on AWS EC2.
• metoffice_aws_lambda: Simple AWS Lambda function to extract specific parts of the UK Met Office's UKV and MOGREPS-UK numerical weather predictions, compress, and save to S3 as Zarr. (We found that AWS Lambda is not a good fit for this task because we actually have to do a bit of heavy-lifting, which gets very expensive on Lambda!)

Operational solar nowcasting

• nowcasting_api: API for hosting nowcasting solar predictions. Will just return 'dummy numbers' until about mid-2022!

For a complete list of all of OCF's repositories tagged with "nowcasting", see this link

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Damien Tanner 📆	lina 💻	AlaaTohamy 💻	Flo 💻	dantravers 🤔	Peter Dudfield 💻	braddf 💻
rachel tipton 👀 💻

This project follows the all-contributors specification. Contributions of any kind welcome!