openclimatefix/graph_weather

Add support for Anemoi Datasets

Opened this issue · 7 comments

jacobbieker commented

ECMWF has now started doing an idea similar to the one in #3
Blog post: https://www.ecmwf.int/en/about/media-centre/aifs-blog/2024/data-driven-regional-modelling

Detailed Description

https://anemoi-datasets.readthedocs.io/en/latest/using/combining.html

Context

Using a high-resolution subset within a global model is quite interesting to get the higher resolution required. Anemoi datasets works to do that as well, making the cutouts.

Possible Implementation

Import the package and use that? Copy over the cutout functionality? As otherwise, its a light wrapper around Zarr stores, which we already support.

0xFrama commented

Hi @jacobbieker, could you give me more details about this?

jacobbieker commented

Hi, we just want to make it so we can train with the anemoi datasets package. It has some nice functionality, so would want to add it to the different ways of loading weather datasets. Probably as another option in the data/ module here, somewhat like the IFS dataloader.

0xFrama commented

Have you a specific weather dataset in mind to start with?

jacobbieker commented

The easiest would probably be either WeatherBench, or get it working with the ICON archive that we've made on Hugging Face.

With the ICON EU and ICON Global datasets on HF, we could get the loading of a low resolution global model + high resolution regional model working together, but would take more work to get up and running with this.

First step would probably try WeatherBench, and just get it to output training examples

  • 👍1
glitch401 commented

hey, is this issue still being worked on?

jacobbieker commented

I'm not sure, but if you want to tackle it, that would be great!

glitch401 commented

alrighty, taking a jab at it :D