[Paper] FenguWu GHR
jacobbieker opened this issue · 3 comments
Arxiv/Blog/Paper Link
https://arxiv.org/abs/2402.00059
Detailed Description
Uses some cool tricks to have vanilla transformer, with LoRA and trained on weather bench data to have high resolution, long lead time forecasts. Trained on era5 mostly, then last 5 years of IFS analysis (since it became 0.09 degree) and has impressive results. The model is trained for only one 6 hour timestep.
LoRA is used per forecast beyond a single 6 hour timestep to correct for biases and such for each future forecast time period. This helped speed up training and reduce memory usage.
Context
Really interesting work, and being able to train on era5 but use the model operationally with higher resolution data. Need to look into it more though.
They also compare to station observations from the NOAA ISD dataset, which I've been working with in the planetary dataset library. Global hourly station observations going back years, and available quite quickly.
Seems somewhat similar to MetNets timestep embedding which resulted in Metnet 1 of one set of model weights for each forecast timestep. LoRA seems more lightweight to use since it's just a small subset of parameters rather than the whole model.