google-deepmind/graphcast

How to train a model by myself

Closed this issue · 6 comments

Trek-on commented

Hello, I have a few questions that I would like to ask you:

  1. Open source code only provides prediction and fine-tuning code, without providing training code, right? If I want to train my own model, it's best to make one by myself?
  2. The reason why we can see all the technical details is because when the model predicts, it predicts by reading the downloaded model parameters. Is this understanding correct?
  3. "Training low resolution random models", I mean the fine-tuning code for the last part of the official Notebook instance. How did he implement it? It seems that he did not download the training data，is it just randomly adjusting the parameters in the parameter file?

Thank you very much!

zlminmin commented

Excuse me, have you implemented training a new graphcast model yourself?

alvarosg commented

Thanks for your message, the open source code provides a "loss" function, which you can use to both train and fine-tune the model if you can fit it in your hardware. However you would need to provide your own data iterators, and implement batch parallelism (to train on multiple devices simultaneously and this way reduce training time) for your specific platform.

zlminmin commented

Thank you very much for your answer, I'm a beginner and it's hard for me to reproduce such a complex model as GraphCast. I want to learn such a good model, but the training details are not mentioned in the paper, which is not enough for me to complete the reproduction independently. So could you please provide an example of training from scratch that I can use as a reference.

alvarosg commented

but the training details are not mentioned in the paper
To the best of our knowledge all training details for minimizing the loss (optimizer, batch size, trajectory sampling, learning rate schedules, etc) are provided in the supplementary materials of the paper (sections 4.4 and 4.5).

If there is something you find is missing, please let us know we will more than happy to clarify!

zhongmengyi commented

Hello, I have some questions about model training. Have you tried training models with different resolutions, GraphCast_small (13levels, 1°) and GraphCast (37levels, 0.25°)? How much time and memory does it take to train these two models?

I look forward to your response. Thank you.

Best regards！

alvarosg commented

@zhongmengyi I have replied in your separate issue #77