ai-forever/ru-dalle

Fine-tuning notebook no longer saves model

StephenHnilica opened this issue · 1 comments

Fine-tuning notebook no longer saves model. Not sure why. It's happening on my private notebooks & the pubic one too.

I did verify that the 2.63 gb model downloads, but /checkpoints/ folder is never created.

`Malevich is 1.3 billion params model from the family GPT3-like, that uses Russian language and text+image multi-modality.

FileNotFoundError Traceback (most recent call last)
in ()
2 model_path = os.path.join('/content/'+args.save_path,f"{args.model_name}_dalle_last.pt")
3 model = get_rudalle_model('Malevich', pretrained=True, fp16=True, device=device)
----> 4 model.load_state_dict(torch.load(model_path))
5 print(f'Loaded from {model_path}')
6 vae = get_vae().to(device)

2 frames
/usr/local/lib/python3.7/dist-packages/torch/serialization.py in init(self, name, mode)
209 class _open_file(_opener):
210 def init(self, name, mode):
--> 211 super(_open_file, self).init(open(name, mode))
212
213 def exit(self, *args):

FileNotFoundError: [Errno 2] No such file or directory: '/content/checkpoints/awesomemodel__dalle_last.pt'`

Never mind, I used cd to change directories and never changed it back.