MIDA-group/CoMIR

Model output nan

rooskraaijveld opened this issue · 2 comments

rooskraaijveld commented

Hi there,

I am trying to use your model on my own dataset, but the model outputs nans even though my data does not include any nans/infs. I have images of size 1x256x256 and in type float32 if that helps.

Do you know what could have caused this?

Thanks in advance!

npielawski commented

Hello @rooskraaijveld,

Thanks for your interest in CoMIR!

When do the nans/infs happen? At the start of training or after a while? Could you share the loss function in the latter case?

What is the range of the intensity of your images? Normalizing or standardizing the images may help as a first step.

Let me know if that works.

Best,
Nicolas

rooskraaijveld commented

Hi Nicolas!

Thanks for your response. I get nan's at the beginning of training and I normalised the images between 0 and 1 and it still happens. Let me know if you need more information!

Best,
Roos