cnguyen10/few_shot_meta_learning

getting NaN's in ABML at about epoch 14

Closed this issue · 4 comments

jeffwillette commented

Thanks for publishing these implementations. I was running ABML on Omniglot for (20/5, 20/1, 5/5, and 5/1) n-way k-shot learning problems. On all four of the above experiments I start getting NaN's for ABML around epoch 12-14. Its weird that they all fail at the same spot. I have looked through the code carefully and I cannot see anything directly which might cause this, but you may have a better idea since you implemented it...

Any ideas where this is coming from? Here are the flags I was using to train. I may have added some flags related to dataloading, but nothing that interferes with the core.

python main.py \
    --datasource=$DATASET \
    --ds-folder $ROOT \
    --run $RUN \
    --ml-algorithm=abml \
    --num-models=2 \
    --minibatch 16 \
    --no-batchnorm \
    --n-way=20 \
    --k-shot=5 \
    --v-shot=$VSHOT \
    --num-epochs=40 \
    --num-episodes-per-epoch 10000 \
    --resume-epoch=0 \
    --train
cnguyen10 commented

Hi Jeff,
Thank you for letting me know.
For the NaN error, what I can think of is the KL divergence (https://github.com/cnguyen10/few_shot_meta_learning/blob/master/_utils.py#L165) (at L166, s1_vec is divided, hence, if s1_vec is zero, it would cause NaN). Another point is the loss prior that regularizes the meta-parameter std (https://github.com/cnguyen10/few_shot_meta_learning/blob/master/Abml.py#L77). If tau = 0, then the log-likelihood of the Gamma distribution for tau would be undefined. I wonder if you can put some print commands to print out the values of the KL divergence and the loss prior to see which one causes the NaN. You also want to check the hyper-parameters used for the Gamma prior. In the ABML paper, they only ran for mini-ImageNet, and hence, I hard-coded those values (https://github.com/cnguyen10/few_shot_meta_learning/blob/master/Abml.py#L20). It might cause problems when running on Omniglot.

jeffwillette commented

It seemed very random and every time I put a print statement in, the nans would come from somewhere else. I finally narrowed it down to the weight sample. The log std of some of the weights were really high such that it caused an inf when exponentiating it https://github.com/cnguyen10/few_shot_meta_learning/blob/master/_utils.py#L240

I have seen this happen with other Bayesian models when using exp instead of something like c + (1 - c) softplus(log_sigma). I guess that changing the KL might help also but I am not sure which way to move it.

I clamped the log_std parameters to be between (1e-8, 5) and it seems to stably train, but there is probably a better solution

cnguyen10 commented

This problem may be caused by using a large meta_lr for log_std, resulting in an overshooting for some values of log_std. The current implementation uses the same meta_lr for both the mean and log_std. It is properly a good idea to separate to have 2 learning rates for the 2 meta-parameters.

jeffwillette commented

I think this was caused by a bug in my code. I somehow cut off the last linear layer of the CNN, but everything trained well without error (for a while), and there were just many 0's at the end of the logit vector. Once I fixed this, ABML trained without error. Sorry for the confusion

  • 👍1