Unstable numeric output for downstream task (moat 4 w/o pos)
edwardyehuang opened this issue · 1 comments
edwardyehuang commented
ckpts : moat4 w/o pos
The output from moat4 can easily result in the following layers (e.g., 3x3 conv) having a NaN output.
The same issue, at least, does not show in moat0.
It is the first time I have met this issue in my career (I met NaN many times, but never like this), so I need to take some time to investigate this issue.
I will update this issue if I have a new finding. Please also check if the provided ckpts are working.
Chenglin-Yang commented
Thanks for your interest. For this issue, I have two suggestions:
- First of all, make sure the weights of all layers are correctly loaded, including the stem layers, etc.
- You can adjust the learning rate, drop path rate and batch size to solve the issue.