Not training in DIGITS
achaiah opened this issue · 4 comments
achaiah commented
Hi, I was wondering if you tried running the models in NVIDIA DIGITS? I imported one of your models (resnet 36) - it runs but does not train. Any ideas?
Thanks!
jay-mahadeokar commented
I haven't tried it. What do you mean by does not train? Can you be more specific?
achaiah commented
It literally doesn't train (i.e. zero progress over 200 epochs). I found a fix here: BVLC/caffe#3919 There's a difference in BatchNorm between bvlc and NVIDIA that is confusing.
mrgloom commented
@achaiah Can you be more specific about differences in BN layer?
Seems realted thread is here ? NVIDIA/DIGITS#629
achaiah commented
The thread you pointed to is correct as well... same issue. The CUDNN BN implementation has changed and that has broken existing nets that use BN.