tensorflow/neural-structured-learning

On adv. robust classifiers having better transfer performance

sayakpaul opened this issue · 2 comments

sayakpaul commented

I am referring to this paper: Do Adversarially Robust ImageNet Models Transfer Better? (Salman et al., NeurIPS'20). The authors show that adversarially robust models have better transfer learning performance. They attribute this gain to the following factors:

  • Adv. robust models have better-informed gradients.
  • Adv. robust models can learn representations that have better and easier invertibility.

I think it might be interesting to use nsl to train models on ImageNet-1k and verify how well these observations hold.

DualityGap commented

Interesting, thanks for the nice suggestion, Sayak!
Meanwhile, if you have any results, feel free to share.

sayakpaul commented

As an individual, it's currently not possible for me to train ImageNet-1k scale models without having the required resources and guidance. This is why I just wanted to pass this suggestion to the NSL team as I think it would be a very nice use case to apply NSL on.