LIP for AlexNet
Closed this issue · 2 comments
Hey there!
I was wondering whether LIP can also replace maxpool layers in AlexNet? Have you done any experiments with that and didn't see any recognizable improvements or just chose to focus on deeper models? I'd appreciate your insights on this issue.
Thanks!
Hi, thank you for your interest in our work.
I have not conducted LIP experiments in AlexNet before because AlexNet is not so popular among these years and instead the recent proposed ResNet and DenseNet become typical archtectures in CNNs (but maybe in the future they'll become old too). Another reason is that the shallow models are somewhat lightweight and logit modules in LIP can take extra computation that is comparable to the cost of original models. That's somehow not economic.
However, when talking back to replacing maxpool with LIP in any CNNs with maxpool, I think it'll be helpful because the maxpool brings prior knowledge into downsampling procedure and maxpool is hard to optimize because gradient is backpropagated into exact one location.
We'll do some experiments on more architectures like MobileNet, SENet and so on to further validate our LIP. We believe that LIP also does well across those architectures ;).
Hi!
Thanks for the quick response. I have a baseline for my task that uses AlexNet, so I kind of have to work on it to compare haha, so I wanted to make sure that this experiment can be feasible and hasn't been done by you already. I might be able to handle the cost of adding it for the task I'm working on, so I will definitely try it out.
Thanks again!