About 1x1 convolution
zmonoid opened this issue · 2 comments
Is there any idea about how to deal with 1x1 convolution for channel alignment for parameter sharing CNN? Since the size changes as the graph change.
There are issues w/ assuring compatible channel dimensions AND spatial dimensions, if you're using skip connections that add/concatenate feature maps. I don't think it's really clear how they handled this in the original paper.
One way would be to only allow connections between layers w/ the same dimensions -- this would be reasonable inside a block in the micro search space, but (probably?) not in the macro search space.
In the paper, there is mention of using 1x1 convs to make sure the number of channels match up, but I don't know whether they were learning a 1x1 conv per layer pair or per dimension pair or whatever.
Let's say there are 4+1+4+1+4 = 14 cells for cifar10, each cell has 5 node, each node has len(conv3x3, conv5x5, sep3x3, sep5x5)=4 parameter weights. My understanding is that there are 1454=280 parameter weights in the shared pool, is it correct?
About the how to use conv1x1, you may check the following code:
https://github.com/Cadene/pretrained-models.pytorch/blob/master/pretrainedmodels/models/nasnet.py
Another way to deal with it may be: for each node to be concatenated, we follow a MxN conv1x1, and for next cell input usage, we add them together, this way we can map (nM) channels to N channels.