Extend this to 1-D
Opened this issue · 8 comments
I'm trying to extend ConvOffset2d to 1d convolution as well.
While this can be easily done by simply consider a length-L 1D tensor as a dimension [1xL] 2D tensor, the problem is that the offset now is for 2D deformation (you can go up or down, left or right); whereas in 1D, I only want the kernel connection to move by left/right.
I'm seeking to change the .c and .cu code so that I no longer need to worry about the up/down offset, and in offset = Conv2d(...) I only need output dimension of num_deformable_groups * kernel_size, instead of num_deformable_groups * 2 * kernel_size`.
Any suggestion on which parts of the .c and .cu that I should change? Thank :-)
These lines controls offsets. You can search for offset_h (vertical) and offset_w (horizontal) across all the functions of .cu.
Let the size of input be inC x inH x inW, the size of kernel be outC x inC x kH x kW, and the size of reshaped kernel be outC x (inC x kH x kW), Conv2d in pytorch is implemented as Conv2d(input, kernel) = GEMM( reshaped_kernel, IM2COL(input) ).
When the input signal is 1D, we don't need to perform the im2col operation on input. It should be much more easier than 2D case. For this reason, I suggest rewriting the kernels (deform_conv_cuda_kernel.cu) instead of removing vertical or horizontal offsets. Then you can write forward and backward based on here.
Thanks for the pointers. Where can I find the documentations for THCudaTensor in PyTorch?
Also, why don't we need im2col anymore? While the input tensor is L x C now (2D indeed), isn't the purpose of im2col here to make sure that the convolution operator can be written in a matrix multiplication form? Since the convolution filter still has dimension C1 x C2 x kL (where C1 and C2 are input an output widths), I guess the im2col is still necessary...
Finally, what is the difference between deformable_col2im_coord and deformable_col2im?
@jerybai1995 did you ever write a 1d version ? pelase share :)
@jerrybai1995 @danFromTelAviv can you share the 1D Deformable Convolution library?
@Anirudh257 this was a 2.5 years ago - naturally I've moved on to other things. I don't think I ever got it to work for 1d. I recommend you look into transformers - they are very similar in nature and much more generic now a days.
@danFromTelAviv Thanks for your response. I am working on transformers but need to understand deformable convolution too. Thanks for responding though.
@Anirudh257 if you still need this I have a small PyTorch library for 1D deformable convolution which may be of use -> https://github.com/jwr1995/dc1d
@jwr1995 Thanks, I will look into it!