has anyone begun to port this to pytorch?
mdering opened this issue · 8 comments
I'd be interested in making this into an extension for pytorch, if anyone is also working on this, or would like to help, please let me know
@mdering my knowledge of Lua is quite limited but I think I can help tough with some guidance -let me know at bagonzalo@gmail.com.
I do not have time at the moment to look deeply into a port of the code, but @davidstutz had a look at it a few months back. Maybe he can share his experience.
I briefly tried to get it working with PyTorch. I started implementing some custom operations and then came to the realization that it is not straight-forward to implement operations on custom data structures - i.e. the octnet structure as provided through the Python interface. However, that was a few day after the release of PyTorch ... So there wasn't much documentation or many people to ask. And I also didn't note everything down, sorry.
I spent more time trying to get it working with Tensorflow. Maybe some of my experiences there are helpful. The problem with Tensorflow is that it is - obviously - built around tensors. Digging a bit in the C++ code, I found it very hard to decouple operations from the underlying tensors. I don't think it is worthwhile adapting Tensorflow to operate on arbitrary or very general data structures. However, an alternative, straight-forward option would have been to express an octnet as a set of tensors (as it is done internally in octnet, i.e. just a set of array). The input to an operation is then a set of tensors encoding the octnet, the octnet is setup inside the operation by just using the memory locations of the tensors, an operation can be performed (e.g. convolution, pooling, etc.) and the operation returns the tensors corresponding to the resulting octnet. However, I didn't end up following this path ... I still believe that in the case of Tensorflow this might be the easiest option - although a bit hacky.
I actually don't have any code in regards to porting OctNet - I mostly followed Extending PyTorch to see how custom operations can be implemented in PyTorch. In the ideal case, you would need to implement PyTorch operations on custom data structures - I mean that's you goal after all when handling OctNets. So what you would like to do is to set up a network, that operates on some sort of custom toy-data structure.
As an example, in Torch, you can define a toy-data structure and simply define a network layer on that data structure:
-- Small example to test forward and packward passes of custom data structures.
require('math')
require('torch')
require('nn')
require('lfs')
package.path = package.path .. ";" .. lfs.currentdir() .. '?.lua'
init = require('init')
--- @class CustomDataStructure
-- This calss will be out simple test data structure.
CustomDataStructure = {}
CustomDataStructure.__index = CustomDataStructure
--- Creates a new CustomDataStructure with 0-vectors of the given size
-- @param n vector size
function CustomDataStructure.create(n)
local cds = {}
setmetatable(cds, CustomDataStructure)
cds.x1 = torch:Tensor(n):fill(0)
cds.x2 = torch:Tensor(n):fill(0)
return cds
end
--- @class CustomLinear
CustomLinear, CustomLinearParent = torch.class('nn.CustomLinear', 'nn.Module')
--- Initialize the layer specifying the number of input and output units.
-- @param nInputUnits number of input units
-- @param nOutputUnits number of output units
function CustomLinear:__init__(nInputUnits, nOutputUnits)
self.nInputUnits = nInputUnits
self.nOutputUnits = nOutputUnits
self.weight = torch.Tensor(nOutputUnits, nInputUnits):fill(0)
end
--- Compute output.
-- @param input input of type CustomDataStructure
-- @return output of type CustomDataStructure
function CustomLinear:updateOutput(input)
self.output = CustomDataStructure.create(self.nOutputUnits)
self.output.x1 = torch.mv(self.weight, input.x1)
self.output.x2 = torch.mv(self.weight, input.x2)
return self.output
end
--- Avoid backward pass.
function CustomLinear:UpdateGradInput(input, gradOutput)
assert(false)
end
N = 10
x1 = torch.Tensor(N):fill(0)
x2 = torch.Tensor(N):fill(1)
x = CustomDataStructure.create(N)
x.x1 = x1
x.x2 = x2
model = nn.Sequential()
module = nn.CustomLinear(10, 1)
module.weight = torch.Tensor(1, 10):fill(1)
model:add(module)
y = model:forward(x)
print(y.x1)
print(y.x2)
The above example is pretty trivial. But it shows that Torch works on any data structure as long as you define your own layers (In my opinion this is partly due to the fact that Torch does not come with any bells and whistles - so no tensor shape inference, no autograd etc.). If you would be able to get something like that working in PyTorch, this would be the first step to work with OctNets, because in the above example, you can simply replace CustomDataStructure by OctNet's LUA interface and you are basically done - because all the essential operations (convolution, pooling etc.) are implemented in OctNet.
So this would be my approach. I didn't go any further when working on PyTorch because it was a few days after its release, so there wasn't a lot of information on extending PyTorch, working with custom, non-tensor data structures and so on ...
Is there any pytorch version available for OctNet?
There is a competing/similar project (implemented in pytorch):
https://github.com/microsoft/O-CNN
perhaps that can motivate someone to convert OctNet to pytorch as well.