demo.lua error on some images

[redserpent7]

Hi,

I just started with multipathnet and I am having issues with some image files. I have tested with a couple images and the test image provided and everything worked fine but some images just give this error:

/home/ubuntu/torch/install/bin/luajit: demo.lua:80: calling 'select' on bad self (empty Tensor at /home/ubuntu/torch/pkg/torch/generic/Tensor.c:380)
stack traceback:
        [C]: in function 'select'
        demo.lua:80: in main chunk
        [C]: in function 'dofile'
        ...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00406670

Here is the image in question: https://1drv.ms/i/s!Av-Yk52R4YupvKRWWmZUdNd4IAciKg

Any suggestions?

[samson-wang]

@redserpent7 Pay attention to transformer setting on the line https://github.com/facebookresearch/multipathnet/blob/master/demo.lua#L43 .
If use wrong transformer, you will get almost zero score for each bounding box candidate. Then after filtering, you will get an empty result which leads to this crash.

Updates: The transformer problem only exists when model gets changed. If some pictures work fine, it may be that the model just can't "recognize" objects correctly, I guess.

[redserpent7]

@samson-wang so how do you determine which transformer to use? I am trying the demo out of the box and as I said it works with most images but some crash

[samson-wang]

@redserpent7 In my case, running demo.lua on the image from the test dataset got very tiny score, i.e. ~1.5e-12, for every candidate bounding box. While, the evaluation code is OK. So I track the data flow through both the evaluation and demo process. I find that the kinds of transformers' diff.

Since most of your images work fine. I'm not sure if the transformer is the reason. Maybe the model just can't recognize the objects in those images which crash.

[outgrabe]

@redserpent7
This error occurs when there are no valid detections. Either all the classes are 1 (the background class, which is ignored) in maxes, or the matching confidence scores in prob are below the threshold.

On this line
local idx = maxes:squeeze():gt(1):cmul(prob:gt(config.thr)):nonzero():select(2,1)
if by the time it gets to "select" you have an empty tensor (no valid detections), it will crash.

You can solve this by building idx (a list of indices of the valid detections) yourself. Something like this:

local indices = torch.LongTensor()
local size = 0
for i=1,maxes:size(1) do
    -- Get non-background detections with a confidence equal to or over the threshold.
    if maxes[i][1] > 1 and prob[i][1] >= config.thr then
        size = size + 1
        indices:resize(size)
        indices[size] = i
    end
end

Then you can check if size > 0, and do something else if you have no detections in the image.

[outgrabe]

It may be worth noting that the demo will also crash on greyscale images, which are included in the COCO dataset (discovered when testing with the 2014 validation set). Copying to three colour channels solves the problem and seems to have decent results.