Detector Descriptor combination error

Question

Detector Descriptor combination error

surendra1234sharma opened this issue 7 years ago · 5 comments

surendra1234sharma commented 7 years ago

I am taking "MSDDetector" for feature detection and "SIFT" for feature description
I am running the following code

detector = cv.FeatureDetector('MSDDetector');
extractor = cv.DescriptorExtractor('SIFT');
Points = detector.detect(img1);

upto this it is running error free but when I run this statement

[d, sf] = extractor.compute(img1,Points);

it shows following error

Unexpected Standard exception from MEX file.
What() is:c:\temp\batserve\win64\opencv\modules\imgproc\src\imgwarp.cpp:3230:
error: (-215) dsize.area() > 0 || (inv_scale_x > 0 && inv_scale_y > 0) in function cv::resize

Error in cv.DescriptorExtractor/compute (line 270)
            [descriptors, keypoints] = DescriptorExtractor_(this.id, 'compute', img, keypoints);

Please resolve this issue
Thanks in advance

Answer 1 · 2017-08-01T05:06:19.000Z

I know that some descriptors like KAZE and AKAZE only work with their own keypoints (this is documented behavior).

So it could be the case the combination of MSDDetector+SIFT is incompatible somehow,
maybe SIFT as a descriptor expects the input keypoints to be a certain way
(see KeyPoint struct fields: angle, octave, class_id, etc.). Not sure...

But it could also be a bug in SIFT, in which case you should report to OpenCV.

Here is a test I did in mexopencv, checks all descriptors against MSD keypoints:

names = {
    'BRISK'
    'ORB'
    'KAZE'
    'AKAZE'
    'SIFT'
    'SURF'
    'FREAK'
    'BriefDescriptorExtractor'
    'LUCID'
    'LATCH'
    'DAISY'
    'VGG'
    'BoostDesc'
};

img = cv.imread(fullfile(mexopencv.root(), 'test', 'blox.jpg'), 'Flags',0);
detector = cv.MSDDetector();
kpts = detector.detect(img);

for i=1:numel(names)
    disp(names{i})
    extractor = cv.DescriptorExtractor(names{i});
    try
        descs = extractor.compute(img, kpts);
    catch ME
        warning(ME.getReport());
    end
end

The failures:

~~`KAZE and AKAZE: as described before~~
~~LUCID: it requires 8-bit color image (on the other hand, if you convert GRAY2RGB, FREAK will complain as it requires grayscale image)~~
SIFT: what you reported

Answer 2 · 2017-08-01T05:34:07.000Z

I don't know if this is related, but a quick test suggests that SIFT keypoints have unusual octave values:

>> obj = cv.SIFT(); k = obj.detect(img);
>> unique([k.octave])'
      459519
      590591
    [..TRUNCATD..]
    16581119
    16712192

A quick search reveals that SIFT "octave" in fact contains three values packed into one int32:

Maybe that has something to do with the MSDDetector+SIFT combination failing?

Answer 3 · 2017-08-01T19:35:53.000Z

Thank you very much for your explanation

I just want to verify that following combinations are also giving errors and If possible please provide some solution.
I got errors in the following combinations of detector and descriptor. Here A+B means:

A: detector
B: descriptor

Any Detector+LUCID

Unexpected Standard exception from MEX file.
What() is:c:\temp\batserve\win64\opencv\modules\core\src\matrix.cpp:2191:
error: (-215) !fixedType() || ((Mat*)obj)->type() == mtype in function cv::_OutputArray::create

Any Detector+ KAZE or AKAZE

Unexpected Standard exception from MEX file.
What() is:c:\temp\batserve\win64\opencv\modules\features2d\src\kaze\akazefeatures.cpp:712:
error: (-215) 0 <= kpts[i].class_id && kpts[i].class_id < static_cast<int>(evolution_.size())
in function cv::AKAZEFeatures::Compute_Descriptors

In Kaze and AKAZE following combinations work correctly
KAZE+KAZE, AKAZE+KAZE, AKAZE+AKZE

SIFT+ORB

Unexpected Standard exception from MEX file.
What() is:c:\temp\batserve\win64\opencv\modules\core\src\matrix.cpp:424:
error: (-215) u != 0 in function cv::Mat::create

MSDDetector+SIFT

Unexpected Standard exception from MEX file.
What() is:c:\temp\batserve\win64\opencv\modules\imgproc\src\imgwarp.cpp:3230:
error: (-215) dsize.area() > 0 || (inv_scale_x > 0 && inv_scale_y > 0) in function cv::resize

Error in cv.DescriptorExtractor/compute (line 270)
            [descriptors, keypoints] = DescriptorExtractor_(this.id, 'compute', img, keypoints);

Thanks

Answer 4 · 2017-08-01T23:49:06.000Z

Right, those combinations gave errors.

1) LUCID

As mentioned in my previous comment, the LUCID implementation in OpenCV requires a color image as input. So a simple GRAY2RGB conversion will do:

gray = imread(..);
kpts = cv.FAST(img);
obj = cv.LUCID();
descs = obj.compute(repmat(img,[1 1 3]), kpts);

2) KAZE and AKAZE

Again, explained before. KAZE/AKAZE descriptors will only work with KAZE/AKAZE keypoints. This is because they store certain information in the class_id field of keypoints, which is then retrieved when computing descriptors. All other detectors set it to -1 (unused).

Bottom line, you shouldn't mix KAZE/AKAZE with other algorithms.

3) SIFT

SIFT is likely to be incompatible with other algorithms as well.

This is because it stores packed values in the octave field of keypoints. So if you pass those keypoints to another descriptor that deals with regular octave values (normally used to indicate the scale-space octave in which the feature has been found), you would get wrong results at best! Same applies in the opposite direction, as other detectors won't know how to fill this field as expected by SIFT.

For reference, you can unpack the values using bit-manipulation:
https://github.com/opencv/opencv_contrib/blob/3.2.0/modules/xfeatures2d/src/sift.cpp#L210-L217

Here is the C++ function translated to MATLAB:

function [octave, layer, scale] = unpackOctave(kpt_octave)
    kpt_octave = int32(kpt_octave);
    octave = bitand(kpt_octave, int32(255));
    layer = bitand(bitshift(kpt_octave, -8), int32(255));
    if octave >= 128
        octave = bitor(octave, int32(-128));
    end
    if octave >= 0
        scale = 1.0 / single(bitshift(int32(1), octave));
    else
        scale = single(bitshift(int32(1), -octave));
    end
end

You might think of processing SIFT keypoints using the above function before passing them to ORB, but you will get another error from ORB (it asserts all octaves >= 0, but some of SIFT octaves will be -1 after unpacking):

https://github.com/opencv/opencv/blob/3.2.0/modules/features2d/src/orb.cpp#L986-L1007

So I don't think SIFT+ORB is a valid combination. Neither is MSDDetector+SIFT...

From my understanding, SIFT should only be used as both descriptor/extractor at the same time, don't combine it with another algorithm..

Of course I could be wrong. You should ask on OpenCV forum for a definitive answer. Good luck :)

Answer 5 · 2017-08-02T14:42:52.000Z

Thanks a lot Amro