Question about HPatches_Sequences_Matching_Benchmark evaluation.
Closed this issue · 8 comments
Dear Mihai Dusmanu!
I am sorry to disturb you again. Can you provide third-party models you used in your paper, such as hesaff, hesaffnet, delf, delf-new, and superpoint? If you upload these models, It will maintain an objective effect as these methods are always updated. In addition, it will help us research your work and others.
In addition, can you provide the code to extract features and descriptors for these third-party models? Because there are less than 2000 features are extracted on the SLAM (simultaneous localization and mapping) system. Usually, there will be less than 1000 features.
Thank you, sincerely.
The code for Hesian Affine is already part of the repository - https://github.com/mihaidusmanu/d2-net/blob/master/extract_hesaff.m.
The code / models for the other methods is available on their respective GitHub pages:
- AffNet: https://github.com/ducha-aiki/affnet
- DELF: https://github.com/tensorflow/models/tree/master/research/delf
- SuperPoint: https://github.com/MagicLeapResearch/SuperPointPretrainedNetwork
I am sorry for the inconvenience, but merging everything into a single repository would be completely unfeasible for multiple reasons (different frameworks, different licensing, need for cleaning / maintenance, ...).
For HesAffNet + HardNet++ I used the default parameters. For SuperPoint I used the default parameters as well (without image downsampling). For DELF I used full descriptors (no PCA / whitening) and lowered the score threshold to 10 without a bound on the maximum number of keypoints. This should be enough to reproduce the cached results, but feel free to open an issue in case you get significantly different results.
Thank you for your response. Only pre-trained models and the part code of extracting features and descriptors are ok. In addition, what is the difference between MMA (mean matching accuracy) and mAP (mean average precision)? If there are different, please tell me the equation of MMA, which is not provided in your paper. Thank you, sincerely.
MMA as the name suggests is the mean matching accuracy over all image pairs defined, for a fixed threshold t in pixels , by:
where
This measure reflects the percent of correct matches / the inlier ratio when varying the threshold.
Thank you for your response. I also to ask two more questions. First, was non-maximal suppression apply to third-party models? If this trick is applied, please give more details. Second, was the ratio test or RANSAC or top_keypoints_selection apply to third-party models? For example, SuperPoint:(https://github.com/MagicLeapResearch/SuperPointPretrainedNetwork) only provides a pre-train model and does not provides more details about Hpatchses evaluation. After all, some methods do not provide complete details. Thank you.
@cumtchenchang For SuperPoint,you can refer to https://github.com/rpautrat/SuperPoint. It provides about Hpatchses evaluation.
I also want to ask a question about the HPatches_Sequences_Matching_Benchmark evaluation.As the Figure 4 shows, what is the meaning about the X axis "threshold[px]" .
@VPRzhou Thank you. I research Superpoint for few months, non-maximal suppression is applied to evaluation on Hpathes datasets in this code (https://github.com/rpautrat/SuperPoint), such as 4 or 8 pixels. But I am not sure other models on d2-net paper. Maybe the author can provide more details @mihaidusmanu. In addition, threshold[px] could be understand as the correctness threshold on Superpoint's original paper. You can also find this parameter in https://github.com/rpautrat/SuperPoint.
As I mentioned above, for SuperPoint I have used the default parameters from the official repository without image downsampling, i.e. NMS distance 4 and confidence threshold 0.015 from https://github.com/MagicLeapResearch/SuperPointPretrainedNetwork/blob/4698c7f001663ff2b13304d699f2cdb3f663c28f/demo_superpoint.py#L601-L604.
Thank you very much for your response and hope your work will be remembered by more people.