Keypoint Scores

[PatrickRowsomeMovidius]

In an attempt to try extract the N best keypoints from a set of results I sorted in descending order using: keypoints = np.asarray([row for _, row in sorted(zip(scores, keypoints), key=lambda pair: pair[0])]).squeeze() after the results are obtained. I then cut this array taking only the top N keypoints: keypoints = keypoints[:N,:]. The results are not what I expected. I am using the assumption that the higher scores are better. Below is an example of the highest 300 keypoints.

I am surprised to see so many keypoints on the texture poor section on the right. I have used the model_ots.pth and model_tf.pth and they both receive similar results. Interestingly if i sort using ascending order I get more reasonable results:

In this case we see the 300 points with the lowest scores. All points are within the texture rich areas. Is this something you would expect or what is your intuition in this case?

Thanks,
Patrick

[tsattler]

@PatrickRowsomeMovidius The keypoint scores of D2-Net have no absolute meaning. We only train for local uniqueness, but not global order. As such, taking the top-n / bottom-n ranked keypoints is not expected to give you the "best" keypoints.

[mihaidusmanu]

Hello @PatrickRowsomeMovidius . @tsattler is right about the scores not being trained for global ordering. Nevertheless, for HPatches sequences, I tried selecting top 2000 highest scores and the results remained similar; this doesn't guarantee that the results will be the same for all datasets, but it is worth investigating further. Thus I have some additional questions:

1. Are you using multi-scale or single-scale? Taking top-k for multi-scale is not expected to work since the feature maps are summed together for higher scales.
2. I think that keypoints = np.asarray([row for _, row in sorted(zip(scores, keypoints), key=lambda pair: pair[0])]).squeeze() sorts in ascending order so you might want to select the last N keypoints (e.g., keypoints[-N :]).

[SkyRiderMike]

I have one question, how can i get the orientation of keypoint ?

[mihaidusmanu]

We do not estimate a feature orientation (only location and eventually scale using an image pyramid). The keypoints can be considered as being upright.

[PatrickRowsomeMovidius]

Hello @PatrickRowsomeMovidius . @tsattler is right about the scores not being trained for global ordering. Nevertheless, for HPatches sequences, I tried selecting top 2000 highest scores and the results remained similar; this doesn't guarantee that the results will be the same for all datasets, but it is worth investigating further. Thus I have some additional questions:

1. Are you using multi-scale or single-scale? Taking top-k for multi-scale is not expected to work since the feature maps are summed together for higher scales.

I am using the single scale option. So this should not cause an issue

2. I think that keypoints = np.asarray([row for _, row in sorted(zip(scores, keypoints), key=lambda pair: pair[0])]).squeeze() sorts in ascending order so you might want to select the last N keypoints (e.g., keypoints[-N :]).

Yes, you are right, this does sort in ascending order, this means that my results are more stable now.

Still I see quite poor performance in terms of repeatability when I attempt to only keep the best N keypoints.

[SkyRiderMike]

@mihaidusmanu Thanks for answering！

[mihaidusmanu]

@PatrickRowsomeMovidius In a recent work related to D2-Net - ASLFeat (https://arxiv.org/pdf/2003.10071.pdf), they seem to be using the soft-detection score for reducing the number of keypoints (section 5.1, paragraph Testing). I suspect this might be helpful for D2-Net features as well, but I didn't experiment with it so far.

I will close the issue. Feel free to open a new one if there are any problems.