Support for open version of Superpoint

Hello,

I have just found out that a new repository to train Lightglue is available (https://github.com/cvg/glue-factory). They have included the open version of Superpoint this time so I was wondering if you are planning to include this also into LightGlue-ONNX.

Thank you very much in advance!

Hi @adricostas

Thank you for the information. I am able to export SuperPoint-Open, however, it seems that when I try to pass its output features to the LightGlue matcher that was trained on SuperPoint-MagicLeap's features, I obtain a blank output (zero matches).

I cannot find any weights for a LightGlue that has been trained on SuperPoint-Open's features. Could you point me to a URL for these weights?

Hi,

Well, I'm not sure, but they say that: "All models and datasets in gluefactory have auto-downloaders, so you can get started right away!". So, inside the code of the model of each extractor/matcher exists an url from which the weights are downloaded. However, as you pointed out, it seems that in the LightGlue case, only the weights for the not open version of SuperPoint are available: https://github.com/cvg/LightGlue/releases.
I guess that it is necessary to train the LightGlue network by yourself using the outputs of the open version of SuperPoint. I don't know if this makes sense.

Thanks

@adricostas It appears that LightGlue weights trained on SuperPoint-Open haven't been released yet. See cvg/glue-factory#22

Ok, thanks for the information! Let's wait for them, then.

From: https://github.com/rpautrat/SuperPoint

Update: We have converted the Tensorflow model into a Pytorch one. This provides a version of SuperPoint with MIT license, which can be used with the Pytorch model defined in superpoint_pytorch.py.

It sounds like those weights might work?
https://github.com/rpautrat/SuperPoint/blob/master/weights/superpoint_v6_from_tf.pth

Hi @adricostas

Thank you for the information. I am able to export SuperPoint-Open, however, it seems that when I try to pass its output features to the LightGlue matcher that was trained on SuperPoint-MagicLeap's features, I obtain a blank output (zero matches).

I cannot find any weights for a LightGlue that has been trained on SuperPoint-Open's features. Could you point me to a URL for these weights?

HI, can you share your code to export superpoint-open? I tried by myself, but I have warnings like this
UserWarning: Exporting aten::index operator of advanced indexing in opset 17 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results. warnings.warn(

UserWarning: The exported ONNX model failed ONNX shape inference. The model will not be executable by the ONNX Runtime. If this is unintended and you believe there is a bug, please report an issue at https://github.com/pytorch/pytorch/issues. Error reported by strict ONNX shape inference: [ShapeInferenceError] Inference error(s): (op_type:MaxPool, node name: /MaxPool): [ShapeInferenceError] Attribute dilations has incorrect size (Triggered internally at ../torch/csrc/jit/serialization/export.cpp:1484.) _C._check_onnx_proto(proto)

Hi @noahzn, thank you for your interest in LightGlue-ONNX.

This is the code I adapted when exporting SuperPoint-Open:

LightGlue-ONNX/lightglue_onnx/superpoint_open.py

Lines 1 to 188 in fc1d67a

    
           # MIT License 
        
           # 
        
           # Copyright (c) 2018 Paul-Edouard Sarlin & Rémi Pautrat 
        
           # 
        
           # Permission is hereby granted, free of charge, to any person obtaining a copy 
        
           # of this software and associated documentation files (the "Software"), to deal 
        
           # in the Software without restriction, including without limitation the rights 
        
           # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 
        
           # copies of the Software, and to permit persons to whom the Software is 
        
           # furnished to do so, subject to the following conditions: 
        
           # 
        
           # The above copyright notice and this permission notice shall be included in all 
        
           # copies or substantial portions of the Software. 
        
           # 
        
           # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 
        
           # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
        
           # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 
        
           # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
        
           # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
        
           # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 
        
           # SOFTWARE. 
        
           # Adapted by Fabio Milentiansen Sim 
        
           """PyTorch implementation of the SuperPoint model, 
        
              derived from the TensorFlow re-implementation (2018). 
        
              Authors: Rémi Pautrat, Paul-Edouard Sarlin 
        
              https://github.com/rpautrat/SuperPoint 
        
              The implementation of this model and its trained weights are made 
        
              available under the MIT license. 
        
           """ 
        
           from collections import OrderedDict 
        
           from types import SimpleNamespace 
        
           from typing import Tuple 
        
           import torch 
        
           import torch.nn as nn 
        
           def sample_descriptors(keypoints, descriptors, s: int = 8): 
        
               """Interpolate descriptors at keypoint locations""" 
        
               b, c, h, w = descriptors.shape 
        
               keypoints = keypoints + 0.5 
        
               keypoints_x = torch.div(keypoints[..., 0], w * s) 
        
               keypoints_y = torch.div(keypoints[..., 1], h * s) 
        
               keypoints = torch.stack((keypoints_x, keypoints_y), dim=-1) 
        
               keypoints = keypoints * 2 - 1  # normalize to (-1, 1) 
        
               descriptors = torch.nn.functional.grid_sample( 
        
                   descriptors, keypoints.view(b, 1, -1, 2), mode="bilinear", align_corners=False 
        
               ) 
        
               descriptors = torch.nn.functional.normalize( 
        
                   descriptors.reshape(b, c, -1), p=2, dim=1 
        
               ) 
        
               return descriptors 
        
           def batched_nms(scores, nms_radius: int): 
        
               assert nms_radius >= 0 
        
               def max_pool(x): 
        
                   return torch.nn.functional.max_pool2d( 
        
                       x, kernel_size=nms_radius * 2 + 1, stride=1, padding=nms_radius 
        
                   ) 
        
               scores = scores[None]  # max_pool bug 
        
               zeros = torch.zeros_like(scores) 
        
               max_mask = scores == max_pool(scores) 
        
               for _ in range(2): 
        
                   supp_mask = max_pool(max_mask.float()) > 0 
        
                   supp_scores = torch.where(supp_mask, zeros, scores) 
        
                   new_max_mask = supp_scores == max_pool(supp_scores) 
        
                   max_mask = max_mask | (new_max_mask & (~supp_mask)) 
        
               return torch.where(max_mask, scores, zeros)[0] 
        
           @torch.jit.script_if_tracing 
        
           def select_top_k_keypoints(keypoints: torch.Tensor, scores: torch.Tensor, k: int): 
        
               if k >= keypoints.shape[0]: 
        
                   return keypoints, scores 
        
               scores, indices = torch.topk(scores, k, dim=0, sorted=True) 
        
               return keypoints[indices], scores 
        
           class VGGBlock(nn.Sequential): 
        
               def __init__(self, c_in, c_out, kernel_size, relu=True): 
        
                   padding = (kernel_size - 1) // 2 
        
                   conv = nn.Conv2d( 
        
                       c_in, c_out, kernel_size=kernel_size, stride=1, padding=padding 
        
                   ) 
        
                   activation = nn.ReLU(inplace=True) if relu else nn.Identity() 
        
                   bn = nn.BatchNorm2d(c_out, eps=0.001) 
        
                   super().__init__( 
        
                       OrderedDict( 
        
                           [ 
        
                               ("conv", conv), 
        
                               ("activation", activation), 
        
                               ("bn", bn), 
        
                           ] 
        
                       ) 
        
                   ) 
        
           class SuperPointOpen(nn.Module): 
        
               default_conf = { 
        
                   "descriptor_dim": 256, 
        
                   "nms_radius": 4, 
        
                   "max_num_keypoints": None, 
        
                   "detection_threshold": 0.005, 
        
                   "remove_borders": 4, 
        
                   "descriptor_dim": 256, 
        
                   "channels": [64, 64, 128, 128, 256], 
        
               } 
        
               checkpoint_url = "https://github.com/rpautrat/SuperPoint/raw/master/weights/superpoint_v6_from_tf.pth" 
        
               def __init__(self, **conf): 
        
                   super().__init__() 
        
                   self.conf = SimpleNamespace(**{**self.default_conf, **conf}) 
        
                   self.stride = 2 ** (len(self.conf.channels) - 2) 
        
                   channels = [1, *self.conf.channels[:-1]] 
        
                   backbone = [] 
        
                   for i, c in enumerate(channels[1:], 1): 
        
                       layers = [VGGBlock(channels[i - 1], c, 3), VGGBlock(c, c, 3)] 
        
                       if i < len(channels) - 1: 
        
                           layers.append(nn.MaxPool2d(kernel_size=2, stride=2)) 
        
                       backbone.append(nn.Sequential(*layers)) 
        
                   self.backbone = nn.Sequential(*backbone) 
        
                   c = self.conf.channels[-1] 
        
                   self.detector = nn.Sequential( 
        
                       VGGBlock(channels[-1], c, 3), 
        
                       VGGBlock(c, self.stride**2 + 1, 1, relu=False), 
        
                   ) 
        
                   self.descriptor = nn.Sequential( 
        
                       VGGBlock(channels[-1], c, 3), 
        
                       VGGBlock(c, self.conf.descriptor_dim, 1, relu=False), 
        
                   ) 
        
                   state_dict = torch.hub.load_state_dict_from_url(self.checkpoint_url) 
        
                   self.load_state_dict(state_dict) 
        
               def forward( 
        
                   self, image: torch.Tensor  # (1, 1, H, W) 
        
               ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: 
        
                   features = self.backbone(image) 
        
                   descriptors_dense = torch.nn.functional.normalize( 
        
                       self.descriptor(features), p=2, dim=1 
        
                   ) 
        
                   # Decode the detection scores 
        
                   scores = self.detector(features) 
        
                   scores = torch.nn.functional.softmax(scores, 1)[:, :-1] 
        
                   b, _, h, w = scores.shape 
        
                   scores = scores.permute(0, 2, 3, 1).reshape(b, h, w, self.stride, self.stride) 
        
                   scores = scores.permute(0, 1, 3, 2, 4).reshape( 
        
                       b, h * self.stride, w * self.stride 
        
                   ) 
        
                   scores = batched_nms(scores, self.conf.nms_radius) 
        
                   # Discard keypoints near the image borders 
        
                   if self.conf.remove_borders: 
        
                       pad = self.conf.remove_borders 
        
                       scores[:, :pad] = -1 
        
                       scores[:, :, :pad] = -1 
        
                       scores[:, -pad:] = -1 
        
                       scores[:, :, -pad:] = -1 
        
                   # Extract keypoints 
        
                   scores = scores.squeeze(0) 
        
                   idxs = torch.where(scores > self.conf.detection_threshold) 
        
                   # Convert (i, j) to (x, y) 
        
                   keypoints = torch.stack(idxs[-2:], dim=-1).flip(1).float() 
        
                   scores = scores[idxs] 
        
                   if self.conf.max_num_keypoints is not None: 
        
                       keypoints, scores = select_top_k_keypoints( 
        
                           keypoints, scores, self.conf.max_num_keypoints 
        
                       ) 
        
                   desc = sample_descriptors(keypoints, descriptors_dense, self.stride) 
        
                   keypoints = keypoints + 0.5 
        
                   scores = scores 
        
                   descriptors = desc.transpose(-1, -2) 
        
                   return keypoints[None], scores[None], descriptors

As for the error/warning, a lot has changed with respect to the frameworks though, so you might have to align to these versions iirc:

torch==2.1.0+cu118
onnx==1.14.1
opset 15/16

Hope this helps!

@fabio-sim Thanks a lot!!

	# MIT License
	#
	# Copyright (c) 2018 Paul-Edouard Sarlin & Rémi Pautrat
	#
	# Permission is hereby granted, free of charge, to any person obtaining a copy
	# of this software and associated documentation files (the "Software"), to deal
	# in the Software without restriction, including without limitation the rights
	# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	# copies of the Software, and to permit persons to whom the Software is
	# furnished to do so, subject to the following conditions:
	#
	# The above copyright notice and this permission notice shall be included in all
	# copies or substantial portions of the Software.
	#
	# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
	# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
	# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
	# SOFTWARE.

	# Adapted by Fabio Milentiansen Sim

	"""PyTorch implementation of the SuperPoint model,
	derived from the TensorFlow re-implementation (2018).
	Authors: Rémi Pautrat, Paul-Edouard Sarlin
	https://github.com/rpautrat/SuperPoint
	The implementation of this model and its trained weights are made
	available under the MIT license.
	"""
	from collections import OrderedDict
	from types import SimpleNamespace
	from typing import Tuple

	import torch
	import torch.nn as nn


	def sample_descriptors(keypoints, descriptors, s: int = 8):
	"""Interpolate descriptors at keypoint locations"""
	b, c, h, w = descriptors.shape
	keypoints = keypoints + 0.5
	keypoints_x = torch.div(keypoints[..., 0], w * s)
	keypoints_y = torch.div(keypoints[..., 1], h * s)
	keypoints = torch.stack((keypoints_x, keypoints_y), dim=-1)
	keypoints = keypoints * 2 - 1 # normalize to (-1, 1)
	descriptors = torch.nn.functional.grid_sample(
	descriptors, keypoints.view(b, 1, -1, 2), mode="bilinear", align_corners=False
	)
	descriptors = torch.nn.functional.normalize(
	descriptors.reshape(b, c, -1), p=2, dim=1
	)
	return descriptors


	def batched_nms(scores, nms_radius: int):
	assert nms_radius >= 0

	def max_pool(x):
	return torch.nn.functional.max_pool2d(
	x, kernel_size=nms_radius * 2 + 1, stride=1, padding=nms_radius
	)

	scores = scores[None] # max_pool bug
	zeros = torch.zeros_like(scores)
	max_mask = scores == max_pool(scores)
	for _ in range(2):
	supp_mask = max_pool(max_mask.float()) > 0
	supp_scores = torch.where(supp_mask, zeros, scores)
	new_max_mask = supp_scores == max_pool(supp_scores)
	max_mask = max_mask \| (new_max_mask & (~supp_mask))
	return torch.where(max_mask, scores, zeros)[0]


	@torch.jit.script_if_tracing
	def select_top_k_keypoints(keypoints: torch.Tensor, scores: torch.Tensor, k: int):
	if k >= keypoints.shape[0]:
	return keypoints, scores
	scores, indices = torch.topk(scores, k, dim=0, sorted=True)
	return keypoints[indices], scores


	class VGGBlock(nn.Sequential):
	def __init__(self, c_in, c_out, kernel_size, relu=True):
	padding = (kernel_size - 1) // 2
	conv = nn.Conv2d(
	c_in, c_out, kernel_size=kernel_size, stride=1, padding=padding
	)
	activation = nn.ReLU(inplace=True) if relu else nn.Identity()
	bn = nn.BatchNorm2d(c_out, eps=0.001)
	super().__init__(
	OrderedDict(
	[
	("conv", conv),
	("activation", activation),
	("bn", bn),
	]
	)
	)


	class SuperPointOpen(nn.Module):
	default_conf = {
	"descriptor_dim": 256,
	"nms_radius": 4,
	"max_num_keypoints": None,
	"detection_threshold": 0.005,
	"remove_borders": 4,
	"descriptor_dim": 256,
	"channels": [64, 64, 128, 128, 256],
	}

	checkpoint_url = "https://github.com/rpautrat/SuperPoint/raw/master/weights/superpoint_v6_from_tf.pth"

	def __init__(self, **conf):
	super().__init__()
	self.conf = SimpleNamespace({self.default_conf, **conf})
	self.stride = 2 ** (len(self.conf.channels) - 2)
	channels = [1, *self.conf.channels[:-1]]

	backbone = []
	for i, c in enumerate(channels[1:], 1):
	layers = [VGGBlock(channels[i - 1], c, 3), VGGBlock(c, c, 3)]
	if i < len(channels) - 1:
	layers.append(nn.MaxPool2d(kernel_size=2, stride=2))
	backbone.append(nn.Sequential(*layers))
	self.backbone = nn.Sequential(*backbone)

	c = self.conf.channels[-1]
	self.detector = nn.Sequential(
	VGGBlock(channels[-1], c, 3),
	VGGBlock(c, self.stride**2 + 1, 1, relu=False),
	)
	self.descriptor = nn.Sequential(
	VGGBlock(channels[-1], c, 3),
	VGGBlock(c, self.conf.descriptor_dim, 1, relu=False),
	)

	state_dict = torch.hub.load_state_dict_from_url(self.checkpoint_url)
	self.load_state_dict(state_dict)

	def forward(
	self, image: torch.Tensor # (1, 1, H, W)
	) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
	features = self.backbone(image)
	descriptors_dense = torch.nn.functional.normalize(
	self.descriptor(features), p=2, dim=1
	)

	# Decode the detection scores
	scores = self.detector(features)
	scores = torch.nn.functional.softmax(scores, 1)[:, :-1]
	b, _, h, w = scores.shape
	scores = scores.permute(0, 2, 3, 1).reshape(b, h, w, self.stride, self.stride)
	scores = scores.permute(0, 1, 3, 2, 4).reshape(
	b, h * self.stride, w * self.stride
	)
	scores = batched_nms(scores, self.conf.nms_radius)

	# Discard keypoints near the image borders
	if self.conf.remove_borders:
	pad = self.conf.remove_borders
	scores[:, :pad] = -1
	scores[:, :, :pad] = -1
	scores[:, -pad:] = -1
	scores[:, :, -pad:] = -1

	# Extract keypoints
	scores = scores.squeeze(0)
	idxs = torch.where(scores > self.conf.detection_threshold)

	# Convert (i, j) to (x, y)
	keypoints = torch.stack(idxs[-2:], dim=-1).flip(1).float()
	scores = scores[idxs]

	if self.conf.max_num_keypoints is not None:
	keypoints, scores = select_top_k_keypoints(
	keypoints, scores, self.conf.max_num_keypoints
	)

	desc = sample_descriptors(keypoints, descriptors_dense, self.stride)

	keypoints = keypoints + 0.5
	scores = scores
	descriptors = desc.transpose(-1, -2)

	return keypoints[None], scores[None], descriptors