LoFTR: Detector-Free Local Feature Matching with Transformers
Jiaming Sun*, Zehong Shen*, Yu'ang Wang*, Hujun Bao, Xiaowei Zhou
CVPR 2021
It is such a brilliant idea to blend the CNN and transformer together for feature matching work. I have to give all the credits to the authors above. What I am specifically doing in this repo is trying to use the LoFTR and test it on the EuRoc dataset and compare to the ground truth for performance checking. This is truly one of the most powerful feature matching networks! If you would like to run it on EuROC dataset, look at "Run on EuRoc" directly after installing the environment.
Want to run LoFTR with custom image pairs without configuring your own GPU environment? Try the Colab demo:
# For full pytorch-lightning trainer features (recommended)
conda env create -f environment.yml
conda activate loftr
# For the LoFTR matcher only
pip install torch einops yacs korniaWe provide the download link to
- the scannet-1500-testset (~1GB).
- the megadepth-1500-testset (~600MB).
- 4 pretrained models of indoor-ds, indoor-ot, outdoor-ds and outdoor-ot (each ~45MB).
By now, the environment is all set and the LoFTR-DS model is ready to go! If you want to run LoFTR-OT, some extra steps are needed:
[Requirements for LoFTR-OT]
We use the code from SuperGluePretrainedNetwork for optimal transport. However, we can't provide the code directly due its strict LICENSE requirements. We recommend downloading it with the following command instead.
cd src/loftr/utils
wget https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/master/models/superglue.py Firstly you need to download the weights from here and place the weights in such directory arrangement: LoFTR/weights. Secondly create an output folder under the LoFTR root directory. Then you can run the following command:
python run_customized.py --model indoor --images /home/your_EuRoc_imageset_directory/ --gtcsv /home/your_gt_dataset_directory/data.csv --match_threshold 0.3
For the images directory, do not forget to add the forward slash at the end of the directory. The model argument can either be indoor or outdoor. The output matching images and evaluation.csv can be found at the output directory upon completion of execution.
After you download the weight and place it in such arrangement: LoFTR/weights, run the following command:
python run_hpatches.py --images /home/path_to_hpatches/hpatches-sequences-release/ --match_threshold 0.2
Output images and evaluation will be placed in output/hpatches folder.
[code snippets]
from src.loftr import LoFTR, default_cfg
# Initialize LoFTR
matcher = LoFTR(config=default_cfg)
matcher.load_state_dict(torch.load("weights/indoor_ds.ckpt")['state_dict'])
matcher = matcher.eval().cuda()
# Inference
with torch.no_grad():
matcher(batch) # batch = {'image0': img0, 'image1': img1}
mkpts0 = batch['mkpts0_f'].cpu().numpy()
mkpts1 = batch['mkpts1_f'].cpu().numpy()An example is given in notebooks/demo_single_pair.ipynb.
Run the online demo with a webcam or video to reproduce the result shown in the GIF above.
cd demo
./run_demo.sh[run_demo.sh]
#!/bin/bash
set -e
# set -x
if [ ! -f utils.py ]; then
echo "Downloading utils.py from the SuperGlue repo."
echo "We cannot provide this file directly due to its strict licence."
wget https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/master/models/utils.py
fi
# Use webcam 0 as input source.
input=0
# or use a pre-recorded video given the path.
# input=/home/sunjiaming/Downloads/scannet_test/$scene_name.mp4
# Toggle indoor/outdoor model here.
model_ckpt=../weights/indoor_ds.ckpt
# model_ckpt=../weights/outdoor_ds.ckpt
# Optionally assign the GPU ID.
# export CUDA_VISIBLE_DEVICES=0
echo "Running LoFTR demo.."
eval "$(conda shell.bash hook)"
conda activate loftr
python demo_loftr.py --weight $model_ckpt --input $input
# To save the input video and output match visualizations.
# python demo_loftr.py --weight $model_ckpt --input $input --save_video --save_input
# Running on remote GPU servers with no GUI.
# Save images first.
# python demo_loftr.py --weight $model_ckpt --input $input --no_display --output_dir="./demo_images/"
# Then convert them to a video.
# ffmpeg -framerate 15 -pattern_type glob -i '*.png' -c:v libx264 -r 30 -pix_fmt yuv420p out.mp4
You need to setup the testing subsets of ScanNet and MegaDepth first. We create symlinks from the previously downloaded datasets to data/{{dataset}}/test.
# set up symlinks
ln -s /path/to/scannet-1500-testset/* /path/to/LoFTR/data/scannet/test
ln -s /path/to/megadepth-1500-testset/* /path/to/LoFTR/data/megadepth/testconda activate loftr
# with shell script
bash ./scripts/reproduce_test/indoor_ds.sh
# or
python test.py configs/data/scannet_test_1500.py configs/loftr/loftr_ds.py --ckpt_path weights/indoor_ds.ckpt --profiler_name inference --gpus=1 --accelerator="ddp"For visualizing the results, please refer to notebooks/visualize_dump_results.ipynb.
See Training LoFTR for more details.
If you find this code useful for your research, please use the following BibTeX entry.
@article{sun2021loftr,
title={{LoFTR}: Detector-Free Local Feature Matching with Transformers},
author={Sun, Jiaming and Shen, Zehong and Wang, Yuang and Bao, Hujun and Zhou, Xiaowei},
journal={{CVPR}},
year={2021}
}This work is affiliated with ZJU-SenseTime Joint Lab of 3D Vision, and its intellectual property belongs to SenseTime Group Ltd.
Copyright SenseTime. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.