Comparative Evaluation of Hand-Crafted and Learned Local Features

This repository contains the instructions and the code for evaluating feature descriptors on our image-based reconstruction benchmark. The details of our local feature benchmark can be found in our paper:

"Comparative Evaluation of Hand-Crafted and Learned Local Features".
J.L. Schönberger, H. Hardmeier, T. Sattler and M. Pollefeys. CVPR 2017.

Paper, Supplementary, Bibtex

You might also be interested in the HPatches benchmark by Balntas and Lenc et al. presented at CVPR 2017.

Benchmark Results

This list is updated with the latest benchmark results. If you want to submit your own results, please open a new issue or pull request in this repository. Note that the below table extends to the right and alternatively can be viewed in a code or text editor.

Metrics:

Dataset	Method	# Images	# Reg. Images	# Sparse Points	# Observations	Track Length	Obs. Per Image	Reproj. Error [px]	# Dense Points	Dense Error [2cm]	Dense Error [10cm]	Mean Pose Error [m]	Median Pose Error [m]	# Inlier Pairs	# Inlier Matches
Fountain	SIFT	11	11	10004	44923	4.49050	4083.91	0.298179	2970239	0.7678	0.8970	0.002412	0.002412	49	76644
	SIFT-PCA		11	14608	70058	4.79587	6368.91	0.389404	3021445	0.7677	0.8969	0.002413	0.002413	55	124066
	DSP-SIFT		11	14785	71041	4.80494	6458.27	0.410291	2999187	0.7677	0.8970	0.002413	0.002413	54	129122
	ConvOpt		11	14179	67308	4.74702	6118.91	0.370262	2999376	0.7677	0.8971	0.002413	0.002413	55	114343
	TFeat		11	13696	64110	4.68093	5828.18	0.352238	2969328	0.7677	0.8969	0.002412	0.002412	54	103260
	DeepDesc		11	13519	61478	4.54753	5588.91	0.353349	2972715	0.7677	0.8969	0.002413	0.002413	55	93708
	LIFT		11	10172	46272	4.54896	4206.55	0.594498	3019888	0.7678	0.8969	0.002413	0.002413	55	83318

Herzjesu	SIFT	8	8	4916	19684	4.00407	2460.50	0.319120	2373266	0.5737	0.7307	0.003533	0.003533	27	28955
	SIFT-PCA		8	7433	31116	4.18620	3889.50	0.421221	2372268	0.5735	0.7306	0.003534	0.003534	28	47384
	DSP-SIFT		8	7760	32494	4.18737	4061.75	0.447413	2376744	0.5734	0.7305	0.003533	0.003533	28	50613
	ConvOpt		8	6939	28638	4.12711	3579.75	0.396862	2375340	0.5737	0.7306	0.003533	0.003533	28	42199
	TFeat		8	6606	27021	4.09037	3377.62	0.381651	2377038	0.5734	0.7304	0.003533	0.003533	28	38573
	DeepDesc		8	6418	25139	3.91695	3142.38	0.379522	2380244	0.5734	0.7307	0.003533	0.003533	28	34591
	LIFT		8	7834	30925	3.94754	3865.62	0.625963	2375055	0.5738	0.7308	0.003533	0.003533	28	46090

South Building	SIFT	128	128	62780	353939	5.63777	2765.15	0.424381	1972543					1851	1003336
	SIFT-PCA		128	107674	650117	6.03783	5079.04	0.540539	1993853					3916	2019148
	DSP-SIFT		128	110394	664533	6.01965	5191.66	0.569184	1994432					3769	2079511
	ConvOpt		128	103602	617078	5.95624	4820.92	0.510579	2007852					4640	1856409
	TFeat		128	94589	566687	5.99105	4427.24	0.486924	1960970					3156	1567873
	DeepDesc		128	101154	558997	5.52620	4367.16	0.483270	2002399					6034	1463340
	LIFT		128	74607	399254	5.35143	3119.17	0.776213	1975540					3441	1168942

Madrid Metropolis	SIFT	1344	440	62729	416727	6.64329	947.107	0.525606	435563					14343	1740200
	SIFT-PCA		465	119244	702936	5.89494	1511.69	0.569103	537758					27664	3597662
	DSP-SIFT		476	107028	681222	6.36490	1431.14	0.639809	570224					21127	3155358
	ConvOpt		455	115134	634011	5.50672	1393.43	0.568558	561697					29765	3148104
	TFeat		439	90274	512470	5.67683	1167.36	0.538515	522327					18450	2135644
	DeepDesc		377	68110	348061	5.11028	923.239	0.526658	516535					19782	1570887
	LIFT		430	52755	337392	6.39545	784.633	0.758943	450562					13337	1498051

Gendarmenmarkt	SIFT	1463	950	169900	1010545	5.94788	1063.73	0.639777	1104976					28683	3292693
	SIFT-PCA		953	272118	1477833	5.43085	1550.72	0.692133	1240706					43413	5137545
	DSP-SIFT		975	321846	1732034	5.38156	1776.45	0.735691	1505886					56470	7648903
	ConvOpt		945	341591	1601383	4.68801	1694.59	0.696203	1342513					56905	6525056
	TFeat		953	297266	1445049	4.86113	1516.32	0.660397	1181279					39115	4685369
	DeepDesc		809	244925	949216	3.87554	1173.32	0.681721	921231					31134	2849341
	LIFT		942	180746	964485	5.33613	1023.87	0.830989	1386731					27879	2495028

Tower of London	SIFT	1576	702	142746	963821	6.75200	1372.96	0.530041	1126600					18716	3211444
	SIFT-PCA		692	137800	1090091	7.91067	1575.28	0.601250	1124538					12154	2455869
	DSP-SIFT		755	236598	1761435	7.44484	2333.03	0.638131	1143471					33500	8056825
	ConvOpt		719	274987	1732771	6.30128	2409.97	0.617079	1129334					39941	7542856
	TFeat		714	206142	1424696	6.91124	1995.37	0.572171	1182746					28388	5333355
	DeepDesc		551	196990	964750	4.89746	1750.91	0.545235	653579					25658	2745700
	LIFT		715	147851	1045724	7.07282	1462.55	0.721916	729060					23058	4079252

Alamo	SIFT	2915	743	120713	1384696	11.47100	1863.66	0.535782	611874					23526	7671821
	SIFT-PCA		746	108553	1377035	12.68540	1845.89	0.550041	564223					12766	4669536
	DSP-SIFT		754	144341	1815879	12.58050	2408.33	0.657329	629061					16925	10115750
	ConvOpt		703	102044	1001340	9.81283	1424.38	0.479573	452541					3962	850327
	TFeat		683	127642	1443116	11.30600	2112.91	0.521289	648970					16764	6356806
	DeepDesc		665	152537	1207394	7.91542	1815.63	0.479996	607091					16691	4196845
	LIFT		768	112984	1477294	13.07520	1923.56	0.734686	607487					23432	9117444

Roman Forum	SIFT	2364	1407	242192	1805253	7.45381	1283.05	0.610871	3097439					25447	6063636
	SIFT-PCA		1463	244556	1834598	7.50175	1254.00	0.613003	2799238					16437	4322039
	DSP-SIFT		1583	372573	2879238	7.72798	1818.85	0.708828	3748342					26416	9685465
	ConvOpt		1376	195305	1173254	6.00729	852.66	0.553454	3043274					11921	2111787
	TFeat		1450	271902	1963303	7.22063	1354.00	0.608724	3477858					19828	5584122
	DeepDesc		1173	174532	1275633	7.30887	1087.49	0.602312	2434123					9831	1834623
	LIFT		1434	220026	1608740	7.31159	1121.85	0.748830	2898383					17322	4732050

Cornell	SIFT	6514	4999	1010544	6317214	6.25130	1263.70	0.527172	12970087			1.536723	0.792612	71919	25603366
	SIFT-PCA		3049	640553	4335971	6.76911	1422.10	0.544811	6135281			11.498496	1.087624	26498	13793332
	DSP-SIFT		4946	1177916	7233500	6.14093	1462.49	0.674990	11066753			2.942793	1.001049	73922	26150621
	ConvOpt		1986	632613	4747658	7.50484	2390.56	0.569796	5321472			5.823939	0.903555	42129	18615334
	TFeat		5428	1499117	9830787	6.55772	1811.13	0.587575	15605086			2.125709	0.593038	89927	40640025
	DeepDesc		3489	1225780	6977970	5.69268	1999.99	0.552574	10159770			3.831561	0.695395	73973	28845684
	LIFT		3798	1455732	7377320	5.06777	1942.42	0.712310	10512321			3.113213	0.712312	81231	39812312

Runtime:

Method	Runtime	Hardware
SIFT	9.3s	(Intel E5-2697 2.60GHz CPU - single-threaded)
SIFT-PCA	10.5s	(Intel E5-2697 2.60GHz CPU - single-threaded)
DSP-SIFT	23.7s	(Intel E5-2697 2.60GHz CPU - single-threaded)
ConvOpt	49.9s	(Intel E5-2697 2.60GHz CPU, NVIDIA Titan X GPU)
DeepDesc	24.3s	(Intel E5-2697 2.60GHz CPU, NVIDIA Titan X GPU)
TFeat	11.8s	(Intel E5-2697 2.60GHz CPU, NVIDIA Titan X GPU)
LIFT	212.3s	(Intel E5-2697 2.60GHz CPU, NVIDIA Titan X GPU)

References:

SIFT: D.G. Lowe: Object Recognition from Local Scale-Invariant Features. ICCV, 1999. R. Arandjelovic and A. Zisserman. Three things everyone should know to improve object retrieval. CVPR, 2012.
SIFT-PCA: A. Bursuc, G. Tolias, and H. Jegou. Kernel local descriptors with implicit rotation matching. ACM Multimedia, 2015.
DSP-SIFT: J.Dong and S.Soatto. Domain-size pooling in local descriptors: DSP-SIFT. CVPR, 2015.
ConvOpt: K. Simonyan, A. Vedaldi, and A. Zisserman. Learning local feature descriptors using convex optimisation. PAMI, 2014.
DeepDesc: E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer. Discriminative learning of deep convolutional feature point descriptors. ICCV, 2015.
TFeat: V.Balntas, E.Riba, D.Ponsa, and K.Mikolajczyk. Learning local feature descriptors with triplets and shallow convolutional neural networks. BMVC, 2016.
LIFT: M. Kwang, E. Trulls, V. Lepetit, and P. Fua. LIFT: Learned Invariant Feature Transform. ECCV, 2016.

bfan/local-feature-evaluation

Comparative Evaluation of Hand-Crafted and Learned Local Features

Benchmark Results