This package allows researches to apply the described networks to match image patches and extract corresponding patches.
We tried to make the code as easy to use as possible. The original models were trained with Torch ( http://torch.ch ) and we release them in Torch7 and binary formats with C++ bindings which do not require Torch installation. Thus we provide example code how to use the models in Torch, MATLAB and with OpenCV http://opencv.org
Copyright © 2015 Ecole des Ponts, Universite Paris-Est
All Rights Reserved. A license to use and copy this software and its documentation solely for your internal research and evaluation purposes, without fee and without a signed licensing agreement, is hereby granted upon your download of the software, through which you agree to the following:
- the above copyright notice, this paragraph and the following three paragraphs will prominently appear in all internal copies and modifications;
- no rights to sublicense or further distribute this software are granted;
- no rights to modify this software are granted; and
- no rights to assign this license are granted.
Please Contact Prof. Nikos Komodakis, 6 Avenue Blaise Pascal - Cite Descartes, Champs-sur-Marne, 77455 Marne-la-Vallee cedex 2, France for commercial licensing opportunities, or for further distribution, modification or license rights.
Created by Sergey Zagoruyko and Nikos Komodakis. http://imagine.enpc.fr/~komodakn/
Please cite the paper below if you use this code in your research.
Sergey Zagoruyko, Nikos Komodakis, "Learning to Compare Image Patches via Convolutional Neural Networks". http://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Zagoruyko_Learning_to_Compare_2015_CVPR_paper.pdf, bib:
@InProceedings{Zagoruyko_2015_CVPR,
author = {Zagoruyko, Sergey and Komodakis, Nikos},
title = {Learning to Compare Image Patches via Convolutional Neural Networks},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2015}
}
Update 4 (April 2017) dead links for models and datasets fixed
Update 3 (July 2016) training code released
Update 2 (February 2016) caffe models released
Update 1 (January 2016) cudnn models removed because cudnn.convert was out
The original dataset website is down, you can still download the files here:
http://icvl.ee.ic.ac.uk/vbalnt/notredame.zip
http://icvl.ee.ic.ac.uk/vbalnt/yosemite.zip
http://icvl.ee.ic.ac.uk/vbalnt/liberty.zip
We provide the models in Torch7 and binary format. The table from the paper is here for convenience.
All models expect input patches to be in [0;1] range before mean subtraction.
The models are not supposed to give outputs in [0;1] range, the outputs are not normalized
| Train set | Test set | 2ch | 2ch2stream | 2chdeep | siam | siam2stream |
|---|---|---|---|---|---|---|
| yosemite | notredame | 2.74 | 2.11 | 2.43 | 5.62 | 5.23 |
| yosemite | liberty | 8.59 | 7.2 | 7.4 | 13.48 | 11.34 |
| notredame | yosemite | 6.04 | 4.09 | 4.38 | 13.23 | 10.44 |
| notredame | liberty | 6.04 | 4.85 | 4.56 | 8.77 | 6.45 |
| liberty | yosemite | 7 | 5 | 6.18 | 14.76 | 9.39 |
| liberty | notredame | 2.76 | 1.9 | 2.77 | 4.04 | 2.82 |
Models in nn format can be loaded and used without CUDA support in Torch. To enable CUDA support model:cuda() call required.
An archive with all models (binary and in torch format) is available at: https://s3.amazonaws.com/modelzoo-networks/cvpr2015matching_networks.tar.gz
To install torch follow http://torch.ch/ Check torch folder for examples. Match patches on CPU:
require 'nn'
N = 76 -- the number of patches to match
patches = torch.rand(N,2,64,64):float()
-- load the network
net = torch.load'../networks/2ch/2ch_liberty.t7'
-- in place mean subtraction
local p = patches:view(N,2,64*64)
p:add(-p:mean(3):expandAs(p))
-- get the output similarities
output = net:forward(patches)Conversion to a faster cudnn backend is done by cudnn.convert(net, cudnn) function.
The code was tested to work in Linux (Ubuntu 14.04) and OS X 10.10, although we release all the source code to enable usage in other operating systems.
We release CUDA code for now, CPU code might be added in the future. To install it you need to have CUDA with the up-to-date CUDA driver (those are separate packages).
Install TH and THC:
cd /tmp
git clone https://github.com/torch/torch7.git
cd torch7/lib/TH
mkdir build; cd build
cmake ..; make -j4 install
cd /tmp
git clone https://github.com/torch/cutorch.git;
cd cutorch/lib/THC
mkdir build; cd build
cmake ..; make -j4 install
Clone and compile this repository it with:
git clone --recursive https://github.com/szagoruyko/cvpr15deepmatch
cd cvpr15deepmatch
mkdir build; cd build;
cmake .. -DCMAKE_INSTALL_PREFIX=../install
make -j4 install
Then you will have loadNetwork function defined in src/loader.h, which expects the state and the path to a network in binary format on input. A simple example:
THCState *state = (THCState*)malloc(sizeof(THCState));
THCudaInit(state);
cunn::Sequential::Ptr net = loadNetwork(state, "networks/siam/siam_notredame.bin");
THCudaTensor *input = THCudaTensor_newWithSize4d(state, 128, 2, 64, 64);
THCudaTensor *output = net->forward(input); // output is 128x1 similarity score tensorOnly 2D and 4D tensors accepted on input.
Again, all binary models expect input patches to be in [0;1] range before mean subtraction.
After you build everything and download the networks run test with run_test.sh. It will download a small test_data.bin file.
Building Matlab bindings requires a little bit of user intervention. Open matlab/make.m file in Matlab and put your paths to Matlab and include/lib paths of TH and THC, then run >> make. Mex file will be created.
To initialize the interface do
deepcompare('init', 'networks/2ch/2ch_notredame.bin');
To reset do
deepcompare('reset')
To propagate through the network:
deepcompare('forward', A)
A can be 2D, 3D or 4D array, which is converted inside to 2D or 4D array (Matlab is col-major and Torch is row-major so the array is transposed):
| #dim | matlab dim | torch dim |
|---|---|---|
| 2d | N x B | B x N |
| 3d | 64 x 64 x N | 1 x N x 64 x 64 |
| 4d | 64 x 64 x N x B | B x N x 64 x 64 |
2D or 4D tensor is returned. In case of full network propagation for example the output will be 2D: 1 x B, if input was B x 2 x 64 x 64.
To set the number of GPU to be used (the numbering starts from 1):
deepcompare('set_device', 2)
Print the network structure:
deepcompare('print')
To enable splitting the computations of descriptor and decision parts in siamese networks we saved their binary parts.
- siamese network:
| Train Set | siam_desc | siam_decision |
|---|---|---|
| yosemite | 3.47 MB,siam_desc_yosemite.bin | 1.00 MB,siam_decision_yosemite.bin |
| notredame | 3.47 MB,siam_desc_notredame.bin | 1.0MB,siam_decision_notredame.bin |
| liberty | 3.47 MB,siam_desc_liberty.bin | 1.00 MB,siam_decision_liberty.bin |
- siam-2stream network:
| Train Set | siam2stream_desc | siam2stream_decision |
|---|---|---|
| yosemite | 9.16 MB,siam2stream_desc_yosemite.bin | 4.01 MB,siam2stream_decision_yosemite.bin |
| notredame | 9.16 MB,siam2stream_desc_notredame.bin | 4.01 MB,siam2stream_decision_notredame.bin |
| liberty | 9.16 MB,siam2stream_desc_liberty.bin | 4.01 MB,siam2stream_decision_liberty.bin |
OpenCV example is here to demonstrate how to use the deep CNN models to match image patches, how to preprocess the patches and use the proposed API.
Depends on OpenCV 3.0. To build the example do
cd build;
cmake -DWITH_OPENCV=ON -DOpenCV_DIR=/opt/opencv .; make -j8
Here /opt/opencv has to be a folder where OpenCV is built. If you have it installed, you don't need to add it, -DWITH_OPENCV=ON will be enough.
To run the example download the images:
wget https://raw.githubusercontent.com/openMVG/ImageDataset_SceauxCastle/master/images/100_7100.JPG
wget https://raw.githubusercontent.com/openMVG/ImageDataset_SceauxCastle/master/images/100_7101.JPG
and run it:
./build/opencv/example networks/siam2stream/siam2stream_desc_notredame.bin 100_7100.JPG 100_7101.JPG
You have to use descriptor matching network in this example. Check the example code for explanation.
Thanks to the awesome @ajtulloch's torch2caffe models were converted to CAFFE format. Unfortunatelly only siam, 2ch and 2chdeep models could be converted at the time, other models will be converted as missing functionality is added to CAFFE.
Download link: https://s3.amazonaws.com/modelzoo-networks/cvpr2015networks-caffe.tar