ArtMiner

Pytorch implementation of Paper "Discovering Visual Patterns in Art Collections with Spatially-consistent Feature Learning"

[PDF] [Project webpage] [Slides]

If our project is helpful for your research, please consider citing :

@inproceedings{shen2019discovery,
          title={Discovering Visual Patterns in Art Collections with Spatially-consistent Feature Learning},
          author={Shen, Xi and Efros, Alexei A and Aubry, Mathieu},
          booktitle={Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
          year={2019}
        }

Table of Content

Installation
Single Shot Detection
Feature Learning
Discovery
Clusters in Dataset

Installation

Dependencies

The code can be used in Linux system with the the following dependencies: Python 2.7, Pytorch 0.3.0.post4, torchvision, tqdm, ujson, cv2, scipy, skimage

We recommend to utilize virtual environment to install all dependencies and test the code. One choice is virtualenv.

To install pytorch 0.3.0.post4 + cuda 8.0 (For other cuda version (9.0, 7.5), the only modification is to change cu80 to your cuda version):

pip install https://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl

To install other dependencies:

bash requirement.sh

Dataset and Model

We release the Brueghel dataset containing 1,587 artworks and our annotations (10 visual details, 273 instances with bounding box annotations).

One can directly download it with the following command:

cd data
bash download_dataset.sh (Brueghel + Ltll + Oxford)

Or click here (~400M) to download the dataset and our annotations.

A full description is provided in our project website.

To download pretrained models (ResNet18 + Brueghel + Ltll + Oxford) :

cd model
bash download_models.sh

Single Shot Detection

To test performance on single shot detection:

cd single_shot_detection
bash run_FeatImageNet.sh # ImageNet feature
bash run_FeatBrueghel.sh # Brueghel feature

You should obtain the results in the table 1 in the paper,

Feature	Cosine Similarity
ImageNet	58.0
Ours (trained on Brueghel)	75.3

The visual results will be saved into the visualDir that you indicate, some examples are shown below:

	Query	Rank 1st	Rank 2nd	Rank 3rd	Rank 4th
ImageNet
Ours

Feature Learning

Visualize Training Data

It is highly recommended to visualize the training data before the training.

One command example is :

cd feature_learning/visualzation/
bash visBrueghelLtllOxford.sh

The training patches will be saved into the output directory, some samples are shown below.

Red / Blue / Green regions indicate Proposal Regions / Verification Regions / Positive Regions.


Brueghel Image 1	Brueghel Image 2


Ltll Image 1	Ltll Image 2

Train

To train on Brueghel / Ltll / Oxford dataset :

cd feature_learning/
bash brughel.sh # training on Brueghel
bash ltll.sh # training on LTLL
bash oxford.sh # training on Oxford

To train on your own dataset, please refer to:

cd feature_learning/
python train.py --help

Discovery

Pair Discovery

To launch discovery between a pair of images, please utilize the script in discovery/pair_discovery.py. One example command is in discovery/pair_discovery.sh :

cd discovery
bash pair_discovery.sh

The results of discovery between the pair of images :


Discovery Image 1 with ImageNet Feature	Discovery Image 2 ImageNet Feature

Discovery Image 1 with Brueghel Feature	Discovery Image 2 Brueghel Feature

Ltll

To get classification results on Ltll :

cd discovery
bash demo_ltll.sh

You should obtain the results in the table 2 in the paper, note that there is RANSAC in the algorithm, but we find very small variation by setting number of iterations to 1000 :

Feature	Disovery
ImageNet	80.9
Ours (trained on LTLL)	88.5

Oxford

To get retrieval results on Oxford5K :

cd discovery
bash demo_oxford.sh

You should obtain the results in the table 2 in the paper :

Feature	Disovery
ImageNet	85.0
Ours (trained on Oxford)	85.7

Clusters in Dataset

To get clusters of images in datasets, we propose the following pipeline (taking LTLL as example):

Discovery scores for all pairs in the dataset

cd discovery
bash LTLLPairScore.sh

Thresholding the pairs and compute connected component in the graph

cd discovery
bash LTLLCluster.sh

The clusters of LTLL can be seen here

XezXey/ArtMiner