This repository contains supplemental scripts and data used in the experiments presented in the paper.
-
Download the imageset - It is recommended that the image filename include the class information. The images can be in class-related subfolders; if not, it can be placed in a single subfolder (ex.
/data/1/*.jpg
) -
Preprocess the imageset - We recommend scaling/center-cropping your images to 227x227 first. We used imagemagick:
$ mogrify -path imageset/# -format jpg -resize "227x227^" -gravity center -crop 227x227+0+0 +repage imageset/#/*.jpg
-
Download/install DEMUD - Available from https://github.com/wkiri/DEMUD
-
Extract features - Extract features from the images by using
DEMUD/scripts/cnn_feat_extraction/feat_csv.py
. The extracted features will be saved as a CSV, with the first column being the image name.
Note: You will need to install Caffe and to specify the trained Caffe model from which the features will be extracted. We used Caffe's pre-trained network called bvlc_reference_caffenet
.
-
Run DEMUD on features - Run DEMUD by adding the path to the feature CSV in
demud.config
atfloatdatafile
and runningpython demud.py -v --init-item=svd --k=50
It will outputrecon-cnn-k=50...
, the reconstructed features,resid-cnn-k50...
, the residual features,select-cnn-k=50...
, the features of the selected image, andselections-k50.csv
, a list of selected image names (in corresponding order with the other outputs). -
Download D&B 2015 - Available from https://lmb.informatik.uni-freiburg.de/resources/software.php
-
Visualize the output features - Using
src/demud_inversion_2015.py
provided in this repo (see below for instructions) -
Plot discovery rates (optional) - Using
src/plot/plot_exp.py
provided in this repo -
Organize your visualizations - Using
src/gen_html.py
provided in this repo
src/gen_html.py
collects the results from DEMUD and Dosovitskiy and Brox inversions into a single HTML file for convenient viewing. It requires the following paths to be edited into the appropriate variables:
outputname
- The path of the html file to be generatedimageset_dir
- The directory containing the full imageset, but not in subdirectoriesdosov_dir
- The directory containing the inversions generated by Dosovitskiy and Brox. It assumes the following subdirectories:dosov_dir/select/
- Inversions of the features in theselect-cnn-*.csv
output from DEMUD-v
dosov_dir/recon/
- Inversions of the features in therecon-cnn-*.csv
output from DEMUD-v
dosov_dir/resid/
- Inversions of the features in theresid-cnn-*.csv
output from DEMUD-v
demud_path
- Theselections-kn.csv
output file from DEMUD-v
perfplot_path
- The path to the discovery performance plot, if it was generated.
src/plot/plot_exp.py
and src/plot/util_plot.py
were used to generate the plots shown in the paper. plot_exp.py
requires the directory containing DEMUD result folders. Note that, since these scripts were used specifically for our set of experiments, they may not be easy to use for other experiments.
src/demud_inversion_2015.py
is a modified version of demo.py
from Dosovitskiy and Brox, 2015. It reads each of the output CSVs from DEMUD -v/--cnn
(selected features, reconstructed features, and residual features), and inverts them for visualization. The required pretrained model definitions are available here
src/build_imageset/ext_from_train.py
builds the balanced ImageNet dataset used in the experiments from the ILSVRC 2012 training set. The classes to be pulled are specified in build_imageset/random_classes.txt
. The files are renamed to match the pattern classID_imageNO.jpg
and placed in their respective subfolders; for example, an image may be /output/n02009912/n02009912_0.jpg
.
data/sun-class/
includes an additional "sun" class to the Mars image dataset available on Zenodo. These 21 additional images were included for the experiments presented in the paper.