This repository contains the official source code used to produce the results reported in the following paper:
Hierarchy-based Image Embeddings for Semantic Image Retrieval.
Björn Barz and Joachim Denzler.
IEEE Winter Conference on Applications of Computer Vision (WACV), 2019.
If you use this code, please cite that paper.
Table of Contents
Features extracted and aggregated from the last convolutional layer of deep neural networks trained for classification have proven to be useful image descriptors for a variety of tasks, e.g., transfer learning and image retrieval. Regarding content-based image retrieval, it is often claimed that visually similar images are clustered in this feature space. However, there are two major problems with this approach:
- Visual similarity does not always correspond to semantic similarity. For example, an orange may appear visually similar to an orange bowl, but they are not related at all from a semantic point of view. Similarly, if users provide a query image of a palm tree, they are probably more interested in semantically similar images of other trees such as oaks and maples than in images of spiders.
- The classification objective does not enforce a high distance between different classes, so that the nearest neighbors of some images may belong to completely different classes.
Hierarchy-based semantic embeddings overcome these issues by embedding images into a feature space where the dot product corresponds directly to semantic similarity.
To this end, the semantic similarity between classes is derived from a class taxonomy specifying is-a relationships between classes. These pair-wise similarities are then used for explicitly computing optimal target locations for all classes in the feature space. Finally, a CNN is trained to maximize the correlation between all images and the embedding of their respective class.
The learning process is divided into two steps:
- Computing target class embeddings based on a given hierarchy.
- Training the CNN to map images onto those embeddings.
In the following, we provide a step-by-step example for the CIFAR-100 dataset.
We derived a class taxonomy for CIFAR-100 from WordNet, but took care that our taxonomy is a tree, which is required for our method.
The hierarchy is encoded in the file Cifar-Hierarchy/cifar.parent-child.txt as a set of parent child tuples.
For example the line 100 53 specifies that class 53 is a direct child of class 100.
A more human-readable version of the hierarchy can be found in Cifar-Hierarchy/hierarchy.txt and can be translated into the encoded version using Cifar-Hierarchy/encode_hierarchy.py
Given this set of parent-child tuples, target class embeddings can be computed as follows:
python compute_class_embedding.py \
--hierarchy Cifar-Hierarchy/cifar.parent-child.txt \
--out embeddings/cifar100.unitsphere.pickleIf your hierarchy contains child-parent instead of parent-child tuples or the class labels are strings instead of integers, you can pass the arguments --is_a or --str_ids, respectively.
By default, the number of features in the embedding space equals the number of classes. If you would like to have an embedding space with less dimensions that approximates the semantic relationships between classes, specify the desired number of feature dimensions with --num_dim and also pass --method approx_sim.
The result will be a pickle file containing a dictionary with the following items:
embedding: a numpy array whose rows are the embeddings of the classes.ind2label: a list with the original labels of all classes, in the order corresponding to the rows ofembedding.label2ind: a dictionary mapping labels to the corresponding row index inembedding.
Hierarchies for the North American Birds and ILSVRC 2012 datasets can be found in NAB-Hierarchy/hierarchy.txt and ILSVRC/wordnet.parent-child.mintree.txt.
The corresponding pre-computed embeddings are stored in embeddings/nab.unitsphere.pickle and embeddings/imagenet_mintree.unitsphere.pickle, respectively.
For ILSVRC, three different taxonomies are provided:
- ILSVRC/wordnet.parent-child.txt: The complete WordNet taxonomy for all synsets contained in ImageNet. This is not a tree! Do not use it directly.
- ILSVRC/wordnet.parent-child.mintree.txt: A tree derived from the complete taxonomy, starting with all classes with a unique root-path and then adding those paths to other classes which would cause the fewest changes to the tree. This is used for training.
- ILSVRC/wordnet.parent-child.pruned.txt: The complete WordNet taxonomy for all synsets, but with all children of ILSVRC classes removed. This is not a tree, but is used for evaluation.
After having computed the target class embeddings based on the hierarchy, we can start with training a CNN for mapping the images from the training dataset onto the embeddings of their respective classes. First, download CIFAR-100 from here and extract it to some directory. Then run:
python learn_image_embeddings.py \
--dataset CIFAR-100 \
--data_root /path/to/your/cifar/directory \
--embedding embeddings/cifar100.unitsphere.pickle \
--architecture resnet-110-fc \
--cls_weight 0.1 \
--model_dump cifar100-embedding.model.h5 \
--feature_dump cifar100-features.pickleThis will train a variant of ResNet-100 with twice the number of channels per block for 372 epochs using Stochastic Gradient Descent with Warm Restarts (SGDR).
Thus, it is normal to see a drop of performance after epochs 12, 36, 84, and 180, where the restarts happen.
The resulting model will be stored as cifar100-embedding.model.h5 and pre-computed features for the test dataset will be written to the pickle file cifar100-features.pickle.
This method trains a network with a combination of two objectives: an embedding loss and a classification loss.
For training with the embedding loss only, just omit the --cls_weight argument.
A set of pre-trained models can be found below.
To evaluate your learned model and features, two scripts are provided.
Several retrieval metrics for the features learned in the previous step can be computed as follows:
python evaluate_retrieval.py \
--dataset CIFAR-100 \
--data_root /path/to/your/cifar/directory \
--hierarchy Cifar-Hierarchy/cifar.parent-child.txt \
--classes_from embeddings/cifar100.unitsphere.pickle \
--feat cifar100-features.pickle \
--label "Semantic Embeddings"This provides hierarchical precision in terms of two different measures for semantic similarity between classes: Wu-Palmer Similarity (WUP) and the height of the lowest common subsumer (LCSH). We used the latter in our paper.
If you want to obtain mAHP@250, as in the paper, instead of mAHP over the entire ranking, pass --clip_ahp 250 in addition.
The classification accuracy can be evaluated as follows:
python evaluate_classification_accuracy.py \
--dataset CIFAR-100 \
--data_root /path/to/your/cifar/directory \
--hierarchy Cifar-Hierarchy/cifar.parent-child.txt \
--classes_from embeddings/cifar100.unitsphere.pickle \
--architecture resnet-100-fc \
--batch_size 100 \
--model cifar100-embedding.model.h5 \
--layer prob \
--prob_features yes \
--label "Semantic Embeddings with Classification"If you have learned a semantic embedding model without classification objective, you can perform classification by assigning samples to the nearest class embedding as follows:
python evaluate_classification_accuracy.py \
--dataset CIFAR-100 \
--data_root /path/to/your/cifar/directory \
--hierarchy Cifar-Hierarchy/cifar.parent-child.txt \
--classes_from embeddings/cifar100.unitsphere.pickle \
--architecture resnet-100-fc \
--batch_size 100 \
--model cifar100-embedding.model.h5 \
--layer l2norm \
--centroids embeddings/cifar100.unitsphere.pickle \
--label "Semantic Embeddings"The following values can be specified for --dataset:
- CIFAR-10: Interface to the CIFAR-10 dataset with 10 classes.
- CIFAR-100: Interface to the CIFAR-100 dataset with 100 classes.
- CIFAR-100-a: The first 50 classes of CIFAR-100.
- CIFAR-100-b: The second 50 classes of CIFAR-100.
- CIFAR-100-b-consec: The second 50 classes of CIFAR-100, but numbered from 0-49 instead of 50-99.
- ILSVRC: Interface to the ILSVRC 2012 dataset.
- ILSVRC-caffe: ILSVRC 2012 dataset with Caffe-style pre-processing (i.e., BGR channel ordering and no normalization of standard deviation).
- NAB: Interface to the NABirds dataset, expecting images in the sub-directory
images. - NAB-caffe: NABirds dataset with Caffe-style pre-processing.
- NAB-ilsvrcmean: NABirds dataset using channel-wise mean and standard deviation computed on ILSVRC.
For ILSVRC, you need to move the test images into sub-directories for each class. This script could be used for this, for example.
Own dataset interfaces can be defined in datasets.py.
For CIFAR:
- simple: The Plain-11 architecture, a strictly sequential, shallow CNN with 11 trainable layers. Good for conducting quick experiments.
For training this network architecture, we used
--max_decay 0.1in addition to the other arguments provided above for the ResNet. This causes a continuous decay of the learning rate so that the final learning rate will be 10 times less than the initial one. - resnet-110: The standard ResNet-110.
- resnet-110-fc: A variant of ResNet-110 with twice the number of channels per block. In contrast to the standard ResNet-110, it will always have a fully-convolutional layer after the global average pooling, even when used for learning embeddings.
- wrn-28-10: A Wide Residual Network with depth 28 and width 10.
- pyramidnet-272-200: A Deep Pyramidal Residual Network. Provides better performance than ResNet, but is also much slower.
For ImageNet and NABirds:
- resnet-50: The standard ResNet-50 implementation from
keras-applications.
For CIFAR:
- resnet-32: The standard ResNet-32.
- densenet-100-12: A Densely Connected Convolutional Network with depth 100 and growth-rate 12.
For ImageNet and NABirds:
- rn18, rn34, rn50, rn101, rn152, rn200: ResNets with different depths. Requires the keras-resnet package, but be aware that batch normalization is broken in version 0.1.0. Thus, you need to either use an earlier or later version or merge this pull request.
- nasnet-a: The NasNet-A implementation from
keras-applications.
The previous sections have shown in detail how to learn semantic image embeddings for CIFAR-100. In the following, we provide the calls to learn_image_embeddings.py that we used to train our semantic embedding models (including classification objective) on the ILSVRC 2012 and NABirds datasets.
# ILSVRC
python learn_image_embeddings.py \
--dataset ILSVRC \
--data_root /path/to/imagenet/ \
--embedding embeddings/imagenet_mintree.unitsphere.pickle \
--architecture resnet-50 \
--loss inv_corr \
--cls_weight 0.1 \
--lr_schedule SGDR \
--sgdr_base_len 80 \
--epochs 80 \
--max_decay 0 \
--batch_size 128 \
--gpus 2 \
--model_dump imagenet_unitsphere-embed+cls_rn50.model.h5
# NAB (from scratch)
python learn_image_embeddings.py \
--dataset NAB \
--data_root /path/to/nab/ \
--embedding embeddings/nab.unitsphere.pickle \
--architecture resnet-50 \
--loss inv_corr \
--cls_weight 0.1 \
--lr_schedule SGDR \
--sgdr_max_lr 0.5 \
--max_decay 0 \
--epochs 180 \
--batch_size 128 \
--gpus 2 \
--read_workers 10 \
--queue_size 20 \
--model_dump nab_unitsphere-embed+cls_rn50.model.h5
# NAB (fine-tuned)
python learn_image_embeddings.py \
--dataset NAB-ilsvrcmean \
--data_root /path/to/nab/ \
--embedding embeddings/nab.unitsphere.pickle \
--architecture resnet-50 \
--loss inv_corr \
--cls_weight 0.1 \
--finetune imagenet_unitsphere-embed+cls_rn50.model.h5 \
--finetune_init 8 \
--lr_schedule SGDR \
--sgd_lr 0.1 \
--sgdr_max_lr 0.5 \
--max_decay 0 \
--epochs 180 \
--batch_size 128 \
--gpus 2 \
--read_workers 10 \
--queue_size 20 \
--model_dump nab_unitsphere-embed+cls_rn50_finetuned.model.h5- Python 3
- numpy
- numexpr
- keras >= 2.2.0
- tensorflow
- sklearn
- scipy
- pillow
- matplotlib
| Dataset | Model | mAHP@250 | Balanced Accuracy |
|---|---|---|---|
| CIFAR-100 | Plain-11 | 82.05% | 74.10% |
| CIFAR-100 | ResNet-110-fc | 83.29% | 76.60% |
| CIFAR-100 | PyramidNet-272-200 | 86.38% | 80.49% |
| NABirds | ResNet-50 (from scratch) | 73.99% | 59.46% |
| NABirds | ResNet-50 (fine-tuned) | 81.46% | 69.49% |
| ILSVRC | ResNet-50 | 82.42% | 69.18% |
The pre-trained models provided above assume input images to be given in RGB color format and standardized by subtracting a dataset-specific channel-wise mean and dividing by a dataset-specific standard deviation. The means and standard deviations for each dataset are provided in the following table.
| Dataset | Mean | Standard Deviation |
|---|---|---|
| CIFAR-100 | [129.30386353, 124.06987, 112.43356323] |
[68.17019653, 65.39176178, 70.4180603] |
| NABirds (from scratch) | [125.30513277, 129.66606421, 118.45121113] |
[57.0045467, 56.70059436, 68.44430446] |
| ILSVRC, NABirds (fine-tuned) | [122.65435242, 116.6545058, 103.99789959] |
[71.40583196, 69.56888997, 73.0440314] |
The default input image size of the ILSVRC and NABirds models is 224x224, cropped from a scaled version of the image, resized so that the smaller side is 256 pixels wide.
Sometimes, loading of the pre-trained models fails with the error message "unknown opcode". In the case of this or other issues, you can still create the architecture yourself and load the pre-trained weights from the model files provided above. For CIFAR-100 and the resnet-110-fc architecture, for example, this can be done as follows:
import keras
import utils
from learn_image_embeddings import cls_model
model = utils.build_network(100, 'resnet-110-fc')
model = keras.models.Model(
model.inputs,
keras.layers.Lambda(utils.l2norm, name = 'l2norm')(model.output)
)
model = cls_model(model, 100)
model.load_weights('cifar_unitsphere-embed+cls_resnet-110-fc.model.h5')