Pytorch code for the classification part of our ICMR 2019 paper Context-Aware Embeddings for Automatic Art Analysis. For the retrieval part, check this other repository.
-
Download dataset from here.
-
Clone the repository:
git clone https://github.com/noagarcia/context-art-classification.git
-
Install dependencies:
- Python 2.7
- pytorch (
conda install pytorch=0.4.1 cuda90 -c pytorch
) - torchvision (
conda install torchvision
) - visdom (check tutorial here)
- pandas (
conda install -c anaconda pandas
) - gensim (
conda install -c anaconda gensim
)
-
For the KGM model, download the pre-computed graph embeddings from here, and save the file into the
Data/
directory.
-
To train MTL multi-classifier run:
python main.py --mode train --model mtl --dir_dataset $semart
-
To train KGM classifier run:
python main.py --mode train --model kgm --att $attribute --dir_dataset $semart
Where $semart
is the path to SemArt dataset and $attribute
is the classifier type (i.e. type
, school
, time
, or author
).
-
To test MTL multi-classifier run:
python main.py --mode test --model mtl --dir_dataset $semart
-
To test KGM classifier run:
python main.py --mode test --model kgm --att $attribute --dir_dataset $semart --model_path $model-file
Where $semart
is the path to SemArt dataset, $attribute
is the classifier type (i.e. type
, school
, time
, or author
), and $model-file
is the path to the trained model.
You can download our pre-trained models from:
Classification results on SemArt:
Model | Type | School | Timeframe | Author |
---|---|---|---|---|
VGG16 pre-trained | 0.706 | 0.502 | 0.418 | 0.482 |
ResNet50 pre-trained | 0.726 | 0.557 | 0.456 | 0.500 |
ResNet152 pre-trained | 0.740 | 0.540 | 0.454 | 0.489 |
VGG16 fine-tuned | 0.768 | 0.616 | 0.559 | 0.520 |
ResNet50 fine-tuned | 0.765 | 0.655 | 0.604 | 0.515 |
ResNet152 fine-tuned | 0.790 | 0.653 | 0.598 | 0.573 |
ResNet50+Attributes | 0.785 | 0.667 | 0.599 | 0.561 |
ResNet50+Captions | 0.799 | 0.649 | 0.598 | 0.607 |
MTL context-aware | 0.791 | 0.691 | 0.632 | 0.603 |
KGM context-aware | 0.815 | 0.671 | 0.613 | 0.615 |
Paintings with the highest scores for each class:
@InProceedings{Garcia2017Context,
author = {Noa Garcia and Benjamin Renoust and Yuta Nakashima},
title = {Context-Aware Embeddings for Automatic Art Analysis},
booktitle = {Proceedings of the ACM International Conference on Multimedia Retrieval},
year = {2019},
}