/doodle2search

Doodle to Search: Practical Zero Shot Sketch Based Image Retrieval

Primary LanguagePython

Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval

PyTorch | Arxiv | Project

PyTorch implementation of our D2S model for zero-shot sketch-based image retrieval:
Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval
Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados and Yi-Zhe Song
CVPR, 2019

Retrieval Results

Sketchy






TU-Berlin






QuickDraw






Prerequisites

  • Linux (tested on Ubuntu 16.04)
  • NVIDIA GPU + CUDA CuDNN
  • 7z
sudo apt-get install p7zip-full

Getting Started

Introduction

We took the first steps to move towards practical zero shot sketch based image retrieval systems (see the paper for more detail). To this end, we have used Quick Draw! to curate the sketches and for the images we would like to thanks Flickr API for such an amazing API.

The structure of this repo is as follows:

  1. Installation
  2. Getting the data
  3. How to train models
  4. At last how to test and evaluate

Installation

  • Clone this repository
git clone https://github.com/sounakdey/doodle2search.git
cd doodle2search
  • Install the requirements (not checked)
pip3 install -r requirements.txt

Download datasets

  • Sketchy
  • TU-Berlin
bash download_datasets.sh

Train

Finally we are ready to train. Magical words are:

python3 src/train.py sketchy_extended --data_path <mention the data path of the dataset>

The first argument is the dataset name, which you can replace it with tuberlin_extend or quickdraw_extend. You can check the options.py for changing a lot of the options such dimension size, different models, hyperparameters, etc.

Test

Sketchy
python3 src/test.py sketchy_extend --data_path <mention the data path of the dataset> --load <path of the trained models>

Citation

@InProceedings{Dey_2019_CVPR,
    author = {Dey, Sounak and Riba, Pau and Dutta, Anjan and Llados, Josep and Song, Yi-Zhe},
    title = {Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval},
    booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {June},
    year = {2019}
}

Conclusion

Thank you and sorry for the bugs!

Author