Multi-Media Similarity Search

Hans Brouwer, Jasper Ketelaar, Jan Willem van Rhenen, and Thomas Pfann

Originally developed for TU Delft CS4065, now maintained by Hans Brouwer

Installation

To install this project we recommend using a conda environment with python version 3.7.

Clone the repository (and its submodules) and install the dependencies with the commands below.

conda create -n mmss python=3.7
git clone --recursive https://gitlab.ewi.tudelft.nl/cs4065/2020-2021/team23/multi-media-similarity-search.git
cd multi-media-similarity-search
conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=10.2
pip install -r requirements.txt

The modules specified in requirements.txt do not always come together nicely, because the specifics of the requirements depend on the operating system you are using and your physical device configuration (whether or not you have a GPU that has CUDA compatible). If you run into errors with the dependencies see the Troubleshooting section below.

Usage

python -m database.insert -h

# add images to search index
python -m database.insert /path/to/directory/of/images
python -m database.insert /path/to/another/directory
...
python -m database.insert /path/to/last/directory

python -m database.search -h

# search database using a text query
python -m database.search

The evaluation results from the report can be reproduced by running the evaluation scripts in evaluation/

python -m evaluation.coco
python -m evaluation.oxbuild
python -m evaluation.performance

# n.b. the dataset must first be downloaded from the link at the top of the file
# text or image queries are supported, for images just supply the path
# multiple queries can be separated with the pipe operator: |
python -m evaluation.qualitative

Architecture

cache/              # intermediate values and database files should be stored here
    modelzoo/       # pretrained model checkpoints etc.

database/
    db.py           # Main database implementation. Keeps track of different features in
                    # individual faiss indices. Provides functionality to search indices

    arguments.py    # Argument parsers for following scripts
    insert.py       # Script to insert new images into the database
    search.py       # Script to search the database based on text queries

processors/
    base.py         # The base Processor interface. This class encapsulates a specific feature processor
                    # or algorithm that transforms a single raw input data point into
                    # its corresponding embedding. Handles loading of feature processor weights and
                    # feature processor-specific preprocessing of raw data

    *.py            # Implementations of specific processors

features/
    data.py         # torch.Dataset implementations to load in raw images and videos

    feature.py      # The base Feature interface. This handles iterating over the data
                    # to transform it to the specified embedding space. Distributes work
                    # using multiprocessing / multi-GPU where applicable

    primitives.py   # Features that wrap around either a single Processor or multiple Processors.
                    # Generally different features can be implemented simply by writing
                    # the corresponding Processor class and wrapping its execution in one of
                    # these primitives

    registry.py     # Stores information on different features for easy access. This is
                    # where insertion and search procedures per feature are specified so
                    # that the rest of the code can rely on one unified API

Troubleshooting

We describe the installation process if your configuration is not directly compatible with the requirements.txt.

If you are working from a Linux distro and have a CUDA capable GPU with CUDA Toolkit installed and configured you can go directly to step 3 and everything should work

1. Installing PyTorch

Since CLIP can cause some awkward dependency issues, installing this first seems to be the way to go without running into issues later on.

conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0

This installation assumes you have a CUDA capable device, replace the cudatoolkit version with the one installed on your device. If your device does not have a GPU with CUDA capabilities, you can replace cudatoolkit with cpuonly.

2. Installing faiss

The next step should be installing faiss. They have GPU and CPU variations of the package however the GPU variation is only available on Linux systems so if you are not installing from a Linux system then you have to install the CPU variation. Below are the three commands to execute depending on your system and specificity.

CPU-only version:

conda install -c pytorch faiss-cpu

GPU without specific CUDA version:

conda install -c pytorch faiss-gpu

GPU for a specific CUDA version (the same version you specified in step 1)

conda install -c pytorch faiss-gpu cudatoolkit=10.0

3. Installing via requirements.txt

Once these steps have been completed you should be able to install any of the missing packages directly. It is also possible that you can do this without the first two steps given the right conditions, as explained in the installation overview.

pip install -r requirements.txt

JCBrouwer/mmss