Deep Unsupervised Contrastive Hashing for Large-Scale Cross-Modal Text-Image Retrieval in Remote Sensing

Unsupervised Contrastive Hashing for Cross-Modal Retrieval in Remote Sensing

G. Mikriukov, M. Ravanbakhsh, and B. Demіr, "Unsupervised Contrastive Hashing for Cross-Modal Retrieval in Remote Sensing", IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.

G. Mikriukov, M. Ravanbakhsh, and B. Demіr, "Deep Unsupervised Contrastive Hashing for Large-Scale Cross-Modal Text-Image Retrieval in Remote Sensing", arXiv:1611.08408, 2022.

If you use the code from this repository in your research, please cite the following paper:

@inproceedings{duch2022icassp,
  title={Unsupervised Contrastive Hashing for Cross-Modal Retrieval in Remote Sensing},
  author={G. {Mikriukov} and M. {Ravanbakhsh} and B. {Demіr}},
  booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2022}
} 

@article{duch2022,
  title={Deep Unsupervised Contrastive Hashing for Large-Scale Cross-Modal Text-Image Retrieval in Remote Sensing},
  author={G. {Mikriukov} and M. {Ravanbakhsh} and B. {Demіr}},
  url={https://arxiv.org/abs/2201.08125},
  journal={arxiv:1611.08408},
  year={2022}
}

Structure

Requirements

Python 3.8
PyTorch 1.8
Torchvision 0.9
Transformers 4.4

Libraries installation:

pip install -r requirements.txt

Data

Augmented and non-augmented image and caption features for UCMerced and RSICD datasets encoded with ResNet18 and BERT respectively. Insert them to ./data/ folder.

Configs

./configs/base_config.py

Base configuration class (inherited by other configs):

CUDA device
seed
data and dataset paths

./configs/config_img_aug.py

Image augmentation configuration:

image augmentation parameters
augmentation transform sets

./configs/config_txt_aug.py

Text augmentation configuration:

text augmentation parameters and translation languages
augmentation type selection

./configs/config.py

DUCH learning configuration:

learning perparameters
learning data presets

Data augmentation

Image augmentation

images_augment.py [-h] [--dataset DATASET_NAME] [--img-aug IMG_AUG_SET]
                         [--crop-size CROP_H CROP_W]
                         [--rot-deg ROT_DEG_MIN ROT_DEG_MAX]
                         [--blur-val KERNEL_W KERNEL_H SIGMA_MIN SIGMA_MAX]
                         [--jit-str JITTER_STRENGTH]

optional arguments:
  -h, --help            show this help message and exit
  --dataset DATASET_NAME
                        ucm or rsicd
  --img-aug IMG_AUG_SET
                        image transform set: see 'image_aug_transform_sets'
                        variable for available sets
  --crop-size CROP_H CROP_W
                        crop size for 'center_crop' and 'random_crop'
  --rot-deg ROT_DEG_MIN ROT_DEG_MAX
                        random rotation degrees range for 'rotation_cc'
  --blur-val KERNEL_W KERNEL_H SIGMA_MIN SIGMA_MAX
                        gaussian blur parameters for 'blur_cc'
  --jit-str JITTER_STRENGTH
                        color jitter strength for 'jitter_cc'

Examples:

No augmentation (only resize to 224x224)

images_augment.py --dataset ucm

Only center crop with default parameters (200x200 crop)

images_augment.py --dataset ucm --img-aug center_crop_only

Augmented center crop setup (crop, gaussian blur and rotation)

images_augment.py --dataset ucm --img-aug aug_center --crop-size 200 200 --rot-deg -10 -5 --blur-val 3 3 1.1 1.3

Random augmentation for each image (one of following: rotation_cc - rotation + center crop, jitter_cc - jitter + center crop, blur_cc - blur + center crop)

images_augment.py --dataset ucm --img-aug each_img_random

Text augmentation

Augmentation

Augments raw sentences from dataset's JSON-file. Augmented sentences are inserted into the same file under other tags (aug_rb, aug_bt_prob, aug_bt_chain for rule-based, backtranslation and chain backtranslation respectively). Check ./configs/config_txt_aug.py for

captions_augment.py [-h] [--dataset DATASET_NAME]
                           [--txt-aug TXT_AUG_TYPE]

optional arguments:
  -h, --help            show this help message and exit
  --dataset DATASET_NAME
                        ucm or rsicd
  --txt-aug TXT_AUG_TYPE
                        image transform set: 'rule-based', 'backtranslation-
                        prob', 'backtranslation-chain'

Example:

captions_augment.py --dataset ucm --txt-aug rule-based

Embedding

Embeds captions (raw) and augmented captions (aug_rb, aug_bt_prob, aug_bt_chain) consequently.

captions_embed.py [-h] [--dataset DATASET_NAME]

optional arguments:
  -h, --help            show this help message and exit
  --dataset DATASET_NAME
                        ucm or rsicd