KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning

Quickstart

pip install -r requirements.txt

Please organize the data directory as following:

REPO ROOT
|
|--data    
|  |--wiki1m_for_simcse.txt  
|  |--flickr30k_ViT_L14.json
|  |--train_coco_ViT_L14.json

Wiki1M

wget https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse/resolve/main/wiki1m_for_simcse.txt

Flickr30k & MS-COCO
Download the preprocessed data:

Or preprocess the data by yourself (using raw images from Flickr and COCO with CLIP).

Prepare the senteval datasets for evaluation:

cd SentEval/data/downstream/
bash download_dataset.sh

Run scripts:

# For example:
sh scripts/run_wiki_flickr.sh

sh scripts/run_wiki_coco.sh