Code for reproducing results from the paper ContraSim -- A Similarity Measure Based on Contrastive Learning.
Install the required libraries by running pip install -r requirements.txt.
Run python layer_prediction.py -dataset DATASET -sim_measure SIM_MEASURE.
This will run the layer prediction benchmark on dataset DATASET with SIM_MEAURE.
- DATASET can be ptb_text_only or wikitext.
- SIM_MEASURE can be one of: CCA, CKA, DeepDot, DeepCKA, contrastive, svcca, contrastive_dis, Dot and Norm.
- In case you want to run all similarity measures at a single run - pass argument
-do_all
Run python multilingual_benchmark.py -sim_measure SIM_MEASURE -faiss FAISS .
This will run the multilnigual benchmark with SIM_MEAURE. If FAISS is True, evaluation will be performed using faiss sampling, otherwise ransom sampling will be used.
- SIM_MEASURE can be one of: CKA, DeepDot, DeepCKA, contrastive, contrastive_dis, Dot and Norm.
- In case you want to run all similarity measures at a single run - pass argument
-do_all
Run python image_caption_benchmark.py .
This will run the image caption benchmark using 4 different model pair (as specified in the paper).
Code used for CKA, CCA and contrastive learning loss is based on publicly available code in the linked repos.
@article{rahamim2023contrasim,
title={ContraSim -- A Similarity Measure Based on Contrastive Learning},
author={Adir Rahamim and Yonatan Belinkov},
journal={arXiv:2303.16992},
year={2023},
}