/document_vector_search_benchmark

Benchmark for Japanese document embedding & vector search

Primary LanguagePythonApache License 2.0Apache-2.0

Document Vector Search Benchmark

Benchmark for Japanese document embedding and vector search

setup

requirements:

  • python >=3.9,<3.12
  • poetry

required environmental variables:

  • OPENAI_API_KEY (to use OpenAI's embedding API)
  • COHERE_API_KEY (to use Cohere's embedding API)

optional:

  • DBSV_CACHE_DIR (for cache directory, default: ~/.dvsb)

to install all dependencies, perform

$ make install

run benchmark

Benchmark settings can be controlled through a yaml file. The default configuration can be found in configs/default.yml. You can use your own config file in the configs directry by setting the DVSB_CONFIG_NAME environmental variable (ex. default).

To run benchmark,

$ make run_benchmark

result

Model Recall@1 JSQuAD-v1.1-valid Recall@3 JSQuAD-v1.1-valid Recall@5 JSQuAD-v1.1-valid Recall@10 JSQuAD-v1.1-valid Recall@3 MIRACL-v1.0-dev Recall@5 MIRACL-v1.0-dev Recall@10 MIRACL-v1.0-dev Recall@100 MIRACL-v1.0-dev
0 ColBERTRetriever-bclavie/JaColBERTv2 0.920756 0.967807 0.976587 0.982665 0.622286 0.713644 0.82424 0.970855
1 SentenceTransformerEmbedding-BAAI/bge-m3 0.849842 0.939217 0.958577 0.975912 0.686322 0.769516 0.85376 0.980984
2 E5Embedding-intfloat/multilingual-e5-large 0.864926 0.952949 0.965781 0.977488 0.658555 0.741041 0.835273 0.982559
3 ColBERTRetriever-bclavie/JaColBERT 0.911301 0.961054 0.970059 0.977262 0.555464 0.639378 0.748628 0.933103
4 E5Embedding-intfloat/multilingual-e5-base 0.838361 0.934039 0.954975 0.972535 0.612025 0.687829 0.798884 0.976868
5 E5Embedding-intfloat/multilingual-e5-small 0.840387 0.933814 0.95385 0.972985 0.59911 0.688636 0.783391 0.972148
6 VertexAITextEmbedding-textembedding-gecko-multilingual@001 0.780729 0.904322 0.932463 0.961054 N/A N/A N/A N/A
7 VertexAITextEmbedding-textembedding-gecko-multilingual@latest 0.780729 0.904548 0.932238 0.960603 N/A N/A N/A N/A
8 OpenAIEmbedding-text-embedding-ada-002 0.75394 0.874606 0.906799 0.937866 N/A N/A N/A N/A
9 SonoisaSentenceLukeJapanese-sonoisa/sentence-luke-japanese-base-lite 0.652634 0.813147 0.861324 0.908825 0.144617 0.211484 0.297427 0.622732
10 SonoisaSentenceBertJapanese-sonoisa/sentence-bert-base-ja-mean-tokens-v2 0.65421 0.810671 0.8629 0.914228 0.170388 0.221441 0.314252 0.660344
11 SentenceTransformerEmbedding-pkshatech/GLuCoSE-base-ja 0.644755 0.798064 0.846466 0.896668 0.472039 0.546962 0.645746 0.861757
12 SentenceTransformerEmbedding-cl-nagoya/sup-simcse-ja-base 0.631923 0.792661 0.848942 0.897118 0.136905 0.185814 0.26734 0.590408
13 SentenceTransformerEmbedding-sonoisa/sentence-bert-base-ja-mean-tokens-v2 0.639802 0.782981 0.841288 0.894867 0.203452 0.26437 0.357611 0.708596
14 SentenceTransformerEmbedding-cl-nagoya/sup-simcse-ja-large 0.603107 0.776452 0.833408 0.889239 0.154618 0.202999 0.293636 0.584585
15 SentenceTransformerEmbedding-cl-nagoya/unsup-simcse-ja-large 0.594777 0.755966 0.8181 0.879559 0.102252 0.142104 0.218686 0.52806
16 SentenceTransformerEmbedding-cl-nagoya/unsup-simcse-ja-base 0.577217 0.746961 0.804142 0.870779 0.0963266 0.121559 0.195299 0.500001