embeddings-benchmark/mteb

Issues

• Flexible post-evaluation filtering

  #1410 opened a month ago by gnatesan
  2

• [mieb] EVA on CVBenchCount fails

  #1409 opened a month ago by Muennighoff
  1

• [mieb] Winoground not working

  #1383 opened 2 months ago by Muennighoff
  2

• RetrievalEvaluator NaN values for similarity scores

  #1389 opened a month ago by gnatesan
  4

• Integrate ArabicMTEB

  #1408 opened a month ago by KennethEnevoldsen
  0

• [mieb] VOC2007 not working

  #1382 opened 2 months ago by Muennighoff
  2

• [mieb] InfoSeekIT2ITRetrieval & InfoSeekIT2TRetrieval fail with BAAI/bge-visualized

  #1386 opened 2 months ago by Muennighoff
  2

• Error when creating task (SICK-BR-STS, XStance)

  #1407 opened a month ago by AlexeyVatolin
  0

• Overview issue: Leaderboard 2.0 release

  #1405 opened 2 months ago by KennethEnevoldsen
  0

• Add zero-shot box to leaderboard

  #1404 opened 2 months ago by KennethEnevoldsen
  0

• How can the settings be adjusted to improve CPU utilization during MTEB execution?

  #1399 opened 2 months ago by eric-yq
  1

• Leaderboard: Add runtime x performance plot

  #1396 opened 2 months ago by KennethEnevoldsen
  2

• standardize descriptive stats

  #1321 opened 2 months ago by KennethEnevoldsen
  2

• Disabling rich output.

  #1385 opened 2 months ago by Napuh
  3

• [MIEB] CVBenchCount with laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K fails

  #1393 opened 2 months ago by Muennighoff
  1

• Leaderboard: Add zero-shot filter to leaderboard

  #1372 opened 2 months ago by KennethEnevoldsen
  1

• [mieb] 'BlipForImageTextRetrieval' object has no attribute 'get_image_features'

  #1392 opened 2 months ago by Muennighoff
  1

• [mieb] kakaobrain/align-base fails on ImageNet10Clustering

  #1387 opened 2 months ago by Muennighoff
  0

• [mieb] Caltech101 not working

  #1381 opened 2 months ago by Muennighoff
  5

• Leaderboard 2.0: should we fetch data from model card?

  #1373 opened 2 months ago by KennethEnevoldsen
  7

• Bugs with verbosity

  #1367 opened 2 months ago by Muennighoff
  0

• Cannot reproduce exact numbers (MTEB French retrieval) from the leaderboard

  #1314 opened 2 months ago by PxYu
  5

• [mieb] Freezes during eval

  #1376 opened 2 months ago by Muennighoff
  0

• [mieb] TIGER-Lab/VLM2Vec-LoRA fails

  #1377 opened 2 months ago by Muennighoff
  1

• MIEB: Remove results-mieb folder and figure out where to store results

  #1338 opened 2 months ago by KennethEnevoldsen
  1

• MIEB: Improve tests coverage

  #1339 opened 2 months ago by KennethEnevoldsen
  0

• MIEB: Define an interface for MIEB encoders

  #1340 opened 2 months ago by KennethEnevoldsen
  0

• [MIEB] Integrate docs

  #1336 opened 2 months ago by KennethEnevoldsen
  0

• [MIEB] Figure out folder structure

  #1337 opened 2 months ago by KennethEnevoldsen
  0

• [MIEB] laion/CLIP-ViT-B-16-DataComp.XL-s13B-b90K not working

  #1332 opened 2 months ago by Muennighoff
  2

• [MIEB] `royokong/e5-v` not runnable

  #1334 opened 2 months ago by Muennighoff
  1

• [MIEB] image file is truncated

  #1326 opened 2 months ago by Muennighoff
  1

• [MIEB] Too many workers?

  #1331 opened 2 months ago by Muennighoff
  3

• Leaderboard 2.0: Re-add model searchbar from 1.0

  #1347 opened 2 months ago by KennethEnevoldsen
  6

• `use_instructions` is spelled `use_instuctions` everywhere

  #1346 opened 2 months ago by x-tabdeveloping
  2

• unable to push mmteb scores to model meta due to file size

  #1368 opened 2 months ago by bwanglzu
  2

• Clarification on samples_per_label Settings for Each Dataset

  #1350 opened 2 months ago by Nieleilei
  2

• No module named 'datasets.tasks'

  #1363 opened 2 months ago by bwanglzu
  2

• How to add a new leaderboard?

  #1358 opened 2 months ago by shizhl-code
  7

• When (manually) caching embeddings this line breaks the order and renders caching useless

  #1366 opened 2 months ago by alexeifigueroa
  1

• BrazilianToxicTweetsClassification broken

  #1324 opened 2 months ago by Samoed
  3

• Increased memory usage for FEVER (and/or other retrieval tasks?)

  #1343 opened 2 months ago by tomaarsen
  4

• Improve leaderboard 2.0 readability

  #1317 opened 2 months ago by KennethEnevoldsen
  9

• Add link to ModelMeta

  #1354 opened 2 months ago by x-tabdeveloping
  1

• Implement borda count and rank in leaderboard

  #1344 opened 2 months ago by x-tabdeveloping
  1

• Leaderboard: Add benchmark description and citation

  #1342 opened 2 months ago by KennethEnevoldsen
  1

• Add Citation to ModelMeta

  #1333 opened 2 months ago by KennethEnevoldsen
  0

• Add promptType Conversation

  #1330 opened 2 months ago by KennethEnevoldsen
  2

• RetrivalEvaluator overrides encode kwargs

  #1325 opened 2 months ago by Samoed
  5

• FSTimeoutError for MLSUMClusteringP2P & MLSUMClusteringS2S

  #1311 opened 2 months ago by bourdoiscatie
  9