LAION-AI/CLIP_benchmark

Issues

• Webdatasets for Linear Probe evaluation

  #127 opened 13 days ago by shrebox
  0

• Questions about example for COCO captions captioning task

  #126 opened 24 days ago by strongweight
  0

• Leaking Validation Data in Few-Shot-Setting

  #125 opened a month ago by MarcoMorik
  0

• Duplicated `imagenet1k` classnames: "missile" & "sunglasses"

  #124 opened 2 months ago by EIFY
  0

• additional argument not recognized in MODEL_TYPES

  #123 opened 3 months ago by arnabk001
  0

• update ImageNet V2 download urls

  #122 opened 3 months ago by shyammarjit
  0

• Sugar-CREPE eval swap_obj error

  #119 opened 4 months ago by samarth4149
  0

• Add compositionality benchmarks

  #97 opened a year ago by mehdidc
  7

• What is the composition of the dataset for model training

  #121 opened 4 months ago by geekchen007
  2

• Update babel image net

  #117 opened 5 months ago by rom1504
  0

• Add support of zero-shot classification/retrieval evaluation of captioning models using log-likelihood scoring, e.g. LLaVA, and others

  #115 opened 5 months ago by mehdidc
  0

• Benchmark siglip

  #108 opened 6 months ago by rom1504
  5

• Overwritten zeroshot templates for webdataset eval

  #109 opened 6 months ago by djghosh13
  2

• WDS classnames and empty lines

  #111 opened 6 months ago by mehdidc
  2

• Support computing metrics using test time distribution normalisation

  #112 opened 7 months ago by rom1504
  0

• Incorrect and missed splits for some datasets

  #104 opened 10 months ago by teasgen
  7

• Variation in the benchmark

  #59 opened a year ago by mehdidc
  1

• Remove benchmark/ files and make it part of core

  #79 opened a year ago by rom1504
  1

• Attribute based eval

  #72 opened a year ago by rom1504
  3

• Add babel imagenet

  #95 opened a year ago by rom1504
  4

• AttributeError: 'JaCLIPForBenchmark' object has no attribute 'eval'

  #93 opened a year ago by pkhk-1
  1

• Error when test on multilingual_mscoco

  #87 opened a year ago by SingL3
  3

• Add ImageNet-W

  #96 opened a year ago by mehdidc
  1

• generate diagram as part of the cli

  #80 opened a year ago by rom1504
  1

• ValueError: Unsupported dataset: multilingual_mscoco_captions,es.

  #70 opened a year ago by mitchellnw
  1

• Add generative benchmarks on mscoco_captions

  #61 opened a year ago by gpucce
  1

• document how to run retrieval and generative eval using wds

  #91 opened a year ago by rom1504
  3

• Performs normalization of the text embedding twice

  #90 opened a year ago by Adamdad
  2

• amp is always set to True

  #83 opened a year ago by SingL3
  1

• Please revise japanese clip's ReadMe and the position of model.eval in cli.py

  #84 opened a year ago by jackee777
  2

• bug in "build_dataset" function

  #85 opened a year ago by JieyuZ2
  1

• Issue of model.txt in the readme example

  #82 opened a year ago by JieyuZ2
  1

• Evaluate mse aligned multilingual clip

  #69 opened a year ago by rom1504
  2

• Organize CLI arguments into groups

  #66 opened a year ago by djghosh13
  0

• add xm3600 for multilingual eval

  #44 opened 2 years ago by rom1504
  0

• add multi30k for eval

  #46 opened 2 years ago by elliottd
  1

• Add forensic/exif tasks

  #65 opened a year ago by rom1504
  0

• Support ImageNet-X

  #41 opened 2 years ago by mehdidc
  0

• add multilingual task from mural

  #35 opened 2 years ago by rom1504
  0

• add chinese retrieval tasks

  #33 opened 2 years ago by rom1504
  1

• Generate pretty circular diagram

  #53 opened a year ago by rom1504
  1

• put all datasets in wds

  #52 opened a year ago by rom1504
  9

• Support retrieval datasets in `clip_benchmark_export_wds`

  #67 opened a year ago by mehdidc
  1

• Clarify recommended evalution command

  #77 opened a year ago by rom1504
  1

• log dataset and model in new cli

  #76 opened a year ago by rom1504
  5

• models should be in .eval() mode for all evaluation

  #73 opened a year ago by rwightman
  0

• Rewrite run.sh as python and expose it

  #43 opened a year ago by rom1504
  1

• fix bug with paths

  #49 opened a year ago by rom1504
  7

• Retrieval results for Flickr8K

  #48 opened 2 years ago by orchidmajumder
  2

• Deploy to pypi

  #32 opened 2 years ago by rom1504
  1