How to get unknown class?
gigasurgeon opened this issue · 2 comments
I have 5 classes, but during inference time there is no guarantee that the image will be from these 5 classes. I am using dino_small_patch16 arch. Is there a way to identify an outlier image, or an unknown class?
@gigasurgeon hello. did you manage to identify outlier image and make a predictor using this repo? I would appreciate any help
@gigasurgeon hello. did you manage to identify outlier image and make a predictor using this repo? I would appreciate any help
@westbalon I generated embeddings for 5 images per class and saved it as .pt files locally. Then, for each test image, I generated embedding and compared it with the vectors already stored, using cosine similarity. It gave around 85% accuracy in my usecase.
I got greater success with CLIP model using this approach. I used its image embeddding as well as the text embeddings and added them together and saved them locally. The text embedding was generated with either some words written on the object or a sentence describing the object. During test time, I generated only image embedding through CLIP and compared the cosine similarity. If it was below a certain threshold, I marked them as unknown or outlier. This gave me 100% accuracy for my usecase. You can use huggingface's openAI clip (giant variant worked best).