VizWiz-Classification is a new test set using 8,900 images taken by people who are blind for which we collected metadata to indicate the presence versus absence of 200 ImageNet object categories. Please read our paper to learn more:
Before you can use our sample code, you must download the dataset (~20GB). Links to download the images and annotations are available here.
Alternatively, you can run this shell script. This will create a directory dataset with all the files in your current directory. This may take a while.
$ sh download_dataset.sh
Please run the following command to install the library for testing pre-trained models from PyTorch Image Models on our dataset.
$ pip install timm
After downloading the dataset, you can run this code.
$ python eval.py
Or, you can set variables in this way.
$ python eval.py --model_name vgg19 --images_path dataset/images --ann_path dataset/annotations.json --batch_size 64 --prediction_path predictions
After the prediction file is generated, you should upload it to our EvalAI server for evaluation. Instructions on how to do it are provided here.
If you make use of our dataset for your research, please be sure to cite our work with the following BibTeX citation.
@InProceedings{Bafghi_2023_CVPR,
author = {Bafghi, Reza Akbarian and Gurari, Danna},
title = {A New Dataset Based on Images Taken by Blind People for Testing the Robustness of Image Classification Models Trained for ImageNet Categories},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2023},
pages = {16261-16270}
}
For any questions, comments, or feedback, please send them to Reza Akbarian Bafghi at reza.akbarianbafghi@colorado.edu.