Calibration sample selection and training logit generation

[night-gale]

Hi,

Thanks for your brilliant work!

We are trying to reproduce your work. However, we run into some troubles. Specifically, we wonder how the calibration set samples are selected (e.g. random with fixed seed, etc.), which could be very important for a fair comparison with other methods. It would be very kind if you could provide some further information about the calibration set split.

Moreover, we're trying to make use of the training set logits but struggle to proceed without pretrained weights of the backbones. We notice that the training scripts are available here for experiments on CIFAR and SVHN yet others are not provided. It would help us a lot if you could share the weights of the pretrained backbones.

Thanks in advance!

[AmirooR]

Hi @night-gale,

Thanks for your interest in our paper. We only used the validation and test logits from https://github.com/markus93/NN_Calibration. For the most of the other models, as far as I remember, we used pre-trained models from https://github.com/osmr/imgclsmob/blob/master/pytorch/README.md. I think the splits were chosen randomly in a way that all classes have similar number of samples for calibration and test splits.

We trained different ResNet type models on CARS dataset using the standard pytorch training script. The ResNet models with (pre) are initialized with pre-trained ImageNet weights.

I currently do not have access to the computers that we trained those models since I have moved to another university in another country! I will try to ask someone to see if I can have access to those computers and let you know. Hope this can answer your question.

Best,
Amir