/ecevecce

Metrics of calibration for probabilistic predictions

Primary LanguageTeXMIT LicenseMIT

The accompanying codes reproduce all figures and statistics presented in the research paper, "Metrics of calibration for probabilistic predictions," by Imanol Arrieta-Ibarra, Paman Gujral, Jonathan Tannen, Mark Tygert, and Cherie Xu. This repository also provides the LaTeX and BibTeX sources required for replicating the paper.

The main files in the repository are the following:

tex/ecevecce.pdf PDF version of the paper

tex/ecevecce.tex LaTeX source for the paper

tex/ecevecce.bib BibTeX source for the paper

codes/calibration.py Functions for plotting calibration, both cumulative and reliability diagrams

codes/imagenetcal.py Python script for processing ImageNet using a pre-trained ResNet-18

codes/imagenet_classes.txt Text file containing a dictionary of the names of the classes in ImageNet

codes/pvals.py Python script which prints P-values for some given values of the ECCEs

codes/dists.py Functions for calculating cumulative distribution functions for Brownian motion (redistributed from Mark Tygert's website)

Regenerating all the figures requires running in the directory codes both calibration.py and imagenetcal.py. Only codes/imagenetcal.py needs a GPU (running on a GPU simultaneously with CPU cores); the other codes need only CPUs.

The command-line tool convert uses ImageMagick for conversion of PDF files to JPEG files; install ImageMagick to enable such conversion.


Copyright license

This ecevecce software is licensed under the (MIT-type) copyright LICENSE file in the root directory of this source tree.