Infotheory

Website: http://mcandadai.com/infotheory/

Introduction

Infotheory, written in C++, and usable in Python as well, is a software package to perform information theoretic analysis on multivariate data. This package implements traditional as well as more recent measures that arise from multivariate extensions to information theory, specifically

Entropy [1]
Mutual Information [2]
Partial Information Decomposition [3]
- Unique Information
- Redundant Information
- Synergistic Information

The main highlights of this package include:

being written in C++ for efficiency
ease of use via python bindings and compatibility with numpy
an API that allows adding the data once to then be able to perform various analyses across different sub-spaces of the dataset very quickly
use of sparse data structures that work well with high-dimensional data
user-controllable estimation of data distribution using averaged shifted histograms [4]
flexibilty to specify binning allows proper estimation of information measures between continuous and discrete variables
perform PI-decomposition over 3 (two sources and 1 target) and 4 (three variables and 1 target) variables

The package can be used in Python or C++. While the C++ headers should function well on all platforms, the python package has currently been tested on MacOS and Linux.

Citing

If you use this package, please cite preprint available at: https://arxiv.org/abs/1907.02339

Candadai, M., & Izquierdo, E. J. (2019). infotheory: A C++/Python package for multivariate information theoretic analysis. arXiv preprint arXiv:1907.02339.

  @article{candadai2019infotheory,
    title={infotheory: A C++/Python package for multivariate information theoretic analysis},
    author={Candadai, Madhavun and Izquierdo, Eduardo J},
    journal={arXiv preprint arXiv:1907.02339},
    year={2019}
  }

Installation

pip install infotheory

On MacOS, upgrade to MacOS-Mojave (or later) and update Xcode. You might have to set these environment variables from your terminal.

export CXXFLAGS="-mmacosx-version-min=10.9"
export LDFLAGS="-mmacosx-version-min=10.9"

For C++, simply download InfoTools.h and VectorMatrix.h and include those header files in your code.

Dependencies

The package doesn't have any dependencies per se but the automated tests require numpy

Usage

Using this package in your own code involves the following steps.

See demos and website for sample programs on how to use this package.

Contact

Created by Madhavun Candadai and Eduardo J. Izquierdo. If you have questions or if you've found a bug, please file an issue or feel free to contact Madhavun at madvncv[at]gmail.com

Contribution

If you'd like a feature to be added to infotheory, please file an issue. Or, better yet, open a pull request. We'll work with you to ensure that all pull requests are in a mergable state.

References

http://www.scholarpedia.org/article/Entropy#Shannon_entropy
http://www.scholarpedia.org/article/Mutual_information
Williams, P. L., & Beer, R. D. (2010). Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515.
Scott, D. W. (1985). Averaged shifted histograms: effective nonparametric density estimators in several dimensions. The Annals of Statistics, 1024-1040.
Timme, N., Alford, W., Flecker, B., & Beggs, J. M. (2014). Synergy, redundancy, and multivariate information measures: an experimentalist’s perspective. Journal of computational neuroscience, 36(2), 119-140.

hih70/infotheory