/PAC_Bayes_Invariance

Code from the paper "PAC-Bayesian Margin Bounds for convolutional neural networks". https://arxiv.org/pdf/1905.09677.pdf

Primary LanguagePythonMIT LicenseMIT

The role of invariance in spectral complexity-based generalization bounds.

Deep convolutional neural networks (CNNs) have been shown to be able to fit a random labeling over data while still being able to generalize well for normal labels. Describing CNN capacity through a posteriory measures of complexity has been recently proposed to tackle this apparent paradox. These complexity measures are usually validated by showing that they correlate empirically with GE; being empirically larger for networks trained on random vs normal labels. Focusing on the case of spectral complexity we investigate theoretically and empirically the insensitivity of the complexity measure to invariances relevant to CNNs, and show several limitations of spectral complexity that occur as a result. For a specific formulation of spectral complexity we show that it results in the same upper bound complexity estimates for convolutional and locally connected architectures (which don’t have the same favorable invariance properties). This is contrary to common intuition and empirical results.

💥 Files 

GE_estimation_dense.py: Trains a dense neural network and estimates the network training and testing accuracy.

GE_estimation_locally_connected.py: Trains a locally connected neural network and estimates the network training and testing accuracy.

GE_estimation_convolutional.py: Trains a convolutional neural network and estimates the network training and testing accuracy.

utils_spectral_norm.py: Includes an implementation of Parseval regularization.

test_spectral_convolutional_variable_ab.py: Calculates the spectral norm for locally connected and convolutional 1d layers and computes empirical averages. Draws the theoretical average as well.

📝 Citation

When citing this repository on your scientific publications please use the following **BibTeX** citation: 
@article{pitas2019role,
  title={The role of invariance in spectral complexity-based generalization bounds},
  author={Pitas, Konstantinos and Loukas, Andreas and Davies, Mike and Vandergheynst, Pierre},
  journal={arXiv preprint arXiv:1905.09677},
  year={2019}
}

✉️ Contact Information

You can contact me at any of my social network profiles:

Or via email at cwstas2007@hotmail.com

⚠️ Disclaimer

This Python package has been made for research purposes.