karimul/awesome-ebm

Collecting research materials on EBM/EBL (Energy Based Models, Energy Based Learning)

Awesome Energy Based Models/Learning (Awesome-EBM)

A comprehensive list of energy based learning papers and materials.

Table of Contents

• Workshops & Symposiums
• Representative Applications
• Papers (Reverse Chronological Order))
  • 2021
  • 2020
  • 2019
  • 2017 ~ 2018
  • 2013 ~ 2016
  • 2007 ~ 2012
  • Early papers (Before 2007)
• Tutorials & Talks & Blogs
• Open Source Libraries

Workshops & Symposiums

• EBM Workshop at ICLR 2021.

Representative Applications

• Data (image/graph/sequence/etc) generation
• Discriminative learning: Classification/regression
• Density estimation
• Maximum entropy reinforcement learning
• Out-of-distribution detection (OOD)/anomaly detection/fraud detection
• Model calibration
• Adversarial robustness
• Image inpainting/denoising/super-resolution
• Prior modeling
• Model-based planning for robotics
• Language/Speech modeling

Papers (Reverse Chronological Order)

2021

• Jaini, P., Holdijk, L., & Welling, M. (2021).
  Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent. arXiv preprint arXiv:2106.07832.

• Yoon, J., Hwang, S. J., & Lee, J. (2021).
  Adversarial purification with Score-based generative models. arXiv preprint arXiv:2106.06041.

• Domingo-Enrich, C., Bietti, A., Vanden-Eijnden, E., & Bruna, J. (2021).
  On Energy-Based Models with Overparametrized Shallow Neural Networks. arXiv preprint arXiv:2104.07531.

• 2021: Durkan, Conor, and Yang Song.
  On Maximum Likelihood Training of Score-Based Generative Models.

• 2021: Song, Y., & Kingma, D. P.
  How to Train Your Energy-Based Models. arXiv preprint arXiv:2101.03288, 2021.

• Shi, C., Luo, S., Xu, M., & Tang, J. (2021).
  Learning Gradient Fields for Molecular Conformation Generation. arXiv preprint arXiv:2105.03902.

• Suhail, M., Mittal, A., Siddiquie, B., Broaddus, C., Eledath, J., Medioni, G., & Sigal, L. (2021).
  Energy-Based Learning for Scene Graph Generation. arXiv preprint arXiv:2103.02221.[code]

• Liu, M., Yan, K., Oztekin, B., & Ji, S. (2021).
  GraphEBM: Molecular Graph Generation with Energy-Based Models. arXiv preprint arXiv:2102.00546.

• Wu, H., Esmaeili, B., Wick, M., Tristan, J. B., & van de Meent, J. W. (2021).
  Conjugate Energy-Based Models.

• Zheng, Z., Xie, J., & Li, P. (2021).
  Patchwise Generative ConvNet: Training Energy-Based Models from a Single Natural Image for Internal Learning.

• Wu, Jiaxiang and Shen, Tao and Lan, Haidong and Bian, Yatao and Huang, Junzhou. (2021)
  SE(3)-Equivariant Energy-based Models for End-to-End Protein Folding.

• Bian, Y., Rong, Y., Xu, T., Wu, J., Krause, A., & Huang, J. (2021).
  Energy-Based Learning for Cooperative Games, with Applications to Feature/Data/Model Valuations. arXiv preprint arXiv:2106.02938.

2020

• Meng, C., Yu, L., Song, Y., Song, J., & Ermon, S. (2020).
  Autoregressive Score Matching. arXiv preprint arXiv:2010.12810.

• Clark, K., Luong, M. T., Le, Q. V., & Manning, C. D. (2020).
  Pre-Training Transformers as Energy-Based Cloze Models. arXiv preprint arXiv:2012.08561.

• Boney, R., Kannala, J., & Ilin, A. (2020, May).
  Regularizing model-based planning with energy-based models. In Conference on Robot Learning (pp. 182-191). PMLR.

• Liu, M., He, T., Xu, M., & Zhang, W. (2020).
  Energy-based imitation learning. arXiv preprint arXiv:2004.09395.

• Xiao, Z., Kreis, K., Kautz, J., & Vahdat, A. (2020, September).
  Vaebm: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations.

• Bao, F., Xu, K., Li, C., Hong, L., Zhu, J., & Zhang, B. (2020).
  Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models. arXiv preprint arXiv:2010.08258.

• Dai, H., Singh, R., Dai, B., Sutton, C., & Schuurmans, D. (2020).
  Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration. arXiv preprint arXiv:2011.05363.

• 2020: Gustafsson, F. K., Danelljan, M., Timofte, R., and Schön, T. B.
  How to Train Your Energy-Based Model for Regression. arXiv preprint arXiv:2005.01698, 2020.

• Gustafsson, F. K., Danelljan, M., Bhat, G., & Schön, T. B. (2020, August).
  Energy-based models for deep probabilistic regression. In European Conference on Computer Vision (pp. 325-343). Springer, Cham.

• 2020: Niu, C., Song, Y., Song, J., Zhao, S., Grover, A., and Ermon, S.
  Permutation invariant graph generation via score-Based generative modeling. In International Conference on Artificial Intelligence and Statistics (pp. 4474-4484). PMLR, 2020. [code]

• 2020: Liu, H., and Abbeel, P.
  Hybrid discriminative-generative training via contrastive learning. arXiv preprint arXiv:2007.09070, 2020.

• Ho, J., Jain, A., & Abbeel, P. (2020).
  Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239.

• 2020: Arbel, Michael, Zhou, Liang, and Gretton, Arthur.
  Generalized energy based models.arXiv e-prints,pp. arXiv–2003, 2020.

• 2020: Song, Yang, Sohl-Dickstein, Jascha, Kingma, Diederik P, Kumar, Abhishek, Ermon, Ste-fano, and Poole, Ben.
  Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020.

• 2020: Song, Yang, and Stefano Ermon.
  Improved techniques for training score-based generative models." arXiv preprint arXiv:2006.09011 (2020).

• 2020: Gao, Ruiqi, et al.
  Flow contrastive estimation of energy-based models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

• 2020: Khemakhem, Ilyes, et al.
  ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA. Advances in Neural Information Processing Systems 33 (2020). [code]

• 2020: Cotta, L., Teixeira, C. H., Swami, A., & Ribeiro, B. (2020).
  Unsupervised Joint $k$-node Graph Representations with Compositional Energy-Based Models. arXiv preprint arXiv:2010.04259.

• 2020: Pang, B., Han, T., Nijkamp, E., Zhu, S. C., & Wu, Y. N. (2020).
  Learning latent space energy-based prior model. arXiv preprint arXiv:2006.08205.

• Liu, W., Wang, X., Owens, J. D., & Li, Y. (2020).
  Energy-based Out-of-distribution Detection. arXiv preprint arXiv:2010.03759.

• Nijkamp, E., Hill, M., Han, T., Zhu, S. C., & Wu, Y. N. (2020, April).
  On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 5272-5280).

• Cai, R., Yang, G., Averbuch-Elor, H., Hao, Z., Belongie, S., Snavely, N., & Hariharan, B. (2020).
  Learning gradient fields for shape generation. arXiv preprint arXiv:2008.06520.

• Wang, Z., Cheng, S., Yueru, L., Zhu, J., & Zhang, B. (2020, June).
  A wasserstein minimum velocity approach to learning unnormalized models. In International Conference on Artificial Intelligence and Statistics (pp. 3728-3738). PMLR.

• Che, T., Zhang, R., Sohl-Dickstein, J., Larochelle, H., Paull, L., Cao, Y., & Bengio, Y. (2020).
  Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. arXiv preprint arXiv:2003.06060.

• Grathwohl, W., Kelly, J., Hashemi, M., Norouzi, M., Swersky, K., & Duvenaud, D. (2020).
  No MCMC for me: Amortized sampling for fast and stable training of energy-based models. arXiv preprint arXiv:2010.04230.

• Yu, L., Song, Y., Song, J., & Ermon, S. (2020, November).
  Training deep energy-based models with f-divergence minimization. In International Conference on Machine Learning (pp. 10957-10967). PMLR.

• Li, S., Du, Y., van de Ven, G. M., Torralba, A., & Mordatch, I. (2020).
  Energy-Based Models for Continual Learning. arXiv preprint arXiv:2011.12216.

• Du, Y., Meier, J., Ma, J., Fergus, R., & Rives, A. (2020).
  Energy-based models for atomic-resolution protein conformations. arXiv preprint arXiv:2004.13167.

• Deng, Y., Bakhtin, A., Ott, M., Szlam, A., & Ranzato, M. A. (2020).
  Residual energy-based models for text generation. arXiv preprint arXiv:2004.11714.

• Sahin, A., Bian, Y., Buhmann, J., & Krause, A. (2020, November).
  From Sets to Multisets: Provable Variational Inference for Probabilistic Integer Submodular Models. In International Conference on Machine Learning (pp. 8388-8397). PMLR.

2019

• Saremi, S., & Hyvärinen, A. (2019).
  Neural empirical bayes. Journal of Machine Learning Research, 20, 1-23.

• 2019: Song, Y., and Ermon, S.
  Generative Modeling by Estimating Gradients of the Data Distribution. In Proceedings of the 33rd Annual Conference on Neural Information Processing Systems, 2019. [code]

• Wenliang, L., Sutherland, D., Strathmann, H., & Gretton, A. (2019, May).
  Learning deep kernels for exponential family densities. In International Conference on Machine Learning (pp. 6737-6746). PMLR.

• 2019: Grathwohl, W., Wang, K. C., Jacobsen, J. H., Duvenaud, D., Norouzi, M., & Swersky, K.
  Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263, 2019. [code]

• 2019: Bian, Y., Buhmann, J., & Krause, A.
  Optimal continuous dr-submodular maximization and applications to provable mean field inference. In International Conference on Machine Learning (pp. 644-653). PMLR. [code]

• 2019: Du, Yilun, and Igor Mordatch.
  Implicit generation and modeling with energy based models. (2019).

• Nijkamp, E., Hill, M., Zhu, S. C., & Wu, Y. N. (2019).
  Learning non-convergent non-persistent short-run MCMC toward energy-based model. arXiv preprint arXiv:1904.09770.

• Dai, B., Liu, Z., Dai, H., He, N., Gretton, A., Song, L., & Schuurmans, D. (2019).
  Exponential family estimation via adversarial dynamics embedding. arXiv preprint arXiv:1904.12083.

• Bartunov, S., Rae, J. W., Osindero, S., & Lillicrap, T. P. (2019).
  Meta-learning deep energy-based memory models. arXiv preprint arXiv:1910.02720.

2017 ~ 2018

• Mordatch, I. (2018).
  Concept learning with energy-based models. arXiv preprint arXiv:1811.02486. [blog]

• Xie, J., Lu, Y., Gao, R., Zhu, S. C., & Wu, Y. N. (2018).
  Cooperative training of descriptor and generator networks. IEEE transactions on pattern analysis and machine intelligence, 42(1), 27-45. [code]

• Gao, R., Lu, Y., Zhou, J., Zhu, S. C., & Wu, Y. N. (2018).
  Learning generative convnets via multi-grid modeling and sampling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 9155-9164). [code]

• Yoon, K., Liao, R., Xiong, Y., Zhang, L., Fetaya, E., Urtasun, R., ... & Pitkow, X. (2018).
  Inference in Probabilistic Graphical Models by Graph Neural Networks. arXiv preprint arXiv:1803.07710.

• Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018, July).
  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning (pp. 1861-1870). PMLR.

• Ceylan, C., & Gutmann, M. U. (2018, July).
  Conditional noise-contrastive estimation of unnormalised models. In International Conference on Machine Learning (pp. 726-734). PMLR.

• Bose, A. J., Ling, H., & Cao, Y. (2018).
  Adversarial contrastive estimation. arXiv preprint arXiv:1805.03642.

• Scellier, B., & Bengio, Y. (2017).
  Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 11, 24.

• Haarnoja, T., Tang, H., Abbeel, P., & Levine, S. (2017, July).
  Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (pp. 1352-1361). PMLR.

2013 ~ 2016

• 2016: Zhao, J., Mathieu, M., & LeCun, Y. (2016).
  Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126.

• Xie, J., Lu, Y., Zhu, S. C., & Wu, Y. (2016, June).
  A theory of generative convnet. In International Conference on Machine Learning (pp. 2635-2644). PMLR.

• Liu, Q., Lee, J., & Jordan, M. (2016, June).
  A kernelized Stein discrepancy for goodness-of-fit tests. In International conference on machine learning (pp. 276-284). PMLR.

• Zhai, S., Cheng, Y., Feris, R., & Zhang, Z. (2016).
  Generative adversarial networks as variational training of energy based models. arXiv preprint arXiv:1611.01799.

• Kim, T., & Bengio, Y. (2016).
  Deep directed generative models with energy-based probability estimation. arXiv preprint arXiv:1606.03439.

• Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. (2015, June).
  Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (pp. 2256-2265). PMLR.

• Gorham, J., & Mackey, L. (2015).
  Measuring sample quality with Stein's method. arXiv preprint arXiv:1506.03039.

2007 ~ 2012

• 2012: Parry, Matthew, Dawid, A Philip, Lauritzen, Steffen, et al.
  Proper local scoring rules.Annals of Statistics, 40(1):561–592, 2012.

• Gutmann, M., & Hirayama, J. I. (2012).
  Bregman divergence as general framework to estimate unnormalized statistical models. arXiv preprint arXiv:1202.3727.

• Welling, M., & Teh, Y. W. (2011).
  Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) (pp. 681-688).

• Raphan, M., & Simoncelli, E. P. (2011).
  Least squares estimation without priors or supervision. Neural computation, 23(2), 374-420.

• Sohl-Dickstein, J., Battaglino, P., & DeWeese, M. R. (2011, January).
  Minimum Probability Flow Learning. In ICML.

• Vincent, P. (2011).
  A connection between score matching and denoising autoencoders. Neural computation, 23(7), 1661-1674.

• Lyu, S. (2011, September).
  Unifying Non-Maximum Likelihood Learning Objectives with Minimum KL Contraction. In NIPS (pp. 64-72).

• Ngiam, J., Chen, Z., Koh, P. W., & Ng, A. Y. (2011, June).
  Learning deep energy models. In Proceedings of the 28th International Conference on International Conference on Machine Learning (pp. 1105-1112).

• Gutmann, M., & Hyvärinen, A. (2010, March).
  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (pp. 297-304). JMLR Workshop and Conference Proceedings.

• Kingma, D. P., & LeCun, Y. (2010, January).
  Regularized estimation of image statistics by Score Matching. In NIPS (Vol. 509, p. 618).

• Lyu, S. (2009, June).
  Interpretation and generalization of score matching. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence (pp. 359-366).

• Javier R Movellan Jan 2008.
  A minimum velocity approach to learning. unpublished draft

• Raphan, M., Simoncelli, E. P., Scholkopf, B., Platt, J., & Hoffman, T. (2007).
  Learning to be Bayesian without supervision. Advances in neural information processing systems, 19, 1145.

• Hyvarinen, A. (2007).
  Connections between score matching, contrastive divergence, and pseudolikelihood for continuous-valued variables. IEEE Transactions on neural networks, 18(5), 1529-1531.

• Hyvarinen, A. (2007).
  Some extensions of score matching. Computational statistics & data analysis, 51(5), 2499-2512.

Early papers (Before 2007)

• Hinton, G. E., Osindero, S., & Teh, Y. W. (2006).
  A fast learning algorithm for deep belief nets. Neural computation, 18(7), 1527-1554.

• LeCun, Y., & Huang, F. J. (2005, January).
  Loss Functions for Discriminative Training of Energy-Based Models. In AIStats (Vol. 6, p. 34).

• Hyvärinen, A., & Dayan, P. (2005).
  Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4).

• Welling, M., Rosen-Zvi, M., & Hinton, G. E. (2004, December).
  Exponential Family Harmoniums with an Application to Information Retrieval. In Nips (Vol. 4, pp. 1481-1488).

• Teh, Y. W., Welling, M., Osindero, S., & Hinton, G. E. (2003).
  Energy-based models for sparse overcomplete representations. Journal of Machine Learning Research, 4(Dec), 1235-1260.

• Hinton, G. E. (2002).
  Training products of experts by minimizing contrastive divergence. Neural computation, 14(8), 1771-1800.

• Dayan, P., Hinton, G. E., Neal, R. M., & Zemel, R. S. (1995).
  The helmholtz machine. Neural computation, 7(5), 889-904.

• Smolensky, P. (1986).
  Information processing in dynamical systems: Foundations of harmony theory. Colorado Univ at Boulder Dept of Computer Science.

• Stein, C. M. (1981).
  Estimation of the mean of a multivariate normal distribution. The annals of Statistics, 1135-1151.

• 1957a: Jaynes, Edwin T.
  Information theory and statistical mechanics.Physical review, 106(4):620,1957a

• 1957b: Jaynes, Edwin T.
  Information theory and statistical mechanics. ii. Physical review, 108(2):171, 1957b.

Tutorials & Talks & Blogs

• 2006: LeCun, Yann, Chopra, Sumit, Hadsell, Raia, Ranzato, M, and Huang, F.
  A tutorial on energy-based learning. Predicting structured data, 1(0), 2006

• Yann LeCun, Sumit Chopra, Raia Hadsell, Fu-Jie Huang, Marc'Aurelio Ranzato (Courant Institute/CBLL), 2003.
  LeCun's research page on EBMs.

• Takayuki Osogami, Sakyasingha Dasgupta, 2017.
  IJCAI-17 Tutorial: Energy-based machine learning.

• 2020 Youtube video: Arthur Gretton.
  On the critic function of implicit generative models.

• 2020 Youtube video: Stefano Ermon.
  Generative Modeling by Estimating Gradients of the Data Distribution

• UvA Deep Learning Tutorials Fall 2020.
  Tutorial 8: Deep Energy-Based Generative Models

Open Source Libraries

• 2020 Mateus Roder and Gustavo Henrique de Rosa and João Paulo Papa.
  Learnergy: Energy-based Machine Learners.

• Maes, F. (2009).
  Nieme: Large-scale energy-based models. The Journal of Machine Learning Research, 10, 743-746.[code (however is missing in the webpage)]