multi-view-AE is a collection of multi-modal autoencoder models for learning joint representations from multiple modalities of data. The package is structured such that all models have fit, predict_latents and predict_reconstruction methods. All models are built in Pytorch and Pytorch-Lightning.
Many of the models implemented in the multi-view-AE library have been benchmarked against previous implementations, with equal or improved results. See below for more details.
For more information on implemented models and how to use the package, please see the documentation.
Below is a table with the models contained within this repository and links to the original papers.
| Model class | Model name | Number of views | Original work |
|---|---|---|---|
| mcVAE | Multi-Channel Variational Autoencoder (mcVAE) | >=1 | link |
| AE | Multi-view Autoencoder | >=1 | |
| mAAE | Multi-view Adversarial Autoencoder | >=1 | |
| DVCCA | Deep Variational CCA | 2 | link |
| mWAE | Multi-view Adversarial Autoencoder with a wasserstein loss | >=1 | |
| mmVAE | Variational mixture-of-experts autoencoder (MMVAE) | >=1 | link |
| mVAE | Multimodal Variational Autoencoder (MVAE) | >=1 | link |
| me_mVAE | Multimodal Variational Autoencoder (MVAE) with separate ELBO terms for each view | >=1 | link |
| JMVAE | Joint Multimodal Variational Autoencoder(JMVAE-kl) | 2 | link |
| MVTCAE | Multi-View Total Correlation Auto-Encoder (MVTCAE) | >=1 | link |
| MoPoEVAE | Mixture-of-Products-of-Experts VAE | >=1 | link |
| mmJSD | Multimodal Jensen-Shannon divergence model (mmJSD) | >=1 | link |
| weighted_mVAE | Generalised Product-of-Experts Variational Autoencoder (gPoE-MVAE) | >=1 | link |
| DMVAE | Disentangled multi-modal variational autoencoder | >=1 | link |
| weighted_DMVAE | Disentangled multi-modal variational autoencoder with gPoE joint posterior | >=1 | |
| mmVAEPlus | Mixture-of-experts multimodal VAE Plus (mmVAE+) | >=1 | link |
To install our package via pip:
pip install multiviewaeOr, clone this repository and move to folder:
git clone https://github.com/alawryaguila/multi-view-AE
cd multi-view-AECreate the customised python environment:
conda create --name mvae python=3.9Activate python environment:
conda activate mvaeInstall the multi-view-AE package:
pip install ./To illustrate the efficacy of the multi-view-AE implementions, we validated some of the implemented models by reproducing a key result of a previous paper. One of the experiments presented in the paper was reproduced using the \texttt{multi-view-AE} implementations using the same network architectures, modelling choices, and training parameters. The code to reproduce the benchmarking experiments is available in the benchmarking folder. We evaluated performance using the joint log likelihood (↑) and conditional coherence accuracy (↑). Summary of the results of the benchmarking experiments using the BinaryMNIST and PolyMNIST datasets:
| Model | Experiment | Metric | Paper | Paper results | multi-view-AE results |
|---|---|---|---|---|---|
| JMVAE | BinaryMNIST | Joint log likelihood | link | -86.86 | -86.76±0.06 |
| me_mVAE | BinaryMNIST | Joint log likelihood | link | -86.26 | -86.31±0.08 |
| MoPoEVAE | PolyMNIST | Conditional Coherence accuracy | link | 63/75/79/81 | 68/79/83/84 |
| mmJSD | PolyMNIST | Conditional Coherence accuracy | link | 69/57/64/67 | 75/74/78/80 |
| mmVAE | PolyMNIST | Conditional Coherence accuracy | link | 71/71/71/71 | 71/71/71/71 |
| MVTCAE | PolyMNIST | Conditional Coherence accuracy | link | 59/77/83/86 | 64/81/87/90 |
| mmVAEPlus | PolyMNIST | Conditional Coherence accuracy | link | 85.2 | 86.6±0.07 |
If you have used multi-view-AE in your research, please consider citing our JOSS paper:
Lawry Aguila et al., (2023). Multi-view-AE: A Python package for multi-view autoencoder models. Journal of Open Source Software, 8(85), 5093, https://doi.org/10.21105/joss.05093
Bibtex entry:
@article{LawryAguila2023,
doi = {10.21105/joss.05093},
url = {https://doi.org/10.21105/joss.05093},
year = {2023},
publisher = {The Open Journal},
volume = {8},
number = {85},
pages = {5093},
author = {Ana Lawry Aguila and Alejandra Jayme and Nina Montaña-Brown and Vincent Heuveline and Andre Altmann},
title = {Multi-view-AE: A Python package for multi-view autoencoder models}, journal = {Journal of Open Source Software}
}Contribution guidelines are available at https://multi-view-ae.readthedocs.io/en/latest/