/awesome-ssm-ml

MIT LicenseMIT

Awesome State-Space Resources for ML

Contributions are welcome! Please read the contribution guidelines before contributing.

Table of Contents

Parameterization and Initialization

  1. Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers (NeurIPS 2021)
  2. Efficiently Modeling Long Sequences with Structured State Spaces (ICLR 2022)
  3. On the Parameterization and Initialization of Diagonal State Space Models (NeurIPS 2022)
  4. Diagonal State Spaces are as Effective as Structured State Spaces (NeurIPS 2022) [code]
  5. How to Train your HIPPO: State Space Models with Generalized Orthogonal Basis Projections (ICLR 2023)
  6. Mamba: Linear-Time Sequence Modeling with Selective State Spaces [code]
  7. Robustifying State-space Models for Long Sequences via Approximate Diagonalization
  8. StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization
  9. Spectral State Space Models

Architecture

  1. S5: Simplified State Space Layers for Sequence Modeling (ICLR 2023) [code]
  2. Long range language modeling via gated state spaces (ICLR 2023)
  3. Pretraining Without Attention
  4. MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  5. LOCOST: State-Space Models for Long Document Abstractive Summarization [code]
  6. BlackMamba: Mixture of Experts for State-Space Models
  7. DenseMamba: State Space Models with Dense Hidden Connection for Efficient Large Language Models
  8. ZigMa: Zigzag Mamba Diffusion Model

Vision

  1. S4ND: Modeling Images and Videos as Multidimensional Signals with State Spaces (NeurIPS 2022)
  2. Long movie clip classification with state-space video models (ECCV 2022) [code]
  3. Efficient Movie Scene Detection using State-Space Transformers (CVPR 2023)
  4. Selective Structured State-Spaces for Long-Form Video Understanding (CVPR 2023)
  5. 2-D SSM: A General Spatial Layer for Visual Transformers [code]
  6. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model [code]
  7. VMamba: Visual State Space Model [code]
  8. U-shaped Vision Mamba for Single Image Dehazing
  9. Res-VMamba: Fine-Grained Food Category Visual Classification Using Selective State Space Models with Deep Residual Learning
  10. Weak-Mamba-UNet: Visual Mamba Makes CNN and ViT Work Better for Scribble-based Medical Image Segmentation

Language

  1. Hungry Hungry Hippos: Towards Language Modeling with State Space Models (ICLR 2023)
  2. Long range language modeling via gated state spaces (ICLR 2023)
  3. Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  4. MambaByte: Token-free Selective State Space Model [code]
  5. Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks

Audio

  1. It's Raw! Audio Generation with State-Space Models (ICML 2022)
  2. Augmenting conformers with structured state space models for online speech recognition
  3. Diagonal State Space Augmented Transformers for Speech Recognition
  4. Structured State Space Decoder for Speech Recognition and Synthesis
  5. Spiking Structured State Space Model for Monaural Speech Enhancement
  6. A Neural State-Space Model Approach to Efficient Speech Separation
  7. Multi-Head State Space Model for Speech Recognition

Time-Series

  1. Deep State Space Models for Time Series Forecasting (NeurIPS 2018)
  2. FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting (NeurIPS 2022)
  3. Effectively modeling time series with simple discrete state spaces (ICLR 2023)
  4. Deep Latent State Space Models for Time-Series Generation (ICML 2023)
  5. Generative AI for End-to-End Limit Order Book Modelling (ICAIF 2023)
  6. On the Performance of Legendre State-Space Models in Short-Term Time Series Forecasting (CCECE 2023)
  7. Neural Continuous-Discrete State Space Models for Irregularly-Sampled Time Series
  8. Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models

Medical

  1. Structured State Space Models for Multiple Instance Learning in Digital Pathology
  2. Modeling Multivariate Biosignals with Graph Neural Networks and Structured State Space
  3. Diffusion-based conditional ECG generation with structured state space models
  4. Improving the Diagnosis of Psychiatric Disorders with Self-Supervised Graph State Space Models
  5. fMRI-S4: learning short- and long-range dynamic fMRI dependencies using 1D Convolutions and State Space Models
  6. Vivim: a Video Vision Mamba for Medical Video Object Segmentation [code]
  7. MambaMorph: a Mamba-based Backbone with Contrastive Feature Learning for Deformable MR-CT Registration [code]
  8. SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation [code]
  9. U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation [code]
  10. nnMamba: 3D Biomedical Image Segmentation, Classification and Landmark Detection with State Space Model
  11. VM-UNet: Vision Mamba UNet for Medical Image Segmentation
  12. MambaMIR: An Arbitrary-Masked Mamba for Joint Medical Image Reconstruction and Uncertainty Estimation

Tabular

  1. MambaTab: A Simple Yet Effective Approach for Handling Tabular Data

Reinforcement Learning

  1. Decision S4: Efficient Sequence-Based RL via State Spaces Layers (ICLR 2023)
  2. Structured State Space Models for In-Context Reinforcement Learning (NeurIPS 2023)
  3. Mastering Memory Tasks with World Models (ICLR 2024 oral)

Books and Surveys

  1. Linear State-Space Control Systems
  2. Modeling Sequences with Structured State Spaces
  3. Principles of System Identification Theory and Practice

Tutorials

  1. The Annotated S4
  2. The Annotated Mamba [code]
  3. A Visual Guide to Mamba and State Space Models

Miscellaneous

  1. Variational learning for switching state-space models (Neural Computation 2000)
  2. Liquid structural state-space models (ICLR 2023)
  3. Resurrecting Recurrent Neural Networks for Long Sequences (ICML 2023)
  4. Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets (ICLR 2023)
  5. Block-State Transformers
  6. Simplifying and Understanding State Space Models with Diagonal Linear RNNs
  7. Never Train from Scratch: Fair Comparison Of Long- Sequence Models Requires Data-Driven Pirors
  8. Structured state-space models are deep Wiener models
  9. Efficient Long Sequence Modeling via State Space Augmented Transformer
  10. State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory
  11. Repeat After Me: Transformers are Better than State Space Models at Copying
  12. Theoretical Foundations of Deep Selective State-Space Models
  13. The Hidden Attention of Mamba Models

Contributions

🎉 Thank you for considering contributing to our Awesome State Space Models for Machine Learning repository! 🚀

Contribute in 3 Steps:

  1. Fork the Repo: Fork this repo to your GitHub account.

  2. Edit Content: Contribute by adding new resources or improving existing content in the README.md file.

  3. Create a Pull Request: Open a pull request (PR) from your branch to the main repository.

Guidelines

  • Follow the existing structure and formatting.
  • Ensure added resources are relevant to State Space Models in Machine Learning.
  • Verify that links work correctly.

Reporting Issues

If you encounter issues or have suggestions, open an issue on the GitHub repository.

Your contributions make this repository awesome! Thank you! 🙌

License

This project is licensed under the MIT License.