This repository gives an overview of the awesome projects created by lucidrains that we as LAION want to share with the community in order to help people train new exciting models and do research with SOTA ML code.
The whole LAION community started with crawling@home that became LAION-400M and later evolved into LAION-5B and at the same time lucidrains' awesome repository DALLE-pytorch, a replication of OpenAI's Dall-E model, that became more and more popular as we trained on CC-3m and CC-12m datasets and later on LAION-400M.
We are very thankful for the great work of lucidrains!
| Repository Name | Date | Description | Projects |
|---|---|---|---|
| coordinate-descent-hierarchical-memory | 27th May, 2023 | Implementation of a hierarchical memory module using coordinate descent routing | |
| flash-genomics-model | 18th May, 2023 | My own attempt at a long context genomics model, leveraging recent advances in long context attention modeling (Flash Attention + other hierarchical methods) | |
| soundstorm-pytorch | 17th May, 2023 | Implementation of SoundStorm, Efficient Parallel Audio Generation from Google Deepmind, in Pytorch | |
| MEGABYTE-pytorch | 15th May, 2023 | Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch | |
| MaMMUT-pytorch | 5th May, 2023 | Implementation of MaMMUT, a simple vision-encoder text-decoder architecture for multimodal tasks from Google, in Pytorch | |
| recurrent-memory-transformer-pytorch | 24th April, 2023 | Implementation of Recurrent Memory Transformer, Neurips 2022 paper, in Pytorch | |
| mixture-of-attention | 21st April, 2023 | Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts | |
| naturalspeech2-pytorch | 19th April, 2023 | Implementation of Natural Speech 2, Zero-shot Speech and Singing Synthesizer, in Pytorch | |
| simple-hierarchical-transformer | 6th April, 2023 | Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT | |
| neural-plexer-pytorch | 4th April, 2023 | Implementation of Nvidia's NeuralPlexer, for end-to-end differentiable design of functional small-molecules and ligand-binding proteins, in Pytorch | |
| nucleotide-transformer | 3rd April, 2023 | 🧬 Nucleotide Transformer: Building and Evaluating Robust Foundation Models for Human Genomics | |
| coordinate-descent-attention | 31st March, 2023 | Implementation of an Attention layer where each head can attend to more than just one token, using coordinate descent to pick topk | |
| RFdiffusion | 31st March, 2023 | Code for running RFdiffusion | |
| st-moe-pytorch | 26th March, 2023 | Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch | |
| CoLT5-attention | 20th March, 2023 | Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch | |
| simplicial-attention | 12th March, 2023 | Explorations into the paper Simplicial Hopfield Networks, to see if any of the learning points can improve upon attention in a transformers setting | |
| gigagan-pytorch | 10th March, 2023 | Implementation of GigaGAN, new SOTA GAN out of Adobe | |
| bitsandbytes | 8th March, 2023 | 8-bit CUDA functions for PyTorch | |
| TPDNE | 4th March, 2023 | Thispersondoesnotexist went down, so this time, while building it back up, I am going to open source all of it. | |
| CLIP | 26th February, 2023 | Contrastive Language-Image Pretraining | |
| lion-pytorch | 15th February, 2023 | 🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch | |
| toolformer-pytorch | 10th February, 2023 | Implementation of Toolformer, Language Models That Can Use Tools, by MetaAI | |
| autoregressive-linear-attention-cuda | 7th February, 2023 | CUDA implementation of autoregressive linear attention, with all the latest research findings | |
| block-recurrent-transformer-pytorch | 7th February, 2023 | Implementation of Block Recurrent Transformer - Pytorch | |
| open_clip | 3rd February, 2023 | An open source implementation of CLIP. | |
| rvq-vae-gpt | 30th January, 2023 | My attempts at applying Soundstream design on learned tokenization of text and then applying hierarchical attention to text generation | |
| musiclm-pytorch | 27th January, 2023 | Implementation of MusicLM, Google's new SOTA model for music generation using attention networks, in Pytorch | |
| zorro-pytorch | 26th January, 2023 | Implementation of Zorro, Masked Multimodal Transformer, in Pytorch | |
| dreamerv3-pytorch | 12th January, 2023 | Implementation of Dreamer v3, Deepmind's first neural network that was able to learn to collect diamonds in Minecraft, in Pytorch | |
| muse-maskgit-pytorch | 3rd January, 2023 | Implementation of Muse: Text-to-Image Generation via Masked Generative Transformers, in Pytorch | |
| nim-tokenizer | 30th December, 2022 | Implementation of a simple BPE tokenizer, but in Nim | |
| equiformer-diffusion | 27th December, 2022 | Implementation of Denoising Diffusion for protein design, but using the new Equiformer (successor to SE3 Transformers) with some additional improvements | |
| recurrent-interface-network-pytorch | 23rd December, 2022 | Implementation of Recurrent Interface Network (RIN), for highly efficient generation of images and video without cascading networks, in Pytorch | |
| Nim | 21st December, 2022 | Nim is a statically typed compiled systems programming language. It combines successful concepts from mature languages like Python, Ada and Modula. Its design focuses on efficiency, expressiveness, and elegance (in that order of priority). | |
| robotic-transformer-pytorch | 13th December, 2022 | Implementation of RT1 (Robotic Transformer) in Pytorch | |
| medical-chatgpt | 10th December, 2022 | Implementation of ChatGPT, but tailored towards primary care medicine, with the reward being able to collect patient histories in a thorough and efficient manner and come up with a reasonable differential diagnosis | |
| PaLM-rlhf-pytorch | 9th December, 2022 | Implementation of RLHF (Reinforcement Learning with Human Feedback) on top of the PaLM architecture. Basically ChatGPT but with PaLM | |
| memory-editable-transformer | 8th December, 2022 | My explorations into editing the knowledge and memories of an attention network | |
| magic3d-pytorch | 5th December, 2022 | Implementation of Magic3D, Text to 3D content synthesis, in Pytorch | |
| classifier-free-guidance-pytorch | 3rd December, 2022 | Implementation of Classifier Free Guidance in Pytorch, with emphasis on text conditioning, and flexibility to include multiple text embedding models | |
| chroma-pytorch | 2nd December, 2022 | Implementation of Chroma, generative models of protein using DDPM and GNNs, in Pytorch | |
| med-seg-diff-pytorch | 23rd November, 2022 | Implementation of MedSegDiff in Pytorch - SOTA medical segmentation using DDPM and filtering of features in fourier space | |
| holodeck-pytorch | 13th November, 2022 | Implementation of a holodeck, written in Pytorch | |
| equiformer-pytorch | 29th October, 2022 | Implementation of the Equiformer, SE3/E3 equivariant attention network that reaches new SOTA, and adopted for use by EquiFold for protein folding | |
| make-a-video-pytorch | 29th September, 2022 | Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch | |
| phenaki-pytorch | 29th September, 2022 | Implementation of Phenaki Video, which uses Mask GIT to produce text guided videos of up to 2 minutes in length, in Pytorch | |
| Mega-pytorch | 23rd September, 2022 | Implementation of Mega, the Single-head Attention with Multi-headed EMA architecture that currently holds SOTA on Long Range Arena | |
| audiolm-pytorch | 9th September, 2022 | Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch | |
| Adan-pytorch | 25th August, 2022 | Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch | |
| JEPA-pytorch | 21st August, 2022 | Implementation of JEPA, Yann LeCun's vision of how AGI would be built, in Pytorch | |
| bit-diffusion | 17th August, 2022 | Implementation of Bit Diffusion, Hinton's group's attempt at discrete denoising diffusion, in Pytorch | |
| transframer-pytorch | 17th August, 2022 | Implementation of Transframer, Deepmind's U-net + Transformer architecture for up to 30 seconds video generation, in Pytorch | |
| pytorch | 16th August, 2022 | Tensors and Dynamic neural networks in Python with strong GPU acceleration | |
| flash-cosine-sim-attention | 4th August, 2022 | Implementation of fused cosine similarity attention in the same style as Flash Attention | |
| discrete-key-value-bottleneck-pytorch | 25th July, 2022 | Implementation of Discrete Key / Value Bottleneck, in Pytorch | |
| flash-attention-jax | 12th July, 2022 | Implementation of Flash Attention in Jax | |
| flash-attention | 7th July, 2022 | Fast and memory-efficient exact attention | |
| RITA | 3rd July, 2022 | RITA is a family of autoregressive protein models, developed by LightOn in collaboration with the OATML group at Oxford and the Debora Marks Lab at Harvard. | |
| gated-state-spaces-pytorch | 29th June, 2022 | Implementation of Gated State Spaces, from the paper "Long Range Language Modeling via Gated State Spaces", in Pytorch | |
| parti-pytorch | 22nd June, 2022 | Implementation of Parti, Google's pure attention-based text-to-image neural network, in Pytorch | |
| ema-pytorch | 20th June, 2022 | A simple way to keep track of an Exponential Moving Average (EMA) version of your pytorch model | |
| bolt | 18th June, 2022 | 10x faster matrix and vector operations | |
| metaformer-gpt | 18th June, 2022 | Implementation of Metaformer, but in an autoregressive manner | |
| perceiver-ar-pytorch | 18th June, 2022 | Implementation of Perceiver AR, Deepmind's new long-context attention network based on Perceiver architecture, in Pytorch | |
| VN-transformer | 12th June, 2022 | A Transformer made of Rotation-equivariant Attention using Vector Neurons | |
| tranception-pytorch | 2nd June, 2022 | Implementation of Tranception, an attention network, paired with retrieval, that is SOTA for protein fitness prediction | |
| ddpm-ipa-protein-generation | 31st May, 2022 | Implementation of the DDPM + IPA (invariant point attention) for protein generation, as outlined in the paper "Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models" | |
| insertion-deletion-ddpm | 31st May, 2022 | Implementation of Insertion-deletion Denoising Diffusion Probabilistic Models | |
| flexible-diffusion-modeling-videos-pytorch | 28th May, 2022 | Implementation of the video diffusion model and training scheme presented in the paper, Flexible Diffusion Modeling of Long Videos, in Pytorch | |
| imagen-pytorch | 23rd May, 2022 | Implementation of Imagen, Google's Text-to-Image Neural Network, in Pytorch | |
| compositional-attention-pytorch | 9th May, 2022 | Implementation of "compositional attention" from MILA, a multi-head attention variant that is reframed as a two-step attention process with disentangled search and retrieval head aggregation, in Pytorch | |
| differentiable-SDF-pytorch | 9th May, 2022 | Implementation of Differentiable Sign-Distance Function Rendering - in Pytorch | |
| CoCa-pytorch | 5th May, 2022 | Implementation of CoCa, Contrastive Captioners are Image-Text Foundation Models, in Pytorch | |
| ResizeRight | 5th May, 2022 | The correct way to resize images or tensors. For Numpy or Pytorch (differentiable). | |
| retrieval-augmented-ddpm | 1st May, 2022 | Implementation of Retrieval-Augmented Denoising Diffusion Probabilistic Models in Pytorch | |
| flamingo-pytorch | 28th April, 2022 | Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch | |
| einops-exts | 9th April, 2022 | Implementation of some personal helper functions for Einops, my most favorite tensor manipulation library ❤️ | |
| PaLM-jax | 8th April, 2022 | Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways - in Jax (Equinox framework) | |
| video-diffusion-pytorch | 8th April, 2022 | Implementation of Video Diffusion Models, Jonathan Ho's new paper extending DDPMs to Video Generation - in Pytorch | |
| DALLE2-pytorch | 7th April, 2022 | Implementation of DALL-E 2, OpenAI's updated text-to-image synthesis neural network, in Pytorch | |
| attention | 5th April, 2022 | This repository will house a visualization that will attempt to convey instant enlightenment of how Attention works to someone not working in artificial intelligence, with 3Blue1Brown as inspiration | |
| RaveForce | 5th April, 2022 | RaveForce - An OpenAI Gym style toolkit for music generation experiments. | |
| PaLM-pytorch | 4th April, 2022 | Implementation of the specific Transformer architecture from PaLM - Scaling Language Modeling with Pathways | |
| ITTR-pytorch | 1st April, 2022 | Implementation of the Hybrid Perception Block and Dual-Pruned Self-Attention block from the ITTR paper for Image to Image Translation using Transformers | |
| neural-sequence-chunkers-pytorch | 1st April, 2022 | Implementation of the Neural Sequence Chunker, Schmidhuber paper back from 1991, in the context of Attention and Transformers | |
| tableformer-pytorch | 29th March, 2022 | Implementation of TableFormer, Robust Transformer Modeling for Table-Text Encoding, in Pytorch | |
| FLASH-pytorch | 28th March, 2022 | Implementation of the Transformer variant proposed in "Transformer Quality in Linear Time" | |
| bidirectional-cross-attention | 27th March, 2022 | A simple cross attention that updates both the source and target in one step | |
| keops | 27th March, 2022 | KErnel OPerationS, on CPUs and GPUs, with autodiff and without memory overflows | |
| x-unet | 23rd March, 2022 | Implementation of a U-net complete with efficient attention as well as the latest research findings | |
| memorizing-transformers-pytorch | 21st March, 2022 | Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch | |
| binding-ddg-predictor | 19th March, 2022 | open source repository | |
| deformable-attention | 17th March, 2022 | Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention" | |
| RQ-Transformer | 11th March, 2022 | Implementation of RQ Transformer, proposed in the paper "Autoregressive Image Generation using Residual Quantization" | |
| ffcv | 9th March, 2022 | FFCV: Fast Forward Computer Vision (and other ML workloads!) | |
| einops | 3rd March, 2022 | Deep learning operations reinvented (for pytorch, tensorflow, jax and others) | |
| memory-efficient-attention-pytorch | 3rd March, 2022 | Implementation of a memory efficient multi-head attention as proposed in the paper, "Self-attention Does Not Need O(n²) Memory" | |
| ColabFold | 18th February, 2022 | Making Protein folding accessible to all via Google Colab! | |
| ETSformer-pytorch | 5th February, 2022 | Implementation of ETSformer, state of the art time-series Transformer, in Pytorch | |
| equinox | 1st February, 2022 | Callable PyTrees and filtered JIT/grad transformations => neural networks in JAX. | |
| logavgexp-torch | 31st January, 2022 | Implementation of LogAvgExp for Pytorch | |
| anymal-belief-state-encoder-decoder-pytorch | 26th January, 2022 | Implementation of the Belief State Encoder / Decoder in the new breakthrough robotics paper from ETH Zürich | |
| RETRO-pytorch | 15th January, 2022 | Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch | |
| rela-transformer | 10th January, 2022 | Implementation of a Transformer using ReLA (Rectified Linear Attention) from https://arxiv.org/abs/2104.07012 | |
| enformer-tensorflow-sonnet-training-script | 5th January, 2022 | The full training script for Enformer - Tensorflow Sonnet | |
| staged-recipes | 22nd December, 2021 | A place to submit conda recipes before they become fully fledged conda-forge feedstocks | |
| DeepBind | 8th December, 2021 | Training and testing of DeepBind models. | |
| tf-bind-transformer | 8th December, 2021 | A repository with exploration into using transformers to predict DNA ↔ transcription factor binding | |
| ANANSE | 7th December, 2021 | Prediction of key transcription factors in cell fate determination using enhancer networks. See full ANANSE documentation for detailed installation instructions and usage examples. http://anansepy.readthedocs.io | |
| x-clip | 1st December, 2021 | A concise but complete implementation of CLIP with various experimental improvements from recent papers | |
| n-grammer-pytorch | 28th November, 2021 | Implementation of N-Grammer, augmenting Transformers with latent n-grams, in Pytorch | |
| nuwa-pytorch | 28th November, 2021 | Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch | |
| panoptic-transformer | 22nd November, 2021 | Another attempt at a long-context / efficient transformer by me | |
| uniformer-pytorch | 13th November, 2021 | Implementation of Uniformer, a simple attention and 3d convolutional net that achieved SOTA in a number of video classification tasks, debuted in ICLR 2022 | |
| hourglass-transformer-pytorch | 8th November, 2021 | Implementation of Hourglass Transformer, in Pytorch, from Google and OpenAI | |
| mujoco | 31st October, 2021 | Multi-Joint dynamics with Contact. A general purpose physics simulator. | |
| deepmind-research | 26th October, 2021 | This repository contains implementations and illustrative code to accompany DeepMind publications | |
| jax2torch | 26th October, 2021 | Use Jax functions in Pytorch | |
| enformer-pytorch | 23rd October, 2021 | Implementation of Enformer, Deepmind's attention network for predicting gene expression, in Pytorch | |
| lucidrains | 1st October, 2021 | Config files for my GitHub profile. | |
| remixer-pytorch | 24th September, 2021 | Implementation of the Remixer Block from the Remixer paper, in Pytorch | |
| triton | 20th September, 2021 | Development repository for the Triton language and compiler | |
| rgn2-replica | 19th September, 2021 | Replication attempt for the Protein Folding Model described in https://www.biorxiv.org/content/10.1101/2021.08.02.454840v1 | |
| HTM-pytorch | 14th September, 2021 | Implementation of Hierarchical Transformer Memory (HTM) for Pytorch | |
| triton-transformer | 8th September, 2021 | Implementation of a Transformer, but completely in Triton | |
| bonito | 1st September, 2021 | A PyTorch Basecaller for Oxford Nanopore Reads | |
| ponder-transformer | 25th August, 2021 | Implementation of a Transformer that Ponders, using the scheme from the PonderNet paper | |
| fast-transformer-pytorch | 23rd August, 2021 | Implementation of Fast Transformer in Pytorch | |
| token-shift-gpt | 17th August, 2021 | Implementation of Token Shift GPT - An autoregressive model that solely relies on shifting the sequence space for mixing | |
| lucidrains.github.io | 3rd August, 2021 | None | |
| multistream-transformers | 29th July, 2021 | Implementation of Multistream Transformers in Pytorch | |
| h-transformer-1d | 28th July, 2021 | Implementation of H-Transformer-1D, Hierarchical Attention for Sequence Learning | |
| triangle-multiplicative-module | 19th July, 2021 | Implementation of the Triangle Multiplicative module, used in Alphafold2 as an efficient way to mix rows or columns of a 2d feature map, as a standalone package for Pytorch | |
| invariant-point-attention | 16th July, 2021 | Implementation of Invariant Point Attention, used for coordinate refinement in the structure module of Alphafold2, as a standalone Pytorch module | |
| long-short-transformer | 7th July, 2021 | Implementation of Long-Short Transformer, combining local and global inductive biases for attention over long sequences, in Pytorch | |
| charformer-pytorch | 30th June, 2021 | Implementation of the GBST block from the Charformer paper, in Pytorch | |
| rotary-embedding-torch | 29th June, 2021 | Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch | |
| graph-transformer-pytorch | 18th June, 2021 | Implementation of Graph Transformer in Pytorch, for potential use in replicating Alphafold2 | |
| uformer-pytorch | 17th June, 2021 | Implementation of Uformer, Attention-based Unet, in Pytorch | |
| ddpm-proteins | 14th June, 2021 | A denoising diffusion probabilistic model (DDPM) tailored for conditional generation of protein distograms | |
| ddpm-jax | 13th June, 2021 | None | |
| NWT-pytorch | 9th June, 2021 | Implementation of NWT, audio-to-video generation, in Pytorch | |
| progen | 9th June, 2021 | Implementation and replication of ProGen, Language Modeling for Protein Generation, in Jax | |
| esbn-transformer | 7th June, 2021 | An attempt to merge ESBN with Transformers, to endow Transformers with the ability to emergently bind symbols | |
| segformer-pytorch | 6th June, 2021 | Implementation of Segformer, Attention + MLP neural network for segmentation, in Pytorch | |
| local-attention-flax | 26th May, 2021 | Local Attention - Flax module for Jax | |
| protein-bert-pytorch | 26th May, 2021 | Implementation of ProteinBERT in Pytorch | |
| mlp-gpt-jax | 21st May, 2021 | A GPT, made only of MLPs, in Jax | |
| g-mlp-gpt | 20th May, 2021 | GPT, but made only out of MLPs | |
| g-mlp-pytorch | 18th May, 2021 | Implementation of gMLP, an all-MLP replacement for Transformers, in Pytorch | |
| res-mlp-pytorch | 10th May, 2021 | Implementation of ResMLP, an all MLP solution to image classification, in Pytorch | |
| mlp-mixer-pytorch | 5th May, 2021 | An All-MLP solution for Vision, from Google AI | |
| CLAP | 16th April, 2021 | Contrastive Language-Audio Pretraining | |
| clasp | 4th April, 2021 | CLASP - Contrastive Language-Aminoacid Sequence Pretraining | |
| DALLE-mtf | 30th March, 2021 | Open-AI's DALL-E for large scale training in mesh-tensorflow. | |
| STAM-pytorch | 28th March, 2021 | Implementation of STAM (Space Time Attention Model), a pure and simple attention model that reaches SOTA for video classification | |
| halonet-pytorch | 24th March, 2021 | Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones | |
| transganformer | 11th March, 2021 | Implementation of TransGanFormer, an all-attention GAN that combines the finding from the recent GanFormer and TransGan paper | |
| perceiver-pytorch | 5th March, 2021 | Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch | |
| mesh | 3rd March, 2021 | Mesh TensorFlow: Model Parallelism Made Easier | |
| coco-lm-pytorch | 2nd March, 2021 | Implementation of COCO-LM, Correcting and Contrasting Text Sequences for Language Model Pretraining, in Pytorch | |
| glom-pytorch | 2nd March, 2021 | An attempt at the implementation of Glom, Geoffrey Hinton's new idea that integrates concepts from neural fields, top-down-bottom-up processing, and attention (consensus between columns), for emergent part-whole heirarchies from data | |
| omninet-pytorch | 2nd March, 2021 | Implementation of OmniNet, Omnidirectional Representations from Transformers, in Pytorch | |
| transformer-in-transformer | 2nd March, 2021 | Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image classification, in Pytorch | |
| En-transformer | 27th February, 2021 | Implementation of E(n)-Transformer, which extends the ideas of Welling's E(n)-Equivariant Graph Neural Network to attention | |
| egnn-pytorch | 26th February, 2021 | Implementation of E(n)-Equivariant Graph Neural Networks, in Pytorch | |
| nystrom-attention | 11th February, 2021 | Implementation of Nyström Self-attention, from the paper Nyströmformer | |
| TimeSformer-pytorch | 11th February, 2021 | Implementation of TimeSformer from Facebook AI, a pure attention-based solution for video classification | |
| tr-rosetta-pytorch | 4th February, 2021 | Implementation of trRosetta and trDesign for Pytorch, made into a convenient package, for protein structure prediction and design | |
| feedback-transformer-pytorch | 2nd February, 2021 | Implementation of Feedback Transformer in Pytorch | |
| bottleneck-transformer-pytorch | 28th January, 2021 | Implementation of Bottleneck Transformer in Pytorch | |
| big-sleep | 18th January, 2021 | A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN. Technique was originally created by https://twitter.com/advadnoun | |
| deep-daze | 17th January, 2021 | Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network). Technique was originally created by https://twitter.com/advadnoun | |
| geometric-vector-perceptron | 13th January, 2021 | Implementation of Geometric Vector Perceptron, a simple circuit for 3d rotation equivariance for learning over large biomolecules, in Pytorch. Idea proposed and accepted at ICLR 2021 | |
| se3-transformer-pytorch | 9th January, 2021 | Implementation of SE3-Transformers for Equivariant Self-Attention, in Pytorch. This specific repository is geared towards integration with eventual Alphafold2 replication. | |
| DALLE-pytorch | 5th January, 2021 | Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch | |
| ESBN-pytorch | 1st January, 2021 | Usable implementation of Emerging Symbol Binding Network (ESBN), in Pytorch | |
| lie-transformer-pytorch | 22nd December, 2020 | Implementation of Lie Transformer, Equivariant Self-Attention, in Pytorch | |
| point-transformer-pytorch | 18th December, 2020 | Implementation of the Point Transformer layer, in Pytorch | |
| tab-transformer-pytorch | 15th December, 2020 | Implementation of TabTransformer, attention network for tabular data, in Pytorch | |
| cross-transformers-pytorch | 11th December, 2020 | Implementation of Cross Transformer for spatially-aware few-shot transfer, in Pytorch | |
| distilled-retriever-pytorch | 11th December, 2020 | Implementation of the retriever distillation procedure as outlined in the paper "Distilling Knowledge from Reader to Retriever" | |
| adjacent-attention-network | 10th December, 2020 | Graph neural network message passing reframed as a Transformer with local attention | |
| pi-GAN-pytorch | 4th December, 2020 | Implementation of π-GAN, for 3d-aware image synthesis, in Pytorch | |
| alphafold2 | 1st December, 2020 | To eventually become an unofficial Pytorch implementation / replication of Alphafold2, as details of the architecture get released | |
| molecule-attention-transformer | 30th November, 2020 | Pytorch reimplementation of Molecule Attention Transformer, which uses a transformer to tackle the graph-like structure of molecules | |
| pixel-level-contrastive-learning | 20th November, 2020 | Implementation of Pixel-level Contrastive Learning, proposed in the paper "Propagate Yourself", in Pytorch | |
| hamburger-pytorch | 11th November, 2020 | Pytorch implementation of the hamburger module from the ICLR 2021 paper "Is Attention Better Than Matrix Decomposition" | |
| lightweight-gan | 11th November, 2020 | Implementation of 'lightweight' GAN, proposed in ICLR 2021, in Pytorch. High resolution image generations that can be trained within a day or two | |
| AoA-pytorch | 7th November, 2020 | A Pytorch implementation of Attention on Attention module (both self and guided variants), for Visual Question Answering | |
| isab-pytorch | 26th October, 2020 | An implementation of (Induced) Set Attention Block, from the Set Transformers paper | |
| x-transformers | 24th October, 2020 | A simple but complete full-attention transformer with a set of promising experimental features from various papers | |
| deep-linear-network | 16th October, 2020 | A simple implementation of a deep linear Pytorch module | |
| learning-to-expire-pytorch | 11th October, 2020 | An implementation of Transformer with Expire-Span, a circuit for learning which memories to retain | |
| memformer | 10th October, 2020 | Implementation of Memformer, a Memory-augmented Transformer, in Pytorch | |
| lambda-networks | 8th October, 2020 | Implementation of LambdaNetworks, a new approach to image recognition that reaches SOTA with less compute | |
| performer-pytorch | 3rd October, 2020 | An implementation of Performer, a linear attention-based transformer, in Pytorch | |
| vit-pytorch | 3rd October, 2020 | Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch | |
| global-self-attention-network | 2nd October, 2020 | A Pytorch implementation of Global Self-Attention Network, a fully-attention backbone for vision tasks | |
| phasic-policy-gradient | 27th September, 2020 | An implementation of Phasic Policy Gradient, a proposed improvement of Proximal Policy Gradients, in Pytorch | |
| all-normalization-transformer | 9th September, 2020 | A simple Transformer where the softmax has been replaced with normalization | |
| kronecker-attention-pytorch | 27th August, 2020 | Implementation of Kronecker Attention in Pytorch | |
| denoising-diffusion-pytorch | 26th August, 2020 | Implementation of Denoising Diffusion Probabilistic Model in Pytorch | |
| marge-pytorch | 24th August, 2020 | Implementation of Marge, Pre-training via Paraphrasing, in Pytorch | |
| omniboard | 20th August, 2020 | Web-based dashboard for Sacred | |
| mlm-pytorch | 14th August, 2020 | An implementation of masked language modeling for Pytorch, made as concise and simple as possible | |
| electra-pytorch | 4th August, 2020 | A simple and working implementation of Electra, the fastest way to pretrain language models from scratch, in Pytorch | |
| attention-tensorflow-mesh | 1st August, 2020 | Tools for building attention networks for Tensorflow Mesh | |
| conformer | 26th July, 2020 | Implementation of the convolutional module from the Conformer paper, for use in Transformers | |
| memory-compressed-attention | 25th July, 2020 | Implementation of Memory-Compressed Attention, from the paper "Generating Wikipedia By Summarizing Long Sequences" | |
| unet-stylegan2 | 25th July, 2020 | A Pytorch implementation of Stylegan2 with UNet Discriminator | |
| mixture-of-experts | 13th July, 2020 | A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models | |
| scattering-compositional-learner | 12th July, 2020 | Implementation of Scattering Compositional Learner in Pytorch | |
| memory-transformer-xl | 10th July, 2020 | A variant of Transformer-XL where the memory is updated not with a queue, but with attention | |
| local-attention | 5th July, 2020 | An implementation of local windowed attention for language modeling | |
| mogrifier | 5th July, 2020 | Usable implementation of Mogrifier, a circuit for enhancing LSTMs and potentially other networks, from Deepmind | |
| slot-attention | 29th June, 2020 | Implementation of Slot Attention from GoogleAI | |
| linformer | 28th June, 2020 | Implementation of Linformer for Pytorch | |
| compressive-transformer-pytorch | 24th June, 2020 | Pytorch implementation of Compressive Transformers, from Deepmind | |
| siren-pytorch | 19th June, 2020 | Pytorch implementation of SIREN - Implicit Neural Representations with Periodic Activation Function | |
| byol-pytorch | 16th June, 2020 | Usable Implementation of "Bootstrap Your Own Latent" self-supervised learning, from Deepmind, in Pytorch | |
| nlp | 16th June, 2020 | 🤗 nlp: datasets and evaluation metrics for Natural Language Processing in NumPy, Pandas, PyTorch and TensorFlow | |
| vector-quantize-pytorch | 9th June, 2020 | Vector Quantization, in Pytorch | |
| axial-positional-embedding | 8th June, 2020 | Axial Positional Embedding for Pytorch | |
| product-key-memory | 6th June, 2020 | Standalone Product Key Memory module in Pytorch - for augmenting Transformer models | |
| linear-attention-transformer | 4th June, 2020 | Transformer based on a variant of attention that is linear complexity in respect to sequence length | |
| axial-attention | 28th May, 2020 | Implementation of Axial attention - attending to multi-dimensional data efficiently | |
| routing-transformer | 22nd May, 2020 | Fully featured implementation of Routing Transformer | |
| contrastive-learner | 28th April, 2020 | A simple to use pytorch wrapper for contrastive self-supervised learning on any neural network | |
| compare_gan | 22nd April, 2020 | Compare GAN code. | |
| sinkhorn-transformer | 3rd April, 2020 | Sinkhorn Transformer - Practical implementation of Sparse Sinkhorn Attention | |
| pytorch-optimizer | 2nd March, 2020 | torch-optimizer -- collection of optimizers for Pytorch | |
| memcnn | 29th January, 2020 | PyTorch Framework for Developing Memory Efficient Deep Invertible Networks | |
| RevTorch | 29th January, 2020 | Framework for creating (partially) reversible neural networks with PyTorch | |
| AdaMod | 24th January, 2020 | Adaptive and Momental Bounds for Adaptive Learning Rate Methods. | |
| openprotein | 24th January, 2020 | A PyTorch framework for prediction of tertiary protein structure | |
| reformer-pytorch | 9th January, 2020 | Reformer, the efficient Transformer, in Pytorch | |
| stylegan2-pytorch | 9th January, 2020 | Simplest working implementation of Stylegan2, state of the art generative adversarial network, in Pytorch. Enabling everyone to experience disentanglement | |
| jax | 7th January, 2020 | Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more | |
| tensorflow | 19th June, 2019 | An Open Source Machine Learning Framework for Everyone | |
| tweet-stance-prediction | 26th April, 2019 | Applying NLP transfer learning techniques to predict Tweet stance | |
| stylegan | 5th February, 2019 | StyleGAN - Official TensorFlow Implementation | |
| tinygbt-js | 16th August, 2018 | A Tiny, Pure Javascript implementation of Gradient Boosted Trees. | |
| liquid-conway | 16th March, 2017 | Liquid simulator based on Conway's Game of Life | |
| arxiv-sanity-preserver | 13th February, 2017 | Web interface for browsing, search and filtering recent arxiv submissions | |
| lb_pool | 3rd October, 2016 | HTTP client load balancer with retries | |
| a-painter | 20th September, 2016 | Paint in VR in your browser. | |
| vectorious | 24th February, 2016 | A high performance linear algebra library. | |
| coffee-genetic-algorithm | 7th July, 2015 | a simple genetic algorithm written in coffeescript | |
| coffee-neural-network | 7th July, 2015 | a simple neural network in coffeescript | |
| recurrentjs | 4th June, 2015 | Deep Recurrent Neural Networks and LSTMs in Javascript. More generally also arbitrary expression graphs with automatic differentiation. |