EPFL Machine Learning and Optimization Laboratory

attention-cnn

Source code for "On the Relationship between Self-Attention and Convolutional Layers"

collaborative-attention

Code for Multi-Head Attention: Collaborate Instead of Concatenate

disco

DISCO is a code-free and installation-free browser platform that allows any non-technical user to collaboratively train machine learning models without sharing any private data.

dynamic-sparse-flash-attention

federated-learning-public-code

landmark-attention

Landmark Attention: Random-Access Infinite Context Length for Transformers

ML_course

EPFL Machine Learning Course, Fall 2024

OptML_course

EPFL Course - Optimization for Machine Learning - CS-439

powersgd

Practical low-rank gradient compression for distributed optimization: https://arxiv.org/abs/1905.13727

sent2vec

General purpose unsupervised sentence representations

epfml/ML_course

EPFL Machine Learning Course, Fall 2024

epfml/OptML_course

EPFL Course - Optimization for Machine Learning - CS-439

epfml/attention-cnn

Source code for "On the Relationship between Self-Attention and Convolutional Layers"

epfml/landmark-attention

Landmark Attention: Random-Access Infinite Context Length for Transformers

epfml/disco

DISCO is a code-free and installation-free browser platform that allows any non-technical user to collaboratively train machine learning models without sharing any private data.

epfml/federated-learning-public-code

epfml/collaborative-attention

Code for Multi-Head Attention: Collaborate Instead of Concatenate

epfml/powersgd

Practical low-rank gradient compression for distributed optimization: https://arxiv.org/abs/1905.13727

epfml/dynamic-sparse-flash-attention

epfml/llm-baselines

nanoGPT-like codebase for LLM training

epfml/DenseFormer

epfml/schedules-and-scaling

Code for NeurIPS 2024 Spotlight: "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations"

epfml/optML-pku

summer school materials

epfml/error-feedback-SGD

SGD with compressed gradients and error-feedback: https://arxiv.org/abs/1901.09847

epfml/getting-started

epfml/REQ

epfml/pam

epfml/relaysgd

Code for the paper “RelaySum for Decentralized Deep Learning on Heterogeneous Data”

epfml/easy-summary

difficulty-guided text summarization

epfml/ghost-noise

epfml/personalized-collaborative-llms

Exploration on-device self-supervised collaborative fine-tuning of large language models with limited local data availability, using Low-Rank Adaptation (LoRA). We introduce three distinct trust-weighted gradient aggregation schemes: weight similarity-based, prediction similarity-based and validation performance-based.

epfml/CoTFormer

epfml/CoMiGS

epfml/cifar

MLO internal cifar 10 / 100 default implementation / reference implementation. single machine, variable batch sizes, allowing maybe gradient compression. need to have clear documentation to make it easy to use, and so that we don't loose time with looking for hyperparameters. we can later keep it in sync with mlbench too, but self-contained is even better

epfml/CoBo

epfml/DoGE

Codebase for ICML submission "DOGE: Domain Reweighting with Generalization Estimation"

epfml/epfml-utils

Tools for experimentation and using run:ai. The aim is for these to be small self-contained utilities that are used by multiple people.

epfml/fineweb2-hq

Code for the paper "Enhancing Multilingual LLM Pretraining with Model-Based Data Selection"

epfml/getting-started-lauzhack

epfml/semester-project-personalization