A curated list of awesome papers on dataset distillation and related applications.
Dataset distillation is the task of synthesizing a small dataset such that models trained on it achieve high performance on the original large dataset. A dataset distillation algorithm takes as input a large real dataset to be distilled (training set), and outputs a small synthetic distilled dataset, which is evaluated via testing models trained on this distilled dataset on a separate real dataset (validation/test set). A good small distilled dataset is not only useful in dataset understanding, but has various applications (e.g., continual learning, privacy, neural architecture search, etc.). This task was first introduced in the paper Dataset Distillation [Tongzhou Wang et al., '18], along with a proposed algorithm using backpropagation through optimization steps. Then the task was first extended to the real-world datasets in the paper Medical Dataset Distillation [Guang Li et al., '19], which also explored the privacy preservation possibilities of dataset distillation. In the paper Dataset Condensation [Bo Zhao et al., '20], gradient matching was first introduced and greatly promoted the development of the dataset distillation field.
In recent years (2022-now), dataset distillation has gained increasing attention in the research community, across many institutes and labs. More papers are now being published each year. These wonderful researches have been constantly improving dataset distillation and exploring its various variants and applications.
This project is curated and maintained by Guang Li, Bo Zhao, and Tongzhou Wang.
The Homepage of Awesome Dataset Distillation was designed and maintained by Longzhen Li.
- ๐ Project Page
Code- ๐
bibtex
- [Call for papers] The First Dataset Distillation Challenge (Kai Wang & Ahmad Sajedi et al., ECCV 2024)
- [2024/05/30] Curriculum Dataset Distillation (Zhiheng Ma & Anjia Cao et al., 2024) ๐
- [2024/05/27] Efficiency for Free: Ideal Data Are Transportable Representations (Peng Sun et al., 2024) ๐
- [2024/05/16] GSDD: Generative Space Dataset Distillation for Image Super-resolution (Haiyu Zhang et al., AAAI 2024) ๐
- [2024/05/11] Multi-Source Domain Adaptation Meets Dataset Distillation through Dataset Dictionary Learning (Montesuma et al., ICASSP 2024) ๐
- [2024/04/12] Self-supervised Dataset Distillation: A Good Compression Is All You Need (Muxin Zhou et al., 2024) ๐
- Main
- Early Work
- Gradient/Trajectory Matching Surrogate Objective
- Distribution/Feature Matching Surrogate Objective
- Better Optimization
- Distilled Dataset Parametrization
- Generative Prior
- Label Distillation
- Dataset Quantization
- Multimodal Distillation
- Self-Supervied Distillation
- Benchmark
- Survey
- Ph.D. Thesis
- Workshop
- Challenge
- Applications
- Dataset Distillation (Tongzhou Wang et al., 2018) ๐ ๐
- Gradient-Based Hyperparameter Optimization Through Reversible Learning (Dougal Maclaurin et al., ICML 2015) ๐
- Dataset Condensation with Gradient Matching (Bo Zhao et al., ICLR 2021) ๐
- Dataset Condensation with Differentiable Siamese Augmentation (Bo Zhao et al., ICML 2021) ๐
- Dataset Distillation by Matching Training Trajectories (George Cazenavette et al., CVPR 2022) ๐ ๐
- Dataset Condensation with Contrastive Signals (Saehyung Lee et al., ICML 2022) ๐
- Loss-Curvature Matching for Dataset Selection and Condensation (Seungjae Shin & Heesun Bae et al., AISTATS 2023) ๐
- Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation (Jiawei Du & Yidi Jiang et al., CVPR 2023) ๐
- Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory (Justin Cui et al., ICML 2023) ๐
- Sequential Subset Matching for Dataset Distillation (Jiawei Du et al., NeurIPS 2023) ๐
- Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matching (Ziyao Guo & Kai Wang et al., ICLR 2024) ๐ ๐
- CAFE: Learning to Condense Dataset by Aligning Features (Kai Wang & Bo Zhao et al., CVPR 2022) ๐
- Dataset Condensation with Distribution Matching (Bo Zhao et al., WACV 2023) ๐
- Improved Distribution Matching for Dataset Condensation (Ganlong Zhao et al., CVPR 2023) ๐
- DataDAM: Efficient Dataset Distillation with Attention Matching (Ahmad Sajedi & Samir Khaki et al., ICCV 2023) ๐ ๐
- Dataset Distillation via the Wasserstein Metric (Haoyang Liu et al., 2023) ๐
- M3D: Dataset Condensation by Minimizing Maximum Mean Discrepancy (Hansong Zhang & Shikun Li et al., AAAI 2024) ๐
- On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm (Peng Sun et al., CVPR 2024) ๐
- Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation (Wenxiao Deng et al., CVPR 2024) ๐
- Optimizing Millions of Hyperparameters by Implicit Differentiation (Jonathan Lorraine et al., AISTATS 2020) ๐
- Dataset Meta-Learning from Kernel Ridge-Regression (Timothy Nguyen et al., ICLR 2021) ๐
- Dataset Distillation with Infinitely Wide Convolutional Networks (Timothy Nguyen et al., NeurIPS 2021) ๐
- On Implicit Bias in Overparameterized Bilevel Optimization (Paul Vicol et al., ICML 2022) ๐
- Dataset Distillation using Neural Feature Regression (Yongchao Zhou et al., NeurIPS 2022) ๐ ๐
- Efficient Dataset Distillation using Random Feature Approximation (Noel Loo et al., NeurIPS 2022) ๐
- Accelerating Dataset Distillation via Model Augmentation (Lei Zhang & Jie Zhang et al., CVPR 2023) ๐
- Dataset Distillation with Convexified Implicit Gradients (Noel Loo et al., ICML 2023) ๐
- DREAM: Efficient Dataset Distillation by Representative Matching (Yanqing Liu & Jianyang Gu & Kai Wang et al., ICCV 2023) ๐
- On the Size and Approximation Error of Distilled Sets (Alaa Maalouf & Murad Tukan et al., NeurIPS 2023) ๐
- Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective (Zeyuan Yin & Zhiqiang Shen et al., NeurIPS 2023) ๐ ๐
- You Only Condense Once: Two Rules for Pruning Condensed Datasets (Yang He et al., NeurIPS 2023) ๐
- MIM4DD: Mutual Information Maximization for Dataset Distillation (Yuzhang Shang et al., NeurIPS 2023) ๐
- MGDD: A Meta Generator for Fast Dataset Distillation (Songhua Liu et al., NeurIPS 2023) ๐
- Distill Gold from Massive Ores: Efficient Dataset Distillation via Critical Samples Selection (Yue Xu et al., 2023) ๐
- Can Pre-Trained Models Assist in Dataset Distillation? (Yao Lu et al., 2023) ๐
- DREAM+: Efficient Dataset Distillation by Bidirectional Representative Matching (Yanqing Liu & Jianyang Gu & Kai Wang et al., 2023) ๐
- Dataset Distillation in Latent Space (Yuxuan Duan et al., 2023) ๐
- Embarassingly Simple Dataset Distillation (Yunzhen Feng et al., ICLR 2024) ๐
- Multisize Dataset Condensation (Yang He et al., ICLR 2024) ๐
- Generalized Large-Scale Data Condensation via Various Backbone and Statistical Matching (Shitong Shao et al., CVPR 2024) ๐
- Curriculum Dataset Distillation (Zhiheng Ma & Anjia Cao et al., 2024) ๐
- Dataset Condensation via Efficient Synthetic-Data Parameterization (Jang-Hyun Kim et al., ICML 2022) ๐
- Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks (Zhiwei Deng et al., NeurIPS 2022) ๐
- On Divergence Measures for Bayesian Pseudocoresets (Balhae Kim et al., NeurIPS 2022) ๐
- Dataset Distillation via Factorization (Songhua Liu et al., NeurIPS 2022) ๐
- PRANC: Pseudo RAndom Networks for Compacting Deep Models (Parsa Nooralinejad et al., 2022) ๐
- Dataset Condensation with Latent Space Knowledge Factorization and Sharing (Hae Beom Lee & Dong Bok Lee et al., 2022) ๐
- Slimmable Dataset Condensation (Songhua Liu et al., CVPR 2023) ๐
- Few-Shot Dataset Distillation via Translative Pre-Training (Songhua Liu et al., ICCV 2023) ๐
- Sparse Parameterization for Epitomic Dataset Distillation (Xing Wei & Anjia Cao et al., NeurIPS 2023) ๐
- Frequency Domain-based Dataset Distillation (Donghyeok Shin & Seungjae Shin et al., NeurIPS 2023) ๐
- Synthesizing Informative Training Samples with GAN (Bo Zhao et al., NeurIPS 2022 Workshop) ๐
- Generalizing Dataset Distillation via Deep Generative Prior (George Cazenavette et al., CVPR 2023) ๐ ๐
- DiM: Distilling Dataset into Generative Model (Kai Wang & Jianyang Gu et al., 2023) ๐
- Dataset Condensation via Generative Model (Junhao Zhang et al., 2023) ๐
- Efficient Dataset Distillation via Minimax Diffusion (Jianyang Gu et al., CVPR 2024) ๐
- Latent Dataset Distillation with Diffusion Models (Brian B. Moser & Federico Raue et al., 2024) ๐
- Flexible Dataset Distillation: Learn Labels Instead of Images (Ondrej Bohdal et al., NeurIPS 2020 Workshop) ๐
- Soft-Label Dataset Distillation and Text Dataset Distillation (Ilia Sucholutsky et al., IJCNN 2021) ๐
- Dataset Quantization (Daquan Zhou & Kai Wang & Jianyang Gu et al., ICCV 2023) ๐
- Vision-Language Dataset Distillation (Xindi Wu et al., 2023) ๐ ๐
- Self-Supervised Dataset Distillation for Transfer Learning (Dong Bok Lee & Seanie Lee et al., ICLR 2024) ๐
- Self-supervised Dataset Distillation: A Good Compression Is All You Need (Muxin Zhou et al., 2024) ๐
- Efficiency for Free: Ideal Data Are Transportable Representations (Peng Sun et al., 2024) ๐
- DC-BENCH: Dataset Condensation Benchmark (Justin Cui et al., NeurIPS 2022) ๐ ๐
- A Comprehensive Study on Dataset Distillation: Performance, Privacy, Robustness and Fairness) (Zongxiong Chen & Jiahui Geng et al., 2023) ๐
- DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation (Yifan Wu et al., 2024) ๐
- Data Distillation: A Survey (Noveen Sachdeva et al., TMLR 2023) ๐
- A Survey on Dataset Distillation: Approaches, Applications and Future Directions (Jiahui Geng & Zongxiong Chen et al., IJCAI 2023) ๐
- A Comprehensive Survey to Dataset Distillation (Shiye Lei et al., TPAMI 2023) ๐
- Dataset Distillation: A Comprehensive Review (Ruonan Yu & Songhua Liu et al., TPAMI 2023) ๐
- Data-efficient Neural Network Training with Dataset Condensation (Bo Zhao, The University of Edinburgh 2023) ๐
- 1st CVPR Workshop on Dataset Distillation (Saeed Vahidian et al., CVPR 2024)
- The First Dataset Distillation Challenge (Kai Wang & Ahmad Sajedi et al., ECCV 2024)
- Reducing Catastrophic Forgetting with Learning on Synthetic Data (Wojciech Masarczyk et al., CVPR 2020 Workshop) ๐
- Condensed Composite Memory Continual Learning (Felix Wiewel et al., IJCNN 2021) ๐
- Distilled Replay: Overcoming Forgetting through Synthetic Samples (Andrea Rosasco et al., IJCAI 2021 Workshop) ๐
- Sample Condensation in Online Continual Learning (Mattia Sangermano et al., IJCNN 2022) ๐
- An Efficient Dataset Condensation Plugin and Its Application to Continual Learning (Enneng Yang et al., NeurIPS 2023) ๐
- Summarizing Stream Data for Memory-Restricted Online Continual Learning (Jianyang Gu et al., AAAI 2024) ๐
- Privacy for Free: How does Dataset Condensation Help Privacy? (Tian Dong et al., ICML 2022) ๐
- Private Set Generation with Discriminative Information (Dingfan Chen et al., NeurIPS 2022) ๐
- No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy" (Nicholas Carlini et al., 2022) ๐
- Backdoor Attacks Against Dataset Distillation (Yugeng Liu et al., NDSS 2023) ๐
- Differentially Private Kernel Inducing Points (DP-KIP) for Privacy-preserving Data Distillation (Margarita Vinaroz et al., 2023) ๐
- Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation (Noel Loo et al., ICLR 2024) ๐
- Rethinking Backdoor Attacks on Dataset Distillation: A Kernel Method Perspective (Ming-Yu Chung et al., ICLR 2024) ๐
- Soft-Label Anonymous Gastric X-ray Image Distillation (Guang Li et al., ICIP 2020) ๐
- Compressed Gastric Image Generation Based on Soft-Label Dataset Distillation for Medical Data Sharing (Guang Li et al., CMPB 2022) ๐
- Dataset Distillation for Medical Dataset Sharing (Guang Li et al., AAAI 2023 Workshop) ๐
- Communication-Efficient Federated Skin Lesion Classification with Generalizable Dataset Distillation (Yuchen Tian & Jiacheng Wang, MICCAI 2023 Workshop) ๐
- Progressive Trajectory Matching for Medical Dataset Distillation (Zhen Yu et al., 2024) ๐
- Federated Learning via Synthetic Data (Jack Goetz et al., 2020) ๐
- Distilled One-Shot Federated Learning (Yanlin Zhou et al., 2020) ๐
- DENSE: Data-Free One-Shot Federated Learning (Jie Zhang & Chen Chen et al., NeurIPS 2022) ๐
- FedSynth: Gradient Compression via Synthetic Data in Federated Learning (Shengyuan Hu et al., 2022) ๐
- Meta Knowledge Condensation for Federated Learning (Ping Liu et al., ICLR 2023) ๐
- DYNAFED: Tackling Client Data Heterogeneity with Global Dynamics (Renjie Pi et al., CVPR 2023) ๐
- FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning (Yuanhao Xiong & Ruochen Wang et al., CVPR 2023) ๐
- Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environments (Rui Song et al., IJCNN 2023) ๐
- Fed-GLOSS-DP: Federated, Global Learning using Synthetic Sets with Record Level Differential Privacy (Hui-Po Wang et al., 2023) ๐
- Federated Virtual Learning on Heterogeneous Data with Local-global Distillation (Chun-Yin Huang et al., 2023) ๐
- Graph Condensation for Graph Neural Networks (Wei Jin et al., ICLR 2022) ๐
- Condensing Graphs via One-Step Gradient Matching (Wei Jin et al., KDD 2022) ๐
- Graph Condensation via Receptive Field Distribution Matching (Mengyang Liu et al., 2022) ๐
- Kernel Ridge Regression-Based Graph Dataset Distillation (Zhe Xu et al., KDD 2023) ๐
- Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Data (Xin Zheng et al., NeurIPS 2023) ๐
- Does Graph Distillation See Like Vision Dataset Counterpart? (Beining Yang & Kai Wang et al., NeurIPS 2023) ๐
- Fair Graph Distillation (Qizhang Feng et al., NeurIPS 2023) ๐
- CaT: Balanced Continual Graph Learning with Graph Condensation (Liu Yilun et al., ICDM 2023) ๐
- Navigating Complexity: Toward Lossless Graph Condensation via Expanding Window Matching (Yuchen Zhang & Tianle Zhang & Kai Wang et al., ICML 2024) ๐
- Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matching (Tianle Zhang & Yuchen Zhang & Kai Wang et al., 2024) ๐
- Graph Data Condensation via Self-expressive Graph Structure Reconstruction (Zhanyu Liu & Chaolv Zeng et al., 2024) ๐
- Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data (Felipe Petroski Such et al., ICML 2020) ๐
- Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiation (Dmitry Medvedev et al., AIST 2021) ๐
- Wearable ImageNet: Synthesizing Tileable Textures via Dataset Distillation (George Cazenavette et al., CVPR 2022 Workshop) ๐ ๐
- Learning from Designers: Fashion Compatibility Analysis Via Dataset Distillation (Yulan Chen et al., ICIP 2022) ๐
- Galaxy Dataset Distillation with Self-Adaptive Trajectory Matching (Haowen Guan et al., NeurIPS 2023 Workshop) ๐
- Knowledge Condensation Distillation (Chenxin Li et al., ECCV 2022) ๐
- Infinite Recommendation Networks: A Data-Centric Approach (Noveen Sachdeva et al., NeurIPS 2022) ๐
- Gradient Matching for Categorical Data Distillation in CTR Prediction (Chen Wang et al., RecSys 2023) ๐
- Bidirectional Learning for Offline Infinite-width Model-based Optimization (Can Chen et al., NeurIPS 2022) ๐
- Bidirectional Learning for Offline Model-based Biological Sequence Design (Can Chen et al., ICML 2023) ๐
- Can We Achieve Robustness from Data Alone? (Nikolaos Tsilivis et al., ICML 2022 Workshop) ๐
- Towards Robust Dataset Learning (Yihan Wu et al., 2022) ๐
- Rethinking Data Distillation: Do Not Overlook Calibration (Dongyao Zhu et al., ICCV 2023) ๐
- Towards Trustworthy Dataset Distillation (Shijie Ma et al., 2023) ๐
- Group Distributionally Robust Dataset Distillation with Risk Minimization (Saeed Vahidian & Mingyu Wang & Jianyang Gu et al., 2024) ๐
- Towards Adversarially Robust Dataset Distillation by Curvature Regularization (Eric Xue et al., 2024) ๐
- Data Distillation for Text Classification (Yongqi Li et al., 2021) ๐
- Dataset Distillation with Attention Labels for Fine-tuning BERT (Aru Maekawa et al., ACL 2023) ๐
- DiLM: Distilling Dataset into Language Model for Text-level Dataset Distillation (Aru Maekawa et al., NAACL 2024) ๐
- New Properties of the Data Distillation Method When Working With Tabular Data (Dmitry Medvedev et al., AIST 2020) ๐
- Towards Efficient Deep Hashing Retrieval: Condensing Your Data via Feature-Embedding Matching (Tao Feng & Jie Zhang et al., 2023) ๐
- Dancing with Still Images: Video Distillation via Static-Dynamic Disentanglement (Ziyu Wang & Yue Xu et al., CVPR 2024) ๐
- Multi-Source Domain Adaptation Meets Dataset Distillation through Dataset Dictionary Learning (Eduardo Montesuma et al., ICASSP 2024) ๐
- GSDD: Generative Space Dataset Distillation for Image Super-resolution (Haiyu Zhang et al., AAAI 2024) ๐
- Dataset Condensation for Time Series Classification via Dual Domain Matching (Zhanyu Liu et al., 2024) ๐
- Beginning of Awesome Dataset Distillation
- Most Popular AI Research Aug 2022
- ไธไธช้กน็ฎๅธฎไฝ ไบ่งฃๆฐๆฎ้่ธ้ฆDataset Distillation
- ๆต็ผฉๅฐฑๆฏ็ฒพๅ๏ผ็จๅคงไธ็ป่ง่ง็ๅพ ๆฐๆฎ้่ธ้ฆ
If you find this project useful for your research, please use the following BibTeX entry.
@misc{li2022awesome,
author={Li, Guang and Zhao, Bo and Wang, Tongzhou},
title={Awesome Dataset Distillation},
howpublished={\url{https://github.com/Guang000/Awesome-Dataset-Distillation}},
year={2022}
}
We would like to express our heartfelt thanks to Nikolaos Tsilivis, Wei Jin, Yongchao Zhou, Noveen Sachdeva, Can Chen, Guangxiang Zhao, Shiye Lei, Xinchao Wang, Dmitry Medvedev, Seungjae Shin, Jiawei Du, Yidi Jiang, Xindi Wu, Guangyi Liu, Yilun Liu, Kai Wang, Yue Xu, Anjia Cao, Jianyang Gu, Yuanzhen Feng, Peng Sun, Ahmad Sajedi, Zhihao Sui, Ziyu Wang, Haoyang Liu and Eduardo Montesuma for their valuable suggestions and contributions.