Awesome Speech Enhancement
Table of contents
Overview
This is a curated list of awesome Speech Enhancement tutorials, papers, libraries, datasets, tools, scripts and results. The purpose of this repo is to organize the world’s resources for speech enhancement, and make them universally accessible and useful.
This repo is jointly contributed by Nana Hou (Nanyang Technoligical University), Meng Ge , Hao Shi (Tianjin University), Chenglin Xu (National University of Singapore), Chen Weiguang (Hunan University).
To add items to this page, simply send a pull request.
Publications
Coming soon...
Survey
A literature survey on single channel speech enhancement, 2020 [paper]
Research Advances and Perspectives on the Cocktail Party Problem and Related Auditory Models, Bo Xu, 2019 [paper (Chinese)]
Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments, Zixing Zhang, 2017 [paper]
Supervised speech separation based on deep learning: An Overview, 2017 [paper]
A review on speech enhancement techniques, 2015 [paper]
Nonlinear speech enhancement: an overview, 2007 [paper]
Feature augmentation
Speech enhancement using self-adaptation and multi-head attention, ICASSP 2020 [paper]
PAN: phoneme-aware network for monaural speech enhancement, ICASSP 2020 [paper]
Noise tokens: learning neural noise templates for environment-aware speech enhancement [paper]
Speaker-aware deep denoising autoencoder with embedded speaker identity for speech enhancement, Interspeech 2019 [paper]
Audio-Visual SE
Lite Audio-Visual Speech Enhancement, INTERSPEECH 2020 [paper]
Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks, TETCI, 2018 (first audio-visual SE) [journal]
Network design
Filter design
Efficient trainable front-ends for neural speech enhancement, ICASSP 2020 [paper]
Fusion techniques
Spectrograms fusion with minimum difference masks estimation for monaural speech dereverberation, ICASSP 2020 [paper]
Masking and inpainting: a two-stage speech enhancement approach for low snr and non-stationary noise, ICASSP 2020 [paper]
A composite dnn architecture for speech enhancement, ICASSP 2020 [paper]
An attention-based neural network approach for single channel speech enhancement, ICASSP 2019 [paper]
Multi-domain processing via hybrid denoising networks for speech enhancement, 2018 [paper]
Attention
Speech enhancement using self-adaptation and multi-head attention, ICASSP 2020 [paper]
Channel-attention dense u-net for multichannel speech enhancement, ICASSP 2020 [paper]
T-GSA: transformer with gaussian-weighted self-attention for speech enhancement, ICASSP 2020 [paper]
U-net
Phase-aware speech enhancement with deep complex u-net, ICLR 2019 [paper] [code]
GAN
PAGAN: a phase-adapted generative adversarial networks for speech enhancement, ICASSP 2020 [paper]
MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement, ICML 2019 [paper]
Time-frequency masking-based speech enhancement using generative adversarial network, ICASSP 2018 [paper]
SEGAN: speech enhancement generative adversarial network, Interspeech 2017 [paper]
Auto-Encoder
Speech Enhancement Based on Deep Denoising Autoencoder, INTERSPEECH 2013 (first deep learning based SE) [paper]
Phase reconstruction
Phase reconstruction based on recurrent phase unwrapping with deep neural networks, ICASSP 2020 [paper]
PAGAN: a phase-adapted generative adversarial networks for speech enhancement, ICASSP 2020 [paper
Invertible dnn-based nonlinear time-frequency transform for speech enhancement, ICASSP 2020 [paper]
Phase-aware speech enhancement with deep complex u-net, ICLR 2019 [paper] [code]
PHASEN: A Phase-and-Harmonics-Aware Speech Enhancement Network, AAAI 2020 [paper]
Learning strategy
Loss function
MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement, ICML 2019 [paper]
Speech denoising with deep feature losses, Interspeech 2019 [paper]
End-to-end multi-task denoising for joint sdr and pesq optimization, Arxiv 2019 [paper]
Multi-task learning
Multi-objective learning and mask-based post-processing for deep neural network based speech enhancement, Arxiv 2017 [paper]
Multiple-target deep learning for LSTM-RNN based speech enhancement, HSCMA 2017 [paper]
Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks, ISCA 2015 [paper]
Curriculum learning
SNR-Based Progressive Learning of Deep Neural Network for Speech Enhancement, INTERSPEECH 2016 [paper]
Transfer learning
Cross-language transfer learning for deep neural network based speech enhancement, ISCSLP 2014 [paper]
Other improvements
Improving robustness of deep learning based monaural speech enhancement against processing artifacts, ICASSP 2020 [paper]
Tools
Framework
Link
Language
Description
SETK
Python & C++
SETK: Speech Enhancement Tools integrated with Kaldi.
pyAudioAnalysis
Python
Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications.
Beamformer
Python
Implementation of the mask-based adaptive beamformer (MVDR, GEVD, MCWF).
Time-frequency Mask
Python
Computation of the time-frequency mask (PSM, IRM, IBM, IAM, ...) as the neural network training labels.
SSL
Python
Implementation of Sound Source Localization.
Data format
Python
Format tranform between Kaldi, Numpy and Matlab.
Evaluation
Link
Language
Description
PESQ etc.
Matlab
Evaluation for PESQ, CSIG, CBAK, COVL, STOI
SNR, LSD
Python
Evaluation for signal-to-noise-ratio and log-spectral-distortion.
SDR
Matlab
Evaluation for signal-to-distortion-ratio.
Audio feature extraction
Link
Language
Description
LPS
Python
Extract log-power-spectrum/magnitude spectrum/log-magnitude spectrum/Cepstral mean and variance normalization.
MFCC
Python
This library provides common speech features for ASR including MFCCs and filterbank energies.
pyAudioAnalysis
Python
Python Audio Analysis Library: Feature Extraction, Classification, Segmentation and Applications.
Audio data augmentation
Link
Language
Description
Data simulation
Python
Add reverberation, noise or mix speaker.
RIR simulation
Python
Generation of the room impluse response (RIR) using image method.
pyroomacoustics
Python
Pyroomacoustics is a package for audio signal processing for indoor applications.
gpuRIR
Python
Python library for Room Impulse Response (RIR) simulation with GPU acceleration
rir_simulator_python
Python
Room impulse response simulator using python
audiomentations
Python
A Python library for audio data augmentation, e.g. time stretch, pitch shift, add noise, add room reverberation
Datasets
Speech ehancement datasets (sorted by usage frequency in paper)
Name
Utterances
Speakers
Language
Pricing
Additional information
Dataset by University of Edinburgh (2016)
35K+
86
English
Free
Noisy speech database for training speech enhancement algorithms and TTS models.
TIMIT (1993)
6K+
630
English
$250.00
The TIMIT corpus of read speech is one of the earliest speaker recognition datasets.
VCTK (2009)
43K+
109
English
Free
Most were selected from a newspaper plus the Rainbow Passage and an elicitation paragraph intended to identify the speaker's accent.
WSJ0 (1993)
--
149
English
$1500
The WSJ database was generated from a machine-readable corpus of Wall Street Journal news text.
REVERB (2014)
-
8K+
English
Free
This corpus is from REVERB 2014 chanllenge. The challenge assumes the scenario of capturing utterances spoken by a single stationary distant-talking speaker with 1-channel (1ch), 2-channel (2ch) or 8-channel (8ch) microphone-arrays in reverberant meeting rooms. It features both real recordings and simulated data, a part of which simulates the real recordings.
LibriSpeech (2015)
292K
2K+
English
Free
Large-scale (1000 hours) corpus of read English speech.
CHiME series (~2020)
--
--
English
Free
The database is published by CHiME Speech Separation and Recognition Challenge.
Augmentation noise sources (sorted by usage frequency in paper)
Name
Noise types
Pricing
Additional information
DEMAND (2013)
18
Free
Diverse Environments Multichannel Acoustic Noise Database provides a set of recordings that allow testing of algorithms using real-world noise in a variety of settings.
115 Noise (2015)
115
Free
The noise bank for simulate noisy data with clean speech. For N1-N100 noises, they were collected by Guoning Hu and the other 15 home-made noise types by USTC.
NoiseX-92 (1996)
15
Free
Database of recording of various noises available on 2 CDROMs.
RIR_Noises (2017)
-
Free
A database of simulated and real room impulse responses, isotropic and point-source noises. The audio files in this data are all in 16k sampling rate and 16-bit precision.This data includes all the room impulse responses (RIRs) and noises we used in our paper "A Study on Data Augmentation of Reverberant Speech for Robust Speech Recognition" submitted to ICASSP 2017. It includes the real RIRs and isotropic noises from the RWCP sound scene database, the 2014 REVERB challenge database and the Aachen impulse response database (AIR); the simulated RIRs generated by ourselves and also the point-source noises that extracted from the MUSAN corpus.
SOTA results
STOA results in dataset by University of Edinburgh . The following methods are all trained by "trainset_28spk" and tested by common testset. ("F" denotes frequency-domain and "T" is time-domain.)
Methods
Publish
Domain
PESQ
CSIG
CBAK
COVL
SegSNR
STOI
Noisy
--
--
1.97
3.35
2.44
2.63
1.68
0.91
Wiener
--
--
2.22
3.23
2.68
2.67
5.07
--
SEGAN
INTERSPEECH 2017
T
2.16
3.48
2.94
2.80
7.73
0.93
CNN-GAN
APSIPA 2018
F
2.34
3.55
2.95
2.92
--
0.93
WaveUnet
arxiv 2018
T
2.40
3.52
3.24
2.96
9.97
--
WaveNet
ICASSP 2018
T
--
3.62
3.24
2.98
--
--
U-net
ISMIR 2017
F
2.48
3.65
3.21
3.05
9.34
--
MSE-GAN
ICASSP 2018
F
2.53
3.80
3.12
3.14
--
0.93
DFL
INTERSPEECH 2019
T
--
3.86
3.33
3.22
--
--
DFL reimplemented
ICLR 2019
T
2.51
3.79
3.27
3.14
9.86
--
TasNet
TASLP 2019
T
2.57
3.80
3.29
3.18
9.65
--
MDPhD
arxiv 2018
T&F
2.70
3.85
3.39
3.27
10.22
--
Complex U-net
INTERSPEECH 2019
F
3.24
4.34
4.10
3.81
16.85
--
Complex U-net reimplemented
arxiv 2019
F
2.87
4.12
3.47
3.51
9.96
--
SDR-PRSQ
arxiv 2019
F
3.01
4.09
3.54
3.55
10.44
T-GSA
ICASSP 2020
F
3.06
4.18
3.59
3.62
10.78
--
RHRnet
ICASSP 2020
T
3.20
4.37
4.02
3.82
14.71
0.98
Applications
Learning materials
Book or thesis
Audio Source Separation and Speech Enhancement, Emmanuel Vincent, 2019 [link]
A Study on WaveNet, GANs and General CNNRNN Architectures, 2019 [link]
Deep learning: method and applications, 2016 [link]
Deep learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville, 2016 [link]
Robust automatic speech recognition by Jinyu Li and Li Deng, 2015 [link]
Video
CCF speech seminar 2020 [link]
Real-time Single-channel Speech Enhancement with Recurrent Neural Networks by Microsoft Research, 2019 [link]
Deep learning in speech by Hongyi Li, 2019 [link]
High-Accuracy Neural-Network Models for Speech Enhancement, 2017 [link]
DNN-Based Online Speech Enhancement Using Multitask Learning and Suppression Rule Estimation, 2015 [link]
Microphone array signal processing: beyond the beamformer,2011 [link]
Slides
Speech Enhancement based on Deep Learning and Intelligibility Evaluation by Yu Tsao, (APSIPA 2019 tutorial, INTERSPEECH 2020 tutorial) [link]
Deep learning in speech by Hongyi Li, 2019 [link]
Learning-based approach to speech enhancement and separation (INTERSPEECH tutorial, 2016) [link]
Deep learning for speech/language processing (INTERSPEECH tutorial by Li Deng, 2015) [link]
Speech enhancement algorithms (Stanford University, 2013) [link]