/gcc-nmf

Real-time GCC-NMF Blind Speech Separation and Enhancement

Primary LanguagePythonMIT LicenseMIT

GCC-NMF

GCC-NMF is a blind source separation and denoising algorithm that combines the GCC spatial localization method with the NMF unsupervised dictionary learning algorithm. GCC-NMF has been used for stereo speech separation and enhancement in both offline and real-time settings. Though we have focused on speech applications so far, GCC-NMF is a generic source separation and denoising algorithm and may well be applicable to other types of signals.

This GitHub repository provides:

  1. A standalone Python executable to execute and visualize GCC-NMF in real-time.

  2. A series of iPython notebooks notebooks presenting GCC-NMF in tutorial style, building towards the low latency, real-time context:

Journal Papers

Conference Papers

Real-time Speech Enhancement: RT-GCC-NMF

The Real-time Speech Enhancement standalone Python executable is an implementation of the RT-GCC-NMF real-time speech enhancement algorithm. Users may interactively modify system parameters including the NMF dictionary size and GCC-NMF masking function parameters, where the effects on speech enhancement quality may be heard in real-time.

png

Offline Speech Separation

The Offline Speech Separation iPython notebook shows how GCC-NMF can be used to separate multiple concurrent speakers in an offline fashion. The NMF dictionary is first learned directly from the mixture signal, and sources are subsequently separated by attributing each atom at each time to a single source based on the dictionary atoms' estimated time delay of arrival (TDOA). Source localization is achieved with GCC-PHAT.

png

Offline Speech Enhancement

The Offline Speech Enhancement iPython notebook demonstrates how GCC-NMF can can be used for offline speech enhancement, where instead of multiple speakers, we have a single speaker plus noise. In this case, individual atoms are attributed either to the speaker or to noise at each point in time base on the the atom TDOAs as above. The target speaker is again localized with GCC-PHAT.

png

Online Speech Enhancement

The Online Speech Enhancement iPython notebook demonstrates an online variant of GCC-NMF that works in a frame-by-frame fashion to perform speech enhancement in real-time. Here, the NMF dictionary is pre-learned from a different dataset than used at test time, NMF coefficients are inferred frame-by-frame, and speaker localization is performed with an accumulated GCC-PHAT method.

png

Low Latency Speech Enhancement

In the Low Latency Speech Enhancement iPython notebook we extend the online GCC-NMF approach to reduce algorithmic latency via asymmetric STFT windowing strategy. Long analysis windows maintain the high spectral resolution required by GCC-NMF, while short synthesis windows drastically reduce algorithmic latency with little effect on speech enhancement quality. Algorithmic latency can be reduced from over 64 ms using traditional symmetric STFT windowing to below 2 ms with the proposed asymmetric STFT windowing, provided sufficient computational power is available.

png