An international research group from Nanyang Technical University Singapore, Hunan University China, BITS Pilani India headed by Dr Eng-Siong Chng, Dr Xionghu Zhong and Dr Van Tung Pham
Speaker Diarization is a process to answer the question of 'who spoke when?' in an audio file. It annotates timeframes in an audio according to the speaker of the frame.
A typical Speaker Diarization pipeline involves solving various subproblems, broadly: Identification of speech regions, Extracting features from speech frames, clustering them, and an optional resegmentation step to refine predictions.
- Clone this project
git clone https://github.com/sehgal-simran/Speaker-Diarization.git
cd Speaker-DiarizationTo find a good baseline, we reviewed many recent works and found 2 common types of systems:
-
Kaldi-based: These are systems built on top of the robust kaldi speaker diarization recipe involving x-vectors, by modifying one or more components in the pipeline.
-
End-to-End/ Neural-based: These are systems which combine many modules of the pipeline within a single (often neural network based) model. They have potential to be/already are end-to-end i.e input is audio and output is speaker labels.
We implemented and reviewed 3 approaches:
a) Kaldi's original x-vector recipe
b) Kaldi's x-vector with an LSTM similarity scoring module
c) Region Proposal Network for Speaker Diarization.
The 3 systems are evaluated on the CALLHOME dataset with a collar of 0.25 secs using the standard md-eval.pl script for scoring. The inference time is simply the proportion of time taken for diarization out of total time of diarized speech. It is meant to give an idea about the speed of the system.
| Pipeline | DER (overlap) | DER (no overlap) | Inference time | Testing Method |
|---|---|---|---|---|
| Original Kaldi x-vector | 16.78% | 7.09% | 19% | PLDA is adapted and tested on Callhome using 2-fold cross validation |
| Kaldi x-vector with LSTM | 16.52% | 6.52% | 18% | LSTM model is trained entirely on Callhome and tested using 5-fold cross validation |
| RPNSD | 18.22% | 12.93% | 4% | RPNSD model is adapted and tested on Callhome using 5-fold cross validation |