sasv-challenge/SASVC2022_Baseline

Reproducing baseline1

yzyouzhang opened this issue · 3 comments

yzyouzhang commented

Thanks for providing the code for pre-trained models and baseline2. I am reproducing baseline1 based on your description in the evaluation plan, but I got very different results on the development set. I am also curious why the SPF-EER on the development set is much worse than that on the evaluation set in your results. Could you please provide the code for reproducing your baseline1 result? Thank you so much!

Jungjee commented

Hello @yzyouzhang,
just a couple of questions first.

  1. Did you extract embeddings using the script yourself or did you use extracted embeddings?
  2. Can you share your results to see how different it is?
yzyouzhang commented
  1. I used your extracted embeddings.
  2. My result for baseline1 is 32.89, 35.32, 0.07, 0.67, 13.05, 19.31 for the six values in the table. The main difference with yours is on the development set.
  • 👍1
Jungjee commented

Thanks for sharing.
I checked the numbers with other organisers.

The difference comes from speaker embedding extraction strategies.
The strategy in this repository coincides with yours.

We will reflect this change in the upcoming evaluation plan v0.2.
Thank you once again for sharing this.

  • 👍3