More detail for package version
Closed this issue · 6 comments
Hello,
I wanna to reproduce this work more accurately.
May I get more details what you used?
Maybe is numpy version or others information?
Hello,
From the environment where I usually ran these experiments I have
numpy 1.19.4
scipy 1.4.1
sklearn 0.20.3
numexpr 2.6.9
h5py 2.9.0
onnxruntime 1.4.0
soundfile 0.10.3
torch 1.6.0
tabulate 0.8.6
In any case, a different version should not produce substantially different results. Differences of up to 0.2 DER can happen just from changing the random seed in the code so I would expect differences because of packages' versions to be even smaller.
@fnlandini Thanks for your answer, I will try it.
@fnlandini we should probably update requirements.txt and setup.py with those versions
@Jamiroquai88 I agree, but I'd like to test that we obtain the same results we report with these particular packages to be 100% sure of them. I'm rather busy lately but I'll keep this issue open to remind me of doing so
Hello @fnlandini , my version environment may be the same as yours.
Now, my VBx DER is 19.07.
And AHC DER is 21.43!
The AHC results are same as yours.
This is my AHC's result_full.
File DER JER B3-Precision B3-Recall B3-F1 GKT(ref, sys) GKT(sys, ref) H(ref|sys) H(sys|ref) MI NMI
--------------- ----- ----- -------------- ----------- ------- --------------- --------------- ------------ ------------ ---- -----
EN2002a 11.70 19.74 0.89 0.82 0.85 0.78 0.86 0.37 0.73 2.08 0.79
EN2002b 14.09 26.64 0.88 0.81 0.84 0.76 0.85 0.38 0.81 2.05 0.78
EN2002c 9.93 27.64 0.89 0.86 0.88 0.82 0.85 0.38 0.55 1.71 0.79
EN2002d 12.94 23.54 0.88 0.81 0.84 0.77 0.85 0.41 0.81 2.04 0.77
ES2004a 9.68 12.35 0.95 0.87 0.91 0.83 0.93 0.18 0.46 1.97 0.86
ES2004b 4.50 5.37 0.96 0.93 0.94 0.91 0.95 0.13 0.30 2.11 0.91
ES2004c 3.93 6.02 0.96 0.93 0.95 0.92 0.95 0.15 0.29 2.09 0.90
ES2004d 7.84 12.42 0.93 0.89 0.91 0.86 0.92 0.23 0.47 1.99 0.85
IS1009a 6.89 14.65 0.94 0.90 0.92 0.84 0.90 0.18 0.36 1.57 0.85
IS1009b 3.96 4.97 0.97 0.93 0.95 0.91 0.96 0.12 0.32 2.13 0.91
IS1009c 3.55 4.86 0.98 0.94 0.96 0.93 0.97 0.09 0.27 2.15 0.92
IS1009d 5.40 7.33 0.96 0.91 0.94 0.88 0.94 0.15 0.37 1.89 0.88
TS3003a 12.24 15.96 0.98 0.85 0.91 0.75 0.96 0.05 0.51 1.12 0.81
TS3003b 6.61 6.45 0.97 0.90 0.93 0.87 0.96 0.09 0.44 2.05 0.89
TS3003c 4.20 3.89 0.98 0.94 0.96 0.92 0.97 0.07 0.28 2.16 0.93
TS3003d 6.35 18.08 0.94 0.91 0.92 0.88 0.91 0.21 0.36 1.91 0.87
*** OVERALL *** 7.60 13.57 0.94 0.89 0.91 0.89 0.94 0.21 0.46 5.91 0.95
Updated requirements.txt and setup.py