Mael-zys/T2M-GPT

95% confidence interval

Closed this issue · 5 comments

exitudio commented

Thank you for your amazing work.
I have a question. Do you have the code for "95% confidence interval" in the evaluation function?

Mael-zys commented

Following t2m, we calculate the 95% confidence interval by: np.std(fid)*1.96/np.sqrt(repeat_time).

The code of "95% confidence interval" can be found in the end of the evaluation function:
https://github.com/Mael-zys/T2M-GPT/blob/main/GPT_eval_multi.py#L120

exitudio commented

Thank you so much.

exitudio commented

I have another related question to double-check my run. Maybe, better to put it here.
I am running "GPT_eval_multi.py" and the evaluation seems to take a very long time (almost 200 hours).
Is this normal?

Mael-zys commented

Yes, following the evaluation protocol of t2m, we repeat the evaluation function 20 times and each time we need to generate 30 motions for each text. This makes the evaluation take a very long time. You can also run all the evaluation 20 times (except MultiModality runs 5 times) as mdm.

exitudio commented

Thank you for the clarification.