Questions about the success rate of experimental results
Closed this issue · 1 comments
In the paper (3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations) The chart appearing in the appendix, which I have shown in the appendix, is entitled TABLE XVII: Full evaluation results of Simple DP3. We evaluate Simple DP3 on 10 tasks and compare it with DP3 and find that Simple DP3 could achieve results very competitive to DP3 and TABLE XVIII: Main results on 72 simulation tasks. Results for each task are provided in this table. A summary across domains is shown in Table I.
Q1: Is the result of TABLE XVII and TABLE XVIII the highest success rate or the average success rate?
Q2: What is the meaning of '+-' for the data in TABLE XVII and TABLE XVIII?
Q3: In the process of reproduction, I found that for the same task, several times running the script, there may be results, for example, the data you provided is 80 (assuming the success rate), whether it is the highest/average/final success rate, I found that there will be fluctuations, such as 78,81,77, etc. May I ask how the data in the final table was processed?
Q4: For the data in TABLE XVII and TABLE XVIII, there may be some decimal phenomena in the process of reproduction, that is, the obta
ined results are not always integers. May I ask how the data in the final table is processed?
Hi, thank you for your interest. The success rate is the average of 3 trials, each trial is the average of 5 best history success rates, as stated in our main paper. The std is reported also by '+-'. The integers are just rounded from decimals.