About reproducing the performance.

Question

About reproducing the performance.

Closed this issue a year ago · 2 comments

Great project! I am very insterested in your work and thanks for the release.

However, after reproducing the experiment with the code(HRDA+PiPa), I was not able to achieve the result reported in the paper. Specifically, I reproduced the experiment by directly run

python run_experiments.py --config configs/pipa/gtaHR2csHR_hrda.py

with the random seed 0, 1, 2, and achieved mean intersection over union score 74.52, 74.34, 74.73 respectively. Here is my logs:

with seed=0, mIoU=74.52 20230109_160110.log
with seed=1, mIoU=74.34 20230208_233817.log
with seed=2, mIoU=74.73 20230210_153346.log

I don't know the reason of the performance drop. Could you please tell me the possible reason or any hint to reproduce the results?

Thanks in advance.

Best,
Yuanbing

Answer 1 · 2023-03-11T03:16:00.000Z

Hi Yuanbing,

Thanks for your interest in our work,
According to my experience, the cuda version will cause some problem, mmcv were compiled with cuda11.0. FYI

Kind Regards
Mu

Answer 2 · 2023-03-11T10:39:10.000Z

Dear Mu@chen742,

Thanks for your kind reply.

As you mentioned in #3, the implemented patch loss in dacs.py is specific for the ablation experiments, and that might be another reason of the performance drop since only pixel loss was applied in my reproduced experiments.

I'm looking forward to the further update.

Best,
Yuanbing