Reproducibility concerns
AndreeaDogaru opened this issue · 7 comments
Hello. Thank you for your amazing work and for sharing the code!
I tried training your model from scratch on some of the benchmark scenes and had problems reproducing your results. It seems like the model is quite susceptible to the random seed, and even after several attempts, the quality I obtained was lower than reported.
| Experiment | Fountain - Full | Fountain - Center | Herzjesu - Full | Herzjesu - Center |
|---|---|---|---|---|
| pre-trained | 7.77 | 1.91 | 8.88 | 2.03 |
| seed 2022 | 8.03 | 2.65 | 7.66 | 2.55 |
| seed 42 | 13.36 | 7.43 | 10.54 | 2.58 |
Could you provide some insights regarding this issue?
Hello,
Sorry about that, I will have a look at it next week. In the meantime and just to be sure, are you sure you have launched with the correct conf file ? baseline_epfl.conf then NeuralWarp_epfl.conf
In order to understand what happens, can you also evaluate the baseline model you have trained and send me the results ?
Yes, I used the baseline_epfl.conf then NeuralWarp_epfl.conf to train the model. Here are the results of the baseline model for the two seeds considered above.
| VolSDF Baseline | Fountain - Full | Fountain - Center | Herzjesu - Full | Herzjesu - Center |
|---|---|---|---|---|
| seed 2022 | 15.55 | 5.41 | 11.61 | 3.12 |
| seed 42 | 18.48 | 11.53 | 16.07 | 7.63 |
Hi, I have given a look to it. Sorry for the late reply
It is true the results are sensible to random seed on epfl dataset. This is because the sphere initialization of the geometry is not suited for EPFL scenes contrary to DTU scenes. It largely influences the "full" metrics and this is the reason why we introduced the "center" metrics
However I have launched 4 different seeds for "fountain" and I have not encountered the same variability as you show
| Seed | Fountain - Full | Fountain - Center |
|---|---|---|
| 2022 | 7.78 | 1.92 |
| 42 | 11.93 | 1.96 |
| 43 | 7.72 | 2.31 |
| 1994 | 5.94 | 1.98 |
Have you launched the mesh extraction with the options --bbox_size 4 --no_one_cc --filter_visible_triangles --min_nb_visible 1 ?
Hi. Yes, I extracted the mesh with the specified options --bbox_size 4 --no_one_cc --filter_visible_triangles --min_nb_visible 1. These are the respective meshes for the fountain scene (seed 42, seed 2022). Also, I believe this issue is not restricted to the EPFL dataset, I noticed similar results on the DTU scene 118.
Your meshes definitely look worse than the ones I got.
I'm sorry I don't understand what may cause the issue... Maybe a package version issue. Do you use the exact same packages as in requirements.txt ? Especially pytorch and pytorch3d ?
No, I was using an older pytorch version. I've rerun the experiments on the fountain scene with torch==1.9.0, as indicated in requirements.txt, and obtained results similar to yours. Thank you!
I'm glad it solved your issue!