How can I reproduce the results?
Closed this issue · 3 comments
Hello author, may I ask if I have neglected some steps, which makes the evaluation result inconsistent with the paper?
09/23 16:42:41 - mmengine - WARNING - The prefix is not set in metric class UnifiedSegMetric.
Loads checkpoint by local backend from path: model-esam/ESAM-E_CA_online_epoch_128.pth
09/23 16:42:49 - mmengine - INFO - Load checkpoint from model-esam/ESAM-E_CA_online_epoch_128.pth
/opt/miniconda3/envs/ESAM/lib/python3.8/site-packages/mmdet3d/structures/points/base_points.py:136: UserWarning: point got color value beyond [0, 255]
warnings.warn('point got color value beyond [0, 255]')
09/23 16:47:24 - mmengine - INFO -
+---------+---------+---------+--------+
| classes | AP_0.25 | AP_0.50 | AP |
+---------+---------+---------+--------+
| object | 0.0009 | 0.0000 | 0.0000 |
+---------+---------+---------+--------+
| Overall | 0.0009 | 0.0000 | 0.0000 |
+---------+---------+---------+--------+
09/23 16:47:24 - mmengine - INFO - Epoch(test) [44/44] all_ap: 0.0000 all_ap_50%: 0.0000 all_ap_25%: 0.0009 data_time: 0.1551 time: 6.2308
For the evaluation of ESAM-E on 3RScan-mv, first, you need to process the 3RScan and 3RScan-mv_fast data according to the dataset preparation readme. Then, download the checkpoint and run the command CUDA_VISIBLE_DEVICES=0 python tools/test.py configs/ESAM-E_CA/ESAM-E_online_3rscan_CA_test.py work_dirs/ESAM-E_online_scannet200_CA/epoch_128.pth --work-dir work_dirs/ESAM-E_online_3rscan_CA_test/.
Please make sure all these steps are completed. If you still encounter issues, please share your log file with us for a more detailed description, so we can work together to resolve the problem.
I am not sure if the processing is completely correct, but I checked according to the file structure you gave and found no problem.
(ESAM) root@gpuserver-8010-24:/home/gx/wtang/ESAM/data/3RScan# ls
3RScan pycache scannet_utils.py
'3RScan.v2 Semantic Classes - Mapping.tsv' batch_load_3rscan_data.py semantic_mask
3rscan_instance_data instance_mask super_points
3rscan_oneformer3d_infos_val.pkl load_scannet_data.py
README.md points
(ESAM) root@gpuserver-8010-24:/home/gx/wtang/ESAM/data/3RScan-mv_fast# ls
3RScan load_3rscan_mv_data.py
'3RScan.v2 Semantic Classes - Mapping.tsv' load_3rscan_mv_data_fast.py
3rscan_mv_oneformer3d_infos_val.pkl load_scannet_data.py
README.md points
pycache scannet_utils.py
axis_align_matrix semantic_mask
'download-3RScan copy.py' super_points
download-3RScan.py test.txt
instance_mask
The evaluation order I executed was as follows：
CUDA_VISIBLE_DEVICES=3 python tools/test.py configs/ESAM-E_CA/ESAM-E_online_3rscan_CA_test.py model-esam/ESAM-E_CA_online_epoch_128.pth --work-dir ./ESAM-E_online_3rscan_CA_test/
The log file is linked below：
20240923_164213.log
It seems there are no apparent issues. We have also tried reproducing the results with the latest code and successfully obtained the correct outcome. Could you please clone the repository again, follow the instructions in the README to download the provided data, process it, and then run the evaluation again? If there are still any issues, please let us know.