val miou high but test miou low
Lizhijian-123 opened this issue · 8 comments
Hello, I try to change your model, but there is a situation where the test proves that the miou is high but the test miou is low, do you know why?
Thank you for your interest in our work! I am happy to answer your question, but I need to understand your experimental setup better. Can you please tell me which dataset you conducted the experiments on? What are the specific values of val mIoU and test mIoU? What changes did you make to the model?
Thank you for your interest in our work! I am happy to answer your question, but I need to understand your experimental setup better. Can you please tell me which dataset you conducted the experiments on? What are the specific values of val mIoU and test mIoU? What changes did you make to the model?
Thank you for your reply, the data set I use is s3dis, the following is the val miou and test miou I ran pointmetabase-l:
val miou:
[02/26 21:44:35] S3DIS INFO: Epoch 94 LR 0.000130 train_miou 89.25, val_miou 68.28, best val miou 68.59
[02/26 22:00:11] S3DIS INFO: Epoch 95 LR 0.000098 train_miou 89.19, val_miou 67.97, best val miou 68.59
[02/26 22:44:08] S3DIS INFO: Epoch 100 LR 0.000012 train_miou 89.21, val_miou 68.45, best val miou 68.59
test miou:
[02/27 00:14:40] S3DIS INFO: Best ckpt @e91, test_oa 90.19, test_macc 75.82, test_miou 69.26,
iou per cls is: [95.62 98.24 83.43 0. 32.92 56.23 76.3 82.43 90.64 75.53 75.32 72.3
61.4 ]
Below are the val miou and test miou of my modified model:
val miou:
[03/09 00:09:44] S3DIS INFO: Epoch 69 LR 0.002329 train_miou 89.58, val_miou 68.96, best val miou 68.96
[03/09 00:09:45] S3DIS INFO: Found the best model and saved in /root/autodl-tmp/PointMetaBase-main_2/examples/segmentation/log/s3dis/s3dis-train-pointmetabase-l-ngpus1-seed2425-20230308-124546-dCRMoh766KbcqeJHyqL74Y/checkpoint/s3dis-train-pointmetabase-l-ngpus1-seed2425-20230308-124546-dCRMoh766KbcqeJHyqL74Y_ckpt_best.pth
[03/09 00:19:39] S3DIS INFO: Epoch 70 LR 0.002197 train_miou 89.59, val_miou 68.07, best val miou 68.96
[03/09 00:29:34] S3DIS INFO: Epoch 71 LR 0.002069 train_miou 89.54, val_miou 68.36, best val miou 68.96
[03/09 00:39:28] S3DIS INFO: Epoch 72 LR 0.001944 train_miou 89.71, val_miou 67.70, best val miou 68.96
[03/09 00:49:24] S3DIS INFO: Epoch 73 LR 0.001821 train_miou 89.74, val_miou 68.36, best val miou 68.96
[03/09 00:59:16] S3DIS INFO: Epoch 74 LR 0.001702 train_miou 89.80, val_miou 67.31, best val miou 68.96
[03/09 01:09:11] S3DIS INFO: Epoch 75 LR 0.001586 train_miou 89.89, val_miou 68.60, best val miou 68.96
[03/09 01:19:02] S3DIS INFO: Epoch 76 LR 0.001473 train_miou 89.86, val_miou 68.22, best val miou 68.96
[03/09 01:28:56] S3DIS INFO: Epoch 77 LR 0.001364 train_miou 89.87, val_miou 67.82, best val miou 68.96
[03/09 01:38:51] S3DIS INFO: Epoch 78 LR 0.001258 train_miou 89.87, val_miou 68.55, best val miou 68.96
[03/09 01:48:46] S3DIS INFO: Epoch 79 LR 0.001156 train_miou 89.84, val_miou 68.11, best val miou 68.96
[03/09 01:58:42] S3DIS INFO: Epoch 80 LR 0.001058 train_miou 89.95, val_miou 68.33, best val miou 68.96
[03/09 02:08:36] S3DIS INFO: Epoch 81 LR 0.000964 train_miou 89.97, val_miou 68.05, best val miou 68.96
[03/09 02:18:32] S3DIS INFO: Epoch 82 LR 0.000874 train_miou 90.00, val_miou 68.08, best val miou 68.96
[03/09 02:28:26] S3DIS INFO: Epoch 83 LR 0.000788 train_miou 90.02, val_miou 67.91, best val miou 68.96
[03/09 02:38:21] S3DIS INFO: Epoch 84 LR 0.000706 train_miou 90.00, val_miou 68.23, best val miou 68.96
[03/09 02:48:13] S3DIS INFO: Epoch 85 LR 0.000628 train_miou 90.07, val_miou 68.37, best val miou 68.96
[03/09 02:58:03] S3DIS INFO: Epoch 86 LR 0.000554 train_miou 90.05, val_miou 68.10, best val miou 68.96
[03/09 03:07:53] S3DIS INFO: Epoch 87 LR 0.000485 train_miou 90.08, val_miou 68.27, best val miou 68.96
[03/09 03:17:44] S3DIS INFO: Epoch 88 LR 0.000421 train_miou 90.10, val_miou 68.47, best val miou 68.96
[03/09 03:27:34] S3DIS INFO: Epoch 89 LR 0.000361 train_miou 90.14, val_miou 68.47, best val miou 68.96
[03/09 03:37:31] S3DIS INFO: Epoch 90 LR 0.000305 train_miou 90.12, val_miou 68.32, best val miou 68.96
[03/09 03:47:22] S3DIS INFO: Epoch 91 LR 0.000254 train_miou 90.16, val_miou 68.65, best val miou 68.96
[03/09 03:57:13] S3DIS INFO: Epoch 92 LR 0.000208 train_miou 90.12, val_miou 68.14, best val miou 68.96
[03/09 04:07:04] S3DIS INFO: Epoch 93 LR 0.000167 train_miou 90.07, val_miou 68.09, best val miou 68.96
[03/09 04:16:56] S3DIS INFO: Epoch 94 LR 0.000130 train_miou 90.05, val_miou 68.34, best val miou 68.96
[03/09 04:26:49] S3DIS INFO: Epoch 95 LR 0.000098 train_miou 90.04, val_miou 68.45, best val miou 68.96
[03/09 04:36:40] S3DIS INFO: Epoch 96 LR 0.000071 train_miou 90.10, val_miou 68.27, best val miou 68.96
[03/09 04:46:30] S3DIS INFO: Epoch 97 LR 0.000049 train_miou 90.11, val_miou 68.34, best val miou 68.96
[03/09 04:56:23] S3DIS INFO: Epoch 98 LR 0.000032 train_miou 90.08, val_miou 68.60, best val miou 68.96
[03/09 05:06:14] S3DIS INFO: Epoch 99 LR 0.000020 train_miou 90.09, val_miou 68.21, best val miou 68.96
[03/09 05:16:10] S3DIS INFO: Epoch 100 LR 0.000012 train_miou 90.09, val_miou 67.96, best val miou 68.96
[03/09 05:16:10] S3DIS INFO: Best ckpt @e69, val_oa 90.09, val_macc 74.81, val_miou 68.96,
iou per cls is: [94.25 97.21 83.58 0. 33.81 58.28 78.98 81.09 90.25 60.86 76.54 79.94
61.64]
test miou:
[03/09 06:03:39] S3DIS INFO: Best ckpt @e69, test_oa 89.96, test_macc 73.03, test_miou 67.23,
iou per cls is: [95.11 97.59 84.47 0. 29.4 56.96 69.21 82.96 91.66 55.58 73.98 79.57
57.52]
As you can see, I don't know why the test miou is so low, my best val miou is higher than pointmetabase-l's best val miou,I have changed many times based on your model, this is the first time that my val miou exceeds yours, I am full of expectations, but the result makes me very sad. Please ignore that the miou of pointmetabase-l is about 0.3-0.5 lower than your accuracy, that's because I added the following code in the test and verification sets to avoid using too much gpu memory, which has a little impact on the actual miou.
for idx, data in pbar:
keys = data.keys() if callable(data.keys) else data.keys
for key in keys:
data[key] = data[key].cuda(non_blocking=True)
print(data[key].shape)
numnn =data[key].shape[1]
target = data['y'].squeeze(-1)
data['x'] = get_features_by_keys(data, cfg.feature_keys)
# print(data[0].shape)
# print("--------------------------------------------------")
# if(numnn>100000):
# continue
if (idx in (5, 12, 15, 51, 52, 55)):
data0 = {}
data1 = {}
num = data['x'].shape[2] // 2
data0['pos'] = data['pos'][:, :num, :]
data0['x'] = data['x'][:, :, :num]
data0['heights'] = data['heights'][:, :num, :]
logits0 = model(data0)
data1['pos'] = data['pos'][:, num:, :]
data1['x'] = data['x'][:, :, num:]
data1['heights'] = data['heights'][:, num:, :]
logits1 = model(data1)
logits = torch.cat((logits0, logits1), dim=2)
else:
print(data['x'].shape)
print("---------------------------")
logits = model(data)
# logits = model(data)
Thank you for your interest in our work! I am happy to answer your question, but I need to understand your experimental setup better. Can you please tell me which dataset you conducted the experiments on? What are the specific values of val mIoU and test mIoU? What changes did you make to the model?
I haven't changed the parameter settings, but I doubt whether the learning rate will be affected. I added some convolutions that integrate pe and fj. Although you said in the paper that this will increase the amount of calculation, I still tried it. At the same time, I tried to add a transformer before this, or further extract the feature f before the group (so as not to increase the amount of parameters too much), but it is worse than pointmetabase-l itself.
I am so amazed at how simple your model is but works so well! it is so amazing.
Your test mIoU is much lower than val mIoU in your experiment, which is not normal. It seems that you have modified the evaluation code to split the point clouds with idx (5, 12, 15, 51, 52, 55) into two inputs for the model. I suggest you check whether your prediction results on these point clouds are lower than expected.
Thank you for your reply, I deleted those split codes to re-verify and test, the result is the same. The following are the val and test results of the same pointmetabase-l and the weight file of the model modified by myself:
pointmeatbase-l:
val miou:68.63
test miou:69.8
changemodel:
val miou:68.88
test miou:67.88
i don't know why
Hi, @Lizhijian-123 . It seems that the issue is not caused by the test code, so the problem may lie in the structure adjustments you made. I suggest you debug in the following way:
- Keep your current structure and run two more experiments to see if the phenomenon of val mIoU being higher than test mIoU still occurs.
- If the phenomenon still exists, gradually ablate the changes you made to your pointmetabase-l and check if the added design caused this phenomenon.
Hi, @Lizhijian-123 . It seems that the issue is not caused by the test code, so the problem may lie in the structure adjustments you made. I suggest you debug in the following way:
- Keep your current structure and run two more experiments to see if the phenomenon of val mIoU being higher than test mIoU still occurs.
- If the phenomenon still exists, gradually ablate the changes you made to your pointmetabase-l and check if the added design caused this phenomenon.
Thank you very much for your reply. I have done some experiments in the past two days. It should be the problem of the model itself, but it is still strange that there is a large gap between val miou and test miou, but I will not get into it, thank you again
@Lizhijian-123 It would be my pleasure to help you. Please feel free to discuss any other issues if you have.