About results
Hello-hubKing opened this issue · 8 comments
Hello !
Thank you for really nice work.
Now, I have trained the pointMLP, but I only obtain the best accuracy is 93.2.
So, I want to know whether is the special parameters.
@Hello-hubKing Thanks for your interest.
Results on ModelNet40 are very unstable, as we discussed here: #1 (comment). In our paper, we report the best result of several runs for ModelNet40 and use voting method.
We would like to suggest run experiments on another dataset, scanobjectNN, and report the mean/std values as shown in Table 3. We believe this would be a better evaluation protocol for point cloud classification task.
Please let me know if you have any further questions.
Hi,
First of all, thanks for your excellent work!
All of the previous issues only talk about the performance on ModelNet40 dataset, but none of them speak about the ScanObjectNN dataset.
Is there such a performance gap on the ScanObjectNN dataset as well?
I trained your work on my workstation (RTX2080Ti) but only got overall accuracy of 78.8% and mean class accuracy of 76.5%
@ja604041062
Thanks for your interest.
First, about the results on ModelNet40, we have clearly explained the gap (unstable training).
Second, there would be a little variance on ScanObjectNN, as shown in Tab. 3 and Fig. 3.
Third, we reported several runs results on ScanObjectNN and the checkpoints can be found in the ReadME file. In case of any mistakes, we also retrain the model based on the release repo today. Here is a simple screenshot.
Clearly, only 38 epochs, PointMLP already achieves 81.575% overall accuracy (much higher than your results at 200 epochs), which is roughly align with our previous release checkpoints/logs at same epochs. We didn't finish the training considering the time, and the screenshot can prove the effectiveness of PointMLP already.
I assume there may be some issues with your training. You can upload your logs/training scripts and I can help debugging. Feel free to let me know if you have further questions or concerns.
Hi,
the argument of my training is:
checkpoint: checkpoints/pointMLP-20220823181946
msg: None
batch_size: 16
model: pointMLP
num_classes: 15
epoch: 200
num_points: 1024
learning_rate: 0.01
weight_decay: 0.0001
smoothing: False
seed: None
workers: 4
The difference is the batch size. Due to hardware limitations (VRAM: 11G), I set batch size of 16.
I don't know if it's causing the performance drop, but I don't think batch size affects that much.
and here is the training log:
Epoch-Num Learning-Rate Train-Loss Train-acc-B Train-acc Valid-Loss Valid-acc-B Valid-acc
0.000000 0.009999 2.770000 8.452000 12.631000 2.568000 9.543000 16.759000
1.000000 0.009998 2.521000 14.481000 19.828000 2.433000 19.117000 25.330000
2.000000 0.009995 2.346000 21.191000 27.761000 2.221000 27.001000 35.357000
3.000000 0.009990 2.232000 27.053000 35.423000 2.105000 36.536000 45.524000
4.000000 0.009985 2.159000 31.827000 40.524000 2.047000 39.960000 48.126000
5.000000 0.009978 2.066000 36.960000 46.099000 1.939000 44.624000 54.511000
6.000000 0.009970 1.996000 41.298000 50.859000 1.892000 49.293000 57.703000
7.000000 0.009961 1.944000 44.847000 53.892000 1.876000 51.325000 57.078000
8.000000 0.009951 1.891000 48.614000 57.205000 1.819000 52.670000 60.444000
9.000000 0.009939 1.830000 52.428000 60.221000 1.815000 53.696000 59.924000
10.000000 0.009926 1.789000 54.785000 62.553000 1.788000 55.547000 62.353000
11.000000 0.009912 1.749000 58.349000 65.138000 1.784000 54.406000 60.826000
12.000000 0.009897 1.709000 60.419000 66.997000 1.720000 60.002000 64.747000
13.000000 0.009881 1.687000 62.589000 68.259000 1.716000 60.645000 64.920000
14.000000 0.009863 1.662000 64.370000 69.705000 1.728000 58.985000 64.539000
15.000000 0.009844 1.635000 65.519000 70.915000 1.699000 62.419000 67.176000
16.000000 0.009825 1.619000 67.284000 71.818000 1.697000 60.206000 66.031000
17.000000 0.009803 1.590000 69.455000 73.659000 1.672000 63.077000 66.586000
18.000000 0.009781 1.588000 69.495000 74.053000 1.667000 62.480000 66.482000
19.000000 0.009758 1.562000 71.319000 75.430000 1.633000 65.760000 69.431000
20.000000 0.009733 1.550000 72.183000 76.017000 1.629000 66.158000 70.125000
21.000000 0.009707 1.524000 73.380000 77.156000 1.598000 68.855000 71.860000
22.000000 0.009680 1.512000 75.134000 78.322000 1.644000 65.350000 69.917000
23.000000 0.009652 1.501000 75.475000 78.734000 1.612000 67.851000 71.201000
24.000000 0.009623 1.481000 76.897000 79.883000 1.693000 65.218000 67.349000
25.000000 0.009593 1.468000 77.086000 80.119000 1.645000 67.303000 69.882000
26.000000 0.009561 1.460000 77.568000 80.531000 1.587000 69.650000 72.241000
27.000000 0.009529 1.448000 78.904000 81.408000 1.566000 71.893000 74.289000
28.000000 0.009495 1.429000 79.793000 82.425000 1.606000 68.598000 70.958000
29.000000 0.009460 1.423000 80.400000 82.819000 1.598000 70.156000 72.866000
30.000000 0.009425 1.405000 81.914000 83.880000 1.592000 69.899000 72.831000
31.000000 0.009388 1.402000 81.598000 83.853000 1.616000 68.758000 70.888000
32.000000 0.009350 1.384000 82.832000 84.756000 1.578000 70.364000 73.005000
33.000000 0.009311 1.382000 83.162000 85.124000 1.550000 71.362000 74.358000
34.000000 0.009271 1.374000 83.105000 85.221000 1.585000 71.795000 72.970000
35.000000 0.009229 1.362000 84.451000 86.343000 1.543000 72.257000 75.295000
36.000000 0.009187 1.349000 85.663000 87.123000 1.574000 71.715000 73.560000
37.000000 0.009144 1.338000 85.360000 87.211000 1.576000 72.016000 73.976000
38.000000 0.009100 1.328000 86.596000 87.894000 1.606000 68.438000 71.548000
39.000000 0.009055 1.320000 87.009000 88.368000 1.541000 74.049000 75.607000
40.000000 0.009008 1.309000 87.704000 89.087000 1.558000 72.649000 74.809000
41.000000 0.008961 1.311000 87.137000 88.745000 1.543000 73.772000 76.544000
42.000000 0.008913 1.305000 87.667000 89.043000 1.554000 72.421000 75.087000
43.000000 0.008864 1.295000 88.888000 89.832000 1.537000 74.100000 75.330000
44.000000 0.008814 1.292000 88.872000 89.884000 1.543000 72.378000 74.601000
45.000000 0.008763 1.291000 88.679000 90.007000 1.531000 74.031000 75.746000
46.000000 0.008711 1.275000 89.943000 90.989000 1.529000 74.160000 76.718000
47.000000 0.008658 1.270000 90.413000 91.339000 1.582000 71.469000 73.838000
48.000000 0.008605 1.260000 90.842000 91.760000 1.553000 72.894000 75.330000
49.000000 0.008550 1.249000 91.841000 92.637000 1.535000 72.674000 75.642000
50.000000 0.008495 1.244000 91.573000 92.435000 1.519000 74.950000 76.544000
51.000000 0.008439 1.241000 92.063000 92.777000 1.556000 73.029000 76.197000
52.000000 0.008381 1.238000 92.220000 92.979000 1.532000 74.137000 75.954000
53.000000 0.008323 1.229000 92.700000 93.426000 1.562000 72.898000 75.434000
54.000000 0.008265 1.226000 93.387000 93.925000 1.555000 70.652000 74.254000
55.000000 0.008205 1.220000 93.420000 94.066000 1.538000 70.986000 74.532000
56.000000 0.008145 1.220000 93.041000 93.706000 1.521000 74.643000 76.475000
57.000000 0.008084 1.220000 93.411000 93.899000 1.580000 71.651000 74.323000
58.000000 0.008022 1.212000 93.960000 94.407000 1.525000 74.978000 76.544000
59.000000 0.007960 1.206000 94.432000 94.846000 1.555000 71.321000 74.636000
60.000000 0.007896 1.200000 94.399000 94.995000 1.552000 73.798000 75.503000
61.000000 0.007832 1.193000 94.969000 95.415000 1.556000 71.148000 74.705000
62.000000 0.007768 1.187000 95.159000 95.678000 1.566000 73.338000 74.115000
63.000000 0.007702 1.184000 95.525000 95.889000 1.539000 74.119000 75.850000
64.000000 0.007636 1.184000 95.420000 95.766000 1.533000 73.160000 75.746000
65.000000 0.007570 1.177000 95.861000 96.117000 1.548000 72.382000 75.156000
66.000000 0.007503 1.172000 96.143000 96.520000 1.518000 74.619000 76.648000
67.000000 0.007435 1.174000 96.002000 96.310000 1.546000 73.519000 76.371000
68.000000 0.007366 1.172000 95.956000 96.292000 1.532000 73.600000 76.960000
69.000000 0.007297 1.166000 96.480000 96.800000 1.567000 73.510000 75.399000
70.000000 0.007228 1.163000 96.750000 96.985000 1.534000 74.070000 76.683000
71.000000 0.007158 1.153000 97.228000 97.423000 1.503000 75.557000 78.036000
72.000000 0.007087 1.160000 96.742000 96.967000 1.573000 72.266000 74.427000
73.000000 0.007016 1.156000 97.053000 97.134000 1.534000 72.011000 75.677000
74.000000 0.006944 1.153000 97.106000 97.326000 1.515000 73.724000 76.787000
75.000000 0.006872 1.148000 97.205000 97.511000 1.527000 74.291000 77.273000
76.000000 0.006800 1.146000 97.497000 97.730000 1.546000 73.116000 75.885000
77.000000 0.006727 1.144000 97.522000 97.589000 1.533000 73.233000 75.815000
78.000000 0.006653 1.141000 97.703000 97.809000 1.540000 74.589000 76.336000
79.000000 0.006580 1.138000 97.852000 97.958000 1.544000 73.658000 76.266000
80.000000 0.006505 1.135000 97.939000 98.159000 1.502000 75.462000 77.654000
81.000000 0.006431 1.140000 97.675000 97.861000 1.538000 74.080000 76.613000
82.000000 0.006356 1.136000 97.941000 98.080000 1.518000 74.596000 76.995000
83.000000 0.006281 1.132000 97.825000 98.133000 1.523000 74.576000 76.787000
84.000000 0.006206 1.128000 98.257000 98.405000 1.517000 74.767000 76.995000
85.000000 0.006130 1.125000 98.326000 98.431000 1.525000 74.817000 76.960000
86.000000 0.006054 1.128000 98.274000 98.413000 1.530000 72.895000 75.434000
87.000000 0.005978 1.126000 98.377000 98.396000 1.521000 73.757000 76.648000
88.000000 0.005901 1.118000 98.560000 98.676000 1.510000 74.901000 77.203000
89.000000 0.005824 1.123000 98.452000 98.519000 1.483000 76.744000 78.695000
90.000000 0.005747 1.120000 98.487000 98.633000 1.521000 75.027000 77.412000
91.000000 0.005670 1.118000 98.677000 98.825000 1.489000 76.064000 78.105000
92.000000 0.005593 1.115000 98.888000 98.896000 1.543000 73.699000 75.885000
93.000000 0.005516 1.119000 98.414000 98.615000 1.515000 74.455000 76.683000
94.000000 0.005438 1.113000 98.973000 99.018000 1.553000 73.630000 75.885000
95.000000 0.005361 1.114000 98.873000 98.983000 1.500000 76.706000 78.661000
96.000000 0.005283 1.115000 98.794000 98.896000 1.543000 73.757000 76.093000
97.000000 0.005205 1.110000 98.992000 99.123000 1.500000 75.272000 77.273000
98.000000 0.005128 1.110000 98.873000 99.045000 1.496000 75.038000 77.759000
99.000000 0.005050 1.107000 99.021000 99.097000 1.527000 74.788000 77.377000
100.000000 0.004972 1.107000 99.028000 99.132000 1.472000 76.833000 79.285000
101.000000 0.004895 1.107000 98.965000 98.992000 1.470000 76.287000 79.042000
102.000000 0.004817 1.107000 99.161000 99.132000 1.495000 75.709000 78.244000
103.000000 0.004739 1.105000 99.130000 99.194000 1.506000 74.970000 77.446000
104.000000 0.004662 1.106000 99.059000 99.097000 1.510000 75.711000 77.897000
105.000000 0.004584 1.103000 99.180000 99.272000 1.488000 76.698000 78.522000
106.000000 0.004507 1.101000 99.233000 99.334000 1.506000 74.709000 77.516000
107.000000 0.004430 1.096000 99.526000 99.509000 1.487000 75.842000 78.626000
108.000000 0.004353 1.097000 99.450000 99.518000 1.489000 75.253000 77.654000
109.000000 0.004276 1.098000 99.291000 99.325000 1.512000 74.381000 77.238000
110.000000 0.004199 1.098000 99.463000 99.483000 1.529000 74.158000 76.371000
111.000000 0.004122 1.095000 99.429000 99.500000 1.507000 74.673000 77.238000
112.000000 0.004046 1.096000 99.566000 99.562000 1.509000 74.667000 77.550000
113.000000 0.003970 1.096000 99.568000 99.553000 1.493000 75.004000 77.377000
114.000000 0.003894 1.093000 99.643000 99.649000 1.473000 76.855000 79.598000
115.000000 0.003819 1.094000 99.505000 99.535000 1.493000 75.508000 77.932000
116.000000 0.003744 1.092000 99.649000 99.649000 1.509000 74.660000 76.891000
117.000000 0.003669 1.089000 99.730000 99.719000 1.531000 75.449000 77.273000
118.000000 0.003595 1.091000 99.607000 99.588000 1.517000 75.665000 77.863000
119.000000 0.003520 1.089000 99.759000 99.711000 1.508000 75.258000 77.689000
120.000000 0.003447 1.091000 99.571000 99.579000 1.514000 74.747000 77.169000
121.000000 0.003373 1.091000 99.528000 99.544000 1.489000 77.106000 78.695000
122.000000 0.003300 1.089000 99.624000 99.658000 1.498000 75.563000 77.897000
123.000000 0.003228 1.086000 99.743000 99.737000 1.481000 76.869000 79.112000
124.000000 0.003156 1.086000 99.723000 99.719000 1.498000 76.318000 78.522000
125.000000 0.003084 1.087000 99.720000 99.693000 1.492000 76.403000 78.314000
126.000000 0.003013 1.086000 99.810000 99.798000 1.515000 75.763000 77.481000
127.000000 0.002942 1.084000 99.824000 99.842000 1.503000 76.112000 78.071000
128.000000 0.002872 1.084000 99.791000 99.833000 1.485000 76.439000 78.210000
129.000000 0.002803 1.086000 99.687000 99.719000 1.509000 75.387000 77.585000
130.000000 0.002734 1.083000 99.784000 99.816000 1.512000 75.990000 78.418000
131.000000 0.002665 1.085000 99.747000 99.772000 1.501000 75.363000 77.654000
132.000000 0.002597 1.083000 99.787000 99.798000 1.497000 75.771000 78.383000
133.000000 0.002530 1.085000 99.671000 99.684000 1.506000 75.462000 77.793000
134.000000 0.002464 1.082000 99.799000 99.816000 1.498000 75.976000 77.967000
135.000000 0.002398 1.081000 99.881000 99.886000 1.488000 75.876000 78.140000
136.000000 0.002332 1.083000 99.822000 99.816000 1.494000 76.605000 78.695000
137.000000 0.002268 1.082000 99.822000 99.816000 1.487000 76.564000 78.383000
138.000000 0.002204 1.082000 99.757000 99.816000 1.499000 75.346000 77.932000
139.000000 0.002140 1.081000 99.840000 99.842000 1.493000 76.684000 78.175000
140.000000 0.002078 1.081000 99.783000 99.825000 1.469000 76.715000 79.146000
141.000000 0.002016 1.080000 99.766000 99.807000 1.503000 75.689000 77.863000
142.000000 0.001955 1.080000 99.827000 99.825000 1.502000 74.935000 77.203000
143.000000 0.001895 1.080000 99.849000 99.860000 1.503000 74.575000 77.620000
144.000000 0.001835 1.079000 99.788000 99.790000 1.482000 77.105000 79.251000
145.000000 0.001777 1.079000 99.829000 99.825000 1.478000 76.389000 78.522000
146.000000 0.001719 1.078000 99.945000 99.930000 1.490000 75.526000 77.793000
147.000000 0.001661 1.078000 99.887000 99.904000 1.489000 76.745000 78.765000
148.000000 0.001605 1.081000 99.832000 99.860000 1.486000 76.690000 78.348000
149.000000 0.001550 1.078000 99.877000 99.860000 1.484000 76.878000 78.661000
150.000000 0.001495 1.078000 99.856000 99.895000 1.473000 77.582000 79.285000
151.000000 0.001442 1.078000 99.910000 99.904000 1.487000 75.778000 78.105000
152.000000 0.001389 1.077000 99.911000 99.921000 1.502000 75.622000 77.897000
153.000000 0.001337 1.078000 99.837000 99.860000 1.500000 75.508000 77.724000
154.000000 0.001286 1.078000 99.864000 99.869000 1.502000 75.423000 77.654000
155.000000 0.001236 1.076000 99.913000 99.912000 1.483000 76.901000 78.869000
156.000000 0.001187 1.077000 99.892000 99.860000 1.482000 76.302000 78.695000
157.000000 0.001139 1.077000 99.868000 99.886000 1.476000 77.172000 79.042000
158.000000 0.001092 1.076000 99.946000 99.947000 1.491000 75.826000 78.591000
159.000000 0.001045 1.077000 99.905000 99.912000 1.501000 75.957000 78.036000
160.000000 0.001000 1.077000 99.893000 99.912000 1.491000 76.007000 78.348000
161.000000 0.000956 1.075000 99.942000 99.930000 1.481000 76.566000 78.661000
162.000000 0.000913 1.077000 99.866000 99.886000 1.490000 76.297000 78.452000
163.000000 0.000871 1.075000 99.884000 99.886000 1.493000 76.239000 78.487000
164.000000 0.000829 1.075000 99.923000 99.904000 1.482000 75.875000 78.244000
165.000000 0.000789 1.076000 99.901000 99.904000 1.483000 76.560000 78.591000
166.000000 0.000750 1.076000 99.847000 99.869000 1.490000 75.955000 78.210000
167.000000 0.000712 1.075000 99.901000 99.912000 1.492000 75.930000 78.279000
168.000000 0.000675 1.075000 99.945000 99.947000 1.491000 76.281000 78.348000
169.000000 0.000640 1.076000 99.838000 99.904000 1.493000 75.966000 78.314000
170.000000 0.000605 1.075000 99.895000 99.912000 1.495000 76.501000 78.418000
171.000000 0.000571 1.075000 99.932000 99.912000 1.486000 77.250000 79.042000
172.000000 0.000539 1.074000 99.912000 99.921000 1.502000 75.722000 77.897000
173.000000 0.000507 1.074000 99.954000 99.939000 1.494000 75.573000 78.140000
174.000000 0.000477 1.076000 99.897000 99.904000 1.490000 76.545000 78.591000
175.000000 0.000448 1.075000 99.914000 99.904000 1.478000 76.988000 78.938000
176.000000 0.000420 1.073000 99.956000 99.947000 1.484000 76.267000 78.695000
177.000000 0.000393 1.074000 99.903000 99.930000 1.482000 77.061000 78.765000
178.000000 0.000367 1.075000 99.934000 99.939000 1.483000 76.449000 78.834000
179.000000 0.000342 1.076000 99.889000 99.895000 1.484000 76.635000 78.834000
180.000000 0.000319 1.076000 99.876000 99.869000 1.480000 77.047000 79.146000
181.000000 0.000297 1.074000 99.902000 99.921000 1.480000 77.385000 79.389000
182.000000 0.000275 1.074000 99.915000 99.939000 1.486000 77.313000 79.251000
183.000000 0.000256 1.075000 99.921000 99.930000 1.477000 76.899000 78.765000
184.000000 0.000237 1.074000 99.937000 99.956000 1.503000 76.059000 78.140000
185.000000 0.000219 1.076000 99.885000 99.877000 1.481000 76.512000 78.557000
186.000000 0.000203 1.075000 99.905000 99.904000 1.497000 75.674000 77.759000
187.000000 0.000188 1.074000 99.921000 99.930000 1.481000 75.918000 78.383000
188.000000 0.000174 1.075000 99.909000 99.904000 1.479000 76.130000 78.452000
189.000000 0.000161 1.073000 99.942000 99.956000 1.480000 76.821000 79.008000
190.000000 0.000149 1.074000 99.920000 99.930000 1.494000 75.823000 77.863000
191.000000 0.000139 1.073000 99.937000 99.912000 1.487000 76.407000 78.557000
192.000000 0.000130 1.073000 99.950000 99.947000 1.485000 76.677000 78.591000
193.000000 0.000122 1.073000 99.934000 99.947000 1.480000 76.471000 78.869000
194.000000 0.000115 1.074000 99.951000 99.956000 1.481000 76.876000 79.112000
195.000000 0.000110 1.073000 99.931000 99.939000 1.476000 76.694000 78.973000
196.000000 0.000105 1.074000 99.858000 99.895000 1.488000 75.980000 78.487000
197.000000 0.000102 1.072000 99.942000 99.947000 1.481000 76.897000 78.765000
198.000000 0.000101 1.074000 99.927000 99.947000 1.496000 75.780000 78.210000
199.000000 0.000100 1.073000 99.945000 99.939000 1.501000 76.206000 77.967000
@ja604041062 Thanks for the information. The only difference is bs. Let me submit a job see what the result would be. I will get back to you by Sunday.
Hi @ja604041062
I just tested batch size 16, there is no problem. This link provides the logs of pointMLP/ pointMLP-elite trained with batchsize16 and batchsize32 (all these models are trained today, see the timestamp). Each experiment is repeated twice. google drive logs.
Here is a summary:
| Model | BatchSize | mAcc. | mAcc. in paper | OA | OA in paper |
|---|---|---|---|---|---|
| PointMLP | 32 | [83.514,83.301] | 83.9±0.5 | [85.253,84.663] | 85.4±0.3 |
| PointMLP | 16 | [83.699,84.457] | - | [85.149,85.947] | - |
| PointMLP-elite | 32 | [82.05,82.019] | 81.8±0.8 | [83.969,84.004] | 83.8±0.6 |
| PointMLP-elite | 16 | [81.045,81.885] | - | [82.929,83.657] | - |
Clearly, there is almost no performance gap between bs16 and bs32.
For your convenience, the sbatch scripts are also included in the google drive link.
Could you please double check the environment or try one more time (maybe on other machines)? The results are wired.
Let me know if you have any further questions.
@ja604041062 @Hello-hubKing
Since there is no further discussion, I will close this issue. Feel free to reopen it if necessary.