The Degeneration of Assign_Matrix.
Opened this issue · 0 comments
CodeLHY commented
I tried to train SGPA on CAMERA25-Composed. However, it seems that it will always drop in a degeneration, where the entropy_loss will drop dramatically to zero. Here is the log:
2022-04-11 07:20:52,242 : Batch 5630 Loss:0.022436, corr_loss:0.015928, cd_loss:0.005587, entropy_loss:0.000209, deform_loss:0.000313, lowrank_loss:0.000399
2022-04-11 07:20:58,618 : Batch 5640 Loss:0.061784, corr_loss:0.045377, cd_loss:0.015464, entropy_loss:0.000174, deform_loss:0.000297, lowrank_loss:0.000472
2022-04-11 07:21:04,251 : Batch 5650 Loss:0.107389, corr_loss:0.086328, cd_loss:0.019944, entropy_loss:0.000153, deform_loss:0.000268, lowrank_loss:0.000695
2022-04-11 07:21:10,448 : Batch 5660 Loss:0.133275, corr_loss:0.105933, cd_loss:0.025799, entropy_loss:0.000137, deform_loss:0.000216, lowrank_loss:0.001190
2022-04-11 07:21:17,287 : Batch 5670 Loss:0.125959, corr_loss:0.103875, cd_loss:0.021085, entropy_loss:0.000160, deform_loss:0.000186, lowrank_loss:0.000652
2022-04-11 07:21:22,624 : Batch 5680 Loss:0.144936, corr_loss:0.126369, cd_loss:0.014371, entropy_loss:0.000087, deform_loss:0.000158, lowrank_loss:0.003951
2022-04-11 07:21:27,534 : Batch 5690 Loss:0.149627, corr_loss:0.127070, cd_loss:0.020821, entropy_loss:0.000024, deform_loss:0.000137, lowrank_loss:0.001574
2022-04-11 07:21:32,971 : Batch 5700 Loss:0.136263, corr_loss:0.121296, cd_loss:0.014015, entropy_loss:0.000021, deform_loss:0.000140, lowrank_loss:0.000791
2022-04-11 07:21:39,646 : Batch 5710 Loss:0.159056, corr_loss:0.132856, cd_loss:0.024632, entropy_loss:0.000019, deform_loss:0.000150, lowrank_loss:0.001400
2022-04-11 07:21:45,321 : Batch 5720 Loss:0.154485, corr_loss:0.128831, cd_loss:0.023682, entropy_loss:0.000012, deform_loss:0.000141, lowrank_loss:0.001818
2022-04-11 07:21:50,838 : Batch 5730 Loss:0.189477, corr_loss:0.158418, cd_loss:0.027742, entropy_loss:0.000011, deform_loss:0.000149, lowrank_loss:0.003158
2022-04-11 07:21:57,104 : Batch 5740 Loss:0.123445, corr_loss:0.107106, cd_loss:0.013482, entropy_loss:0.000011, deform_loss:0.000129, lowrank_loss:0.002717
2022-04-11 07:22:03,384 : Batch 5750 Loss:0.133284, corr_loss:0.111759, cd_loss:0.019112, entropy_loss:0.000010, deform_loss:0.000132, lowrank_loss:0.002271
2022-04-11 07:22:08,641 : Batch 5760 Loss:0.123830, corr_loss:0.107223, cd_loss:0.013527, entropy_loss:0.000012, deform_loss:0.000139, lowrank_loss:0.002929
2022-04-11 07:22:14,902 : Batch 5770 Loss:0.282613, corr_loss:0.088271, cd_loss:0.015968, entropy_loss:0.000015, deform_loss:0.000141, lowrank_loss:0.178218
2022-04-11 07:22:19,834 : Batch 5780 Loss:0.140194, corr_loss:0.119629, cd_loss:0.018370, entropy_loss:0.000006, deform_loss:0.000141, lowrank_loss:0.002047
2022-04-11 07:22:26,458 : Batch 5790 Loss:0.147937, corr_loss:0.121299, cd_loss:0.024671, entropy_loss:0.000005, deform_loss:0.000153, lowrank_loss:0.001809
2022-04-11 07:22:31,845 : Batch 5800 Loss:0.124483, corr_loss:0.105013, cd_loss:0.017752, entropy_loss:0.000004, deform_loss:0.000135, lowrank_loss:0.001578
2022-04-11 07:22:38,269 : Batch 5810 Loss:0.136623, corr_loss:0.119391, cd_loss:0.015243, entropy_loss:0.000004, deform_loss:0.000133, lowrank_loss:0.001852
2022-04-11 07:22:43,064 : Batch 5820 Loss:0.159818, corr_loss:0.133095, cd_loss:0.024861, entropy_loss:0.000004, deform_loss:0.000162, lowrank_loss:0.001696
2022-04-11 07:22:49,981 : Batch 5830 Loss:0.159613, corr_loss:0.131518, cd_loss:0.026577, entropy_loss:0.000004, deform_loss:0.000117, lowrank_loss:0.001396
2022-04-11 07:22:55,398 : Batch 5840 Loss:0.127762, corr_loss:0.107937, cd_loss:0.018362, entropy_loss:0.000004, deform_loss:0.000152, lowrank_loss:0.001308
2022-04-11 07:23:00,962 : Batch 5850 Loss:0.127322, corr_loss:0.111806, cd_loss:0.014305, entropy_loss:0.000005, deform_loss:0.000112, lowrank_loss:0.001096
2022-04-11 07:23:08,073 : Batch 5860 Loss:0.129297, corr_loss:0.108440, cd_loss:0.019306, entropy_loss:0.000005, deform_loss:0.000124, lowrank_loss:0.001422
2022-04-11 07:23:15,776 : Batch 5870 Loss:0.167962, corr_loss:0.145520, cd_loss:0.021149, entropy_loss:0.000006, deform_loss:0.000134, lowrank_loss:0.001153
2022-04-11 07:23:21,810 : Batch 5880 Loss:0.197241, corr_loss:0.174802, cd_loss:0.021063, entropy_loss:0.000003, deform_loss:0.000139, lowrank_loss:0.001234
2022-04-11 07:23:27,260 : Batch 5890 Loss:0.216616, corr_loss:0.177835, cd_loss:0.036533, entropy_loss:0.000003, deform_loss:0.000244, lowrank_loss:0.002000
2022-04-11 07:23:32,242 : Batch 5900 Loss:0.163611, corr_loss:0.136137, cd_loss:0.026894, entropy_loss:0.000003, deform_loss:0.000138, lowrank_loss:0.000438
2022-04-11 07:23:39,344 : Batch 5910 Loss:0.268385, corr_loss:0.159586, cd_loss:0.011610, entropy_loss:0.000003, deform_loss:0.000119, lowrank_loss:0.097068
2022-04-11 07:23:44,910 : Batch 5920 Loss:0.350295, corr_loss:0.149164, cd_loss:0.016909, entropy_loss:0.000003, deform_loss:0.000231, lowrank_loss:0.183987
2022-04-11 07:23:50,413 : Batch 5930 Loss:0.920049, corr_loss:0.178489, cd_loss:0.022725, entropy_loss:0.000004, deform_loss:0.000168, lowrank_loss:0.718663
2022-04-11 07:23:56,567 : Batch 5940 Loss:0.179104, corr_loss:0.153191, cd_loss:0.023358, entropy_loss:0.000005, deform_loss:0.000153, lowrank_loss:0.002396
2022-04-11 07:24:02,931 : Batch 5950 Loss:0.152043, corr_loss:0.133964, cd_loss:0.015110, entropy_loss:0.000004, deform_loss:0.000144, lowrank_loss:0.002821
2022-04-11 07:24:10,418 : Batch 5960 Loss:0.175748, corr_loss:0.152540, cd_loss:0.020229, entropy_loss:0.000002, deform_loss:0.000125, lowrank_loss:0.002852
2022-04-11 07:24:15,222 : Batch 5970 Loss:0.557693, corr_loss:0.180952, cd_loss:0.024084, entropy_loss:0.000002, deform_loss:0.000139, lowrank_loss:0.352516
2022-04-11 07:24:20,400 : Batch 5980 Loss:0.164646, corr_loss:0.142908, cd_loss:0.019423, entropy_loss:0.000003, deform_loss:0.000136, lowrank_loss:0.002176
2022-04-11 07:24:27,004 : Batch 5990 Loss:0.167778, corr_loss:0.150446, cd_loss:0.014688, entropy_loss:0.000002, deform_loss:0.000142, lowrank_loss:0.002499
2022-04-11 07:24:32,110 : Batch 6000 Loss:0.151957, corr_loss:0.132350, cd_loss:0.017354, entropy_loss:0.000003, deform_loss:0.000154, lowrank_loss:0.002096
2022-04-11 07:24:38,853 : Batch 6010 Loss:0.161332, corr_loss:0.143597, cd_loss:0.015208, entropy_loss:0.000003, deform_loss:0.000141, lowrank_loss:0.002383
2022-04-11 07:24:44,674 : Batch 6020 Loss:0.146042, corr_loss:0.124359, cd_loss:0.019413, entropy_loss:0.000004, deform_loss:0.000153, lowrank_loss:0.002113
2022-04-11 07:24:50,090 : Batch 6030 Loss:0.167414, corr_loss:0.148456, cd_loss:0.016587, entropy_loss:0.000004, deform_loss:0.000136, lowrank_loss:0.002231
2022-04-11 07:24:56,130 : Batch 6040 Loss:0.245671, corr_loss:0.131757, cd_loss:0.022333, entropy_loss:0.000004, deform_loss:0.000146, lowrank_loss:0.091431
2022-04-11 07:25:02,632 : Batch 6050 Loss:0.158918, corr_loss:0.137141, cd_loss:0.019525, entropy_loss:0.000002, deform_loss:0.000154, lowrank_loss:0.002096
2022-04-11 07:25:08,729 : Batch 6060 Loss:0.144677, corr_loss:0.128974, cd_loss:0.012056, entropy_loss:0.000001, deform_loss:0.000139, lowrank_loss:0.003506
2022-04-11 07:25:14,971 : Batch 6070 Loss:0.163134, corr_loss:0.141428, cd_loss:0.018614, entropy_loss:0.000001, deform_loss:0.000136, lowrank_loss:0.002955
2022-04-11 07:25:20,348 : Batch 6080 Loss:0.178839, corr_loss:0.161275, cd_loss:0.014825, entropy_loss:0.000001, deform_loss:0.000150, lowrank_loss:0.002589
2022-04-11 07:25:28,763 : Batch 6090 Loss:0.177177, corr_loss:0.154025, cd_loss:0.020273, entropy_loss:0.000001, deform_loss:0.000151, lowrank_loss:0.002728
2022-04-11 07:25:34,005 : Batch 6100 Loss:0.205118, corr_loss:0.180742, cd_loss:0.021688, entropy_loss:0.000001, deform_loss:0.000155, lowrank_loss:0.002531
2022-04-11 07:25:39,137 : Batch 6110 Loss:0.140715, corr_loss:0.120587, cd_loss:0.017866, entropy_loss:0.000001, deform_loss:0.000132, lowrank_loss:0.002129
2022-04-11 07:25:44,117 : Batch 6120 Loss:0.121930, corr_loss:0.104700, cd_loss:0.014153, entropy_loss:0.000001, deform_loss:0.000138, lowrank_loss:0.002938
2022-04-11 07:25:51,788 : Batch 6130 Loss:0.159974, corr_loss:0.130274, cd_loss:0.022835, entropy_loss:0.000001, deform_loss:0.000146, lowrank_loss:0.006718
2022-04-11 07:25:57,249 : Batch 6140 Loss:0.192346, corr_loss:0.166914, cd_loss:0.022647, entropy_loss:0.000001, deform_loss:0.000156, lowrank_loss:0.002628
2022-04-11 07:26:02,346 : Batch 6150 Loss:0.189960, corr_loss:0.163396, cd_loss:0.024124, entropy_loss:0.000001, deform_loss:0.000140, lowrank_loss:0.002298
2022-04-11 07:26:07,871 : Batch 6160 Loss:0.145737, corr_loss:0.124859, cd_loss:0.017940, entropy_loss:0.000002, deform_loss:0.000154, lowrank_loss:0.002782
2022-04-11 07:26:15,325 : Batch 6170 Loss:0.130483, corr_loss:0.114265, cd_loss:0.012683, entropy_loss:0.000001, deform_loss:0.000130, lowrank_loss:0.003403
2022-04-11 07:26:21,753 : Batch 6180 Loss:0.147745, corr_loss:0.130941, cd_loss:0.014452, entropy_loss:0.000001, deform_loss:0.000145, lowrank_loss:0.002206
2022-04-11 07:26:27,523 : Batch 6190 Loss:0.179217, corr_loss:0.148284, cd_loss:0.027994, entropy_loss:0.000001, deform_loss:0.000142, lowrank_loss:0.002797
2022-04-11 07:26:32,387 : Batch 6200 Loss:0.177200, corr_loss:0.152850, cd_loss:0.022009, entropy_loss:0.000001, deform_loss:0.000142, lowrank_loss:0.002198
2022-04-11 07:26:40,009 : Batch 6210 Loss:0.172815, corr_loss:0.145709, cd_loss:0.024553, entropy_loss:0.000002, deform_loss:0.000142, lowrank_loss:0.002410
2022-04-11 07:26:44,898 : Batch 6220 Loss:0.169863, corr_loss:0.131859, cd_loss:0.017734, entropy_loss:0.000002, deform_loss:0.000137, lowrank_loss:0.020130
2022-04-11 07:26:51,031 : Batch 6230 Loss:0.192499, corr_loss:0.164317, cd_loss:0.024804, entropy_loss:0.000002, deform_loss:0.000145, lowrank_loss:0.003232
2022-04-11 07:26:56,936 : Batch 6240 Loss:0.145409, corr_loss:0.120767, cd_loss:0.021301, entropy_loss:0.000002, deform_loss:0.000129, lowrank_loss:0.003209
2022-04-11 07:27:04,429 : Batch 6250 Loss:0.161274, corr_loss:0.134427, cd_loss:0.023983, entropy_loss:0.000002, deform_loss:0.000143, lowrank_loss:0.002719
2022-04-11 07:27:09,629 : Batch 6260 Loss:0.197566, corr_loss:0.171911, cd_loss:0.022160, entropy_loss:0.000001, deform_loss:0.000148, lowrank_loss:0.003346
2022-04-11 07:27:14,962 : Batch 6270 Loss:0.449405, corr_loss:0.187930, cd_loss:0.023193, entropy_loss:0.000001, deform_loss:0.000150, lowrank_loss:0.238131
2022-04-11 07:27:21,387 : Batch 6280 Loss:0.286750, corr_loss:0.180243, cd_loss:0.018598, entropy_loss:0.000002, deform_loss:0.000141, lowrank_loss:0.087766
2022-04-11 07:27:27,619 : Batch 6290 Loss:0.224757, corr_loss:0.202550, cd_loss:0.018794, entropy_loss:0.000001, deform_loss:0.000152, lowrank_loss:0.003260
2022-04-11 07:27:33,079 : Batch 6300 Loss:0.144296, corr_loss:0.127126, cd_loss:0.013946, entropy_loss:0.000001, deform_loss:0.000147, lowrank_loss:0.003075
2022-04-11 07:27:39,827 : Batch 6310 Loss:0.174579, corr_loss:0.153421, cd_loss:0.017926, entropy_loss:0.000002, deform_loss:0.000130, lowrank_loss:0.003100
2022-04-11 07:27:45,572 : Batch 6320 Loss:0.240689, corr_loss:0.215314, cd_loss:0.020162, entropy_loss:0.000006, deform_loss:0.000137, lowrank_loss:0.005070
2022-04-11 07:27:51,225 : Batch 6330 Loss:0.206055, corr_loss:0.188849, cd_loss:0.013425, entropy_loss:0.000002, deform_loss:0.000144, lowrank_loss:0.003634
2022-04-11 07:27:56,587 : Batch 6340 Loss:0.191800, corr_loss:0.171729, cd_loss:0.016815, entropy_loss:0.000001, deform_loss:0.000131, lowrank_loss:0.003124
2022-04-11 07:28:04,025 : Batch 6350 Loss:0.241368, corr_loss:0.211277, cd_loss:0.026727, entropy_loss:0.000001, deform_loss:0.000145, lowrank_loss:0.003218
2022-04-11 07:28:09,405 : Batch 6360 Loss:0.217268, corr_loss:0.195405, cd_loss:0.017690, entropy_loss:0.000001, deform_loss:0.000133, lowrank_loss:0.004039
2022-04-11 07:28:14,819 : Batch 6370 Loss:0.182568, corr_loss:0.158691, cd_loss:0.020242, entropy_loss:0.000001, deform_loss:0.000124, lowrank_loss:0.003511
2022-04-11 07:28:20,880 : Batch 6380 Loss:0.185456, corr_loss:0.168582, cd_loss:0.012798, entropy_loss:0.000000, deform_loss:0.000130, lowrank_loss:0.003947\