The Training problem
shi-10 opened this issue · 0 comments
Hello, I would like to ask a question, when I train train_classification_with_puzzle.py, why my mIOU is always 4.23 and can’t improve? I used two 2080Ti for training.
Like this
[i] iteration=66, learning_rate=0.0994, alpha=0.03, loss=1.2468, class_loss=0.6384, p_class_loss=0.6042, re_loss=0.3179, conf_loss=0.0000, time=65sec
[i] iteration=132, learning_rate=0.0988, alpha=0.08, loss=0.5721, class_loss=0.2831, p_class_loss=0.2832, re_loss=0.0733, conf_loss=0.0000, time=54sec
[i] iteration=198, learning_rate=0.0982, alpha=0.13, loss=0.5701, class_loss=0.2822, p_class_loss=0.2793, re_loss=0.0651, conf_loss=0.0000, time=54sec
[i] iteration=264, learning_rate=0.0976, alpha=0.19, loss=0.5488, class_loss=0.2696, p_class_loss=0.2712, re_loss=0.0434, conf_loss=0.0000, time=54sec
[i] iteration=330, learning_rate=0.0970, alpha=0.24, loss=0.5321, class_loss=0.2615, p_class_loss=0.2606, re_loss=0.0416, conf_loss=0.0000, time=54sec
[i] iteration=396, learning_rate=0.0964, alpha=0.29, loss=0.5358, class_loss=0.2632, p_class_loss=0.2591, re_loss=0.0462, conf_loss=0.0000, time=53sec
[i] iteration=462, learning_rate=0.0958, alpha=0.35, loss=0.5387, class_loss=0.2635, p_class_loss=0.2608, re_loss=0.0417, conf_loss=0.0000, time=54sec
[i] iteration=528, learning_rate=0.0952, alpha=0.40, loss=0.5292, class_loss=0.2578, p_class_loss=0.2573, re_loss=0.0351, conf_loss=0.0000, time=54sec
[i] iteration=594, learning_rate=0.0946, alpha=0.45, loss=0.5279, class_loss=0.2573, p_class_loss=0.2545, re_loss=0.0356, conf_loss=0.0000, time=53sec
[i] iteration=660, learning_rate=0.0940, alpha=0.51, loss=0.5173, class_loss=0.2518, p_class_loss=0.2496, re_loss=0.0313, conf_loss=0.0000, time=53sec
[i] save model
[i] iteration=661, threshold=0.10, train_mIoU=4.23%, best_train_mIoU=4.23%, time=28sec
[i] iteration=726, learning_rate=0.0934, alpha=0.56, loss=0.5259, class_loss=0.2554, p_class_loss=0.2537, re_loss=0.0303, conf_loss=0.0000, time=83sec
[i] iteration=792, learning_rate=0.0928, alpha=0.61, loss=0.5114, class_loss=0.2484, p_class_loss=0.2483, re_loss=0.0241, conf_loss=0.0000, time=53sec
[i] iteration=858, learning_rate=0.0922, alpha=0.67, loss=0.5194, class_loss=0.2523, p_class_loss=0.2526, re_loss=0.0219, conf_loss=0.0000, time=53sec
[i] iteration=924, learning_rate=0.0916, alpha=0.72, loss=0.5110, class_loss=0.2479, p_class_loss=0.2472, re_loss=0.0221, conf_loss=0.0000, time=53sec
[i] iteration=990, learning_rate=0.0910, alpha=0.77, loss=0.5185, class_loss=0.2515, p_class_loss=0.2500, re_loss=0.0220, conf_loss=0.0000, time=53sec
[i] iteration=1,056, learning_rate=0.0904, alpha=0.83, loss=0.5102, class_loss=0.2470, p_class_loss=0.2465, re_loss=0.0202, conf_loss=0.0000, time=53sec
[i] iteration=1,122, learning_rate=0.0898, alpha=0.88, loss=0.5295, class_loss=0.2542, p_class_loss=0.2514, re_loss=0.0271, conf_loss=0.0000, time=53sec
[i] iteration=1,188, learning_rate=0.0892, alpha=0.93, loss=0.5289, class_loss=0.2525, p_class_loss=0.2503, re_loss=0.0280, conf_loss=0.0000, time=53sec
[i] iteration=1,254, learning_rate=0.0886, alpha=0.98, loss=0.5302, class_loss=0.2542, p_class_loss=0.2531, re_loss=0.0233, conf_loss=0.0000, time=53sec
[i] iteration=1,320, learning_rate=0.0879, alpha=1.04, loss=0.5196, class_loss=0.2501, p_class_loss=0.2486, re_loss=0.0201, conf_loss=0.0000, time=53sec
[i] iteration=1,322, threshold=0.10, train_mIoU=4.19%, best_train_mIoU=4.23%, time=29sec
[i] iteration=1,386, learning_rate=0.0873, alpha=1.09, loss=0.5252, class_loss=0.2533, p_class_loss=0.2521, re_loss=0.0182, conf_loss=0.0000, time=84sec