Ground Truth Not Found
howardchina opened this issue · 4 comments
howardchina commented
My command for training
./tools/dist_train.sh configs/queryinst/queryinst_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco.py 4
The training was stucked when model met an empty croped gt.
My log is shown as below:
2021-08-26 10:39:36,940 - mmdet - INFO - workflow: [('train', 1)], max: 36 epochs
2021-08-26 10:41:30,531 - mmdet - INFO - Epoch [1][50/14659] lr: 1.249e-06, eta: 13 days, 20:51:37, time: 2.271, data_time: 0.978, memory: 9683, stage0_loss_cls: 2.2784, stage0_pos_acc: 2.0437, stage0_loss_bbox: 2.9789, stage0_loss_iou: 1.6788, stage0_loss_mask: 5.9337, stage1_loss_cls: 2.3385, stage1_pos_acc: 2.0824, stage1_loss_bbox: 4.2721, stage1_loss_iou: 1.8811, stage1_loss_mask: 6.2975, stage2_loss_cls: 2.3301, stage2_pos_acc: 0.9363, stage2_loss_bbox: 2.9707, stage2_loss_iou: 1.9285, stage2_loss_mask: 5.8670, stage3_loss_cls: 2.1712, stage3_pos_acc: 3.0306, stage3_loss_bbox: 3.1920, stage3_loss_iou: 2.2616, stage3_loss_mask: 6.0022, stage4_loss_cls: 2.2627, stage4_pos_acc: 1.7393, stage4_loss_bbox: 2.8403, stage4_loss_iou: 2.3233, stage4_loss_mask: 5.8660, stage5_loss_cls: 2.2882, stage5_pos_acc: 1.5126, stage5_loss_bbox: 2.8825, stage5_loss_iou: 2.3851, stage5_loss_mask: 5.8457, loss: 81.0761, grad_norm: 9063.7358
2021-08-26 10:42:34,279 - mmdet - INFO - Epoch [1][100/14659] lr: 2.498e-06, eta: 10 days, 19:53:52, time: 1.276, data_time: 0.029, memory: 9683, stage0_loss_cls: 2.2928, stage0_pos_acc: 2.2991, stage0_loss_bbox: 2.4200, stage0_loss_iou: 1.5812, stage0_loss_mask: 5.5926, stage1_loss_cls: 2.3128, stage1_pos_acc: 1.5530, stage1_loss_bbox: 2.8068, stage1_loss_iou: 1.7299, stage1_loss_mask: 5.6873, stage2_loss_cls: 2.2738, stage2_pos_acc: 1.3359, stage2_loss_bbox: 2.0444, stage2_loss_iou: 1.7757, stage2_loss_mask: 5.1849, stage3_loss_cls: 2.0966, stage3_pos_acc: 3.8337, stage3_loss_bbox: 1.9937, stage3_loss_iou: 1.9307, stage3_loss_mask: 5.0764, stage4_loss_cls: 2.2168, stage4_pos_acc: 2.2209, stage4_loss_bbox: 1.8457, stage4_loss_iou: 1.9443, stage4_loss_mask: 4.9774, stage5_loss_cls: 2.1422, stage5_pos_acc: 2.2032, stage5_loss_bbox: 1.8217, stage5_loss_iou: 1.9223, stage5_loss_mask: 4.9423, loss: 68.6125, grad_norm: 5779.0945
2021-08-26 10:43:39,684 - mmdet - INFO - Epoch [1][150/14659] lr: 3.746e-06, eta: 9 days, 21:06:05, time: 1.307, data_time: 0.030, memory: 9683, stage0_loss_cls: 2.1774, stage0_pos_acc: 2.4055, stage0_loss_bbox: 1.7417, stage0_loss_iou: 1.5327, stage0_loss_mask: 5.2476, stage1_loss_cls: 2.1912, stage1_pos_acc: 1.9714, stage1_loss_bbox: 1.5765, stage1_loss_iou: 1.6106, stage1_loss_mask: 4.8875, stage2_loss_cls: 2.1093, stage2_pos_acc: 3.1017, stage2_loss_bbox: 1.4834, stage2_loss_iou: 1.7319, stage2_loss_mask: 4.6163, stage3_loss_cls: 1.9555, stage3_pos_acc: 9.2308, stage3_loss_bbox: 1.4211, stage3_loss_iou: 1.7616, stage3_loss_mask: 4.4452, stage4_loss_cls: 2.0144, stage4_pos_acc: 13.1486, stage4_loss_bbox: 1.3939, stage4_loss_iou: 1.7237, stage4_loss_mask: 4.5496, stage5_loss_cls: 1.9050, stage5_pos_acc: 18.0571, stage5_loss_bbox: 1.4082, stage5_loss_iou: 1.6820, stage5_loss_mask: 4.6783, loss: 59.8446, grad_norm: 2041.8116
2021-08-26 10:44:43,974 - mmdet - INFO - Epoch [1][200/14659] lr: 4.995e-06, eta: 9 days, 8:55:17, time: 1.286, data_time: 0.027, memory: 9683, stage0_loss_cls: 2.1020, stage0_pos_acc: 2.5095, stage0_loss_bbox: 1.4044, stage0_loss_iou: 1.4558, stage0_loss_mask: 4.7985, stage1_loss_cls: 2.0902, stage1_pos_acc: 3.7327, stage1_loss_bbox: 1.2853, stage1_loss_iou: 1.5602, stage1_loss_mask: 4.4117, stage2_loss_cls: 1.9464, stage2_pos_acc: 14.1383, stage2_loss_bbox: 1.2960, stage2_loss_iou: 1.5834, stage2_loss_mask: 4.4650, stage3_loss_cls: 1.9040, stage3_pos_acc: 18.7631, stage3_loss_bbox: 1.3171, stage3_loss_iou: 1.5608, stage3_loss_mask: 4.4906, stage4_loss_cls: 1.9529, stage4_pos_acc: 19.2329, stage4_loss_bbox: 1.4226, stage4_loss_iou: 1.6032, stage4_loss_mask: 4.6667, stage5_loss_cls: 1.8324, stage5_pos_acc: 21.7396, stage5_loss_bbox: 1.4392, stage5_loss_iou: 1.6493, stage5_loss_mask: 4.6460, loss: 56.8839, grad_norm: 1372.3742
2021-08-26 10:45:50,304 - mmdet - INFO - Epoch [1][250/14659] lr: 6.244e-06, eta: 9 days, 2:47:51, time: 1.327, data_time: 0.028, memory: 9694, stage0_loss_cls: 2.0222, stage0_pos_acc: 5.2418, stage0_loss_bbox: 1.2542, stage0_loss_iou: 1.4832, stage0_loss_mask: 4.7497, stage1_loss_cls: 1.9436, stage1_pos_acc: 10.3808, stage1_loss_bbox: 1.1948, stage1_loss_iou: 1.5693, stage1_loss_mask: 4.5709, stage2_loss_cls: 1.9026, stage2_pos_acc: 18.4786, stage2_loss_bbox: 1.2399, stage2_loss_iou: 1.5680, stage2_loss_mask: 4.6608, stage3_loss_cls: 1.8653, stage3_pos_acc: 21.0621, stage3_loss_bbox: 1.2361, stage3_loss_iou: 1.5910, stage3_loss_mask: 4.5865, stage4_loss_cls: 1.8833, stage4_pos_acc: 22.2822, stage4_loss_bbox: 1.2228, stage4_loss_iou: 1.5849, stage4_loss_mask: 4.6193, stage5_loss_cls: 1.8216, stage5_pos_acc: 20.0130, stage5_loss_bbox: 1.2325, stage5_loss_iou: 1.5996, stage5_loss_mask: 4.6086, loss: 56.0107, grad_norm: 612.7333
2021-08-26 10:46:54,223 - mmdet - INFO - Epoch [1][300/14659] lr: 7.493e-06, eta: 8 days, 21:30:13, time: 1.277, data_time: 0.028, memory: 9704, stage0_loss_cls: 1.9121, stage0_pos_acc: 13.8155, stage0_loss_bbox: 1.2645, stage0_loss_iou: 1.4868, stage0_loss_mask: 4.5622, stage1_loss_cls: 1.8526, stage1_pos_acc: 22.7398, stage1_loss_bbox: 1.2319, stage1_loss_iou: 1.5474, stage1_loss_mask: 4.4435, stage2_loss_cls: 1.8383, stage2_pos_acc: 23.9177, stage2_loss_bbox: 1.2318, stage2_loss_iou: 1.5391, stage2_loss_mask: 4.4818, stage3_loss_cls: 1.7926, stage3_pos_acc: 25.6336, stage3_loss_bbox: 1.2685, stage3_loss_iou: 1.5620, stage3_loss_mask: 4.5198, stage4_loss_cls: 1.8395, stage4_pos_acc: 25.4535, stage4_loss_bbox: 1.2195, stage4_loss_iou: 1.5786, stage4_loss_mask: 4.4744, stage5_loss_cls: 1.7601, stage5_pos_acc: 26.6780, stage5_loss_bbox: 1.2468, stage5_loss_iou: 1.5819, stage5_loss_mask: 4.4257, loss: 54.6615, grad_norm: 539.8647
2021-08-26 10:47:59,050 - mmdet - INFO - Epoch [1][350/14659] lr: 8.741e-06, eta: 8 days, 18:07:15, time: 1.297, data_time: 0.027, memory: 10247, stage0_loss_cls: 1.8577, stage0_pos_acc: 21.1812, stage0_loss_bbox: 1.1807, stage0_loss_iou: 1.4646, stage0_loss_mask: 4.5113, stage1_loss_cls: 1.8366, stage1_pos_acc: 22.9034, stage1_loss_bbox: 1.1725, stage1_loss_iou: 1.5198, stage1_loss_mask: 4.4354, stage2_loss_cls: 1.8283, stage2_pos_acc: 24.0835, stage2_loss_bbox: 1.1699, stage2_loss_iou: 1.5183, stage2_loss_mask: 4.4518, stage3_loss_cls: 1.7548, stage3_pos_acc: 25.1505, stage3_loss_bbox: 1.1876, stage3_loss_iou: 1.5402, stage3_loss_mask: 4.3562, stage4_loss_cls: 1.7753, stage4_pos_acc: 25.4366, stage4_loss_bbox: 1.1960, stage4_loss_iou: 1.5559, stage4_loss_mask: 4.4155, stage5_loss_cls: 1.7162, stage5_pos_acc: 25.3584, stage5_loss_bbox: 1.1832, stage5_loss_iou: 1.5339, stage5_loss_mask: 4.4058, loss: 53.5674, grad_norm: 324.1903
2021-08-26 10:49:02,791 - mmdet - INFO - Epoch [1][400/14659] lr: 9.990e-06, eta: 8 days, 15:11:47, time: 1.276, data_time: 0.029, memory: 10247, stage0_loss_cls: 1.8611, stage0_pos_acc: 25.3944, stage0_loss_bbox: 1.1805, stage0_loss_iou: 1.4688, stage0_loss_mask: 4.4686, stage1_loss_cls: 1.8132, stage1_pos_acc: 24.9371, stage1_loss_bbox: 1.1515, stage1_loss_iou: 1.4919, stage1_loss_mask: 4.3721, stage2_loss_cls: 1.7563, stage2_pos_acc: 27.1302, stage2_loss_bbox: 1.1182, stage2_loss_iou: 1.4570, stage2_loss_mask: 4.3585, stage3_loss_cls: 1.7043, stage3_pos_acc: 28.2956, stage3_loss_bbox: 1.1452, stage3_loss_iou: 1.4661, stage3_loss_mask: 4.3051, stage4_loss_cls: 1.7711, stage4_pos_acc: 28.3810, stage4_loss_bbox: 1.1635, stage4_loss_iou: 1.5052, stage4_loss_mask: 4.2992, stage5_loss_cls: 1.6963, stage5_pos_acc: 27.6587, stage5_loss_bbox: 1.2358, stage5_loss_iou: 1.5787, stage5_loss_mask: 4.3334, loss: 52.7017, grad_norm: 268.2526
2021-08-26 10:50:07,148 - mmdet - INFO - Epoch [1][450/14659] lr: 1.124e-05, eta: 8 days, 13:06:45, time: 1.288, data_time: 0.028, memory: 10247, stage0_loss_cls: 1.8659, stage0_pos_acc: 21.0187, stage0_loss_bbox: 1.1456, stage0_loss_iou: 1.4673, stage0_loss_mask: 4.4616, stage1_loss_cls: 1.8124, stage1_pos_acc: 20.8431, stage1_loss_bbox: 1.1410, stage1_loss_iou: 1.5024, stage1_loss_mask: 4.3939, stage2_loss_cls: 1.7415, stage2_pos_acc: 22.2469, stage2_loss_bbox: 1.0791, stage2_loss_iou: 1.4552, stage2_loss_mask: 4.3218, stage3_loss_cls: 1.6844, stage3_pos_acc: 23.0229, stage3_loss_bbox: 1.0540, stage3_loss_iou: 1.4710, stage3_loss_mask: 4.2206, stage4_loss_cls: 1.7414, stage4_pos_acc: 23.0507, stage4_loss_bbox: 1.0820, stage4_loss_iou: 1.4799, stage4_loss_mask: 4.2415, stage5_loss_cls: 1.6787, stage5_pos_acc: 22.9782, stage5_loss_bbox: 1.1479, stage5_loss_iou: 1.5137, stage5_loss_mask: 4.2180, loss: 51.9207, grad_norm: 213.7757
2021-08-26 10:51:12,068 - mmdet - INFO - Epoch [1][500/14659] lr: 1.249e-05, eta: 8 days, 11:35:48, time: 1.298, data_time: 0.028, memory: 10247, stage0_loss_cls: 1.8008, stage0_pos_acc: 26.0289, stage0_loss_bbox: 1.1152, stage0_loss_iou: 1.4964, stage0_loss_mask: 4.5369, stage1_loss_cls: 1.7177, stage1_pos_acc: 26.9426, stage1_loss_bbox: 1.0998, stage1_loss_iou: 1.4901, stage1_loss_mask: 4.4598, stage2_loss_cls: 1.6178, stage2_pos_acc: 28.2533, stage2_loss_bbox: 1.0045, stage2_loss_iou: 1.4458, stage2_loss_mask: 4.2719, stage3_loss_cls: 1.5983, stage3_pos_acc: 28.3737, stage3_loss_bbox: 0.9817, stage3_loss_iou: 1.4316, stage3_loss_mask: 4.1732, stage4_loss_cls: 1.6095, stage4_pos_acc: 28.0076, stage4_loss_bbox: 1.0100, stage4_loss_iou: 1.4404, stage4_loss_mask: 4.1736, stage5_loss_cls: 1.5825, stage5_pos_acc: 27.7242, stage5_loss_bbox: 1.0086, stage5_loss_iou: 1.4421, stage5_loss_mask: 4.1600, loss: 50.6682, grad_norm: 161.7756
2021-08-26 10:52:17,094 - mmdet - INFO - Epoch [1][550/14659] lr: 1.374e-05, eta: 8 days, 10:22:02, time: 1.299, data_time: 0.031, memory: 10247, stage0_loss_cls: 1.7570, stage0_pos_acc: 26.8588, stage0_loss_bbox: 1.1083, stage0_loss_iou: 1.4725, stage0_loss_mask: 4.3113, stage1_loss_cls: 1.6231, stage1_pos_acc: 28.9210, stage1_loss_bbox: 1.0594, stage1_loss_iou: 1.4476, stage1_loss_mask: 4.2218, stage2_loss_cls: 1.5860, stage2_pos_acc: 28.7443, stage2_loss_bbox: 0.9882, stage2_loss_iou: 1.3822, stage2_loss_mask: 4.0420, stage3_loss_cls: 1.5382, stage3_pos_acc: 29.1749, stage3_loss_bbox: 0.9445, stage3_loss_iou: 1.3533, stage3_loss_mask: 3.9423, stage4_loss_cls: 1.5706, stage4_pos_acc: 29.3126, stage4_loss_bbox: 0.9957, stage4_loss_iou: 1.3977, stage4_loss_mask: 3.9466, stage5_loss_cls: 1.5491, stage5_pos_acc: 30.3116, stage5_loss_bbox: 1.0134, stage5_loss_iou: 1.4122, stage5_loss_mask: 3.9807, loss: 48.6438, grad_norm: 165.5006
2021-08-26 10:53:22,121 - mmdet - INFO - Epoch [1][600/14659] lr: 1.499e-05, eta: 8 days, 9:22:05, time: 1.302, data_time: 0.029, memory: 10247, stage0_loss_cls: 1.8014, stage0_pos_acc: 24.0576, stage0_loss_bbox: 1.0937, stage0_loss_iou: 1.4583, stage0_loss_mask: 4.4055, stage1_loss_cls: 1.6502, stage1_pos_acc: 25.7283, stage1_loss_bbox: 1.0336, stage1_loss_iou: 1.4023, stage1_loss_mask: 4.2823, stage2_loss_cls: 1.6193, stage2_pos_acc: 25.9719, stage2_loss_bbox: 0.9329, stage2_loss_iou: 1.3439, stage2_loss_mask: 4.0294, stage3_loss_cls: 1.5818, stage3_pos_acc: 26.1097, stage3_loss_bbox: 0.9050, stage3_loss_iou: 1.3209, stage3_loss_mask: 3.9482, stage4_loss_cls: 1.6150, stage4_pos_acc: 25.8387, stage4_loss_bbox: 0.9309, stage4_loss_iou: 1.3462, stage4_loss_mask: 3.9348, stage5_loss_cls: 1.5781, stage5_pos_acc: 26.1663, stage5_loss_bbox: 0.9603, stage5_loss_iou: 1.3778, stage5_loss_mask: 3.9012, loss: 48.4532, grad_norm: 135.0550
2021-08-26 10:54:27,236 - mmdet - INFO - Epoch [1][650/14659] lr: 1.623e-05, eta: 8 days, 8:30:35, time: 1.301, data_time: 0.029, memory: 10247, stage0_loss_cls: 1.7764, stage0_pos_acc: 24.2415, stage0_loss_bbox: 1.0727, stage0_loss_iou: 1.4545, stage0_loss_mask: 4.3936, stage1_loss_cls: 1.6191, stage1_pos_acc: 26.6580, stage1_loss_bbox: 0.9463, stage1_loss_iou: 1.4043, stage1_loss_mask: 4.0678, stage2_loss_cls: 1.5642, stage2_pos_acc: 26.5745, stage2_loss_bbox: 0.8532, stage2_loss_iou: 1.3563, stage2_loss_mask: 3.7875, stage3_loss_cls: 1.5627, stage3_pos_acc: 25.8296, stage3_loss_bbox: 0.8517, stage3_loss_iou: 1.3410, stage3_loss_mask: 3.7530, stage4_loss_cls: 1.5756, stage4_pos_acc: 27.2208, stage4_loss_bbox: 0.8791, stage4_loss_iou: 1.3547, stage4_loss_mask: 3.7740, stage5_loss_cls: 1.5621, stage5_pos_acc: 27.6374, stage5_loss_bbox: 0.8854, stage5_loss_iou: 1.3647, stage5_loss_mask: 3.7638, loss: 46.9637, grad_norm: 121.3612
2021-08-26 10:55:31,504 - mmdet - INFO - Epoch [1][700/14659] lr: 1.748e-05, eta: 8 days, 7:37:41, time: 1.287, data_time: 0.026, memory: 10247, stage0_loss_cls: 1.7435, stage0_pos_acc: 26.5202, stage0_loss_bbox: 1.0671, stage0_loss_iou: 1.4283, stage0_loss_mask: 4.2096, stage1_loss_cls: 1.6122, stage1_pos_acc: 26.1169, stage1_loss_bbox: 0.9308, stage1_loss_iou: 1.3396, stage1_loss_mask: 3.8757, stage2_loss_cls: 1.5824, stage2_pos_acc: 28.1921, stage2_loss_bbox: 0.8364, stage2_loss_iou: 1.2766, stage2_loss_mask: 3.7162, stage3_loss_cls: 1.5652, stage3_pos_acc: 27.4108, stage3_loss_bbox: 0.8119, stage3_loss_iou: 1.2662, stage3_loss_mask: 3.6062, stage4_loss_cls: 1.5751, stage4_pos_acc: 28.9362, stage4_loss_bbox: 0.8734, stage4_loss_iou: 1.3164, stage4_loss_mask: 3.6785, stage5_loss_cls: 1.5723, stage5_pos_acc: 29.3925, stage5_loss_bbox: 0.9343, stage5_loss_iou: 1.3846, stage5_loss_mask: 3.6585, loss: 45.8611, grad_norm: 108.1211
2021-08-26 10:56:35,840 - mmdet - INFO - Epoch [1][750/14659] lr: 1.873e-05, eta: 8 days, 6:51:16, time: 1.286, data_time: 0.025, memory: 10247, stage0_loss_cls: 1.7365, stage0_pos_acc: 25.9305, stage0_loss_bbox: 1.1240, stage0_loss_iou: 1.4425, stage0_loss_mask: 4.2803, stage1_loss_cls: 1.6124, stage1_pos_acc: 27.2223, stage1_loss_bbox: 0.9277, stage1_loss_iou: 1.3490, stage1_loss_mask: 3.8425, stage2_loss_cls: 1.5820, stage2_pos_acc: 27.2568, stage2_loss_bbox: 0.8401, stage2_loss_iou: 1.2861, stage2_loss_mask: 3.6427, stage3_loss_cls: 1.5647, stage3_pos_acc: 28.4669, stage3_loss_bbox: 0.8065, stage3_loss_iou: 1.2604, stage3_loss_mask: 3.5016, stage4_loss_cls: 1.5462, stage4_pos_acc: 28.9539, stage4_loss_bbox: 0.8224, stage4_loss_iou: 1.2827, stage4_loss_mask: 3.5201, stage5_loss_cls: 1.5790, stage5_pos_acc: 28.0625, stage5_loss_bbox: 0.8625, stage5_loss_iou: 1.2923, stage5_loss_mask: 3.5762, loss: 45.2801, grad_norm: 99.4212
2021-08-26 10:57:40,493 - mmdet - INFO - Epoch [1][800/14659] lr: 1.998e-05, eta: 8 days, 6:14:34, time: 1.294, data_time: 0.029, memory: 10247, stage0_loss_cls: 1.7001, stage0_pos_acc: 27.6462, stage0_loss_bbox: 1.0656, stage0_loss_iou: 1.4303, stage0_loss_mask: 4.2057, stage1_loss_cls: 1.5513, stage1_pos_acc: 28.8365, stage1_loss_bbox: 0.8659, stage1_loss_iou: 1.3150, stage1_loss_mask: 3.6463, stage2_loss_cls: 1.5046, stage2_pos_acc: 29.7402, stage2_loss_bbox: 0.7838, stage2_loss_iou: 1.2468, stage2_loss_mask: 3.4932, stage3_loss_cls: 1.4943, stage3_pos_acc: 30.3968, stage3_loss_bbox: 0.7764, stage3_loss_iou: 1.2411, stage3_loss_mask: 3.4376, stage4_loss_cls: 1.4802, stage4_pos_acc: 31.3031, stage4_loss_bbox: 0.8314, stage4_loss_iou: 1.2847, stage4_loss_mask: 3.5074, stage5_loss_cls: 1.4886, stage5_pos_acc: 33.1703, stage5_loss_bbox: 0.8857, stage5_loss_iou: 1.3279, stage5_loss_mask: 3.5649, loss: 44.1290, grad_norm: 99.3313
2021-08-26 10:58:45,254 - mmdet - INFO - Epoch [1][850/14659] lr: 2.123e-05, eta: 8 days, 5:42:30, time: 1.295, data_time: 0.028, memory: 10247, stage0_loss_cls: 1.7266, stage0_pos_acc: 25.7188, stage0_loss_bbox: 1.0323, stage0_loss_iou: 1.4061, stage0_loss_mask: 4.1405, stage1_loss_cls: 1.5495, stage1_pos_acc: 26.3889, stage1_loss_bbox: 0.8425, stage1_loss_iou: 1.2886, stage1_loss_mask: 3.5787, stage2_loss_cls: 1.5065, stage2_pos_acc: 28.2313, stage2_loss_bbox: 0.7561, stage2_loss_iou: 1.2177, stage2_loss_mask: 3.4587, stage3_loss_cls: 1.4781, stage3_pos_acc: 29.3448, stage3_loss_bbox: 0.7380, stage3_loss_iou: 1.2051, stage3_loss_mask: 3.3795, stage4_loss_cls: 1.4777, stage4_pos_acc: 29.7030, stage4_loss_bbox: 0.7641, stage4_loss_iou: 1.2305, stage4_loss_mask: 3.4189, stage5_loss_cls: 1.4723, stage5_pos_acc: 30.1511, stage5_loss_bbox: 0.7887, stage5_loss_iou: 1.2514, stage5_loss_mask: 3.3970, loss: 43.1052, grad_norm: 91.7964
2021-08-26 10:59:50,234 - mmdet - INFO - Epoch [1][900/14659] lr: 2.248e-05, eta: 8 days, 5:16:40, time: 1.300, data_time: 0.028, memory: 10247, stage0_loss_cls: 1.6663, stage0_pos_acc: 27.5676, stage0_loss_bbox: 1.0314, stage0_loss_iou: 1.3817, stage0_loss_mask: 4.0167, stage1_loss_cls: 1.5113, stage1_pos_acc: 29.0569, stage1_loss_bbox: 0.8031, stage1_loss_iou: 1.2455, stage1_loss_mask: 3.4317, stage2_loss_cls: 1.4575, stage2_pos_acc: 30.5629, stage2_loss_bbox: 0.7264, stage2_loss_iou: 1.1641, stage2_loss_mask: 3.2658, stage3_loss_cls: 1.4436, stage3_pos_acc: 32.9249, stage3_loss_bbox: 0.7004, stage3_loss_iou: 1.1511, stage3_loss_mask: 3.1839, stage4_loss_cls: 1.4497, stage4_pos_acc: 32.7798, stage4_loss_bbox: 0.7274, stage4_loss_iou: 1.1699, stage4_loss_mask: 3.2156, stage5_loss_cls: 1.4516, stage5_pos_acc: 32.8581, stage5_loss_bbox: 0.7502, stage5_loss_iou: 1.1905, stage5_loss_mask: 3.2463, loss: 41.3817, grad_norm: 82.7916
2021-08-26 11:00:55,297 - mmdet - INFO - Epoch [1][950/14659] lr: 2.373e-05, eta: 8 days, 4:53:37, time: 1.301, data_time: 0.029, memory: 10247, stage0_loss_cls: 1.6926, stage0_pos_acc: 24.8302, stage0_loss_bbox: 1.0278, stage0_loss_iou: 1.4474, stage0_loss_mask: 4.1217, stage1_loss_cls: 1.5174, stage1_pos_acc: 26.3150, stage1_loss_bbox: 0.7981, stage1_loss_iou: 1.3047, stage1_loss_mask: 3.5163, stage2_loss_cls: 1.4542, stage2_pos_acc: 29.7413, stage2_loss_bbox: 0.6940, stage2_loss_iou: 1.2135, stage2_loss_mask: 3.3908, stage3_loss_cls: 1.4345, stage3_pos_acc: 30.6539, stage3_loss_bbox: 0.6771, stage3_loss_iou: 1.1896, stage3_loss_mask: 3.3288, stage4_loss_cls: 1.4294, stage4_pos_acc: 31.8011, stage4_loss_bbox: 0.6830, stage4_loss_iou: 1.1847, stage4_loss_mask: 3.3438, stage5_loss_cls: 1.4342, stage5_pos_acc: 30.7768, stage5_loss_bbox: 0.6908, stage5_loss_iou: 1.1846, stage5_loss_mask: 3.3271, loss: 42.0862, grad_norm: 80.4314
2021-08-26 11:01:59,821 - mmdet - INFO - Exp name: queryinst_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco.py
2021-08-26 11:01:59,821 - mmdet - INFO - Epoch [1][1000/14659] lr: 2.498e-05, eta: 8 days, 4:28:13, time: 1.290, data_time: 0.029, memory: 10247, stage0_loss_cls: 1.7185, stage0_pos_acc: 23.4859, stage0_loss_bbox: 1.0140, stage0_loss_iou: 1.3786, stage0_loss_mask: 3.8208, stage1_loss_cls: 1.5292, stage1_pos_acc: 27.5284, stage1_loss_bbox: 0.8249, stage1_loss_iou: 1.2268, stage1_loss_mask: 3.2860, stage2_loss_cls: 1.4810, stage2_pos_acc: 28.5340, stage2_loss_bbox: 0.7470, stage2_loss_iou: 1.1554, stage2_loss_mask: 3.1576, stage3_loss_cls: 1.4664, stage3_pos_acc: 31.6244, stage3_loss_bbox: 0.7178, stage3_loss_iou: 1.1216, stage3_loss_mask: 3.0968, stage4_loss_cls: 1.4632, stage4_pos_acc: 31.3865, stage4_loss_bbox: 0.7182, stage4_loss_iou: 1.1125, stage4_loss_mask: 3.1103, stage5_loss_cls: 1.4789, stage5_pos_acc: 31.1255, stage5_loss_bbox: 0.7303, stage5_loss_iou: 1.1205, stage5_loss_mask: 3.1161, loss: 40.5925, grad_norm: 91.0885
2021-08-26 11:03:05,687 - mmdet - INFO - Epoch [1][1050/14659] lr: 2.500e-05, eta: 8 days, 4:16:47, time: 1.318, data_time: 0.028, memory: 10260, stage0_loss_cls: 1.6991, stage0_pos_acc: 26.2447, stage0_loss_bbox: 1.0155, stage0_loss_iou: 1.4066, stage0_loss_mask: 3.8848, stage1_loss_cls: 1.4980, stage1_pos_acc: 27.5659, stage1_loss_bbox: 0.7647, stage1_loss_iou: 1.2471, stage1_loss_mask: 3.2741, stage2_loss_cls: 1.4364, stage2_pos_acc: 29.7703, stage2_loss_bbox: 0.6777, stage2_loss_iou: 1.1569, stage2_loss_mask: 3.1341, stage3_loss_cls: 1.4320, stage3_pos_acc: 30.3810, stage3_loss_bbox: 0.6392, stage3_loss_iou: 1.1158, stage3_loss_mask: 3.0552, stage4_loss_cls: 1.4235, stage4_pos_acc: 30.6780, stage4_loss_bbox: 0.6377, stage4_loss_iou: 1.1067, stage4_loss_mask: 3.0394, stage5_loss_cls: 1.4280, stage5_pos_acc: 30.8946, stage5_loss_bbox: 0.6443, stage5_loss_iou: 1.1061, stage5_loss_mask: 3.0546, loss: 39.8778, grad_norm: 81.4806
2021-08-26 11:04:09,375 - mmdet - INFO - Epoch [1][1100/14659] lr: 2.500e-05, eta: 8 days, 3:48:30, time: 1.274, data_time: 0.026, memory: 10260, stage0_loss_cls: 1.6987, stage0_pos_acc: 23.8894, stage0_loss_bbox: 1.0367, stage0_loss_iou: 1.3789, stage0_loss_mask: 3.7901, stage1_loss_cls: 1.4886, stage1_pos_acc: 28.3727, stage1_loss_bbox: 0.7877, stage1_loss_iou: 1.2206, stage1_loss_mask: 3.2159, stage2_loss_cls: 1.4322, stage2_pos_acc: 30.4321, stage2_loss_bbox: 0.6857, stage2_loss_iou: 1.1131, stage2_loss_mask: 3.1053, stage3_loss_cls: 1.4224, stage3_pos_acc: 32.5551, stage3_loss_bbox: 0.6554, stage3_loss_iou: 1.0726, stage3_loss_mask: 3.0383, stage4_loss_cls: 1.4080, stage4_pos_acc: 33.1763, stage4_loss_bbox: 0.6439, stage4_loss_iou: 1.0641, stage4_loss_mask: 3.0504, stage5_loss_cls: 1.4335, stage5_pos_acc: 32.6161, stage5_loss_bbox: 0.6368, stage5_loss_iou: 1.0557, stage5_loss_mask: 3.0440, loss: 39.4782, grad_norm: 78.5361
2021-08-26 11:05:14,090 - mmdet - INFO - Epoch [1][1150/14659] lr: 2.500e-05, eta: 8 days, 3:30:28, time: 1.294, data_time: 0.029, memory: 10260, stage0_loss_cls: 1.6715, stage0_pos_acc: 27.3082, stage0_loss_bbox: 1.0007, stage0_loss_iou: 1.4006, stage0_loss_mask: 3.7686, stage1_loss_cls: 1.4621, stage1_pos_acc: 29.3495, stage1_loss_bbox: 0.7468, stage1_loss_iou: 1.2150, stage1_loss_mask: 3.1715, stage2_loss_cls: 1.4031, stage2_pos_acc: 32.7003, stage2_loss_bbox: 0.6451, stage2_loss_iou: 1.1141, stage2_loss_mask: 2.9907, stage3_loss_cls: 1.3903, stage3_pos_acc: 34.3246, stage3_loss_bbox: 0.6216, stage3_loss_iou: 1.0744, stage3_loss_mask: 2.9547, stage4_loss_cls: 1.3798, stage4_pos_acc: 34.3448, stage4_loss_bbox: 0.6325, stage4_loss_iou: 1.0721, stage4_loss_mask: 2.9729, stage5_loss_cls: 1.4005, stage5_pos_acc: 33.8655, stage5_loss_bbox: 0.6101, stage5_loss_iou: 1.0476, stage5_loss_mask: 2.9774, loss: 38.7239, grad_norm: 78.3448
2021-08-26 11:06:18,885 - mmdet - INFO - Epoch [1][1200/14659] lr: 2.500e-05, eta: 8 days, 3:14:17, time: 1.295, data_time: 0.024, memory: 10260, stage0_loss_cls: 1.6741, stage0_pos_acc: 26.5082, stage0_loss_bbox: 1.0016, stage0_loss_iou: 1.3997, stage0_loss_mask: 3.8476, stage1_loss_cls: 1.4358, stage1_pos_acc: 29.2393, stage1_loss_bbox: 0.7487, stage1_loss_iou: 1.2229, stage1_loss_mask: 3.2792, stage2_loss_cls: 1.3819, stage2_pos_acc: 30.4406, stage2_loss_bbox: 0.6568, stage2_loss_iou: 1.1234, stage2_loss_mask: 3.1402, stage3_loss_cls: 1.3637, stage3_pos_acc: 33.2124, stage3_loss_bbox: 0.6305, stage3_loss_iou: 1.0950, stage3_loss_mask: 3.1075, stage4_loss_cls: 1.3658, stage4_pos_acc: 32.1867, stage4_loss_bbox: 0.6289, stage4_loss_iou: 1.0902, stage4_loss_mask: 3.0865, stage5_loss_cls: 1.3765, stage5_pos_acc: 33.1551, stage5_loss_bbox: 0.6318, stage5_loss_iou: 1.1011, stage5_loss_mask: 3.1301, loss: 39.5195, grad_norm: 76.9653
2021-08-26 11:07:25,167 - mmdet - INFO - Epoch [1][1250/14659] lr: 2.500e-05, eta: 8 days, 3:10:12, time: 1.326, data_time: 0.035, memory: 10260, stage0_loss_cls: 1.6800, stage0_pos_acc: 25.1354, stage0_loss_bbox: 0.9629, stage0_loss_iou: 1.3814, stage0_loss_mask: 3.5957, stage1_loss_cls: 1.4227, stage1_pos_acc: 28.0095, stage1_loss_bbox: 0.7004, stage1_loss_iou: 1.1653, stage1_loss_mask: 3.0698, stage2_loss_cls: 1.3556, stage2_pos_acc: 31.2645, stage2_loss_bbox: 0.6103, stage2_loss_iou: 1.0676, stage2_loss_mask: 2.9250, stage3_loss_cls: 1.3398, stage3_pos_acc: 33.5218, stage3_loss_bbox: 0.5740, stage3_loss_iou: 1.0287, stage3_loss_mask: 2.8877, stage4_loss_cls: 1.3407, stage4_pos_acc: 34.1780, stage4_loss_bbox: 0.5645, stage4_loss_iou: 1.0191, stage4_loss_mask: 2.8721, stage5_loss_cls: 1.3556, stage5_pos_acc: 33.6427, stage5_loss_bbox: 0.5609, stage5_loss_iou: 1.0110, stage5_loss_mask: 2.8753, loss: 37.3662, grad_norm: 75.3087
2021-08-26 11:08:30,769 - mmdet - INFO - Epoch [1][1300/14659] lr: 2.500e-05, eta: 8 days, 3:01:29, time: 1.312, data_time: 0.031, memory: 10260, stage0_loss_cls: 1.6479, stage0_pos_acc: 28.9296, stage0_loss_bbox: 0.9668, stage0_loss_iou: 1.4114, stage0_loss_mask: 3.8106, stage1_loss_cls: 1.3825, stage1_pos_acc: 32.6764, stage1_loss_bbox: 0.6959, stage1_loss_iou: 1.2048, stage1_loss_mask: 3.2734, stage2_loss_cls: 1.3125, stage2_pos_acc: 35.9537, stage2_loss_bbox: 0.6092, stage2_loss_iou: 1.1096, stage2_loss_mask: 3.1411, stage3_loss_cls: 1.3076, stage3_pos_acc: 37.4952, stage3_loss_bbox: 0.5831, stage3_loss_iou: 1.0698, stage3_loss_mask: 3.0958, stage4_loss_cls: 1.2939, stage4_pos_acc: 38.1969, stage4_loss_bbox: 0.5723, stage4_loss_iou: 1.0563, stage4_loss_mask: 3.0967, stage5_loss_cls: 1.3119, stage5_pos_acc: 37.5753, stage5_loss_bbox: 0.5783, stage5_loss_iou: 1.0544, stage5_loss_mask: 3.1193, loss: 38.7049, grad_norm: 69.7411
2021-08-26 11:09:37,150 - mmdet - INFO - Epoch [1][1350/14659] lr: 2.500e-05, eta: 8 days, 2:58:17, time: 1.327, data_time: 0.031, memory: 10260, stage0_loss_cls: 1.6179, stage0_pos_acc: 29.8083, stage0_loss_bbox: 0.9617, stage0_loss_iou: 1.4000, stage0_loss_mask: 3.6221, stage1_loss_cls: 1.3624, stage1_pos_acc: 32.9784, stage1_loss_bbox: 0.7101, stage1_loss_iou: 1.1954, stage1_loss_mask: 3.0473, stage2_loss_cls: 1.3282, stage2_pos_acc: 34.8804, stage2_loss_bbox: 0.6097, stage2_loss_iou: 1.0835, stage2_loss_mask: 2.9072, stage3_loss_cls: 1.3201, stage3_pos_acc: 36.4125, stage3_loss_bbox: 0.5731, stage3_loss_iou: 1.0404, stage3_loss_mask: 2.8688, stage4_loss_cls: 1.3002, stage4_pos_acc: 37.3596, stage4_loss_bbox: 0.5613, stage4_loss_iou: 1.0244, stage4_loss_mask: 2.8589, stage5_loss_cls: 1.3196, stage5_pos_acc: 37.2872, stage5_loss_bbox: 0.5531, stage5_loss_iou: 1.0206, stage5_loss_mask: 2.8500, loss: 37.1359, grad_norm: 74.0133
2021-08-26 11:10:42,106 - mmdet - INFO - Epoch [1][1400/14659] lr: 2.500e-05, eta: 8 days, 2:46:29, time: 1.299, data_time: 0.027, memory: 10260, stage0_loss_cls: 1.6639, stage0_pos_acc: 24.5354, stage0_loss_bbox: 0.9730, stage0_loss_iou: 1.3973, stage0_loss_mask: 3.6086, stage1_loss_cls: 1.4075, stage1_pos_acc: 27.8628, stage1_loss_bbox: 0.7055, stage1_loss_iou: 1.1755, stage1_loss_mask: 3.0696, stage2_loss_cls: 1.3575, stage2_pos_acc: 31.3315, stage2_loss_bbox: 0.6009, stage2_loss_iou: 1.0746, stage2_loss_mask: 2.9750, stage3_loss_cls: 1.3346, stage3_pos_acc: 35.5107, stage3_loss_bbox: 0.5721, stage3_loss_iou: 1.0375, stage3_loss_mask: 2.9445, stage4_loss_cls: 1.3261, stage4_pos_acc: 34.3865, stage4_loss_bbox: 0.5639, stage4_loss_iou: 1.0271, stage4_loss_mask: 2.9395, stage5_loss_cls: 1.3379, stage5_pos_acc: 34.6709, stage5_loss_bbox: 0.5691, stage5_loss_iou: 1.0290, stage5_loss_mask: 2.9497, loss: 37.6400, grad_norm: 71.6296
2021-08-26 11:11:47,091 - mmdet - INFO - Epoch [1][1450/14659] lr: 2.500e-05, eta: 8 days, 2:35:03, time: 1.298, data_time: 0.026, memory: 10260, stage0_loss_cls: 1.6694, stage0_pos_acc: 25.8043, stage0_loss_bbox: 0.9478, stage0_loss_iou: 1.3652, stage0_loss_mask: 3.5024, stage1_loss_cls: 1.4135, stage1_pos_acc: 30.3336, stage1_loss_bbox: 0.6791, stage1_loss_iou: 1.1441, stage1_loss_mask: 2.9723, stage2_loss_cls: 1.3526, stage2_pos_acc: 33.2251, stage2_loss_bbox: 0.6031, stage2_loss_iou: 1.0438, stage2_loss_mask: 2.8801, stage3_loss_cls: 1.3430, stage3_pos_acc: 35.2084, stage3_loss_bbox: 0.5677, stage3_loss_iou: 1.0006, stage3_loss_mask: 2.8446, stage4_loss_cls: 1.3277, stage4_pos_acc: 35.9694, stage4_loss_bbox: 0.5671, stage4_loss_iou: 0.9964, stage4_loss_mask: 2.8584, stage5_loss_cls: 1.3488, stage5_pos_acc: 35.4904, stage5_loss_bbox: 0.5653, stage5_loss_iou: 0.9986, stage5_loss_mask: 2.8676, loss: 36.8592, grad_norm: 73.8468
2021-08-26 11:12:52,427 - mmdet - INFO - Epoch [1][1500/14659] lr: 2.500e-05, eta: 8 days, 2:27:06, time: 1.308, data_time: 0.028, memory: 10260, stage0_loss_cls: 1.6548, stage0_pos_acc: 25.4055, stage0_loss_bbox: 0.9172, stage0_loss_iou: 1.3403, stage0_loss_mask: 3.5027, stage1_loss_cls: 1.3948, stage1_pos_acc: 30.2240, stage1_loss_bbox: 0.6886, stage1_loss_iou: 1.1475, stage1_loss_mask: 2.9771, stage2_loss_cls: 1.3370, stage2_pos_acc: 33.6155, stage2_loss_bbox: 0.5852, stage2_loss_iou: 1.0394, stage2_loss_mask: 2.8556, stage3_loss_cls: 1.3124, stage3_pos_acc: 37.1978, stage3_loss_bbox: 0.5461, stage3_loss_iou: 0.9921, stage3_loss_mask: 2.8452, stage4_loss_cls: 1.3030, stage4_pos_acc: 38.8616, stage4_loss_bbox: 0.5495, stage4_loss_iou: 0.9866, stage4_loss_mask: 2.8609, stage5_loss_cls: 1.3085, stage5_pos_acc: 38.3760, stage5_loss_bbox: 0.5476, stage5_loss_iou: 0.9827, stage5_loss_mask: 2.8690, loss: 36.5439, grad_norm: 70.3628
2021-08-26 11:13:57,001 - mmdet - INFO - Epoch [1][1550/14659] lr: 2.500e-05, eta: 8 days, 2:15:13, time: 1.292, data_time: 0.027, memory: 10260, stage0_loss_cls: 1.6851, stage0_pos_acc: 24.5149, stage0_loss_bbox: 0.9397, stage0_loss_iou: 1.3625, stage0_loss_mask: 3.4770, stage1_loss_cls: 1.4110, stage1_pos_acc: 28.2324, stage1_loss_bbox: 0.6584, stage1_loss_iou: 1.1252, stage1_loss_mask: 2.8640, stage2_loss_cls: 1.3431, stage2_pos_acc: 32.4330, stage2_loss_bbox: 0.5522, stage2_loss_iou: 1.0091, stage2_loss_mask: 2.7325, stage3_loss_cls: 1.3222, stage3_pos_acc: 34.9220, stage3_loss_bbox: 0.5196, stage3_loss_iou: 0.9702, stage3_loss_mask: 2.7097, stage4_loss_cls: 1.3133, stage4_pos_acc: 35.7501, stage4_loss_bbox: 0.5125, stage4_loss_iou: 0.9522, stage4_loss_mask: 2.7075, stage5_loss_cls: 1.3295, stage5_pos_acc: 35.7375, stage5_loss_bbox: 0.5176, stage5_loss_iou: 0.9538, stage5_loss_mask: 2.7439, loss: 35.7117, grad_norm: 72.5540
2021-08-26 11:15:02,440 - mmdet - INFO - Epoch [1][1600/14659] lr: 2.500e-05, eta: 8 days, 2:08:33, time: 1.309, data_time: 0.031, memory: 10260, stage0_loss_cls: 1.6480, stage0_pos_acc: 26.2355, stage0_loss_bbox: 0.9417, stage0_loss_iou: 1.3586, stage0_loss_mask: 3.4764, stage1_loss_cls: 1.3551, stage1_pos_acc: 31.0848, stage1_loss_bbox: 0.6607, stage1_loss_iou: 1.1333, stage1_loss_mask: 2.8353, stage2_loss_cls: 1.2772, stage2_pos_acc: 35.2232, stage2_loss_bbox: 0.5610, stage2_loss_iou: 1.0170, stage2_loss_mask: 2.7561, stage3_loss_cls: 1.2661, stage3_pos_acc: 36.9457, stage3_loss_bbox: 0.5299, stage3_loss_iou: 0.9758, stage3_loss_mask: 2.7287, stage4_loss_cls: 1.2547, stage4_pos_acc: 38.4415, stage4_loss_bbox: 0.5165, stage4_loss_iou: 0.9627, stage4_loss_mask: 2.7384, stage5_loss_cls: 1.2716, stage5_pos_acc: 39.2369, stage5_loss_bbox: 0.5176, stage5_loss_iou: 0.9582, stage5_loss_mask: 2.7461, loss: 35.4867, grad_norm: 70.7494
2021-08-26 11:16:07,147 - mmdet - INFO - Epoch [1][1650/14659] lr: 2.500e-05, eta: 8 days, 1:58:19, time: 1.294, data_time: 0.027, memory: 10260, stage0_loss_cls: 1.6747, stage0_pos_acc: 25.6228, stage0_loss_bbox: 0.9274, stage0_loss_iou: 1.3638, stage0_loss_mask: 3.3953, stage1_loss_cls: 1.3673, stage1_pos_acc: 29.8191, stage1_loss_bbox: 0.6477, stage1_loss_iou: 1.1207, stage1_loss_mask: 2.8112, stage2_loss_cls: 1.3010, stage2_pos_acc: 33.2384, stage2_loss_bbox: 0.5509, stage2_loss_iou: 1.0006, stage2_loss_mask: 2.6553, stage3_loss_cls: 1.2719, stage3_pos_acc: 36.5612, stage3_loss_bbox: 0.5095, stage3_loss_iou: 0.9532, stage3_loss_mask: 2.6669, stage4_loss_cls: 1.2616, stage4_pos_acc: 38.9800, stage4_loss_bbox: 0.5050, stage4_loss_iou: 0.9380, stage4_loss_mask: 2.6746, stage5_loss_cls: 1.2729, stage5_pos_acc: 38.0502, stage5_loss_bbox: 0.4997, stage5_loss_iou: 0.9334, stage5_loss_mask: 2.6741, loss: 34.9768, grad_norm: 71.5673
2021-08-26 11:17:11,737 - mmdet - INFO - Epoch [1][1700/14659] lr: 2.500e-05, eta: 8 days, 1:48:00, time: 1.292, data_time: 0.028, memory: 10260, stage0_loss_cls: 1.6531, stage0_pos_acc: 24.8506, stage0_loss_bbox: 0.9111, stage0_loss_iou: 1.3768, stage0_loss_mask: 3.3537, stage1_loss_cls: 1.3289, stage1_pos_acc: 29.6029, stage1_loss_bbox: 0.6354, stage1_loss_iou: 1.1256, stage1_loss_mask: 2.7265, stage2_loss_cls: 1.2492, stage2_pos_acc: 35.0890, stage2_loss_bbox: 0.5676, stage2_loss_iou: 1.0286, stage2_loss_mask: 2.6394, stage3_loss_cls: 1.2338, stage3_pos_acc: 40.2179, stage3_loss_bbox: 0.5471, stage3_loss_iou: 0.9936, stage3_loss_mask: 2.5983, stage4_loss_cls: 1.2161, stage4_pos_acc: 41.3352, stage4_loss_bbox: 0.5345, stage4_loss_iou: 0.9823, stage4_loss_mask: 2.5976, stage5_loss_cls: 1.2298, stage5_pos_acc: 40.0981, stage5_loss_bbox: 0.5292, stage5_loss_iou: 0.9787, stage5_loss_mask: 2.6037, loss: 34.6408, grad_norm: 74.2182
2021-08-26 11:18:17,203 - mmdet - INFO - Epoch [1][1750/14659] lr: 2.500e-05, eta: 8 days, 1:42:19, time: 1.308, data_time: 0.029, memory: 10260, stage0_loss_cls: 1.6493, stage0_pos_acc: 27.4200, stage0_loss_bbox: 0.9352, stage0_loss_iou: 1.3797, stage0_loss_mask: 3.4129, stage1_loss_cls: 1.3249, stage1_pos_acc: 33.7146, stage1_loss_bbox: 0.6363, stage1_loss_iou: 1.1272, stage1_loss_mask: 2.7844, stage2_loss_cls: 1.2501, stage2_pos_acc: 38.5490, stage2_loss_bbox: 0.5377, stage2_loss_iou: 1.0149, stage2_loss_mask: 2.6762, stage3_loss_cls: 1.2411, stage3_pos_acc: 42.5941, stage3_loss_bbox: 0.5090, stage3_loss_iou: 0.9733, stage3_loss_mask: 2.6403, stage4_loss_cls: 1.2254, stage4_pos_acc: 42.4388, stage4_loss_bbox: 0.4923, stage4_loss_iou: 0.9571, stage4_loss_mask: 2.6582, stage5_loss_cls: 1.2398, stage5_pos_acc: 42.0586, stage5_loss_bbox: 0.4944, stage5_loss_iou: 0.9531, stage5_loss_mask: 2.6798, loss: 34.7925, grad_norm: 70.7024
2021-08-26 11:19:22,128 - mmdet - INFO - Epoch [1][1800/14659] lr: 2.500e-05, eta: 8 days, 1:34:55, time: 1.300, data_time: 0.030, memory: 10379, stage0_loss_cls: 1.6151, stage0_pos_acc: 28.5230, stage0_loss_bbox: 0.8962, stage0_loss_iou: 1.3379, stage0_loss_mask: 3.3015, stage1_loss_cls: 1.3039, stage1_pos_acc: 33.2951, stage1_loss_bbox: 0.6194, stage1_loss_iou: 1.0892, stage1_loss_mask: 2.7000, stage2_loss_cls: 1.2418, stage2_pos_acc: 38.5387, stage2_loss_bbox: 0.5380, stage2_loss_iou: 0.9809, stage2_loss_mask: 2.5582, stage3_loss_cls: 1.2145, stage3_pos_acc: 42.3104, stage3_loss_bbox: 0.5076, stage3_loss_iou: 0.9360, stage3_loss_mask: 2.5354, stage4_loss_cls: 1.1984, stage4_pos_acc: 43.3533, stage4_loss_bbox: 0.5003, stage4_loss_iou: 0.9203, stage4_loss_mask: 2.5300, stage5_loss_cls: 1.2125, stage5_pos_acc: 42.7439, stage5_loss_bbox: 0.4876, stage5_loss_iou: 0.9110, stage5_loss_mask: 2.5421, loss: 33.6781, grad_norm: 74.8392
2021-08-26 11:20:26,881 - mmdet - INFO - Epoch [1][1850/14659] lr: 2.500e-05, eta: 8 days, 1:26:36, time: 1.295, data_time: 0.026, memory: 10379, stage0_loss_cls: 1.6402, stage0_pos_acc: 26.3269, stage0_loss_bbox: 0.9378, stage0_loss_iou: 1.3596, stage0_loss_mask: 3.3698, stage1_loss_cls: 1.3408, stage1_pos_acc: 31.4008, stage1_loss_bbox: 0.6353, stage1_loss_iou: 1.0938, stage1_loss_mask: 2.7399, stage2_loss_cls: 1.2689, stage2_pos_acc: 35.9966, stage2_loss_bbox: 0.5364, stage2_loss_iou: 0.9779, stage2_loss_mask: 2.6163, stage3_loss_cls: 1.2561, stage3_pos_acc: 37.8304, stage3_loss_bbox: 0.5033, stage3_loss_iou: 0.9412, stage3_loss_mask: 2.5846, stage4_loss_cls: 1.2425, stage4_pos_acc: 38.1376, stage4_loss_bbox: 0.5033, stage4_loss_iou: 0.9371, stage4_loss_mask: 2.6020, stage5_loss_cls: 1.2553, stage5_pos_acc: 40.5602, stage5_loss_bbox: 0.4952, stage5_loss_iou: 0.9264, stage5_loss_mask: 2.6149, loss: 34.3788, grad_norm: 69.8435
2021-08-26 11:21:31,601 - mmdet - INFO - Epoch [1][1900/14659] lr: 2.500e-05, eta: 8 days, 1:18:42, time: 1.295, data_time: 0.029, memory: 10379, stage0_loss_cls: 1.6763, stage0_pos_acc: 23.5548, stage0_loss_bbox: 0.9613, stage0_loss_iou: 1.3327, stage0_loss_mask: 3.2624, stage1_loss_cls: 1.3659, stage1_pos_acc: 28.5197, stage1_loss_bbox: 0.6521, stage1_loss_iou: 1.0761, stage1_loss_mask: 2.6724, stage2_loss_cls: 1.3015, stage2_pos_acc: 33.7910, stage2_loss_bbox: 0.5524, stage2_loss_iou: 0.9623, stage2_loss_mask: 2.5465, stage3_loss_cls: 1.2769, stage3_pos_acc: 37.5364, stage3_loss_bbox: 0.5105, stage3_loss_iou: 0.9124, stage3_loss_mask: 2.5400, stage4_loss_cls: 1.2598, stage4_pos_acc: 38.8722, stage4_loss_bbox: 0.5056, stage4_loss_iou: 0.8961, stage4_loss_mask: 2.5365, stage5_loss_cls: 1.2736, stage5_pos_acc: 40.1570, stage5_loss_bbox: 0.4917, stage5_loss_iou: 0.8863, stage5_loss_mask: 2.5463, loss: 33.9976, grad_norm: 70.5757
2021-08-26 11:22:36,294 - mmdet - INFO - Epoch [1][1950/14659] lr: 2.500e-05, eta: 8 days, 1:10:51, time: 1.293, data_time: 0.027, memory: 10379, stage0_loss_cls: 1.6400, stage0_pos_acc: 26.8736, stage0_loss_bbox: 0.9084, stage0_loss_iou: 1.3418, stage0_loss_mask: 3.4105, stage1_loss_cls: 1.3184, stage1_pos_acc: 31.7751, stage1_loss_bbox: 0.6120, stage1_loss_iou: 1.0903, stage1_loss_mask: 2.7483, stage2_loss_cls: 1.2402, stage2_pos_acc: 37.1692, stage2_loss_bbox: 0.5336, stage2_loss_iou: 0.9890, stage2_loss_mask: 2.6649, stage3_loss_cls: 1.2257, stage3_pos_acc: 41.1638, stage3_loss_bbox: 0.5075, stage3_loss_iou: 0.9536, stage3_loss_mask: 2.6287, stage4_loss_cls: 1.2063, stage4_pos_acc: 42.5924, stage4_loss_bbox: 0.5062, stage4_loss_iou: 0.9484, stage4_loss_mask: 2.6098, stage5_loss_cls: 1.2333, stage5_pos_acc: 43.1168, stage5_loss_bbox: 0.5024, stage5_loss_iou: 0.9465, stage5_loss_mask: 2.6365, loss: 34.4023, grad_norm: 68.9732
2021-08-26 11:23:41,363 - mmdet - INFO - Exp name: queryinst_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco.py
2021-08-26 11:23:41,364 - mmdet - INFO - Epoch [1][2000/14659] lr: 2.500e-05, eta: 8 days, 1:05:07, time: 1.302, data_time: 0.033, memory: 10379, stage0_loss_cls: 1.6157, stage0_pos_acc: 28.0797, stage0_loss_bbox: 0.8930, stage0_loss_iou: 1.3461, stage0_loss_mask: 3.2833, stage1_loss_cls: 1.2790, stage1_pos_acc: 33.3812, stage1_loss_bbox: 0.5989, stage1_loss_iou: 1.0778, stage1_loss_mask: 2.6179, stage2_loss_cls: 1.1971, stage2_pos_acc: 40.0282, stage2_loss_bbox: 0.5161, stage2_loss_iou: 0.9692, stage2_loss_mask: 2.4923, stage3_loss_cls: 1.1864, stage3_pos_acc: 41.1632, stage3_loss_bbox: 0.4747, stage3_loss_iou: 0.9215, stage3_loss_mask: 2.4593, stage4_loss_cls: 1.1722, stage4_pos_acc: 44.4289, stage4_loss_bbox: 0.4639, stage4_loss_iou: 0.9089, stage4_loss_mask: 2.4508, stage5_loss_cls: 1.1835, stage5_pos_acc: 44.9918, stage5_loss_bbox: 0.4557, stage5_loss_iou: 0.8996, stage5_loss_mask: 2.4687, loss: 32.9316, grad_norm: 67.7723
2021-08-26 11:24:46,291 - mmdet - INFO - Epoch [1][2050/14659] lr: 2.500e-05, eta: 8 days, 0:58:58, time: 1.299, data_time: 0.025, memory: 10379, stage0_loss_cls: 1.6159, stage0_pos_acc: 28.6850, stage0_loss_bbox: 0.9238, stage0_loss_iou: 1.3584, stage0_loss_mask: 3.3194, stage1_loss_cls: 1.2727, stage1_pos_acc: 34.6746, stage1_loss_bbox: 0.6231, stage1_loss_iou: 1.1023, stage1_loss_mask: 2.7153, stage2_loss_cls: 1.1760, stage2_pos_acc: 40.8597, stage2_loss_bbox: 0.5384, stage2_loss_iou: 0.9938, stage2_loss_mask: 2.5986, stage3_loss_cls: 1.1673, stage3_pos_acc: 42.9596, stage3_loss_bbox: 0.5067, stage3_loss_iou: 0.9520, stage3_loss_mask: 2.5876, stage4_loss_cls: 1.1475, stage4_pos_acc: 46.5867, stage4_loss_bbox: 0.4939, stage4_loss_iou: 0.9400, stage4_loss_mask: 2.5573, stage5_loss_cls: 1.1630, stage5_pos_acc: 44.7366, stage5_loss_bbox: 0.4903, stage5_loss_iou: 0.9357, stage5_loss_mask: 2.5647, loss: 33.7436, grad_norm: 69.3526
2021-08-26 11:25:52,366 - mmdet - INFO - Epoch [1][2100/14659] lr: 2.500e-05, eta: 8 days, 0:57:52, time: 1.322, data_time: 0.031, memory: 10379, stage0_loss_cls: 1.5835, stage0_pos_acc: 29.9282, stage0_loss_bbox: 0.8743, stage0_loss_iou: 1.3925, stage0_loss_mask: 3.4292, stage1_loss_cls: 1.2144, stage1_pos_acc: 36.4214, stage1_loss_bbox: 0.6154, stage1_loss_iou: 1.1387, stage1_loss_mask: 2.7780, stage2_loss_cls: 1.1328, stage2_pos_acc: 42.5968, stage2_loss_bbox: 0.5381, stage2_loss_iou: 1.0282, stage2_loss_mask: 2.6629, stage3_loss_cls: 1.1280, stage3_pos_acc: 46.1657, stage3_loss_bbox: 0.5008, stage3_loss_iou: 0.9818, stage3_loss_mask: 2.6260, stage4_loss_cls: 1.1079, stage4_pos_acc: 47.8904, stage4_loss_bbox: 0.4936, stage4_loss_iou: 0.9680, stage4_loss_mask: 2.6379, stage5_loss_cls: 1.1114, stage5_pos_acc: 49.2550, stage5_loss_bbox: 0.4914, stage5_loss_iou: 0.9609, stage5_loss_mask: 2.6372, loss: 34.0325, grad_norm: 67.0531
2021-08-26 11:26:55,927 - mmdet - INFO - Epoch [1][2150/14659] lr: 2.500e-05, eta: 8 days, 0:46:29, time: 1.271, data_time: 0.022, memory: 10379, stage0_loss_cls: 1.6465, stage0_pos_acc: 27.2861, stage0_loss_bbox: 0.9112, stage0_loss_iou: 1.3809, stage0_loss_mask: 3.2773, stage1_loss_cls: 1.2806, stage1_pos_acc: 32.4545, stage1_loss_bbox: 0.5916, stage1_loss_iou: 1.0747, stage1_loss_mask: 2.6087, stage2_loss_cls: 1.2013, stage2_pos_acc: 37.1091, stage2_loss_bbox: 0.5088, stage2_loss_iou: 0.9662, stage2_loss_mask: 2.4953, stage3_loss_cls: 1.1934, stage3_pos_acc: 40.7695, stage3_loss_bbox: 0.4824, stage3_loss_iou: 0.9193, stage3_loss_mask: 2.4814, stage4_loss_cls: 1.1760, stage4_pos_acc: 41.7712, stage4_loss_bbox: 0.4698, stage4_loss_iou: 0.9015, stage4_loss_mask: 2.4674, stage5_loss_cls: 1.1843, stage5_pos_acc: 42.3751, stage5_loss_bbox: 0.4690, stage5_loss_iou: 0.8993, stage5_loss_mask: 2.4923, loss: 33.0790, grad_norm: 72.1277
2021-08-26 11:28:00,678 - mmdet - INFO - Epoch [1][2200/14659] lr: 2.500e-05, eta: 8 days, 0:40:20, time: 1.295, data_time: 0.028, memory: 10379, stage0_loss_cls: 1.6484, stage0_pos_acc: 26.1184, stage0_loss_bbox: 0.9199, stage0_loss_iou: 1.3046, stage0_loss_mask: 3.1031, stage1_loss_cls: 1.2997, stage1_pos_acc: 33.3425, stage1_loss_bbox: 0.6182, stage1_loss_iou: 1.0203, stage1_loss_mask: 2.4607, stage2_loss_cls: 1.2138, stage2_pos_acc: 39.4199, stage2_loss_bbox: 0.5175, stage2_loss_iou: 0.9011, stage2_loss_mask: 2.3768, stage3_loss_cls: 1.1978, stage3_pos_acc: 42.3183, stage3_loss_bbox: 0.4882, stage3_loss_iou: 0.8593, stage3_loss_mask: 2.3531, stage4_loss_cls: 1.1829, stage4_pos_acc: 43.8389, stage4_loss_bbox: 0.4770, stage4_loss_iou: 0.8437, stage4_loss_mask: 2.3511, stage5_loss_cls: 1.1914, stage5_pos_acc: 43.3682, stage5_loss_bbox: 0.4723, stage5_loss_iou: 0.8356, stage5_loss_mask: 2.3603, loss: 31.9969, grad_norm: 71.5349
2021-08-26 11:29:05,539 - mmdet - INFO - Epoch [1][2250/14659] lr: 2.500e-05, eta: 8 days, 0:34:48, time: 1.297, data_time: 0.026, memory: 10379, stage0_loss_cls: 1.6389, stage0_pos_acc: 26.4171, stage0_loss_bbox: 0.8738, stage0_loss_iou: 1.3478, stage0_loss_mask: 3.2712, stage1_loss_cls: 1.2597, stage1_pos_acc: 36.5073, stage1_loss_bbox: 0.5825, stage1_loss_iou: 1.0741, stage1_loss_mask: 2.5992, stage2_loss_cls: 1.1765, stage2_pos_acc: 42.4178, stage2_loss_bbox: 0.5101, stage2_loss_iou: 0.9813, stage2_loss_mask: 2.5105, stage3_loss_cls: 1.1682, stage3_pos_acc: 44.9137, stage3_loss_bbox: 0.4719, stage3_loss_iou: 0.9277, stage3_loss_mask: 2.4942, stage4_loss_cls: 1.1524, stage4_pos_acc: 47.3806, stage4_loss_bbox: 0.4631, stage4_loss_iou: 0.9198, stage4_loss_mask: 2.4841, stage5_loss_cls: 1.1615, stage5_pos_acc: 47.7888, stage5_loss_bbox: 0.4551, stage5_loss_iou: 0.9104, stage5_loss_mask: 2.5115, loss: 32.9455, grad_norm: 70.3203
2021-08-26 11:30:10,491 - mmdet - INFO - Epoch [1][2300/14659] lr: 2.500e-05, eta: 8 days, 0:29:51, time: 1.299, data_time: 0.025, memory: 10379, stage0_loss_cls: 1.6181, stage0_pos_acc: 27.4700, stage0_loss_bbox: 0.8644, stage0_loss_iou: 1.3378, stage0_loss_mask: 3.2031, stage1_loss_cls: 1.2365, stage1_pos_acc: 34.6672, stage1_loss_bbox: 0.5978, stage1_loss_iou: 1.0615, stage1_loss_mask: 2.6021, stage2_loss_cls: 1.1551, stage2_pos_acc: 41.1225, stage2_loss_bbox: 0.5164, stage2_loss_iou: 0.9623, stage2_loss_mask: 2.5070, stage3_loss_cls: 1.1470, stage3_pos_acc: 43.9128, stage3_loss_bbox: 0.4866, stage3_loss_iou: 0.9210, stage3_loss_mask: 2.4700, stage4_loss_cls: 1.1324, stage4_pos_acc: 45.1144, stage4_loss_bbox: 0.4751, stage4_loss_iou: 0.9079, stage4_loss_mask: 2.4605, stage5_loss_cls: 1.1379, stage5_pos_acc: 45.8629, stage5_loss_bbox: 0.4707, stage5_loss_iou: 0.9008, stage5_loss_mask: 2.4844, loss: 32.6565, grad_norm: 66.8825
Ground Truth Not Found!
Ground Truth Not Found!
Ground Truth Not Found!
Ground Truth Not Found!
Ground Truth Not Found!
Ground Truth Not Found!
^CTraceback (most recent call last):
File "/mnt/home1/programs/miniconda3/envs/usd/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/mnt/home1/programs/miniconda3/envs/usd/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/mnt/home1/programs/miniconda3/envs/usd/lib/python3.8/site-packages/torch/distributed/launch.py", line 260, in <module>
main()
File "/mnt/home1/programs/miniconda3/envs/usd/lib/python3.8/site-packages/torch/distributed/launch.py", line 253, in main
process.wait()
File "/mnt/home1/programs/miniconda3/envs/usd/lib/python3.8/subprocess.py", line 1083, in wait
return self._wait(timeout=timeout)
File "/mnt/home1/programs/miniconda3/envs/usd/lib/python3.8/subprocess.py", line 1806, in _wait
(pid, sts) = self._try_wait(0)
File "/mnt/home1/programs/miniconda3/envs/usd/lib/python3.8/subprocess.py", line 1764, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
Yuxin-CV commented
Hello, thanks for your interest in our work.
Can you successfully train mask rcnn with this codebase?
iFighting commented
i also meet this problem when training configs/queryinst/queryinst_r101_fpn_300_proposals_crop_mstrain_480-800_3x_coco.py
vealocia commented
Hi, @iFighting ! If you find your code stucked during training phase with a log Ground Truth Not Found! get out, please just kill this process and training from the latest checkpoint. We will update this repo recently to fix this problem. Please stay tuned for our next update!
Yuxin-CV commented
QueryInst is now officially included by mmdetection library, with new checkpoints, corresponding logs, and augmented training settings.
https://github.com/open-mmlab/mmdetection/blob/master/configs/queryinst/README.md