LiWentomng/OrientedRepPoints

训练时在计算giou时报错

Closed this issue · 2 comments

windyjunfeng commented

在训练时一个epoch没有跑完就会出现计算giou的错误，但是检查了下数据集应该没有问题。在有的数据集会出现这种问题，在大部分数据集上都能正常训练，请问这可能是什么原因呢？应该怎么解决呢？谢谢~

Traceback (most recent call last):
  File "tools/train.py", line 154, in <module>
    main()
  File "tools/train.py", line 150, in main
    meta=meta)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/apis/train.py", line 112, in train_detector
    meta=meta)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/apis/train.py", line 245, in _non_dist_train
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/mmcv/runner/runner.py", line 373, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/mmcv/runner/runner.py", line 275, in train
    self.model, data_batch, train_mode=True, **kwargs)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/apis/train.py", line 75, in batch_processor
    losses = model(**data)
  File "/home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/core/fp16/decorators.py", line 49, in new_func
    return old_func(*args, **kwargs)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/models/detectors/base.py", line 147, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/models/detectors/orientedreppoints_detector.py", line 36, in forward_train
    *loss_inputs, gt_rbboxes_ignore=gt_rbboxes_ignore)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/models/anchor_heads/orientedreppoints_head.py", line 465, in loss
    rbox_weights_list_refine, pos_inds_list_refine)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/core/utils/misc.py", line 24, in multi_apply
    return tuple(map(list, zip(*map_results)))
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/models/anchor_heads/orientedreppoints_head.py", line 567, in points_quality_assessment
    reduction_override='none')
  File "/home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/models/losses/iou_loss.py", line 134, in forward
    self.loss_weight)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/models/losses/iou_loss.py", line 78, in forward
    convex_gious, grad = convex_giou(pred, target)
  File "/home/f523/guazai/sdb/zhoujunfeng/OrientedRepPoints-main/mmdet/ops/iou/iou_wrapper.py", line 13, in convex_giou
    convex_giou_grad = convex_giou_cuda.convex_giou(pred, target)
RuntimeError: cuda runtime error (700) : an illegal memory access was encountered at mmdet/ops/iou/src/convex_giou_kernel.cu:856
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered (insert_events at /opt/conda/conda-bld/pytorch_1573049304260/work/c10/cuda/CUDACachingAllocator.cpp:569)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x47 (0x7f3b4d9a5687 in /home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x17044 (0x7f3b4dbe1044 in /home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: <unknown function> + 0x1cccb (0x7f3b4dbe6ccb in /home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x4d (0x7f3b4d992e9d in /home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0x1c86c9 (0x7f3b4eb2d6c9 in /home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x4aff6b (0x7f3b4ee14f6b in /home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x4affa6 (0x7f3b4ee14fa6 in /home/f523/anaconda3/envs/redet/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #26: __libc_start_main + 0xf0 (0x7f3b53406840 in /lib/x86_64-linux-gnu/libc.so.6)

已放弃 (核心已转储)
xinsirBUPT2016 commented

请问这个解决了吗，我也碰到同样的问题，在一个普通的检测数据集上，会出现RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at mmdet/ops/iou/src/convex_giou_kernel.cu:856，运行一段时间后就会随机出现

windyjunfeng commented

请问这个解决了吗，我也碰到同样的问题，在一个普通的检测数据集上，会出现RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at mmdet/ops/iou/src/convex_giou_kernel.cu:856，运行一段时间后就会随机出现

你好，可以检查下数据集的标注是否存在问题。我的是因为数据集标注本身存在问题，有的标注框宽度或高度为0，这种情况会导致计算GIOU时报错。