inference cascade_rcnn has error

Question

inference cascade_rcnn has error

lqyp0420 opened this issue 3 years ago · 0 comments

when i do a test, it has error like this.
trt_detector = create_wrap_detector(args.save_path, cfg_path, args.device)
image_path = args.img
result = inference_detector(trt_detector, image_path)

inference log
[10/06/2021-17:23:04] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.2.1
[10/06/2021-17:23:05] [TRT] [W] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.2.0
[10/06/2021-17:23:05] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.2.1
[10/06/2021-17:23:05] [TRT] [W] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.2.0
/home/lq/anaconda3/lib/python3.7/site-packages/mmdet/datasets/utils.py:68: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.
'data pipeline in your config file.', UserWarning)

进程已结束，退出代码为 139 (interrupted by signal 11: SIGSEGV)

mmdet2trt log:
[10/06/2021-18:24:35] [TRT] [I] [MemUsageChange] Init CUDA: CPU +316, GPU +0, now: CPU 3138, GPU 3735 (MiB)
Warning: Encountered known unsupported method torch.Tensor.new_tensor
Warning: Encountered known unsupported method torch.Tensor.new_tensor
Warning: Encountered known unsupported method torch.Tensor.new_tensor
Warning: Encountered known unsupported method torch.Tensor.new_tensor
Warning: Encountered known unsupported method torch.Tensor.new_tensor
Warning: Encountered known unsupported method torch.Tensor.new_tensor
[10/06/2021-18:24:39] [TRT] [I] [MemUsageSnapshot] Builder begin: CPU 3776 MiB, GPU 2705 MiB
[10/06/2021-18:24:41] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.2.1
[10/06/2021-18:24:41] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +287, GPU +158, now: CPU 4354, GPU 2863 (MiB)
[10/06/2021-18:24:41] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +405, GPU +170, now: CPU 4759, GPU 3033 (MiB)
[10/06/2021-18:24:41] [TRT] [W] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.2.0
[10/06/2021-18:24:41] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
[10/06/2021-18:24:58] [TRT] [I] Some tactics do not have sufficient workspace memory to run. Increasing workspace size may increase performance, please check verbose output.
[10/06/2021-18:27:22] [TRT] [I] [BlockAssignment] Algorithm Linear took 0.841195ms to assign 563 blocks to 563 nodes requiring 2687037004 bytes.
[10/06/2021-18:27:22] [TRT] [I] Total Activation Memory: -1607930292
[10/06/2021-18:27:22] [TRT] [I] Detected 1 inputs and 4 output network tensors.
[10/06/2021-18:27:22] [TRT] [I] Total Host Persistent Memory: 328928
[10/06/2021-18:27:22] [TRT] [I] Total Device Persistent Memory: 266317312
[10/06/2021-18:27:22] [TRT] [I] Total Scratch Memory: 180652032
[10/06/2021-18:27:22] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 559 MiB, GPU 23 MiB
[10/06/2021-18:27:24] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 2053.62ms to assign 51 blocks to 599 nodes requiring 335122447 bytes.
[10/06/2021-18:27:24] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.2.1
[10/06/2021-18:27:24] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 5410, GPU 3477 (MiB)
[10/06/2021-18:27:24] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 5410, GPU 3487 (MiB)
[10/06/2021-18:27:24] [TRT] [W] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.2.0
[10/06/2021-18:27:25] [TRT] [I] [MemUsageSnapshot] Builder end: CPU 5119 MiB, GPU 3455 MiB
[10/06/2021-18:27:25] [TRT] [I] [MemUsageSnapshot] ExecutionContext creation begin: CPU 5119 MiB, GPU 3455 MiB
[10/06/2021-18:27:25] [TRT] [W] TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.2.1
[10/06/2021-18:27:25] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 5119, GPU 3463 (MiB)
[10/06/2021-18:27:25] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 5119, GPU 3471 (MiB)
[10/06/2021-18:27:25] [TRT] [W] TensorRT was linked against cuDNN 8.2.1 but loaded cuDNN 8.2.0
[10/06/2021-18:27:25] [TRT] [I] [MemUsageSnapshot] ExecutionContext creation end: CPU 5120 MiB, GPU 4045 MiB

my config file:

model = dict(
type='CascadeRCNN',
pretrained='open-mmlab://resnext101_64x4d',
backbone=dict(
type='ResNeXt',
depth=101,
num_stages=4,
out_indices=(0, 1, 2, 3),
frozen_stages=1,
norm_cfg=dict(type='BN', requires_grad=True),
norm_eval=True,
style='pytorch',
groups=64,
base_width=4,
dcn=dict(type='DCN', deform_groups=1, fallback_on_stride=False),
stage_with_dcn=(False, True, True, True)),
neck=dict(
type='FPN',
in_channels=[256, 512, 1024, 2048],
out_channels=256,
num_outs=5),
rpn_head=dict(
type='RPNHead',
in_channels=256,
feat_channels=256,
anchor_generator=dict(
type='AnchorGenerator',
scales=[8],
ratios=[0.5, 1.0, 2],
strides=[4, 8, 16, -1, -1]),
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[1.0, 1.0, 1.0, 1.0]),
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
reg_decoded_bbox=True,
loss_bbox=dict(type='GIoULoss', loss_weight=5.0)),
roi_head=dict(
type='CascadeRoIHead',
num_stages=3,
mask_head=None,
stage_loss_weights=[1, 0.5, 0.25],
bbox_roi_extractor=dict(
type='SingleRoIExtractor',
roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),
out_channels=256,
featmap_strides=[4, 8, 16, 32]),
bbox_head=[
dict(
type='Shared2FCBBoxHead',
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=1,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
reg_class_agnostic=True,
loss_cls=dict(
type='CrossEntropyLoss',
use_sigmoid=False,
loss_weight=1.0),
reg_decoded_bbox=True,
loss_bbox=dict(type='GIoULoss', loss_weight=5.0)),
dict(
type='Shared2FCBBoxHead',
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=1,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.05, 0.05, 0.1, 0.1]),
reg_class_agnostic=True,
loss_cls=dict(
type='CrossEntropyLoss',
use_sigmoid=False,
loss_weight=1.0),
reg_decoded_bbox=True,
loss_bbox=dict(type='GIoULoss', loss_weight=5.0)),
dict(
type='Shared2FCBBoxHead',
in_channels=256,
fc_out_channels=1024,
roi_feat_size=7,
num_classes=1,
bbox_coder=dict(
type='DeltaXYWHBBoxCoder',
target_means=[0.0, 0.0, 0.0, 0.0],
target_stds=[0.033, 0.033, 0.067, 0.067]),
reg_class_agnostic=True,
loss_cls=dict(
type='CrossEntropyLoss',
use_sigmoid=False,
loss_weight=1.0),
reg_decoded_bbox=True,
loss_bbox=dict(type='GIoULoss', loss_weight=5.0))
]),
train_cfg=dict(
rpn=dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.6,
neg_iou_thr=0.3,
min_pos_iou=0.3,
match_low_quality=True,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=256,
pos_fraction=0.5,
neg_pos_ub=-1,
add_gt_as_proposals=False),
allowed_border=0,
pos_weight=-1,
debug=False),
rpn_proposal=dict(
nms_across_levels=False,
nms_pre=4000,
max_per_img=2000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=[
dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.5,
neg_iou_thr=0.5,
min_pos_iou=0.5,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False),
dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.6,
neg_iou_thr=0.6,
min_pos_iou=0.6,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False),
dict(
assigner=dict(
type='MaxIoUAssigner',
pos_iou_thr=0.7,
neg_iou_thr=0.7,
min_pos_iou=0.7,
match_low_quality=False,
ignore_iof_thr=-1),
sampler=dict(
type='RandomSampler',
num=512,
pos_fraction=0.25,
neg_pos_ub=-1,
add_gt_as_proposals=True),
pos_weight=-1,
debug=False)
],
stage_loss_weights=[1, 0.5, 0.25]),
test_cfg=dict(
rpn=dict(
nms_across_levels=False,
nms_pre=2000,
max_per_img=1000,
nms=dict(type='nms', iou_threshold=0.7),
min_bbox_size=0),
rcnn=dict(
score_thr=0.1,
nms=dict(type='soft_nms', iou_threshold=0.5),
max_per_img=100)))
classes = ('threat', )
data_root = '/home/lq/mmdet_torch/mmdetection-2.10.0/data/v12'
img_norm_cfg = dict(
mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Resize',
img_scale=[(2000, 360), (2000, 440)],
keep_ratio=True,
multiscale_mode='range'),
dict(
type='Albu',
transforms=[
dict(
type='RandomBrightnessContrast',
brightness_limit=0.1,
contrast_limit=0.1,
p=0.2)
]),
dict(type='RandomFlip', flip_ratio=0.5),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(760, 400),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=2,
workers_per_gpu=2,
train=dict(
type='CocoDataset',
classes=('threat', ),
ann_file=
'/home/lq/mmdet_torch/mmdetection-2.10.0/data/v12/annotations/train.json',
img_prefix=
'/home/lq/mmdet_torch/mmdetection-2.10.0/data/v12/train/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations', with_bbox=True),
dict(
type='Resize',
img_scale=[(2000, 360), (2000, 440)],
keep_ratio=True,
multiscale_mode='range'),
dict(type='RandomFlip', flip_ratio=0.5),
dict(
type='Albu',
transforms=[
dict(
type='RandomBrightnessContrast',
brightness_limit=0.1,
contrast_limit=0.1,
p=0.2)
]),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels'])
]),
val=dict(
type='CocoDataset',
classes=('threat', ),
ann_file=
'/home/lq/mmdet_torch/mmdetection-2.10.0/data/v12/annotations/val.json',
img_prefix=
'/home/lq/mmdet_torch/mmdetection-2.10.0/data/v12/val/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(760, 400),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]),
test=dict(
type='CocoDataset',
classes=('threat', ),
ann_file=
'/home/lq/mmdet_torch/mmdetection-2.10.0/data/v12/annotations/val.json',
img_prefix=
'/home/lq/mmdet_torch/mmdetection-2.10.0/data/v12/val/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(760, 400),
flip=False,
transforms=[
# dict(type='Resize', keep_ratio=True),
# dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]))
evaluation = dict(interval=1, metric='bbox')
optimizer = dict(
type='AdamW',
lr=0.0002,
weight_decay=0.0001,
paramwise_cfg=dict(
custom_keys=dict(
backbone=dict(lr_mult=0.1),
sampling_offsets=dict(lr_mult=0.1),
reference_points=dict(lr_mult=0.1))))
optimizer_config = dict(grad_clip=dict(max_norm=0.1, norm_type=2))
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=5000,
warmup_ratio=0.001,
step=[16, 19])
total_epochs = 20
checkpoint_config = dict(interval=1)
log_config = dict(
interval=100,
hooks=[dict(type='TextLoggerHook'),
dict(type='TensorboardLoggerHook')])
custom_hooks = [dict(type='NumClassCheckHook')]
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = '/home/lq/mmdet_torch/mmdetection-2.10.0/data/v12/epoch_20.pth'
resume_from = None
workflow = [('train', 1)]
work_dir = '../data/v12'
gpu_ids = range(0, 2)

thanks.

enviroment:

OS: Ubuntu 20.04
python_version: 3.7
pytorch_version: 1.9.1
cuda_version: cuda-11.1
cudnn_version: 8.2.0
tensorrt_version: 8.2
mmdetection_version: 2.10

Additional context
Add any other context about the problem here.