zf020114/DARDet

Uable to get repr for <class 'dict'>

shenhaibb opened this issue · 0 comments

shenhaibb commented

pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/henu/anaconda3/envs/dardet/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py (will have no effect)
/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/torch/cuda/init.py:106: UserWarning:
NVIDIA GeForce RTX 3070 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3070 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
fatal: not a git repository (or any parent up to mount point /media/henu)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
2022-10-16 13:31:40,380 - mmdet - INFO - Environment info:

sys.platform: linux
Python: 3.7.13 (default, Mar 29 2022, 02:18:16) [GCC 7.5.0]
CUDA available: True
GPU 0: NVIDIA GeForce RTX 3070
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.4.r11.4/compiler.30300941_0
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PyTorch: 1.9.0+cu102
PyTorch compiling details: PyTorch built with:

  • GCC 7.3
  • C++ Version: 201402
  • Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  • Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
  • OpenMP 201511 (a.k.a. OpenMP 4.5)
  • NNPACK is enabled
  • CPU capability usage: AVX2
  • CUDA Runtime 10.2
  • NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70
  • CuDNN 7.6.5
  • Magma 2.5.2
  • Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=7.6.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
    TorchVision: 0.10.0+cu102
    OpenCV: 4.6.0
    MMCV: 1.3.3
    MMCV Compiler: GCC 9.4
    MMCV CUDA Compiler: 11.4
    MMDetection: 2.13.0+

2022-10-16 13:31:43,812 - mmdet - INFO - Distributed training: False
2022-10-16 13:31:47,265 - mmdet - INFO - Config:
model = dict(
type='DARDet',
pretrained='torchvision://resnet50',
backbone=dict(
type='ResNet',
depth=50,
num_stages=4,
out_indices=(0, 1, 2, 3),
frozen_stages=1,
norm_cfg=dict(type='BN', requires_grad=True),
norm_eval=True,
style='pytorch'),
neck=dict(
type='FPN',
in_channels=[256, 512, 1024, 2048],
out_channels=256,
start_level=1,
add_extra_convs=True,
extra_convs_on_inputs=False,
num_outs=5,
relu_before_extra_convs=True),
bbox_head=dict(
type='DARDetHead',
num_classes=15,
in_channels=256,
stacked_convs=3,
feat_channels=256,
strides=[8, 16, 32, 64, 128],
center_sampling=False,
dcn_on_last_conv=False,
use_atss=True,
use_vfl=True,
loss_cls=dict(
type='VarifocalLoss',
use_sigmoid=True,
alpha=0.75,
gamma=2.0,
iou_weighted=True,
loss_weight=1.0),
loss_rbox=dict(type='RotatedIoULoss', loss_weight=1.5),
loss_rbox_refine=dict(type='RotatedIoULoss', loss_weight=2.0)),
train_cfg=dict(
assigner=dict(type='ATSSAssigner', topk=9),
allowed_border=-1,
pos_weight=-1,
debug=False),
test_cfg=dict(
rotate_test=True,
nms_pre=2000,
min_bbox_size=0,
score_thr=0.05,
nms=dict(type='nms', iou_threshold=0.1),
max_per_img=1500))
dataset_type = 'DotaKDataset'
data_root = '/media/henu/新加卷1/rxh/DARDet-master/data/dota15_1024/'
img_norm_cfg = dict(
mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='LoadAnnotations',
with_bbox=True,
with_mask=True,
poly2mask=False),
dict(type='Resize', img_scale=(1024, 1024), keep_ratio=True),
dict(
type='RandomFlip',
direction=['horizontal', 'vertical', 'diagonal'],
flip_ratio=0.5),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks'])
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1024, 1024),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=2,
workers_per_gpu=2,
train=dict(
delete=True,
type='ClassBalancedDataset',
oversample_thr=0.06,
dataset=dict(
type='DotaKDataset',
ann_file=
'/media/henu/新加卷1/rxh/DARDet-master/data/dota15_1024/trainval1024/DOTA1_5_trainval1024.json',
img_prefix=
'/media/henu/新加卷1/rxh/DARDet-master/data/dota15_1024/trainval1024/images/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='LoadAnnotations',
with_bbox=True,
with_mask=True,
poly2mask=False),
dict(type='Resize', img_scale=(1024, 1024), keep_ratio=True),
dict(
type='RandomFlip',
direction=['horizontal', 'vertical', 'diagonal'],
flip_ratio=0.5),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(
type='Collect',
keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks'])
])),
val=dict(
type='DotaKDataset',
ann_file=
'/media/henu/新加卷1/rxh/DARDet-master/data/dota15_1024/trainval1024/DOTA1_5_trainval1024.json',
img_prefix=
'/media/henu/新加卷1/rxh/DARDet-master/data/dota15_1024/trainval1024/images/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1024, 1024),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img'])
])
]),
test=dict(
type='DotaKDataset',
ann_file=
'/media/henu/新加卷1/rxh/DARDet-master/data/dota15_1024/test1024/DOTA1_5_test1024.json',
img_prefix=
'/media/henu/新加卷1/rxh/DARDet-master/data/dota15_1024/test1024/images/',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(1024, 1024),
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size_divisor=32),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img'])
])
]))
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
optimizer_config = dict(grad_clip=None)
lr_config = dict(
policy='step',
warmup='linear',
warmup_iters=500,
warmup_ratio=0.001,
step=[8, 11])
runner = dict(type='EpochBasedRunner', max_epochs=12)
work_dir = '/media/henu/新加卷1/rxh/DARDet-master/workdir/DARDet_r50_DCN_rotate'
load_from = None
resume_from = None
evaluation = dict(
interval=3,
metric='bbox',
eval_dir='/media/henu/新加卷1/rxh/DARDet-master/workdir/DARDet_r50_DCN_rotate',
gt_dir='/media/zf/E/Dataset/dota_1024_s2anet2/valGTtxt/')
checkpoint_config = dict(interval=1)
log_config = dict(interval=50, hooks=[dict(type='TextLoggerHook')])
custom_hooks = [dict(type='NumClassCheckHook')]
dist_params = dict(backend='nccl')
log_level = 'INFO'
workflow = [('train', 1)]
gpu_ids = [0]
/media/henu/新加卷1/rxh/DARDet-master/mmdet/models/backbones/resnet.py:400: UserWarning: DeprecationWarning: pretrained is a deprecated, please use "init_cfg" instead
warnings.warn('DeprecationWarning: pretrained is a deprecated, '
2022-10-16 13:31:47,479 - mmcv - INFO - load model from: torchvision://resnet50
2022-10-16 13:31:47,479 - mmcv - INFO - Use load_from_torchvision loader
2022-10-16 13:31:47,558 - mmcv - WARNING - The model and loaded state dict do not match exactly
unexpected key in source state_dict: fc.weight, fc.bias
/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/mmcv/cnn/utils/weight_init.py:100: UserWarning: init_cfg without layer key, if you do not define override key either, this init_cfg will do nothing
'init_cfg without layer key, if you do not define override'
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
fatal: not a git repository (or any parent up to mount point /media/henu)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
2022-10-16 13:31:48,151 - mmdet - INFO - Start running, host: henu@henu-Super-Server, work_dir: /media/henu/新加卷1/rxh/DARDet-master/workdir/DARDet_r50_DCN_rotate
2022-10-16 13:31:48,151 - mmdet - INFO - workflow: [('train', 1)], max: 12 epochs
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/henu/anaconda3/envs/dardet/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py (will have no effect)
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/henu/anaconda3/envs/dardet/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py (will have no effect)
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/henu/anaconda3/envs/dardet/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py (will have no effect)
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/henu/anaconda3/envs/dardet/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py (will have no effect)
pydev debugger: warning: trying to add breakpoint to file that does not exist: /home/henu/anaconda3/envs/dardet/lib/python3.8/site-packages/mmcv/parallel/data_parallel.py (will have no effect)
Traceback (most recent call last):
File "/home/henu/.pycharm_helpers/pydev/pydevd.py", line 1483, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/henu/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/media/henu/新加卷1/rxh/DARDet-master/tools/train.py", line 191, in
main()
File "/media/henu/新加卷1/rxh/DARDet-master/tools/train.py", line 188, in main
meta=meta)
File "/media/henu/新加卷1/rxh/DARDet-master/mmdet/apis/train.py", line 175, in train_detector
runner.run(data_loaders, cfg.workflow)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
self.run_iter(data_batch, train_mode=True, **kwargs)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 30, in run_iter
**kwargs)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
return self.module.train_step(*inputs[0], **kwargs[0])
File "/media/henu/新加卷1/rxh/DARDet-master/mmdet/models/detectors/base.py", line 237, in train_step
losses = self(**data)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 95, in new_func
return old_func(*args, **kwargs)
File "/media/henu/新加卷1/rxh/DARDet-master/mmdet/models/detectors/base.py", line 171, in forward
return self.forward_train(img, img_metas, **kwargs)
File "/media/henu/新加卷1/rxh/DARDet-master/mmdet/models/detectors/dardet.py", line 143, in forward_train
x = self.extract_feat(img)
File "/media/henu/新加卷1/rxh/DARDet-master/mmdet/models/detectors/single_stage.py", line 37, in extract_feat
x = self.backbone(img)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/media/henu/新加卷1/rxh/DARDet-master/mmdet/models/backbones/resnet.py", line 637, in forward
x = self.relu(x)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in call_impl
return forward_call(*input, **kwargs)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 102, in forward
return F.relu(input, inplace=self.inplace)
File "/home/henu/anaconda3/envs/dardet/lib/python3.7/site-packages/torch/nn/functional.py", line 1296, in relu
result = torch.relu(input)
RuntimeError: CUDA error: no kernel image is available for execution on the device

After debugging, I found the variable 'data' in mmdet/models/detectors/base.py, Line237 shows that 'Uable to get repr for <class 'dict>'
Specially, the variable 'img' in 'data' shows that 'Uable to get repr for <class 'torch.Tensor'>
and the variable 'gt_bboxes' in 'data' shows that 'Uable to get repr for <class 'list'>

What's the probable reason?