xmed-lab/URN

RuntimeError: DataLoader worker (pid(s) 58594) exited unexpectedly

Dream-ai opened this issue · 3 comments

Dream-ai commented

I don't know why this is？
please reply, thanks

Eli-YiLi commented

When it happened?
If it was at the beginning, check the image and gt.
If happened after iterations, may be the device problem. You can try to reduce the worker number.

Dream-ai commented

Thanks for your reply, but now there is a new question：
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
2022-07-13 09:43:16,604 - mmseg - INFO - Environment info:

sys.platform: linux
Python: 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0]
CUDA available: True
GPU 0: Tesla V100-PCIE-16GB
CUDA_HOME: /apps/compilers/nvidia/cuda/cuda-11.3
NVCC: Build cuda_11.3.r11.3/compiler.29920130_0
GCC: gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)
PyTorch: 1.7.0
PyTorch compiling details: PyTorch built with:

  • GCC 7.3
  • C++ Version: 201402
  • Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
  • Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
  • OpenMP 201511 (a.k.a. OpenMP 4.5)
  • NNPACK is enabled
  • CPU capability usage: AVX2
  • CUDA Runtime 10.1
  • NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  • CuDNN 7.6.3
  • Magma 2.5.2
  • Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.8.0
OpenCV: 4.6.0
MMCV: 1.2.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: 10.1
MMSegmentation: 0.8.0+

2022-07-13 09:43:16,605 - mmseg - INFO - Distributed training: True
2022-07-13 09:43:16,880 - mmseg - INFO - Config:
norm_cfg = dict(type='SyncBN', requires_grad=True)
model = dict(
type='EncoderDecoder',
pretrained=
'/home/imu_zhengyuan/zhx/URN-main/models/res2net101_v1b_26w_4s-0812c246.pth',
decode_head=dict(
type='PSPHead',
in_channels=2048,
in_index=3,
channels=512,
pool_scales=(1, 2, 3, 6),
dropout_ratio=0.1,
num_classes=21,
norm_cfg=dict(type='SyncBN', requires_grad=True),
align_corners=False,
loss_decode=dict(
type='CrossEntropyLoss',
use_sigmoid=False,
loss_weight=1.0,
weight_thresh=0.05)),
auxiliary_head=dict(
type='FCNHead',
in_channels=1024,
in_index=2,
channels=256,
num_convs=1,
concat_input=False,
dropout_ratio=0.1,
num_classes=21,
norm_cfg=dict(type='SyncBN', requires_grad=True),
align_corners=False,
loss_decode=dict(
type='CrossEntropyLoss',
use_sigmoid=False,
loss_weight=0.4,
weight_thresh=0.05)),
backbone=dict(
type='Res2Net',
layers=[3, 4, 23, 3],
out_indices=(0, 1, 2, 3),
strides=(1, 2, 1, 1),
dilations=(1, 1, 2, 4),
norm_eval=False))
train_cfg = dict()
test_cfg = dict(
mode='slide', crop_size=(512, 512), stride=(480, 480), crf=True)
dataset_type = 'PascalVOCDataset'
data_root = '/home/imu_zhengyuan/zhx/URN-main/data/voc12/VOC2012/'
img_norm_cfg = dict(
mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
crop_size = (512, 512)
train_pipeline = [
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations'),
dict(type='Resize', img_scale=(2048, 512), ratio_range=(0.5, 2.0)),
dict(type='RandomCrop', crop_size=(512, 512), cat_max_ratio=0.75),
dict(type='RandomFlip', prob=0.5),
dict(type='PhotoMetricDistortion'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_semantic_seg'])
]
test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(2048, 512),
img_ratios=[0.5, 0.75, 1.0, 1.25, 1.5, 1.75],
flip=True,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]
data = dict(
samples_per_gpu=2,
workers_per_gpu=4,
train=dict(
type='PascalVOCDataset',
data_root='/home/imu_zhengyuan/zhx/URN-main/data/voc12/VOC2012/',
img_dir='JPEGImages',
ann_dir='urn_r2n',
split='ImageSets/Segmentation/trainaug.txt',
pipeline=[
dict(type='LoadImageFromFile'),
dict(type='LoadAnnotations'),
dict(type='Resize', img_scale=(2048, 512), ratio_range=(0.5, 2.0)),
dict(type='RandomCrop', crop_size=(512, 512), cat_max_ratio=0.75),
dict(type='RandomFlip', prob=0.5),
dict(type='PhotoMetricDistortion'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='Pad', size=(512, 512), pad_val=0, seg_pad_val=255),
dict(type='DefaultFormatBundle'),
dict(type='Collect', keys=['img', 'gt_semantic_seg'])
]),
val=dict(
type='PascalVOCDataset',
data_root='/home/imu_zhengyuan/zhx/URN-main/data/voc12/VOC2012/',
img_dir='JPEGImages',
ann_dir='SegmentationClass',
split='ImageSets/Segmentation/val.txt',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(2048, 512),
img_ratios=[0.5, 0.75, 1.0, 1.25, 1.5, 1.75],
flip=True,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]),
test=dict(
type='PascalVOCDataset',
data_root='/home/imu_zhengyuan/zhx/URN-main/data/voc12/VOC2012/',
img_dir='JPEGImages',
ann_dir='SegmentationClass',
split='ImageSets/Segmentation/val.txt',
pipeline=[
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=(2048, 512),
img_ratios=[0.5, 0.75, 1.0, 1.25, 1.5, 1.75],
flip=True,
transforms=[
dict(type='Resize', keep_ratio=True),
dict(type='RandomFlip'),
dict(
type='Normalize',
mean=[123.675, 116.28, 103.53],
std=[58.395, 57.12, 57.375],
to_rgb=True),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img'])
])
]))
log_config = dict(
interval=50, hooks=[dict(type='TextLoggerHook', by_epoch=False)])
dist_params = dict(backend='nccl')
log_level = 'INFO'
load_from = None
resume_from = None
workflow = [('train', 1)]
cudnn_benchmark = True
optimizer = dict(type='SGD', lr=0.005, momentum=0.9, weight_decay=0.0005)
optimizer_config = dict()
lr_config = dict(policy='poly', power=0.9, min_lr=0.0001, by_epoch=False)
runner = dict(type='IterBasedRunner', max_iters=20000)
checkpoint_config = dict(by_epoch=False, interval=2000)
evaluation = dict(interval=2000, metric='mIoU')
work_dir = './work_dirs/pspnet_res2net_20k_voc12aug_urn'
gpu_ids = range(0, 1)

2022-07-13 09:43:18,752 - mmseg - WARNING - The model and loaded state dict do not match exactly

unexpected key in source state_dict: fc.weight, fc.bias

2022-07-13 09:43:18,754 - mmseg - INFO - EncoderDecoder(
(backbone): Res2Net(
(conv1): Sequential(
(0): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(1): SyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
(3): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(4): SyncBatchNorm(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU(inplace=True)
(6): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
)
(bn1): SyncBatchNorm(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU()
(maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
(layer1): Sequential(
(0): Bottle2neck(
(conv1): Conv2d(64, 104, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
(convs): ModuleList(
(0): Conv2d(26, 26, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(1): Conv2d(26, 26, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(2): Conv2d(26, 26, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(26, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(26, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(26, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(104, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn3): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(downsample): Sequential(
(0): AvgPool2d(kernel_size=1, stride=1, padding=0)
(1): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(2): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): Bottle2neck(
(conv1): Conv2d(256, 104, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(26, 26, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(1): Conv2d(26, 26, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(2): Conv2d(26, 26, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(26, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(26, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(26, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(104, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn3): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(2): Bottle2neck(
(conv1): Conv2d(256, 104, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(26, 26, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(1): Conv2d(26, 26, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(2): Conv2d(26, 26, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(26, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(26, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(26, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(104, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn3): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
)
(layer2): Sequential(
(0): Bottle2neck(
(conv1): Conv2d(256, 208, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn1): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=3, stride=2, padding=1)
(convs): ModuleList(
(0): Conv2d(52, 52, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(1): Conv2d(52, 52, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
(2): Conv2d(52, 52, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(208, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn3): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(downsample): Sequential(
(0): AvgPool2d(kernel_size=2, stride=2, padding=0)
(1): Conv2d(256, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(2): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): Bottle2neck(
(conv1): Conv2d(512, 208, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn1): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(52, 52, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(1): Conv2d(52, 52, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(2): Conv2d(52, 52, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(208, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn3): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(2): Bottle2neck(
(conv1): Conv2d(512, 208, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn1): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(52, 52, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(1): Conv2d(52, 52, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(2): Conv2d(52, 52, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(208, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn3): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(3): Bottle2neck(
(conv1): Conv2d(512, 208, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn1): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(52, 52, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(1): Conv2d(52, 52, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(2): Conv2d(52, 52, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(52, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(208, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn3): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
)
(layer3): Sequential(
(0): Bottle2neck(
(conv1): Conv2d(512, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(downsample): Sequential(
(0): AvgPool2d(kernel_size=1, stride=1, padding=0)
(1): Conv2d(512, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
(2): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(2): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(3): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(4): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(5): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(6): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(7): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(8): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(9): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(10): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(11): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(12): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(13): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(14): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(15): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(16): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(17): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(18): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(19): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(20): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(21): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(22): Bottle2neck(
(conv1): Conv2d(1024, 416, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn1): SyncBatchNorm(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(1): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
(2): Conv2d(104, 104, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(104, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(416, 1024, kernel_size=(1, 1), stride=(1, 1), dilation=(2, 2), bias=False)
(bn3): SyncBatchNorm(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
)
(layer4): Sequential(
(0): Bottle2neck(
(conv1): Conv2d(1024, 832, kernel_size=(1, 1), stride=(1, 1), dilation=(4, 4), bias=False)
(bn1): SyncBatchNorm(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(pool): AvgPool2d(kernel_size=3, stride=1, padding=1)
(convs): ModuleList(
(0): Conv2d(208, 208, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
(1): Conv2d(208, 208, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
(2): Conv2d(208, 208, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(832, 2048, kernel_size=(1, 1), stride=(1, 1), dilation=(4, 4), bias=False)
(bn3): SyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
(downsample): Sequential(
(0): AvgPool2d(kernel_size=1, stride=1, padding=0)
(1): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
(2): SyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
)
(1): Bottle2neck(
(conv1): Conv2d(2048, 832, kernel_size=(1, 1), stride=(1, 1), dilation=(4, 4), bias=False)
(bn1): SyncBatchNorm(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(208, 208, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
(1): Conv2d(208, 208, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
(2): Conv2d(208, 208, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(832, 2048, kernel_size=(1, 1), stride=(1, 1), dilation=(4, 4), bias=False)
(bn3): SyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
(2): Bottle2neck(
(conv1): Conv2d(2048, 832, kernel_size=(1, 1), stride=(1, 1), dilation=(4, 4), bias=False)
(bn1): SyncBatchNorm(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(convs): ModuleList(
(0): Conv2d(208, 208, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
(1): Conv2d(208, 208, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
(2): Conv2d(208, 208, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
)
(bns): ModuleList(
(0): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): SyncBatchNorm(208, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
)
(conv3): Conv2d(832, 2048, kernel_size=(1, 1), stride=(1, 1), dilation=(4, 4), bias=False)
(bn3): SyncBatchNorm(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(relu): ReLU(inplace=True)
)
)
)
(decode_head): PSPHead(
input_transform=None, ignore_index=255, align_corners=False
(loss_decode): CrossEntropyLoss()
(conv_seg): Conv2d(512, 21, kernel_size=(1, 1), stride=(1, 1))
(dropout): Dropout2d(p=0.1, inplace=False)
(psp_modules): PPM(
(0): Sequential(
(0): AdaptiveAvgPool2d(output_size=1)
(1): ConvModule(
(conv): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activate): ReLU(inplace=True)
)
)
(1): Sequential(
(0): AdaptiveAvgPool2d(output_size=2)
(1): ConvModule(
(conv): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activate): ReLU(inplace=True)
)
)
(2): Sequential(
(0): AdaptiveAvgPool2d(output_size=3)
(1): ConvModule(
(conv): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activate): ReLU(inplace=True)
)
)
(3): Sequential(
(0): AdaptiveAvgPool2d(output_size=6)
(1): ConvModule(
(conv): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
(bn): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activate): ReLU(inplace=True)
)
)
)
(bottleneck): ConvModule(
(conv): Conv2d(4096, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): SyncBatchNorm(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activate): ReLU(inplace=True)
)
)
(auxiliary_head): FCNHead(
input_transform=None, ignore_index=255, align_corners=False
(loss_decode): CrossEntropyLoss()
(conv_seg): Conv2d(256, 21, kernel_size=(1, 1), stride=(1, 1))
(dropout): Dropout2d(p=0.1, inplace=False)
(convs): Sequential(
(0): ConvModule(
(conv): Conv2d(1024, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
(bn): SyncBatchNorm(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(activate): ReLU(inplace=True)
)
)
)
)
2022-07-13 09:43:18,772 - mmseg - INFO - Loaded 10582 images
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
2022-07-13 09:43:19,425 - mmseg - INFO - Loaded 1449 images
2022-07-13 09:43:19,425 - mmseg - INFO - Start running, host: imu_zhengyuan@r02c12s03, work_dir: /home/imu_zhengyuan/zhx/URN-main/work_dirs/pspnet_res2net_20k_voc12aug_urn
2022-07-13 09:43:19,425 - mmseg - INFO - workflow: [('train', 1)], max: 20000 iters
2022-07-13 09:44:00,086 - mmseg - INFO - Iter [50/20000] lr: 4.989e-03, eta: 3:09:44, time: 0.571, data_time: 0.010, memory: 8672, decode.loss_seg: 1.3815, decode.acc_seg: 54.1316, aux.loss_seg: 0.5912, aux.acc_seg: 55.4183, loss: 1.9727
2022-07-13 09:44:21,508 - mmseg - INFO - Iter [100/20000] lr: 4.978e-03, eta: 2:45:41, time: 0.428, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3725, decode.acc_seg: 54.1527, aux.loss_seg: 0.5239, aux.acc_seg: 55.8106, loss: 1.8964
2022-07-13 09:44:42,932 - mmseg - INFO - Iter [150/20000] lr: 4.967e-03, eta: 2:37:25, time: 0.428, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3100, decode.acc_seg: 49.1515, aux.loss_seg: 0.4790, aux.acc_seg: 52.8697, loss: 1.7890
2022-07-13 09:45:04,325 - mmseg - INFO - Iter [200/20000] lr: 4.956e-03, eta: 2:33:04, time: 0.428, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2577, decode.acc_seg: 53.6081, aux.loss_seg: 0.4646, aux.acc_seg: 55.4392, loss: 1.7223
2022-07-13 09:45:25,711 - mmseg - INFO - Iter [250/20000] lr: 4.945e-03, eta: 2:30:18, time: 0.428, data_time: 0.004, memory: 8672, decode.loss_seg: 1.4712, decode.acc_seg: 52.1280, aux.loss_seg: 0.5523, aux.acc_seg: 53.3996, loss: 2.0235
2022-07-13 09:45:47,107 - mmseg - INFO - Iter [300/20000] lr: 4.934e-03, eta: 2:28:21, time: 0.428, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2138, decode.acc_seg: 52.5085, aux.loss_seg: 0.4622, aux.acc_seg: 53.8356, loss: 1.6760
2022-07-13 09:46:08,442 - mmseg - INFO - Iter [350/20000] lr: 4.923e-03, eta: 2:26:48, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.4514, decode.acc_seg: 50.1414, aux.loss_seg: 0.5332, aux.acc_seg: 50.7504, loss: 1.9846
2022-07-13 09:46:29,876 - mmseg - INFO - Iter [400/20000] lr: 4.912e-03, eta: 2:25:37, time: 0.429, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3240, decode.acc_seg: 50.5130, aux.loss_seg: 0.4780, aux.acc_seg: 51.8052, loss: 1.8021
2022-07-13 09:46:51,304 - mmseg - INFO - Iter [450/20000] lr: 4.901e-03, eta: 2:24:37, time: 0.429, data_time: 0.005, memory: 8672, decode.loss_seg: 1.2666, decode.acc_seg: 56.4055, aux.loss_seg: 0.4730, aux.acc_seg: 57.5181, loss: 1.7396
2022-07-13 09:47:12,729 - mmseg - INFO - Iter [500/20000] lr: 4.890e-03, eta: 2:23:45, time: 0.429, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2310, decode.acc_seg: 55.7776, aux.loss_seg: 0.4693, aux.acc_seg: 56.8210, loss: 1.7002
2022-07-13 09:47:34,071 - mmseg - INFO - Iter [550/20000] lr: 4.879e-03, eta: 2:22:56, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.5322, decode.acc_seg: 51.0567, aux.loss_seg: 0.5923, aux.acc_seg: 51.0930, loss: 2.1245
2022-07-13 09:47:55,410 - mmseg - INFO - Iter [600/20000] lr: 4.868e-03, eta: 2:22:11, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3260, decode.acc_seg: 59.7565, aux.loss_seg: 0.5044, aux.acc_seg: 60.2340, loss: 1.8304
2022-07-13 09:48:16,733 - mmseg - INFO - Iter [650/20000] lr: 4.857e-03, eta: 2:21:29, time: 0.426, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2470, decode.acc_seg: 58.0231, aux.loss_seg: 0.4696, aux.acc_seg: 58.5524, loss: 1.7167
2022-07-13 09:48:38,069 - mmseg - INFO - Iter [700/20000] lr: 4.846e-03, eta: 2:20:50, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2786, decode.acc_seg: 56.9571, aux.loss_seg: 0.4740, aux.acc_seg: 57.1457, loss: 1.7525
2022-07-13 09:48:59,378 - mmseg - INFO - Iter [750/20000] lr: 4.835e-03, eta: 2:20:14, time: 0.426, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3741, decode.acc_seg: 51.3448, aux.loss_seg: 0.5306, aux.acc_seg: 51.7686, loss: 1.9047
2022-07-13 09:49:20,725 - mmseg - INFO - Iter [800/20000] lr: 4.823e-03, eta: 2:19:40, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.1703, decode.acc_seg: 54.6150, aux.loss_seg: 0.4550, aux.acc_seg: 54.8517, loss: 1.6253
2022-07-13 09:49:42,089 - mmseg - INFO - Iter [850/20000] lr: 4.812e-03, eta: 2:19:07, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2358, decode.acc_seg: 50.0286, aux.loss_seg: 0.4795, aux.acc_seg: 50.3952, loss: 1.7154
2022-07-13 09:50:03,459 - mmseg - INFO - Iter [900/20000] lr: 4.801e-03, eta: 2:18:37, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3101, decode.acc_seg: 53.9602, aux.loss_seg: 0.5157, aux.acc_seg: 53.8506, loss: 1.8257
2022-07-13 09:50:24,776 - mmseg - INFO - Iter [950/20000] lr: 4.790e-03, eta: 2:18:06, time: 0.426, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3940, decode.acc_seg: 52.2593, aux.loss_seg: 0.5454, aux.acc_seg: 52.2852, loss: 1.9394
2022-07-13 09:50:46,020 - mmseg - INFO - Exp name: pspnet_res2net_20k_voc12aug_urn.py
2022-07-13 09:50:46,020 - mmseg - INFO - Iter [1000/20000] lr: 4.779e-03, eta: 2:17:34, time: 0.425, data_time: 0.004, memory: 8672, decode.loss_seg: 1.4098, decode.acc_seg: 53.9361, aux.loss_seg: 0.5444, aux.acc_seg: 53.8450, loss: 1.9542
2022-07-13 09:51:07,388 - mmseg - INFO - Iter [1050/20000] lr: 4.768e-03, eta: 2:17:06, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3503, decode.acc_seg: 50.4923, aux.loss_seg: 0.5220, aux.acc_seg: 50.9986, loss: 1.8724
2022-07-13 09:51:28,719 - mmseg - INFO - Iter [1100/20000] lr: 4.757e-03, eta: 2:16:38, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2335, decode.acc_seg: 53.1151, aux.loss_seg: 0.4909, aux.acc_seg: 53.3112, loss: 1.7243
2022-07-13 09:51:50,093 - mmseg - INFO - Iter [1150/20000] lr: 4.746e-03, eta: 2:16:11, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2979, decode.acc_seg: 52.2914, aux.loss_seg: 0.5125, aux.acc_seg: 52.3643, loss: 1.8104
2022-07-13 09:52:11,400 - mmseg - INFO - Iter [1200/20000] lr: 4.735e-03, eta: 2:15:44, time: 0.426, data_time: 0.004, memory: 8672, decode.loss_seg: 1.1889, decode.acc_seg: 56.3883, aux.loss_seg: 0.4746, aux.acc_seg: 55.8908, loss: 1.6635
2022-07-13 09:52:32,710 - mmseg - INFO - Iter [1250/20000] lr: 4.724e-03, eta: 2:15:17, time: 0.426, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3048, decode.acc_seg: 55.8444, aux.loss_seg: 0.5123, aux.acc_seg: 55.9071, loss: 1.8171
2022-07-13 09:52:54,017 - mmseg - INFO - Iter [1300/20000] lr: 4.713e-03, eta: 2:14:50, time: 0.426, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3150, decode.acc_seg: 56.4902, aux.loss_seg: 0.5120, aux.acc_seg: 56.3866, loss: 1.8271
2022-07-13 09:53:15,323 - mmseg - INFO - Iter [1350/20000] lr: 4.702e-03, eta: 2:14:24, time: 0.426, data_time: 0.005, memory: 8672, decode.loss_seg: 1.2056, decode.acc_seg: 52.2403, aux.loss_seg: 0.4816, aux.acc_seg: 52.4360, loss: 1.6872
2022-07-13 09:53:36,662 - mmseg - INFO - Iter [1400/20000] lr: 4.690e-03, eta: 2:13:59, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2012, decode.acc_seg: 53.8220, aux.loss_seg: 0.4784, aux.acc_seg: 54.5836, loss: 1.6796
2022-07-13 09:53:57,991 - mmseg - INFO - Iter [1450/20000] lr: 4.679e-03, eta: 2:13:33, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.1250, decode.acc_seg: 60.5095, aux.loss_seg: 0.4463, aux.acc_seg: 60.5863, loss: 1.5713
2022-07-13 09:54:19,396 - mmseg - INFO - Iter [1500/20000] lr: 4.668e-03, eta: 2:13:09, time: 0.428, data_time: 0.005, memory: 8672, decode.loss_seg: 1.3107, decode.acc_seg: 57.5408, aux.loss_seg: 0.5130, aux.acc_seg: 57.7037, loss: 1.8237
2022-07-13 09:54:40,720 - mmseg - INFO - Iter [1550/20000] lr: 4.657e-03, eta: 2:12:45, time: 0.426, data_time: 0.005, memory: 8672, decode.loss_seg: 1.3367, decode.acc_seg: 53.2623, aux.loss_seg: 0.5148, aux.acc_seg: 53.2135, loss: 1.8516
2022-07-13 09:55:02,148 - mmseg - INFO - Iter [1600/20000] lr: 4.646e-03, eta: 2:12:21, time: 0.429, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2944, decode.acc_seg: 52.9241, aux.loss_seg: 0.5086, aux.acc_seg: 52.9352, loss: 1.8029
2022-07-13 09:55:23,502 - mmseg - INFO - Iter [1650/20000] lr: 4.635e-03, eta: 2:11:57, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.1610, decode.acc_seg: 53.5977, aux.loss_seg: 0.4588, aux.acc_seg: 53.8807, loss: 1.6198
2022-07-13 09:55:44,881 - mmseg - INFO - Iter [1700/20000] lr: 4.624e-03, eta: 2:11:33, time: 0.428, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3522, decode.acc_seg: 57.4431, aux.loss_seg: 0.5248, aux.acc_seg: 57.3154, loss: 1.8771
2022-07-13 09:56:06,286 - mmseg - INFO - Iter [1750/20000] lr: 4.613e-03, eta: 2:11:10, time: 0.428, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3023, decode.acc_seg: 54.0453, aux.loss_seg: 0.5157, aux.acc_seg: 53.9612, loss: 1.8180
2022-07-13 09:56:27,658 - mmseg - INFO - Iter [1800/20000] lr: 4.601e-03, eta: 2:10:47, time: 0.427, data_time: 0.005, memory: 8672, decode.loss_seg: 1.3982, decode.acc_seg: 54.4469, aux.loss_seg: 0.5554, aux.acc_seg: 54.4404, loss: 1.9537
2022-07-13 09:56:48,997 - mmseg - INFO - Iter [1850/20000] lr: 4.590e-03, eta: 2:10:23, time: 0.427, data_time: 0.004, memory: 8672, decode.loss_seg: 1.3316, decode.acc_seg: 55.1616, aux.loss_seg: 0.5251, aux.acc_seg: 55.5972, loss: 1.8567
2022-07-13 09:57:10,264 - mmseg - INFO - Iter [1900/20000] lr: 4.579e-03, eta: 2:09:59, time: 0.425, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2509, decode.acc_seg: 54.3581, aux.loss_seg: 0.5051, aux.acc_seg: 54.3940, loss: 1.7560
2022-07-13 09:57:31,525 - mmseg - INFO - Iter [1950/20000] lr: 4.568e-03, eta: 2:09:35, time: 0.425, data_time: 0.004, memory: 8672, decode.loss_seg: 1.2188, decode.acc_seg: 57.2662, aux.loss_seg: 0.4870, aux.acc_seg: 57.4907, loss: 1.7058
2022-07-13 09:57:52,862 - mmseg - INFO - Saving checkpoint at 2000 iterations
Traceback (most recent call last):
File "tools/train.py", line 161, in
main()
File "tools/train.py", line 150, in main
train_segmentor(
File "/home/imu_zhengyuan/zhx/URN-main/mmseg/apis/train.py", line 116, in train_segmentor
runner.run(data_loaders, cfg.workflow)
File "/home/imu_zhengyuan/.conda/envs/urn/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 130, in run
iter_runner(iter_loaders[i], **kwargs)
File "/home/imu_zhengyuan/.conda/envs/urn/lib/python3.8/site-packages/mmcv/runner/iter_based_runner.py", line 66, in train
self.call_hook('after_train_iter')
File "/home/imu_zhengyuan/.conda/envs/urn/lib/python3.8/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook
getattr(hook, fn_name)(self)
File "/home/imu_zhengyuan/zhx/URN-main/mmseg/core/evaluation/eval_hooks.py", line 85, in after_train_iter
results = multi_gpu_test(
File "/home/imu_zhengyuan/zhx/URN-main/mmseg/apis/test.py", line 98, in multi_gpu_test
result = model(return_loss=False, rescale=True, **data)
File "/home/imu_zhengyuan/.conda/envs/urn/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in call_impl
result = self.forward(*input, **kwargs)
File "/home/imu_zhengyuan/.conda/envs/urn/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 606, in forward
if self.reducer.rebuild_buckets():
RuntimeError: replicas[0].size() == rebuilt_param_indices.size() INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1603729009598/work/torch/csrc/distributed/c10d/reducer.cpp":1326, please report a bug to PyTorch. rebuilt parameter indices size is not same as original model parameters size.538 versus 1076000
srun: error: r02c12s03: task 0: Exited with exit code 1
I don't know if it's the wrong version of pytorch，Looking forward to your reply！！！

Eli-YiLi commented

Sorry for late reply.
The pytorch version may be the reason: pytorch/pytorch#47050