/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda [](int)->auto::operator()(int)->auto: block: [0,0,0], thread: [121,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
zcl912 opened this issue · 10 comments
hello, when i train the model on 4 GPU, i met the following error, if train it on 1 gpu, the error disappear:
[04/08 14:39:50 c2.utils.dump.events]: eta: 4:24:06 iter: 6960/22500 total_loss: 0.748 loss_cls: 0.333 loss_box_reg: 0.420 time: 1.0219 data_time: 0.6507 lr: 0.010000 max_mem: 5233M
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [121,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
ERROR [04/08 14:39:54 c2.engine.base_runner]: Exception during training:
Traceback (most recent call last):
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 84, in train
self.run_step()
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 185, in run_step
loss_dict = self.model(data)
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "../yolof_base/yolof.py", line 135, in forward
pred_logits, pred_anchor_deltas)
File "../yolof_base/yolof.py", line 216, in losses
pred_class_logits[valid_idxs],
RuntimeError: copy_if failed to synchronize: device-side assert triggered
[04/08 14:39:54 c2.engine.hooks]: Overall training speed: 6961 iterations in 1:58:34 (1.0221 s / it)
[04/08 14:39:54 c2.engine.hooks]: Total training time: 2:04:42 (0:06:08 on hooks)
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:764)
Could you provide the full log file?
Could you provide the full log file?
yeah, the full log is as follw:
[04/08 19:21:13 c2.utils.dump.events]: eta: 5:54:29 iter: 1580/22500 total_loss: 0.915 loss_cls: 0.425 loss_box_reg: 0.483 time: 0.9901 data_time: 0.6297 lr: 0.026332 max_mem: 5223M
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [20,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:764)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7f1a20023193 in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: + 0x17f66 (0x7f1a20260f66 in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: + 0x19cbd (0x7f1a20262cbd in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x4d (0x7f1a2001363d in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #4: + 0x67a902 (0x7f1a69402902 in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x67a9a6 (0x7f1a694029a6 in /home/env/python3.6env/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x19f636 (0x55ee290af636 in /home/env/python3.6env/bin/python3)
frame #7: + 0xeef28 (0x55ee28ffef28 in /home/env/python3.6env/bin/python3)
frame #8: + 0xeeb1f (0x55ee28ffeb1f in /home/env/python3.6env/bin/python3)
frame #9: + 0xef0da (0x55ee28fff0da in /home/env/python3.6env/bin/python3)
frame #10: + 0xef0da (0x55ee28fff0da in /home/env/python3.6env/bin/python3)
frame #11: + 0xee917 (0x55ee28ffe917 in /home/env/python3.6env/bin/python3)
frame #12: + 0xee7a7 (0x55ee28ffe7a7 in /home/env/python3.6env/bin/python3)
frame #13: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3)
frame #14: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3)
frame #15: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3)
frame #16: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3)
frame #17: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3)
frame #18: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3)
frame #19: + 0xee7bd (0x55ee28ffe7bd in /home/env/python3.6env/bin/python3)
frame #20: + 0xe1047 (0x55ee28ff1047 in /home/env/python3.6env/bin/python3)
frame #21: + 0xe10b3 (0x55ee28ff10b3 in /home/env/python3.6env/bin/python3)
frame #22: + 0xe1036 (0x55ee28ff1036 in /home/env/python3.6env/bin/python3)
frame #23: + 0x1c8393 (0x55ee290d8393 in /home/env/python3.6env/bin/python3)
frame #24: _PyEval_EvalFrameDefault + 0x4378 (0x55ee290d4d38 in /home/env/python3.6env/bin/python3)
frame #25: + 0x19870b (0x55ee290a870b in /home/env/python3.6env/bin/python3)
frame #26: + 0x19e755 (0x55ee290ae755 in /home/env/python3.6env/bin/python3)
frame #27: _PyEval_EvalFrameDefault + 0x2fa (0x55ee290d0cba in /home/env/python3.6env/bin/python3)
frame #28: + 0x19870b (0x55ee290a870b in /home/env/python3.6env/bin/python3)
frame #29: + 0x19e755 (0x55ee290ae755 in /home/env/python3.6env/bin/python3)
frame #30: _PyEval_EvalFrameDefault + 0x2fa (0x55ee290d0cba in /home/env/python3.6env/bin/python3)
frame #31: + 0x197a94 (0x55ee290a7a94 in /home/env/python3.6env/bin/python3)
frame #32: + 0x198941 (0x55ee290a8941 in /home/env/python3.6env/bin/python3)
frame #33: + 0x19e755 (0x55ee290ae755 in /home/env/python3.6env/bin/python3)
frame #34: _PyEval_EvalFrameDefault + 0x10ba (0x55ee290d1a7a in /home/env/python3.6env/bin/python3)
frame #35: PyEval_EvalCodeEx + 0x329 (0x55ee290a9459 in /home/env/python3.6env/bin/python3)
frame #36: PyEval_EvalCode + 0x1c (0x55ee290aa1ec in /home/env/python3.6env/bin/python3)
frame #37: + 0x2149a4 (0x55ee291249a4 in /home/env/python3.6env/bin/python3)
frame #38: PyRun_StringFlags + 0x7d (0x55ee29124a3d in /home/env/python3.6env/bin/python3)
frame #39: PyRun_SimpleStringFlags + 0x3f (0x55ee29124a9f in /home/env/python3.6env/bin/python3)
frame #40: Py_Main + 0x43b (0x55ee2912889b in /home/env/python3.6env/bin/python3)
frame #41: main + 0xee (0x55ee28ff04be in /home/env/python3.6env/bin/python3)
frame #42: __libc_start_main + 0xf0 (0x7f1a735fa840 in /lib/x86_64-linux-gnu/libc.so.6)
frame #43: + 0x1c7773 (0x55ee290d7773 in /home/env/python3.6env/bin/python3)
len(valid_idxs):38000
pred_class_logits:torch.Size([38000, 80])
len(valid_idxs):42000
pred_class_logits:torch.Size([42000, 80])
len(valid_idxs):39000
pred_class_logits:torch.Size([39000, 80])
Traceback (most recent call last):
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/tools/train_net.py", line 114, in
args=(args,),
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/launch.py", line 53, in launch
daemon=False,
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
while not spawn_context.join():
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
raise Exception(msg)
Exception:
-- Process 2 terminated with the following error:
Traceback (most recent call last):
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
fn(i, *args)
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/launch.py", line 88, in _distributed_worker
main_func(*args)
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/tools/train_net.py", line 100, in main
runner.train()
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/runner.py", line 270, in train
super().train(self.start_iter, self.start_epoch, self.max_iter)
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 84, in train
self.run_step()
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 185, in run_step
loss_dict = self.model(data)
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "../yolof_base/yolof.py", line 135, in forward
pred_logits, pred_anchor_deltas)
File "../yolof_base/yolof.py", line 218, in losses
pred_class_logits[valid_idxs],
RuntimeError: copy_if failed to synchronize: device-side assert triggered
Which model are you training with? and what command do you use?
It will be more clear if you post your training log file in the log directory.
okay, thanks for your reply. i just follw the commond:
cd playground/detection/coco/yolof/yolof.res50.C5.1x
pods_train --num-gpus 4
the log is:
[04/08 19:32:57] cvpods INFO: Rank of current process: 0. World size: 4
[04/08 19:32:57] cvpods INFO: Environment info:
sys.platform linux
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) [GCC 7.2.0]
numpy 1.18.4
cvpods 0.1 @/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods
cvpods compiler GCC 5.4
cvpods CUDA compiler 10.0
cvpods arch flags sm_75
cvpods_ENV_MODULE
PyTorch 1.4.0+cu100 @/home/env/python3.6env/lib/python3.6/site-packages/torch
PyTorch debug build False
CUDA available True
GPU 0,1,2,3 GeForce RTX 2080 Ti
CUDA_HOME /usr/local/cuda-10.0
NVCC Cuda compilation tools, release 10.0, V10.0.130
Pillow 8.0.1
torchvision 0.5.0+cu100 @/home/env/python3.6env/lib/python3.6/site-packages/torchvision
torchvision arch flags sm_35, sm_50, sm_60, sm_70, sm_75
cv2 4.5.1
PyTorch built with:
- GCC 7.3
- Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- NNPACK is enabled
- CUDA Runtime 10.0
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
- CuDNN 7.6.3
- Magma 2.5.1
- Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,
[04/08 19:32:57] cvpods INFO: Command line arguments: Namespace(dist_url='tcp://127.0.0.1:50155', eval_only=False, machine_rank=0, num_gpus=4, num_machines=1, opts=[], resume=False)
[04/08 19:32:57] cvpods INFO: Running with full config:
╒═════════════════╤═══════════════════════════════════════════════════════════════════════════╕
│ config params │ values │
╞═════════════════╪═══════════════════════════════════════════════════════════════════════════╡
│ MODEL │ {'ANCHOR_GENERATOR': {'ANGLES': [[-90, 0, 90]], │
│ │ 'ASPECT_RATIOS': [[1.0]], │
│ │ 'OFFSET': 0.0, │
│ │ 'SIZES': [[32, 64, 128, 256, 512]]}, │
│ │ 'AS_PRETRAIN': False, │
│ │ 'BACKBONE': {'FREEZE_AT': 2}, │
│ │ 'DEVICE': 'cuda', │
│ │ 'FPN': {'BLOCK_IN_FEATURES': 'p5', │
│ │ 'FUSE_TYPE': 'sum', │
│ │ 'IN_FEATURES': [], │
│ │ 'NORM': '', │
│ │ 'OUT_CHANNELS': 256}, │
│ │ 'KEYPOINT_ON': False, │
│ │ 'LOAD_PROPOSALS': False, │
│ │ 'MASK_ON': False, │
│ │ 'NMS_TYPE': 'normal', │
│ │ 'PIXEL_MEAN': [103.53, 116.28, 123.675], │
│ │ 'PIXEL_STD': [1.0, 1.0, 1.0], │
│ │ 'RESNETS': {'ACTIVATION': {'INPLACE': True, 'NAME': 'ReLU'}, │
│ │ 'DEEP_STEM': False, │
│ │ 'DEPTH': 50, │
│ │ 'NORM': 'FrozenBN', │
│ │ 'NUM_CLASSES': None, │
│ │ 'NUM_GROUPS': 1, │
│ │ 'OUT_FEATURES': ['res5'], │
│ │ 'RES2_OUT_CHANNELS': 256, │
│ │ 'RES5_DILATION': 1, │
│ │ 'STEM_OUT_CHANNELS': 64, │
│ │ 'STRIDE_IN_1X1': True, │
│ │ 'WIDTH_PER_GROUP': 64, │
│ │ 'ZERO_INIT_RESIDUAL': False}, │
│ │ 'WEIGHTS': 'detectron2://ImageNetPretrained/MSRA/R-50.pkl', │
│ │ 'YOLOF': {'ADD_CTR_CLAMP': True, │
│ │ 'BBOX_REG_WEIGHTS': [1.0, 1.0, 1.0, 1.0], │
│ │ 'CTR_CLAMP': 32, │
│ │ 'DECODER': {'ACTIVATION': 'ReLU', │
│ │ 'CLS_NUM_CONVS': 2, │
│ │ 'IN_CHANNELS': 512, │
│ │ 'NORM': 'BN', │
│ │ 'NUM_ANCHORS': 5, │
│ │ 'NUM_CLASSES': 80, │
│ │ 'PRIOR_PROB': 0.01, │
│ │ 'REG_NUM_CONVS': 4}, │
│ │ 'ENCODER': {'ACTIVATION': 'ReLU', │
│ │ 'BLOCK_DILATIONS': [2, 4, 6, 8], │
│ │ 'BLOCK_MID_CHANNELS': 128, │
│ │ 'IN_FEATURES': ['res5'], │
│ │ 'NORM': 'BN', │
│ │ 'NUM_CHANNELS': 512, │
│ │ 'NUM_RESIDUAL_BLOCKS': 4}, │
│ │ 'FOCAL_LOSS_ALPHA': 0.25, │
│ │ 'FOCAL_LOSS_GAMMA': 2.0, │
│ │ 'MATCHER_TOPK': 4, │
│ │ 'NEG_IGNORE_THRESHOLD': 0.7, │
│ │ 'NMS_THRESH_TEST': 0.6, │
│ │ 'POS_IGNORE_THRESHOLD': 0.15, │
│ │ 'SCORE_THRESH_TEST': 0.05, │
│ │ 'TOPK_CANDIDATES_TEST': 1000}} │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ INPUT │ {'AUG': {'TEST_PIPELINES': [('ResizeShortestEdge', │
│ │ {'max_size': 1333, │
│ │ 'sample_style': 'choice', │
│ │ 'short_edge_length': 800})], │
│ │ 'TRAIN_PIPELINES': [('ResizeShortestEdge', │
│ │ {'max_size': 1333, │
│ │ 'sample_style': 'choice', │
│ │ 'short_edge_length': (800,)}), │
│ │ ('RandomFlip', {}), │
│ │ ('RandomShift', {'max_shifts': 32})]}, │
│ │ 'FORMAT': 'BGR', │
│ │ 'MASK_FORMAT': 'polygon'} │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ DATASETS │ {'CUSTOM_TYPE': ['ConcatDataset', {}], │
│ │ 'PRECOMPUTED_PROPOSAL_TOPK_TEST': 1000, │
│ │ 'PRECOMPUTED_PROPOSAL_TOPK_TRAIN': 2000, │
│ │ 'PROPOSAL_FILES_TEST': [], │
│ │ 'PROPOSAL_FILES_TRAIN': [], │
│ │ 'TEST': ['coco_2017_val'], │
│ │ 'TRAIN': ['coco_2017_train']} │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ DATALOADER │ {'ASPECT_RATIO_GROUPING': True, │
│ │ 'FILTER_EMPTY_ANNOTATIONS': True, │
│ │ 'NUM_WORKERS': 0, │
│ │ 'REPEAT_THRESHOLD': 0.0, │
│ │ 'SAMPLER_TRAIN': 'DistributedGroupSampler'} │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ SOLVER │ {'BATCH_SUBDIVISIONS': 1, │
│ │ 'CHECKPOINT_PERIOD': 5000, │
│ │ 'CLIP_GRADIENTS': {'CLIP_TYPE': 'value', │
│ │ 'CLIP_VALUE': 1.0, │
│ │ 'ENABLED': False, │
│ │ 'NORM_TYPE': 2.0}, │
│ │ 'IMS_PER_BATCH': 32, │
│ │ 'IMS_PER_DEVICE': 8, │
│ │ 'LR_SCHEDULER': {'GAMMA': 0.1, │
│ │ 'MAX_EPOCH': None, │
│ │ 'MAX_ITER': 22500, │
│ │ 'NAME': 'WarmupMultiStepLR', │
│ │ 'STEPS': [15000, 20000], │
│ │ 'WARMUP_FACTOR': 0.00066667, │
│ │ 'WARMUP_ITERS': 3000, │
│ │ 'WARMUP_METHOD': 'linear'}, │
│ │ 'OPTIMIZER': {'BACKBONE_LR_FACTOR': 0.334, │
│ │ 'BASE_LR': 0.0005, │
│ │ 'BIAS_LR_FACTOR': 1.0, │
│ │ 'MOMENTUM': 0.9, │
│ │ 'NAME': 'D2SGD', │
│ │ 'WEIGHT_DECAY': 0.0001, │
│ │ 'WEIGHT_DECAY_NORM': 0.0}} │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ TEST │ {'AUG': {'ENABLED': False, │
│ │ 'EXTRA_SIZES': [], │
│ │ 'FLIP': True, │
│ │ 'MAX_SIZE': 4000, │
│ │ 'MIN_SIZES': [400, 500, 600, 700, 800, 900, 1000, 1100, 1200], │
│ │ 'SCALE_FILTER': False, │
│ │ 'SCALE_RANGES': []}, │
│ │ 'DETECTIONS_PER_IMAGE': 100, │
│ │ 'EVAL_PERIOD': 0, │
│ │ 'EXPECTED_RESULTS': [], │
│ │ 'KEYPOINT_OKS_SIGMAS': [], │
│ │ 'PRECISE_BN': {'ENABLED': False, 'NUM_ITER': 200}} │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ TRAINER │ {'FP16': {'ENABLED': False, 'OPTS': {'OPT_LEVEL': 'O1'}, 'TYPE': 'APEX'}, │
│ │ 'NAME': 'DefaultRunner', │
│ │ 'WINDOW_SIZE': 20} │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ OUTPUT_DIR │ './output' │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ SEED │ -1 │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ CUDNN_BENCHMARK │ False │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ VIS_PERIOD │ 0 │
├─────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ GLOBAL │ {'DUMP_TEST': False, 'DUMP_TRAIN': True, 'HACK': 1.0} │
╘═════════════════╧═══════════════════════════════════════════════════════════════════════════╛
[04/08 19:32:57] cvpods INFO: different config with base class:
╒═════════════════╤═════════════════════════════════════════════════════════════════════╕
│ config params │ values │
╞═════════════════╪═════════════════════════════════════════════════════════════════════╡
│ MODEL │ {'ANCHOR_GENERATOR': {'ASPECT_RATIOS': [[1.0]]}, │
│ │ 'RESNETS': {'DEPTH': 50, 'OUT_FEATURES': ['res5']}, │
│ │ 'WEIGHTS': 'detectron2://ImageNetPretrained/MSRA/R-50.pkl', │
│ │ 'YOLOF': {'ADD_CTR_CLAMP': True, │
│ │ 'BBOX_REG_WEIGHTS': [1.0, 1.0, 1.0, 1.0], │
│ │ 'CTR_CLAMP': 32, │
│ │ 'DECODER': {'ACTIVATION': 'ReLU', │
│ │ 'CLS_NUM_CONVS': 2, │
│ │ 'IN_CHANNELS': 512, │
│ │ 'NORM': 'BN', │
│ │ 'NUM_ANCHORS': 5, │
│ │ 'NUM_CLASSES': 80, │
│ │ 'PRIOR_PROB': 0.01, │
│ │ 'REG_NUM_CONVS': 4}, │
│ │ 'ENCODER': {'ACTIVATION': 'ReLU', │
│ │ 'BLOCK_DILATIONS': [2, 4, 6, 8], │
│ │ 'BLOCK_MID_CHANNELS': 128, │
│ │ 'IN_FEATURES': ['res5'], │
│ │ 'NORM': 'BN', │
│ │ 'NUM_CHANNELS': 512, │
│ │ 'NUM_RESIDUAL_BLOCKS': 4}, │
│ │ 'FOCAL_LOSS_ALPHA': 0.25, │
│ │ 'FOCAL_LOSS_GAMMA': 2.0, │
│ │ 'MATCHER_TOPK': 4, │
│ │ 'NEG_IGNORE_THRESHOLD': 0.7, │
│ │ 'NMS_THRESH_TEST': 0.6, │
│ │ 'POS_IGNORE_THRESHOLD': 0.15, │
│ │ 'SCORE_THRESH_TEST': 0.05, │
│ │ 'TOPK_CANDIDATES_TEST': 1000}} │
├─────────────────┼─────────────────────────────────────────────────────────────────────┤
│ INPUT │ {'AUG': {'TRAIN_PIPELINES': [('ResizeShortestEdge', │
│ │ {'max_size': 1333, │
│ │ 'sample_style': 'choice', │
│ │ 'short_edge_length': (800,)}), │
│ │ ('RandomFlip', {}), │
│ │ ('RandomShift', {'max_shifts': 32})]}} │
├─────────────────┼─────────────────────────────────────────────────────────────────────┤
│ DATASETS │ {'TEST': ['coco_2017_val'], 'TRAIN': ['coco_2017_train']} │
├─────────────────┼─────────────────────────────────────────────────────────────────────┤
│ DATALOADER │ {'NUM_WORKERS': 0} │
├─────────────────┼─────────────────────────────────────────────────────────────────────┤
│ SOLVER │ {'IMS_PER_BATCH': 32, │
│ │ 'IMS_PER_DEVICE': 8, │
│ │ 'LR_SCHEDULER': {'MAX_ITER': 22500, │
│ │ 'STEPS': [15000, 20000], │
│ │ 'WARMUP_FACTOR': 0.00066667, │
│ │ 'WARMUP_ITERS': 3000}, │
│ │ 'OPTIMIZER': {'BACKBONE_LR_FACTOR': 0.334, 'BASE_LR': 0.0005}} │
╘═════════════════╧═════════════════════════════════════════════════════════════════════╛
[04/08 19:32:57] c2.utils.env.env INFO: Using a generated random seed 57776492
[04/08 19:32:57] c2.data.build INFO: TransformGens used: [ResizeShortestEdge(short_edge_length=(800,), max_size=1333, sample_style='choice'), RandomFlip(), RandomShift(max_shifts=32)] in training
[04/08 19:33:09] c2.data.datasets.coco INFO: Loading /media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/datasets/coco/annotations/instances_train2017.json takes 11.49 seconds.
[04/08 19:33:10] c2.data.datasets.coco INFO: Loaded 118287 images in COCO format from /media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/datasets/coco/annotations/instances_train2017.json
[04/08 19:33:16] c2.data.base_dataset INFO: Removed 1021 images with no usable annotations. 117266 images left.
[04/08 19:33:18] c2.data.base_dataset INFO: Distribution of instances among all 80 categories:
�[36m| category | #instances | category | #instances | category | #instances |
|:-------------:|:-------------|:------------:|:-------------|:-------------:|:-------------|
| person | 257253 | bicycle | 7056 | car | 43533 |
| motorcycle | 8654 | airplane | 5129 | bus | 6061 |
| train | 4570 | truck | 9970 | boat | 10576 |
| traffic light | 12842 | fire hydrant | 1865 | stop sign | 1983 |
| parking meter | 1283 | bench | 9820 | bird | 10542 |
| cat | 4766 | dog | 5500 | horse | 6567 |
| sheep | 9223 | cow | 8014 | elephant | 5484 |
| bear | 1294 | zebra | 5269 | giraffe | 5128 |
| backpack | 8714 | umbrella | 11265 | handbag | 12342 |
| tie | 6448 | suitcase | 6112 | frisbee | 2681 |
| skis | 6623 | snowboard | 2681 | sports ball | 6299 |
| kite | 8802 | baseball bat | 3273 | baseball gl.. | 3747 |
| skateboard | 5536 | surfboard | 6095 | tennis racket | 4807 |
| bottle | 24070 | wine glass | 7839 | cup | 20574 |
| fork | 5474 | knife | 7760 | spoon | 6159 |
| bowl | 14323 | banana | 9195 | apple | 5776 |
| sandwich | 4356 | orange | 6302 | broccoli | 7261 |
| carrot | 7758 | hot dog | 2884 | pizza | 5807 |
| donut | 7005 | cake | 6296 | chair | 38073 |
| couch | 5779 | potted plant | 8631 | bed | 4192 |
| dining table | 15695 | toilet | 4149 | tv | 5803 |
| laptop | 4960 | mouse | 2261 | remote | 5700 |
| keyboard | 2854 | cell phone | 6422 | microwave | 1672 |
| oven | 3334 | toaster | 225 | sink | 5609 |
| refrigerator | 2634 | book | 24077 | clock | 6320 |
| vase | 6577 | scissors | 1464 | teddy bear | 4729 |
| hair drier | 198 | toothbrush | 1945 | | |
| total | 849949 | | | | |�[0m
[04/08 19:33:19] c2.data.build INFO: Using training sampler DistributedGroupSampler
[04/08 19:33:19] cvpods INFO: Model:
YOLOF(
(backbone): ResNet(
(stem): BasicStem(
(conv1): Conv2d(
3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(activation): ReLU(inplace=True)
(max_pool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
)
(res2): Sequential(
(0): BottleneckBlock(
(shortcut): Conv2d(
64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(activation): ReLU(inplace=True)
(conv1): Conv2d(
64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv2): Conv2d(
64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv3): Conv2d(
64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
)
(1): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv2): Conv2d(
64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv3): Conv2d(
64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
)
(2): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv2): Conv2d(
64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv3): Conv2d(
64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
)
)
(res3): Sequential(
(0): BottleneckBlock(
(shortcut): Conv2d(
256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(activation): ReLU(inplace=True)
(conv1): Conv2d(
256, 128, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv2): Conv2d(
128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv3): Conv2d(
128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
)
(1): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv2): Conv2d(
128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv3): Conv2d(
128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
)
(2): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv2): Conv2d(
128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv3): Conv2d(
128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
)
(3): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv2): Conv2d(
128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv3): Conv2d(
128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
)
)
(res4): Sequential(
(0): BottleneckBlock(
(shortcut): Conv2d(
512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
(activation): ReLU(inplace=True)
(conv1): Conv2d(
512, 256, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(1): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(2): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(3): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(4): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(5): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
)
(res5): Sequential(
(0): BottleneckBlock(
(shortcut): Conv2d(
1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05)
)
(activation): ReLU(inplace=True)
(conv1): Conv2d(
1024, 512, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv2): Conv2d(
512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv3): Conv2d(
512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05)
)
)
(1): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv2): Conv2d(
512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv3): Conv2d(
512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05)
)
)
(2): BottleneckBlock(
(activation): ReLU(inplace=True)
(conv1): Conv2d(
2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv2): Conv2d(
512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv3): Conv2d(
512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05)
)
)
)
)
(encoder): DilatedEncoder(
(lateral_conv): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1))
(lateral_norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(fpn_conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(fpn_norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(dilated_encoder_blocks): Sequential(
(0): Bottleneck(
(conv1): Sequential(
(0): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1))
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
(conv2): Sequential(
(0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2))
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
(conv3): Sequential(
(0): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
)
(1): Bottleneck(
(conv1): Sequential(
(0): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1))
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
(conv2): Sequential(
(0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4))
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
(conv3): Sequential(
(0): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
)
(2): Bottleneck(
(conv1): Sequential(
(0): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1))
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
(conv2): Sequential(
(0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(6, 6), dilation=(6, 6))
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
(conv3): Sequential(
(0): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
)
(3): Bottleneck(
(conv1): Sequential(
(0): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1))
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
(conv2): Sequential(
(0): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(8, 8), dilation=(8, 8))
(1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
(conv3): Sequential(
(0): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1))
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
)
)
)
)
(decoder): Decoder(
(cls_subnet): Sequential(
(0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
(3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU(inplace=True)
)
(bbox_subnet): Sequential(
(0): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(2): ReLU(inplace=True)
(3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(5): ReLU(inplace=True)
(6): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(7): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(8): ReLU(inplace=True)
(9): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(10): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(11): ReLU(inplace=True)
)
(cls_score): Conv2d(512, 400, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(bbox_pred): Conv2d(512, 20, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(object_pred): Conv2d(512, 5, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
)
(anchor_generator): DefaultAnchorGenerator(
(cell_anchors): BufferList()
)
(matcher): UniformMatcher()
)
[04/08 19:33:21] c2.checkpoint.checkpoint INFO: Loading checkpoint from detectron2://ImageNetPretrained/MSRA/R-50.pkl
[04/08 19:33:21] c2.utils.file.file_io INFO: URL https://dl.fbaipublicfiles.com/detectron2/ImageNetPretrained/MSRA/R-50.pkl cached in /home/.torch/cvpods_cache/detectron2/ImageNetPretrained/MSRA/R-50.pkl
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: Remapping C2 weights ......
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.norm.bias loaded from res2_0_branch2a_bn_beta of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.norm.running_mean loaded from res2_0_branch2a_bn_running_mean of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.norm.running_var loaded from res2_0_branch2a_bn_running_var of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.norm.weight loaded from res2_0_branch2a_bn_gamma of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv1.weight loaded from res2_0_branch2a_w of shape (64, 64, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.norm.bias loaded from res2_0_branch2b_bn_beta of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.norm.running_mean loaded from res2_0_branch2b_bn_running_mean of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.norm.running_var loaded from res2_0_branch2b_bn_running_var of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.norm.weight loaded from res2_0_branch2b_bn_gamma of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv2.weight loaded from res2_0_branch2b_w of shape (64, 64, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.norm.bias loaded from res2_0_branch2c_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.norm.running_mean loaded from res2_0_branch2c_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.norm.running_var loaded from res2_0_branch2c_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.norm.weight loaded from res2_0_branch2c_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.conv3.weight loaded from res2_0_branch2c_w of shape (256, 64, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.norm.bias loaded from res2_0_branch1_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.norm.running_mean loaded from res2_0_branch1_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.norm.running_var loaded from res2_0_branch1_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.norm.weight loaded from res2_0_branch1_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.0.shortcut.weight loaded from res2_0_branch1_w of shape (256, 64, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.norm.bias loaded from res2_1_branch2a_bn_beta of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.norm.running_mean loaded from res2_1_branch2a_bn_running_mean of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.norm.running_var loaded from res2_1_branch2a_bn_running_var of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.norm.weight loaded from res2_1_branch2a_bn_gamma of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv1.weight loaded from res2_1_branch2a_w of shape (64, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.norm.bias loaded from res2_1_branch2b_bn_beta of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.norm.running_mean loaded from res2_1_branch2b_bn_running_mean of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.norm.running_var loaded from res2_1_branch2b_bn_running_var of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.norm.weight loaded from res2_1_branch2b_bn_gamma of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv2.weight loaded from res2_1_branch2b_w of shape (64, 64, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.norm.bias loaded from res2_1_branch2c_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.norm.running_mean loaded from res2_1_branch2c_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.norm.running_var loaded from res2_1_branch2c_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.norm.weight loaded from res2_1_branch2c_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.1.conv3.weight loaded from res2_1_branch2c_w of shape (256, 64, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.norm.bias loaded from res2_2_branch2a_bn_beta of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.norm.running_mean loaded from res2_2_branch2a_bn_running_mean of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.norm.running_var loaded from res2_2_branch2a_bn_running_var of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.norm.weight loaded from res2_2_branch2a_bn_gamma of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv1.weight loaded from res2_2_branch2a_w of shape (64, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.norm.bias loaded from res2_2_branch2b_bn_beta of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.norm.running_mean loaded from res2_2_branch2b_bn_running_mean of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.norm.running_var loaded from res2_2_branch2b_bn_running_var of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.norm.weight loaded from res2_2_branch2b_bn_gamma of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv2.weight loaded from res2_2_branch2b_w of shape (64, 64, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.norm.bias loaded from res2_2_branch2c_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.norm.running_mean loaded from res2_2_branch2c_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.norm.running_var loaded from res2_2_branch2c_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.norm.weight loaded from res2_2_branch2c_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res2.2.conv3.weight loaded from res2_2_branch2c_w of shape (256, 64, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.norm.bias loaded from res3_0_branch2a_bn_beta of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.norm.running_mean loaded from res3_0_branch2a_bn_running_mean of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.norm.running_var loaded from res3_0_branch2a_bn_running_var of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.norm.weight loaded from res3_0_branch2a_bn_gamma of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv1.weight loaded from res3_0_branch2a_w of shape (128, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.norm.bias loaded from res3_0_branch2b_bn_beta of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.norm.running_mean loaded from res3_0_branch2b_bn_running_mean of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.norm.running_var loaded from res3_0_branch2b_bn_running_var of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.norm.weight loaded from res3_0_branch2b_bn_gamma of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv2.weight loaded from res3_0_branch2b_w of shape (128, 128, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.norm.bias loaded from res3_0_branch2c_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.norm.running_mean loaded from res3_0_branch2c_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.norm.running_var loaded from res3_0_branch2c_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.norm.weight loaded from res3_0_branch2c_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.conv3.weight loaded from res3_0_branch2c_w of shape (512, 128, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.norm.bias loaded from res3_0_branch1_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.norm.running_mean loaded from res3_0_branch1_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.norm.running_var loaded from res3_0_branch1_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.norm.weight loaded from res3_0_branch1_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.0.shortcut.weight loaded from res3_0_branch1_w of shape (512, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.norm.bias loaded from res3_1_branch2a_bn_beta of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.norm.running_mean loaded from res3_1_branch2a_bn_running_mean of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.norm.running_var loaded from res3_1_branch2a_bn_running_var of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.norm.weight loaded from res3_1_branch2a_bn_gamma of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv1.weight loaded from res3_1_branch2a_w of shape (128, 512, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.norm.bias loaded from res3_1_branch2b_bn_beta of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.norm.running_mean loaded from res3_1_branch2b_bn_running_mean of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.norm.running_var loaded from res3_1_branch2b_bn_running_var of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.norm.weight loaded from res3_1_branch2b_bn_gamma of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv2.weight loaded from res3_1_branch2b_w of shape (128, 128, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.norm.bias loaded from res3_1_branch2c_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.norm.running_mean loaded from res3_1_branch2c_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.norm.running_var loaded from res3_1_branch2c_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.norm.weight loaded from res3_1_branch2c_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.1.conv3.weight loaded from res3_1_branch2c_w of shape (512, 128, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.norm.bias loaded from res3_2_branch2a_bn_beta of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.norm.running_mean loaded from res3_2_branch2a_bn_running_mean of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.norm.running_var loaded from res3_2_branch2a_bn_running_var of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.norm.weight loaded from res3_2_branch2a_bn_gamma of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv1.weight loaded from res3_2_branch2a_w of shape (128, 512, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.norm.bias loaded from res3_2_branch2b_bn_beta of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.norm.running_mean loaded from res3_2_branch2b_bn_running_mean of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.norm.running_var loaded from res3_2_branch2b_bn_running_var of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.norm.weight loaded from res3_2_branch2b_bn_gamma of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv2.weight loaded from res3_2_branch2b_w of shape (128, 128, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.norm.bias loaded from res3_2_branch2c_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.norm.running_mean loaded from res3_2_branch2c_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.norm.running_var loaded from res3_2_branch2c_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.norm.weight loaded from res3_2_branch2c_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.2.conv3.weight loaded from res3_2_branch2c_w of shape (512, 128, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.norm.bias loaded from res3_3_branch2a_bn_beta of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.norm.running_mean loaded from res3_3_branch2a_bn_running_mean of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.norm.running_var loaded from res3_3_branch2a_bn_running_var of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.norm.weight loaded from res3_3_branch2a_bn_gamma of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv1.weight loaded from res3_3_branch2a_w of shape (128, 512, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.norm.bias loaded from res3_3_branch2b_bn_beta of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.norm.running_mean loaded from res3_3_branch2b_bn_running_mean of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.norm.running_var loaded from res3_3_branch2b_bn_running_var of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.norm.weight loaded from res3_3_branch2b_bn_gamma of shape (128,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv2.weight loaded from res3_3_branch2b_w of shape (128, 128, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.norm.bias loaded from res3_3_branch2c_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.norm.running_mean loaded from res3_3_branch2c_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.norm.running_var loaded from res3_3_branch2c_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.norm.weight loaded from res3_3_branch2c_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res3.3.conv3.weight loaded from res3_3_branch2c_w of shape (512, 128, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.norm.bias loaded from res4_0_branch2a_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.norm.running_mean loaded from res4_0_branch2a_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.norm.running_var loaded from res4_0_branch2a_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.norm.weight loaded from res4_0_branch2a_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv1.weight loaded from res4_0_branch2a_w of shape (256, 512, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.norm.bias loaded from res4_0_branch2b_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.norm.running_mean loaded from res4_0_branch2b_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.norm.running_var loaded from res4_0_branch2b_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.norm.weight loaded from res4_0_branch2b_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv2.weight loaded from res4_0_branch2b_w of shape (256, 256, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.norm.bias loaded from res4_0_branch2c_bn_beta of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.norm.running_mean loaded from res4_0_branch2c_bn_running_mean of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.norm.running_var loaded from res4_0_branch2c_bn_running_var of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.norm.weight loaded from res4_0_branch2c_bn_gamma of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.conv3.weight loaded from res4_0_branch2c_w of shape (1024, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.norm.bias loaded from res4_0_branch1_bn_beta of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.norm.running_mean loaded from res4_0_branch1_bn_running_mean of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.norm.running_var loaded from res4_0_branch1_bn_running_var of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.norm.weight loaded from res4_0_branch1_bn_gamma of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.0.shortcut.weight loaded from res4_0_branch1_w of shape (1024, 512, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.norm.bias loaded from res4_1_branch2a_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.norm.running_mean loaded from res4_1_branch2a_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.norm.running_var loaded from res4_1_branch2a_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.norm.weight loaded from res4_1_branch2a_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv1.weight loaded from res4_1_branch2a_w of shape (256, 1024, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.norm.bias loaded from res4_1_branch2b_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.norm.running_mean loaded from res4_1_branch2b_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.norm.running_var loaded from res4_1_branch2b_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.norm.weight loaded from res4_1_branch2b_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv2.weight loaded from res4_1_branch2b_w of shape (256, 256, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.norm.bias loaded from res4_1_branch2c_bn_beta of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.norm.running_mean loaded from res4_1_branch2c_bn_running_mean of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.norm.running_var loaded from res4_1_branch2c_bn_running_var of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.norm.weight loaded from res4_1_branch2c_bn_gamma of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.1.conv3.weight loaded from res4_1_branch2c_w of shape (1024, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.norm.bias loaded from res4_2_branch2a_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.norm.running_mean loaded from res4_2_branch2a_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.norm.running_var loaded from res4_2_branch2a_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.norm.weight loaded from res4_2_branch2a_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv1.weight loaded from res4_2_branch2a_w of shape (256, 1024, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.norm.bias loaded from res4_2_branch2b_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.norm.running_mean loaded from res4_2_branch2b_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.norm.running_var loaded from res4_2_branch2b_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.norm.weight loaded from res4_2_branch2b_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv2.weight loaded from res4_2_branch2b_w of shape (256, 256, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.norm.bias loaded from res4_2_branch2c_bn_beta of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.norm.running_mean loaded from res4_2_branch2c_bn_running_mean of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.norm.running_var loaded from res4_2_branch2c_bn_running_var of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.norm.weight loaded from res4_2_branch2c_bn_gamma of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.2.conv3.weight loaded from res4_2_branch2c_w of shape (1024, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.norm.bias loaded from res4_3_branch2a_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.norm.running_mean loaded from res4_3_branch2a_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.norm.running_var loaded from res4_3_branch2a_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.norm.weight loaded from res4_3_branch2a_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv1.weight loaded from res4_3_branch2a_w of shape (256, 1024, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.norm.bias loaded from res4_3_branch2b_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.norm.running_mean loaded from res4_3_branch2b_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.norm.running_var loaded from res4_3_branch2b_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.norm.weight loaded from res4_3_branch2b_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv2.weight loaded from res4_3_branch2b_w of shape (256, 256, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.norm.bias loaded from res4_3_branch2c_bn_beta of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.norm.running_mean loaded from res4_3_branch2c_bn_running_mean of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.norm.running_var loaded from res4_3_branch2c_bn_running_var of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.norm.weight loaded from res4_3_branch2c_bn_gamma of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.3.conv3.weight loaded from res4_3_branch2c_w of shape (1024, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.norm.bias loaded from res4_4_branch2a_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.norm.running_mean loaded from res4_4_branch2a_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.norm.running_var loaded from res4_4_branch2a_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.norm.weight loaded from res4_4_branch2a_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv1.weight loaded from res4_4_branch2a_w of shape (256, 1024, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.norm.bias loaded from res4_4_branch2b_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.norm.running_mean loaded from res4_4_branch2b_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.norm.running_var loaded from res4_4_branch2b_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.norm.weight loaded from res4_4_branch2b_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv2.weight loaded from res4_4_branch2b_w of shape (256, 256, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.norm.bias loaded from res4_4_branch2c_bn_beta of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.norm.running_mean loaded from res4_4_branch2c_bn_running_mean of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.norm.running_var loaded from res4_4_branch2c_bn_running_var of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.norm.weight loaded from res4_4_branch2c_bn_gamma of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.4.conv3.weight loaded from res4_4_branch2c_w of shape (1024, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.norm.bias loaded from res4_5_branch2a_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.norm.running_mean loaded from res4_5_branch2a_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.norm.running_var loaded from res4_5_branch2a_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.norm.weight loaded from res4_5_branch2a_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv1.weight loaded from res4_5_branch2a_w of shape (256, 1024, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.norm.bias loaded from res4_5_branch2b_bn_beta of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.norm.running_mean loaded from res4_5_branch2b_bn_running_mean of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.norm.running_var loaded from res4_5_branch2b_bn_running_var of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.norm.weight loaded from res4_5_branch2b_bn_gamma of shape (256,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv2.weight loaded from res4_5_branch2b_w of shape (256, 256, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.norm.bias loaded from res4_5_branch2c_bn_beta of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.norm.running_mean loaded from res4_5_branch2c_bn_running_mean of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.norm.running_var loaded from res4_5_branch2c_bn_running_var of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.norm.weight loaded from res4_5_branch2c_bn_gamma of shape (1024,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res4.5.conv3.weight loaded from res4_5_branch2c_w of shape (1024, 256, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.norm.bias loaded from res5_0_branch2a_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.norm.running_mean loaded from res5_0_branch2a_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.norm.running_var loaded from res5_0_branch2a_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.norm.weight loaded from res5_0_branch2a_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv1.weight loaded from res5_0_branch2a_w of shape (512, 1024, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.norm.bias loaded from res5_0_branch2b_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.norm.running_mean loaded from res5_0_branch2b_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.norm.running_var loaded from res5_0_branch2b_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.norm.weight loaded from res5_0_branch2b_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv2.weight loaded from res5_0_branch2b_w of shape (512, 512, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.norm.bias loaded from res5_0_branch2c_bn_beta of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.norm.running_mean loaded from res5_0_branch2c_bn_running_mean of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.norm.running_var loaded from res5_0_branch2c_bn_running_var of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.norm.weight loaded from res5_0_branch2c_bn_gamma of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.conv3.weight loaded from res5_0_branch2c_w of shape (2048, 512, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.norm.bias loaded from res5_0_branch1_bn_beta of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.norm.running_mean loaded from res5_0_branch1_bn_running_mean of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.norm.running_var loaded from res5_0_branch1_bn_running_var of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.norm.weight loaded from res5_0_branch1_bn_gamma of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.0.shortcut.weight loaded from res5_0_branch1_w of shape (2048, 1024, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.norm.bias loaded from res5_1_branch2a_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.norm.running_mean loaded from res5_1_branch2a_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.norm.running_var loaded from res5_1_branch2a_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.norm.weight loaded from res5_1_branch2a_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv1.weight loaded from res5_1_branch2a_w of shape (512, 2048, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.norm.bias loaded from res5_1_branch2b_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.norm.running_mean loaded from res5_1_branch2b_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.norm.running_var loaded from res5_1_branch2b_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.norm.weight loaded from res5_1_branch2b_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv2.weight loaded from res5_1_branch2b_w of shape (512, 512, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.norm.bias loaded from res5_1_branch2c_bn_beta of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.norm.running_mean loaded from res5_1_branch2c_bn_running_mean of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.norm.running_var loaded from res5_1_branch2c_bn_running_var of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.norm.weight loaded from res5_1_branch2c_bn_gamma of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.1.conv3.weight loaded from res5_1_branch2c_w of shape (2048, 512, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.norm.bias loaded from res5_2_branch2a_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.norm.running_mean loaded from res5_2_branch2a_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.norm.running_var loaded from res5_2_branch2a_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.norm.weight loaded from res5_2_branch2a_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv1.weight loaded from res5_2_branch2a_w of shape (512, 2048, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.norm.bias loaded from res5_2_branch2b_bn_beta of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.norm.running_mean loaded from res5_2_branch2b_bn_running_mean of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.norm.running_var loaded from res5_2_branch2b_bn_running_var of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.norm.weight loaded from res5_2_branch2b_bn_gamma of shape (512,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv2.weight loaded from res5_2_branch2b_w of shape (512, 512, 3, 3)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.norm.bias loaded from res5_2_branch2c_bn_beta of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.norm.running_mean loaded from res5_2_branch2c_bn_running_mean of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.norm.running_var loaded from res5_2_branch2c_bn_running_var of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.norm.weight loaded from res5_2_branch2c_bn_gamma of shape (2048,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.res5.2.conv3.weight loaded from res5_2_branch2c_w of shape (2048, 512, 1, 1)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.norm.bias loaded from res_conv1_bn_beta of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.norm.running_mean loaded from res_conv1_bn_running_mean of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.norm.running_var loaded from res_conv1_bn_running_var of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.norm.weight loaded from res_conv1_bn_gamma of shape (64,)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: backbone.stem.conv1.weight loaded from conv1_w of shape (64, 3, 7, 7)
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: Some model parameters are not in the checkpoint:
�[34manchor_generator.cell_anchors.0�[0m
�[34mdecoder.bbox_pred.{bias, weight}�[0m
�[34mdecoder.bbox_subnet.0.{bias, weight}�[0m
�[34mdecoder.bbox_subnet.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mdecoder.bbox_subnet.10.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mdecoder.bbox_subnet.3.{bias, weight}�[0m
�[34mdecoder.bbox_subnet.4.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mdecoder.bbox_subnet.6.{bias, weight}�[0m
�[34mdecoder.bbox_subnet.7.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mdecoder.bbox_subnet.9.{bias, weight}�[0m
�[34mdecoder.cls_score.{bias, weight}�[0m
�[34mdecoder.cls_subnet.0.{bias, weight}�[0m
�[34mdecoder.cls_subnet.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mdecoder.cls_subnet.3.{bias, weight}�[0m
�[34mdecoder.cls_subnet.4.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mdecoder.object_pred.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.0.conv1.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.0.conv1.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.0.conv2.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.0.conv2.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.0.conv3.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.0.conv3.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.1.conv1.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.1.conv1.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.1.conv2.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.1.conv2.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.1.conv3.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.1.conv3.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.2.conv1.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.2.conv1.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.2.conv2.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.2.conv2.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.2.conv3.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.2.conv3.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.3.conv1.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.3.conv1.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.3.conv2.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.3.conv2.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.dilated_encoder_blocks.3.conv3.0.{bias, weight}�[0m
�[34mencoder.dilated_encoder_blocks.3.conv3.1.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.fpn_conv.{bias, weight}�[0m
�[34mencoder.fpn_norm.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mencoder.lateral_conv.{bias, weight}�[0m
�[34mencoder.lateral_norm.{bias, num_batches_tracked, running_mean, running_var, weight}�[0m
�[34mpixel_mean�[0m
�[34mpixel_std�[0m
[04/08 19:33:21] c2.checkpoint.c2_model_loading INFO: The checkpoint contains parameters not used by the model:
�[35mfc1000_b�[0m
�[35mfc1000_w�[0m
�[35mconv1_b�[0m
[04/08 19:33:21] cvpods INFO: Running with full config:
╒════════════════════════╤═══════════════════════════════════════════════════════════════════════════╕
│ config params │ values │
╞════════════════════════╪═══════════════════════════════════════════════════════════════════════════╡
│ MODEL │ {'ANCHOR_GENERATOR': {'ANGLES': [[-90, 0, 90]], │
│ │ 'ASPECT_RATIOS': [[1.0]], │
│ │ 'OFFSET': 0.0, │
│ │ 'SIZES': [[32, 64, 128, 256, 512]]}, │
│ │ 'AS_PRETRAIN': False, │
│ │ 'BACKBONE': {'FREEZE_AT': 2}, │
│ │ 'DEVICE': 'cuda', │
│ │ 'FPN': {'BLOCK_IN_FEATURES': 'p5', │
│ │ 'FUSE_TYPE': 'sum', │
│ │ 'IN_FEATURES': [], │
│ │ 'NORM': '', │
│ │ 'OUT_CHANNELS': 256}, │
│ │ 'KEYPOINT_ON': False, │
│ │ 'LOAD_PROPOSALS': False, │
│ │ 'MASK_ON': False, │
│ │ 'NMS_TYPE': 'normal', │
│ │ 'PIXEL_MEAN': [103.53, 116.28, 123.675], │
│ │ 'PIXEL_STD': [1.0, 1.0, 1.0], │
│ │ 'RESNETS': {'ACTIVATION': {'INPLACE': True, 'NAME': 'ReLU'}, │
│ │ 'DEEP_STEM': False, │
│ │ 'DEPTH': 50, │
│ │ 'NORM': 'FrozenBN', │
│ │ 'NUM_CLASSES': None, │
│ │ 'NUM_GROUPS': 1, │
│ │ 'OUT_FEATURES': ['res5'], │
│ │ 'RES2_OUT_CHANNELS': 256, │
│ │ 'RES5_DILATION': 1, │
│ │ 'STEM_OUT_CHANNELS': 64, │
│ │ 'STRIDE_IN_1X1': True, │
│ │ 'WIDTH_PER_GROUP': 64, │
│ │ 'ZERO_INIT_RESIDUAL': False}, │
│ │ 'WEIGHTS': 'detectron2://ImageNetPretrained/MSRA/R-50.pkl', │
│ │ 'YOLOF': {'ADD_CTR_CLAMP': True, │
│ │ 'BBOX_REG_WEIGHTS': [1.0, 1.0, 1.0, 1.0], │
│ │ 'CTR_CLAMP': 32, │
│ │ 'DECODER': {'ACTIVATION': 'ReLU', │
│ │ 'CLS_NUM_CONVS': 2, │
│ │ 'IN_CHANNELS': 512, │
│ │ 'NORM': 'BN', │
│ │ 'NUM_ANCHORS': 5, │
│ │ 'NUM_CLASSES': 80, │
│ │ 'PRIOR_PROB': 0.01, │
│ │ 'REG_NUM_CONVS': 4}, │
│ │ 'ENCODER': {'ACTIVATION': 'ReLU', │
│ │ 'BLOCK_DILATIONS': [2, 4, 6, 8], │
│ │ 'BLOCK_MID_CHANNELS': 128, │
│ │ 'IN_FEATURES': ['res5'], │
│ │ 'NORM': 'BN', │
│ │ 'NUM_CHANNELS': 512, │
│ │ 'NUM_RESIDUAL_BLOCKS': 4}, │
│ │ 'FOCAL_LOSS_ALPHA': 0.25, │
│ │ 'FOCAL_LOSS_GAMMA': 2.0, │
│ │ 'MATCHER_TOPK': 4, │
│ │ 'NEG_IGNORE_THRESHOLD': 0.7, │
│ │ 'NMS_THRESH_TEST': 0.6, │
│ │ 'POS_IGNORE_THRESHOLD': 0.15, │
│ │ 'SCORE_THRESH_TEST': 0.05, │
│ │ 'TOPK_CANDIDATES_TEST': 1000}} │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ INPUT │ {'AUG': {'TEST_PIPELINES': [('ResizeShortestEdge', │
│ │ {'max_size': 1333, │
│ │ 'sample_style': 'choice', │
│ │ 'short_edge_length': 800})], │
│ │ 'TRAIN_PIPELINES': [('ResizeShortestEdge', │
│ │ {'max_size': 1333, │
│ │ 'sample_style': 'choice', │
│ │ 'short_edge_length': (800,)}), │
│ │ ('RandomFlip', {}), │
│ │ ('RandomShift', {'max_shifts': 32})]}, │
│ │ 'FORMAT': 'BGR', │
│ │ 'MASK_FORMAT': 'polygon'} │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ DATASETS │ {'CUSTOM_TYPE': ['ConcatDataset', {}], │
│ │ 'PRECOMPUTED_PROPOSAL_TOPK_TEST': 1000, │
│ │ 'PRECOMPUTED_PROPOSAL_TOPK_TRAIN': 2000, │
│ │ 'PROPOSAL_FILES_TEST': [], │
│ │ 'PROPOSAL_FILES_TRAIN': [], │
│ │ 'TEST': ['coco_2017_val'], │
│ │ 'TRAIN': ['coco_2017_train']} │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ DATALOADER │ {'ASPECT_RATIO_GROUPING': True, │
│ │ 'FILTER_EMPTY_ANNOTATIONS': True, │
│ │ 'NUM_WORKERS': 0, │
│ │ 'REPEAT_THRESHOLD': 0.0, │
│ │ 'SAMPLER_TRAIN': 'DistributedGroupSampler'} │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ SOLVER │ {'BATCH_SUBDIVISIONS': 1, │
│ │ 'CHECKPOINT_PERIOD': 5000, │
│ │ 'CLIP_GRADIENTS': {'CLIP_TYPE': 'value', │
│ │ 'CLIP_VALUE': 1.0, │
│ │ 'ENABLED': False, │
│ │ 'NORM_TYPE': 2.0}, │
│ │ 'IMS_PER_BATCH': 32, │
│ │ 'IMS_PER_DEVICE': 8, │
│ │ 'LR_SCHEDULER': {'EPOCH_ITERS': -1, │
│ │ 'GAMMA': 0.1, │
│ │ 'MAX_EPOCH': None, │
│ │ 'MAX_ITER': 22500, │
│ │ 'NAME': 'WarmupMultiStepLR', │
│ │ 'STEPS': [15000, 20000], │
│ │ 'WARMUP_FACTOR': 0.00066667, │
│ │ 'WARMUP_ITERS': 3000, │
│ │ 'WARMUP_METHOD': 'linear'}, │
│ │ 'OPTIMIZER': {'BACKBONE_LR_FACTOR': 0.334, │
│ │ 'BASE_LR': 0.0005, │
│ │ 'BIAS_LR_FACTOR': 1.0, │
│ │ 'MOMENTUM': 0.9, │
│ │ 'NAME': 'D2SGD', │
│ │ 'WEIGHT_DECAY': 0.0001, │
│ │ 'WEIGHT_DECAY_NORM': 0.0}} │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ TEST │ {'AUG': {'ENABLED': False, │
│ │ 'EXTRA_SIZES': [], │
│ │ 'FLIP': True, │
│ │ 'MAX_SIZE': 4000, │
│ │ 'MIN_SIZES': [400, 500, 600, 700, 800, 900, 1000, 1100, 1200], │
│ │ 'SCALE_FILTER': False, │
│ │ 'SCALE_RANGES': []}, │
│ │ 'DETECTIONS_PER_IMAGE': 100, │
│ │ 'EVAL_PERIOD': 0, │
│ │ 'EXPECTED_RESULTS': [], │
│ │ 'KEYPOINT_OKS_SIGMAS': [], │
│ │ 'PRECISE_BN': {'ENABLED': False, 'NUM_ITER': 200}} │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ TRAINER │ {'FP16': {'ENABLED': False, 'OPTS': {'OPT_LEVEL': 'O1'}, 'TYPE': 'APEX'}, │
│ │ 'NAME': 'DefaultRunner', │
│ │ 'WINDOW_SIZE': 20} │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ OUTPUT_DIR │ './output' │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ SEED │ 57776492 │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ CUDNN_BENCHMARK │ False │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ VIS_PERIOD │ 0 │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ GLOBAL │ {'DUMP_TEST': False, 'DUMP_TRAIN': True, 'HACK': 1.0} │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ build_backbone │ <function build_backbone at 0x7f128cfb3488> │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ build_anchor_generator │ <function build_anchor_generator at 0x7f128cfb3510> │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ build_encoder │ <function build_encoder at 0x7f128cfb3598> │
├────────────────────────┼───────────────────────────────────────────────────────────────────────────┤
│ build_decoder │ <function build_decoder at 0x7f128cfb3b70> │
╘════════════════════════╧═══════════════════════════════════════════════════════════════════════════╛
[04/08 19:33:21] cvpods INFO: different config with base class:
╒════════════════════════╤═════════════════════════════════════════════════════════════════════╕
│ config params │ values │
╞════════════════════════╪═════════════════════════════════════════════════════════════════════╡
│ MODEL │ {'ANCHOR_GENERATOR': {'ASPECT_RATIOS': [[1.0]]}, │
│ │ 'RESNETS': {'DEPTH': 50, 'OUT_FEATURES': ['res5']}, │
│ │ 'WEIGHTS': 'detectron2://ImageNetPretrained/MSRA/R-50.pkl', │
│ │ 'YOLOF': {'ADD_CTR_CLAMP': True, │
│ │ 'BBOX_REG_WEIGHTS': [1.0, 1.0, 1.0, 1.0], │
│ │ 'CTR_CLAMP': 32, │
│ │ 'DECODER': {'ACTIVATION': 'ReLU', │
│ │ 'CLS_NUM_CONVS': 2, │
│ │ 'IN_CHANNELS': 512, │
│ │ 'NORM': 'BN', │
│ │ 'NUM_ANCHORS': 5, │
│ │ 'NUM_CLASSES': 80, │
│ │ 'PRIOR_PROB': 0.01, │
│ │ 'REG_NUM_CONVS': 4}, │
│ │ 'ENCODER': {'ACTIVATION': 'ReLU', │
│ │ 'BLOCK_DILATIONS': [2, 4, 6, 8], │
│ │ 'BLOCK_MID_CHANNELS': 128, │
│ │ 'IN_FEATURES': ['res5'], │
│ │ 'NORM': 'BN', │
│ │ 'NUM_CHANNELS': 512, │
│ │ 'NUM_RESIDUAL_BLOCKS': 4}, │
│ │ 'FOCAL_LOSS_ALPHA': 0.25, │
│ │ 'FOCAL_LOSS_GAMMA': 2.0, │
│ │ 'MATCHER_TOPK': 4, │
│ │ 'NEG_IGNORE_THRESHOLD': 0.7, │
│ │ 'NMS_THRESH_TEST': 0.6, │
│ │ 'POS_IGNORE_THRESHOLD': 0.15, │
│ │ 'SCORE_THRESH_TEST': 0.05, │
│ │ 'TOPK_CANDIDATES_TEST': 1000}} │
├────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ INPUT │ {'AUG': {'TRAIN_PIPELINES': [('ResizeShortestEdge', │
│ │ {'max_size': 1333, │
│ │ 'sample_style': 'choice', │
│ │ 'short_edge_length': (800,)}), │
│ │ ('RandomFlip', {}), │
│ │ ('RandomShift', {'max_shifts': 32})]}} │
├────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ DATASETS │ {'TEST': ['coco_2017_val'], 'TRAIN': ['coco_2017_train']} │
├────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ DATALOADER │ {'NUM_WORKERS': 0} │
├────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ SOLVER │ {'IMS_PER_BATCH': 32, │
│ │ 'IMS_PER_DEVICE': 8, │
│ │ 'LR_SCHEDULER': {'EPOCH_ITERS': -1, │
│ │ 'MAX_ITER': 22500, │
│ │ 'STEPS': [15000, 20000], │
│ │ 'WARMUP_FACTOR': 0.00066667, │
│ │ 'WARMUP_ITERS': 3000}, │
│ │ 'OPTIMIZER': {'BACKBONE_LR_FACTOR': 0.334, 'BASE_LR': 0.0005}} │
├────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ SEED │ 57776492 │
├────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ build_backbone │ <function build_backbone at 0x7f128cfb3488> │
├────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ build_anchor_generator │ <function build_anchor_generator at 0x7f128cfb3510> │
├────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ build_encoder │ <function build_encoder at 0x7f128cfb3598> │
├────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ build_decoder │ <function build_decoder at 0x7f128cfb3b70> │
╘════════════════════════╧═════════════════════════════════════════════════════════════════════╛
[04/08 19:33:21] c2.engine.runner INFO: Starting training from iteration 0
[04/08 19:33:23] c2.utils.dump.events INFO: eta: N/A iter: 1/22500 total_loss: 2.216 loss_cls: 1.290 loss_box_reg: 0.926 data_time: 1.1023 lr: 0.000000 max_mem: 4760M
[04/08 19:33:41] c2.utils.dump.events INFO: eta: 5:20:30 iter: 20/22500 total_loss: 2.150 loss_cls: 1.287 loss_box_reg: 0.864 time: 0.8982 data_time: 0.5389 lr: 0.000003 max_mem: 5186M
[04/08 19:34:00] c2.utils.dump.events INFO: eta: 5:37:23 iter: 40/22500 total_loss: 2.165 loss_cls: 1.284 loss_box_reg: 0.882 time: 0.9086 data_time: 0.5233 lr: 0.000007 max_mem: 5190M
[04/08 19:34:20] c2.utils.dump.events INFO: eta: 5:38:19 iter: 60/22500 total_loss: 2.116 loss_cls: 1.271 loss_box_reg: 0.851 time: 0.9096 data_time: 0.5163 lr: 0.000010 max_mem: 5190M
[04/08 19:34:39] c2.utils.dump.events INFO: eta: 5:36:14 iter: 80/22500 total_loss: 2.102 loss_cls: 1.262 loss_box_reg: 0.842 time: 0.9070 data_time: 0.4970 lr: 0.000013 max_mem: 5200M
[04/08 19:34:58] c2.utils.dump.events INFO: eta: 5:38:46 iter: 100/22500 total_loss: 2.100 loss_cls: 1.250 loss_box_reg: 0.854 time: 0.9112 data_time: 0.5313 lr: 0.000017 max_mem: 5200M
[04/08 19:35:18] c2.utils.dump.events INFO: eta: 5:39:41 iter: 120/22500 total_loss: 2.056 loss_cls: 1.230 loss_box_reg: 0.846 time: 0.9126 data_time: 0.5210 lr: 0.000020 max_mem: 5200M
[04/08 19:35:37] c2.utils.dump.events INFO: eta: 5:40:35 iter: 140/22500 total_loss: 2.055 loss_cls: 1.211 loss_box_reg: 0.842 time: 0.9142 data_time: 0.5304 lr: 0.000023 max_mem: 5200M
[04/08 19:35:56] c2.utils.dump.events INFO: eta: 5:39:39 iter: 160/22500 total_loss: 1.986 loss_cls: 1.193 loss_box_reg: 0.799 time: 0.9133 data_time: 0.5155 lr: 0.000027 max_mem: 5200M
[04/08 19:36:16] c2.utils.dump.events INFO: eta: 5:40:21 iter: 180/22500 total_loss: 1.994 loss_cls: 1.167 loss_box_reg: 0.819 time: 0.9144 data_time: 0.5241 lr: 0.000030 max_mem: 5200M
[04/08 19:36:35] c2.utils.dump.events INFO: eta: 5:39:42 iter: 200/22500 total_loss: 1.966 loss_cls: 1.152 loss_box_reg: 0.808 time: 0.9146 data_time: 0.5258 lr: 0.000033 max_mem: 5203M
[04/08 19:36:54] c2.utils.dump.events INFO: eta: 5:38:09 iter: 220/22500 total_loss: 1.916 loss_cls: 1.127 loss_box_reg: 0.789 time: 0.9121 data_time: 0.5016 lr: 0.000037 max_mem: 5203M
[04/08 19:37:13] c2.utils.dump.events INFO: eta: 5:38:45 iter: 240/22500 total_loss: 1.911 loss_cls: 1.114 loss_box_reg: 0.798 time: 0.9122 data_time: 0.5127 lr: 0.000040 max_mem: 5203M
[04/08 19:37:33] c2.utils.dump.events INFO: eta: 5:37:12 iter: 260/22500 total_loss: 1.869 loss_cls: 1.097 loss_box_reg: 0.779 time: 0.9119 data_time: 0.5214 lr: 0.000043 max_mem: 5203M
[04/08 19:37:52] c2.utils.dump.events INFO: eta: 5:36:26 iter: 280/22500 total_loss: 1.874 loss_cls: 1.068 loss_box_reg: 0.788 time: 0.9105 data_time: 0.4854 lr: 0.000047 max_mem: 5203M
[04/08 19:38:11] c2.utils.dump.events INFO: eta: 5:36:16 iter: 300/22500 total_loss: 1.859 loss_cls: 1.069 loss_box_reg: 0.794 time: 0.9115 data_time: 0.5283 lr: 0.000050 max_mem: 5203M
[04/08 19:38:26] c2.engine.base_runner ERROR: Exception during training:
Traceback (most recent call last):
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 84, in train
self.run_step()
File "/media/6855ca5f-2432-4ace-ab31-3877011231fc/CODE_detection/YOLOF/cvpods/engine/base_runner.py", line 185, in run_step
loss_dict = self.model(data)
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 447, in forward
output = self.module(*inputs[0], **kwargs[0])
File "/home/env/python3.6env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "../yolof_base/yolof.py", line 135, in forward
pred_logits, pred_anchor_deltas)
File "../yolof_base/yolof.py", line 218, in losses
pred_class_logits[valid_idxs],
RuntimeError: copy_if failed to synchronize: device-side assert triggered
[04/08 19:38:26] c2.engine.hooks INFO: Overall training speed: 312 iterations in 0:04:45 (0.9148 s / it)
[04/08 19:38:26] c2.engine.hooks INFO: Total training time: 0:05:01 (0:00:16 on hooks)
It seems that you modify the BASE_LR. But it's weird. We have not encountered this error before.
Could you try the standard 8 GPUs setting and train YOLOF again?
It seems that you modify the
BASE_LR. But it's weird. We have not encountered this error before.
Could you try the standard 8 GPUs setting and train YOLOF again?
thx, i will try it later
hello, the error is about the unusual value as follw:
It seems that there is somewhere the code performances ZeroDivision. Does this error occurs definitely or randomly?
It seems that there is somewhere the code performances ZeroDivision. Does this error occurs definitely or randomly?
it occurs randomly
I met the same problem? Have you fixed it?