ValueError: min() arg is an empty sequence & t1_clustering_with_save

Question

ValueError: min() arg is an empty sequence & t1_clustering_with_save

CtCCtV opened this issue 3 years ago · 5 comments

When runing t1_val, it comes out this wrong result. And after t1_train, folder "t1_clustering_with_save" is not exist in folder "output/t1". Looking forward your responding! Thanks very much!

nohup: ignoring input
Command Line Args: Namespace(config_file='./configs/OWOD/t1/t1_val.yaml', dist_url='tcp://127.0.0.1:52133', eval_only=False, machine_rank=0, num_gpus=2, num_machines=1, opts=['SOLVER.IMS_PER_BATCH', '8', 'SOLVER.BASE_LR', '0.02', 'OWOD.TEMPERATURE', '1.5', 'OUTPUT_DIR', './output/t1_final', 'MODEL.WEIGHTS', './output/t1/model_final.pth'], resume=False)
�[32m[12/10 12:52:59 detectron2]: �[0mRank of current process: 0. World size: 2
�[32m[12/10 12:53:01 detectron2]: �[0mEnvironment info:

sys.platform linux
Python 3.6.13 |Anaconda, Inc.| (default, Jun 4 2021, 14:25:59) [GCC 7.5.0]
numpy 1.19.2
detectron2 0.2.1 @/root/data/MSTAR/detectron2
Compiler GCC 7.4
CUDA compiler CUDA 10.1
detectron2 arch flags 7.0
DETECTRON2_ENV_MODULE
PyTorch 1.6.0 @/opt/conda/envs/owod/lib/python3.6/site-packages/torch
PyTorch debug build False
GPU available True
GPU 0,1 Tesla V100-SXM2-32GB-LS (arch=7.0)
CUDA_HOME /usr/local/cuda
Pillow 8.3.1
torchvision 0.7.0 @/opt/conda/envs/owod/lib/python3.6/site-packages/torchvision
torchvision arch flags 3.5, 5.0, 6.0, 7.0, 7.5
fvcore 0.1.5.post20211023
cv2 Not found

PyTorch built with:

GCC 7.3
C++ Version: 201402
Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
Intel(R) MKL-DNN v1.5.0 (Git Hash e2ac1fac44c5078ca927cb9b90e1b3066a0b2ed0)
OpenMP 201511 (a.k.a. OpenMP 4.5)
NNPACK is enabled
CPU capability usage: AVX2
CUDA Runtime 10.1
NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
CuDNN 7.6.3
Magma 2.5.2
Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

�[32m[12/10 12:53:01 detectron2]: �[0mCommand line arguments: Namespace(config_file='./configs/OWOD/t1/t1_val.yaml', dist_url='tcp://127.0.0.1:52133', eval_only=False, machine_rank=0, num_gpus=2, num_machines=1, opts=['SOLVER.IMS_PER_BATCH', '8', 'SOLVER.BASE_LR', '0.02', 'OWOD.TEMPERATURE', '1.5', 'OUTPUT_DIR', './output/t1_final', 'MODEL.WEIGHTS', './output/t1/model_final.pth'], resume=False)
�[32m[12/10 12:53:01 detectron2]: �[0mContents of args.config_file=./configs/OWOD/t1/t1_val.yaml:
BASE: "../../Base-RCNN-FPN-OWOD.yaml"
MODEL:
#WEIGHTS: "/root/data/MSTAR/output/t1_clustering_with_save/model_final.pth"
WEIGHTS: "./output/t1/model_final.pth"
DATASETS:
TRAIN: ('voc_coco_2007_val', ) # t1_voc_coco_2007_train, t1_voc_coco_2007_ft
TEST: ('voc_coco_2007_val', ) # voc_coco_2007_test
SOLVER:
STEPS: (12000, 16000)
MAX_ITER: 500
WARMUP_ITERS: 0
OUTPUT_DIR: "./output/temp_3"
OWOD:
PREV_INTRODUCED_CLS: 0
#CUR_INTRODUCED_CLS: 20
CUR_INTRODUCED_CLS: 10
COMPUTE_ENERGY: True
ENERGY_SAVE_PATH: 'energy'
SKIP_TRAINING_WHILE_EVAL: False
#ENABLE_CLUSTERING: False
ENABLE_CLUSTERING: True
TEMPERATURE: 1.5
�[32m[12/10 12:53:01 detectron2]: �[0mRunning with full config:
CUDNN_BENCHMARK: False
DATALOADER:
ASPECT_RATIO_GROUPING: True
FILTER_EMPTY_ANNOTATIONS: True
NUM_WORKERS: 4
REPEAT_THRESHOLD: 0.0
SAMPLER_TRAIN: TrainingSampler
DATASETS:
PRECOMPUTED_PROPOSAL_TOPK_TEST: 1000
PRECOMPUTED_PROPOSAL_TOPK_TRAIN: 2000
PROPOSAL_FILES_TEST: ()
PROPOSAL_FILES_TRAIN: ()
TEST: ('voc_coco_2007_val',)
TRAIN: ('voc_coco_2007_val',)
GLOBAL:
HACK: 1.0
INPUT:
CROP:
ENABLED: False
SIZE: [0.9, 0.9]
TYPE: relative_range
FORMAT: BGR
MASK_FORMAT: polygon
MAX_SIZE_TEST: 1333
MAX_SIZE_TRAIN: 1333
MIN_SIZE_TEST: 800
MIN_SIZE_TRAIN: (640, 672, 704, 736, 768, 800)
MIN_SIZE_TRAIN_SAMPLING: choice
RANDOM_FLIP: horizontal
MODEL:
ANCHOR_GENERATOR:
ANGLES: [[-90, 0, 90]]
ASPECT_RATIOS: [[0.5, 1.0, 2.0]]
NAME: DefaultAnchorGenerator
OFFSET: 0.0
SIZES: [[32], [64], [128], [256], [512]]
BACKBONE:
FREEZE_AT: 2
NAME: build_resnet_fpn_backbone
DEVICE: cuda
FPN:
FUSE_TYPE: sum
IN_FEATURES: ['res2', 'res3', 'res4', 'res5']
NORM:
OUT_CHANNELS: 256
KEYPOINT_ON: False
LOAD_PROPOSALS: False
MASK_ON: False
META_ARCHITECTURE: GeneralizedRCNN
PANOPTIC_FPN:
COMBINE:
ENABLED: True
INSTANCES_CONFIDENCE_THRESH: 0.5
OVERLAP_THRESH: 0.5
STUFF_AREA_LIMIT: 4096
INSTANCE_LOSS_WEIGHT: 1.0
PIXEL_MEAN: [103.53, 116.28, 123.675]
PIXEL_STD: [1.0, 1.0, 1.0]
PROPOSAL_GENERATOR:
MIN_SIZE: 0
NAME: RPN
RESNETS:
DEFORM_MODULATED: False
DEFORM_NUM_GROUPS: 1
DEFORM_ON_PER_STAGE: [False, False, False, False]
DEPTH: 50
NORM: FrozenBN
NUM_GROUPS: 1
OUT_FEATURES: ['res2', 'res3', 'res4', 'res5']
RES2_OUT_CHANNELS: 256
RES5_DILATION: 1
STEM_OUT_CHANNELS: 64
STRIDE_IN_1X1: True
WIDTH_PER_GROUP: 64
RETINANET:
BBOX_REG_LOSS_TYPE: smooth_l1
BBOX_REG_WEIGHTS: (1.0, 1.0, 1.0, 1.0)
FOCAL_LOSS_ALPHA: 0.25
FOCAL_LOSS_GAMMA: 2.0
IN_FEATURES: ['p3', 'p4', 'p5', 'p6', 'p7']
IOU_LABELS: [0, -1, 1]
IOU_THRESHOLDS: [0.4, 0.5]
NMS_THRESH_TEST: 0.5
NORM:
NUM_CLASSES: 80
NUM_CONVS: 4
PRIOR_PROB: 0.01
SCORE_THRESH_TEST: 0.05
SMOOTH_L1_LOSS_BETA: 0.1
TOPK_CANDIDATES_TEST: 1000
ROI_BOX_CASCADE_HEAD:
BBOX_REG_WEIGHTS: ((10.0, 10.0, 5.0, 5.0), (20.0, 20.0, 10.0, 10.0), (30.0, 30.0, 15.0, 15.0))
IOUS: (0.5, 0.6, 0.7)
ROI_BOX_HEAD:
BBOX_REG_LOSS_TYPE: smooth_l1
BBOX_REG_LOSS_WEIGHT: 1.0
BBOX_REG_WEIGHTS: (10.0, 10.0, 5.0, 5.0)
CLS_AGNOSTIC_BBOX_REG: False
CONV_DIM: 256
FC_DIM: 1024
NAME: FastRCNNConvFCHead
NORM:
NUM_CONV: 0
NUM_FC: 2
POOLER_RESOLUTION: 7
POOLER_SAMPLING_RATIO: 0
POOLER_TYPE: ROIAlignV2
SMOOTH_L1_BETA: 0.0
TRAIN_ON_PRED_BOXES: False
ROI_HEADS:
BATCH_SIZE_PER_IMAGE: 512
IN_FEATURES: ['p2', 'p3', 'p4', 'p5']
IOU_LABELS: [0, 1]
IOU_THRESHOLDS: [0.5]
NAME: StandardROIHeads
NMS_THRESH_TEST: 0.5
NUM_CLASSES: 11
POSITIVE_FRACTION: 0.25
PROPOSAL_APPEND_GT: True
SCORE_THRESH_TEST: 0.05
ROI_KEYPOINT_HEAD:
CONV_DIMS: (512, 512, 512, 512, 512, 512, 512, 512)
LOSS_WEIGHT: 1.0
MIN_KEYPOINTS_PER_IMAGE: 1
NAME: KRCNNConvDeconvUpsampleHead
NORMALIZE_LOSS_BY_VISIBLE_KEYPOINTS: True
NUM_KEYPOINTS: 17
POOLER_RESOLUTION: 14
POOLER_SAMPLING_RATIO: 0
POOLER_TYPE: ROIAlignV2
ROI_MASK_HEAD:
CLS_AGNOSTIC_MASK: False
CONV_DIM: 256
NAME: MaskRCNNConvUpsampleHead
NORM:
NUM_CONV: 4
POOLER_RESOLUTION: 14
POOLER_SAMPLING_RATIO: 0
POOLER_TYPE: ROIAlignV2
RPN:
BATCH_SIZE_PER_IMAGE: 256
BBOX_REG_LOSS_TYPE: smooth_l1
BBOX_REG_LOSS_WEIGHT: 1.0
BBOX_REG_WEIGHTS: (1.0, 1.0, 1.0, 1.0)
BOUNDARY_THRESH: -1
HEAD_NAME: StandardRPNHead
IN_FEATURES: ['p2', 'p3', 'p4', 'p5', 'p6']
IOU_LABELS: [0, -1, 1]
IOU_THRESHOLDS: [0.3, 0.7]
LOSS_WEIGHT: 1.0
NMS_THRESH: 0.7
POSITIVE_FRACTION: 0.5
POST_NMS_TOPK_TEST: 1000
POST_NMS_TOPK_TRAIN: 1000
PRE_NMS_TOPK_TEST: 1000
PRE_NMS_TOPK_TRAIN: 2000
SMOOTH_L1_BETA: 0.0
SEM_SEG_HEAD:
COMMON_STRIDE: 4
CONVS_DIM: 128
IGNORE_VALUE: 255
IN_FEATURES: ['p2', 'p3', 'p4', 'p5']
LOSS_WEIGHT: 1.0
NAME: SemSegFPNHead
NORM: GN
NUM_CLASSES: 54
WEIGHTS: ./output/t1/model_final.pth
OUTPUT_DIR: ./output/t1_final
OWOD:
CLUSTERING:
ITEMS_PER_CLASS: 20
MARGIN: 10.0
MOMENTUM: 0.99
START_ITER: 1000
UPDATE_MU_ITER: 3000
Z_DIMENSION: 128
COMPUTE_ENERGY: True
CUR_INTRODUCED_CLS: 10
ENABLE_CLUSTERING: True
ENABLE_THRESHOLD_AUTOLABEL_UNK: True
ENABLE_UNCERTAINITY_AUTOLABEL_UNK: False
ENERGY_SAVE_PATH: energy
FEATURE_STORE_SAVE_PATH: feature_store
NUM_UNK_PER_IMAGE: 1
PREV_INTRODUCED_CLS: 0
SKIP_TRAINING_WHILE_EVAL: False
TEMPERATURE: 1.5
SEED: -1
SOLVER:
BASE_LR: 0.02
BIAS_LR_FACTOR: 1.0
CHECKPOINT_PERIOD: 5000
CLIP_GRADIENTS:
CLIP_TYPE: value
CLIP_VALUE: 1.0
ENABLED: False
NORM_TYPE: 2.0
GAMMA: 0.1
IMS_PER_BATCH: 8
LR_SCHEDULER_NAME: WarmupMultiStepLR
MAX_ITER: 500
MOMENTUM: 0.9
NESTEROV: False
REFERENCE_WORLD_SIZE: 0
STEPS: (12000, 16000)
WARMUP_FACTOR: 0.001
WARMUP_ITERS: 0
WARMUP_METHOD: linear
WEIGHT_DECAY: 0.0001
WEIGHT_DECAY_BIAS: 0.0001
WEIGHT_DECAY_NORM: 0.0
TEST:
AUG:
ENABLED: False
FLIP: True
MAX_SIZE: 4000
MIN_SIZES: (400, 500, 600, 700, 800, 900, 1000, 1100, 1200)
DETECTIONS_PER_IMAGE: 100
EVAL_PERIOD: 0
EXPECTED_RESULTS: []
KEYPOINT_OKS_SIGMAS: []
PRECISE_BN:
ENABLED: False
NUM_ITER: 200
VERSION: 2
VIS_PERIOD: 0
�[32m[12/10 12:53:01 detectron2]: �[0mFull config saved to ./output/t1_final/config.yaml
�[32m[12/10 12:53:01 d2.utils.env]: �[0mUsing a generated random seed 1446765
�[32m[12/10 12:53:02 d2.modeling.roi_heads.fast_rcnn]: �[0mInvalid class range: []
�[32m[12/10 12:53:02 d2.modeling.roi_heads.fast_rcnn]: �[0mFeature store not found in ./output/t1_final/feature_store/feat.pt. Creating new feature store.
�[32m[12/10 12:53:02 d2.engine.defaults]: �[0mModel:
GeneralizedRCNN(
(backbone): FPN(
(fpn_lateral2): Conv2d(256, 256, kernel_size=(1, 1), stride=(1, 1))
(fpn_output2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(fpn_lateral3): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1))
(fpn_output3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(fpn_lateral4): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1))
(fpn_output4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(fpn_lateral5): Conv2d(2048, 256, kernel_size=(1, 1), stride=(1, 1))
(fpn_output5): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(top_block): LastLevelMaxPool()
(bottom_up): ResNet(
(stem): BasicStem(
(conv1): Conv2d(
3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
)
(res2): Sequential(
(0): BottleneckBlock(
(shortcut): Conv2d(
64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv1): Conv2d(
64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv2): Conv2d(
64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv3): Conv2d(
64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
)
(1): BottleneckBlock(
(conv1): Conv2d(
256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv2): Conv2d(
64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv3): Conv2d(
64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
)
(2): BottleneckBlock(
(conv1): Conv2d(
256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv2): Conv2d(
64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=64, eps=1e-05)
)
(conv3): Conv2d(
64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
)
)
(res3): Sequential(
(0): BottleneckBlock(
(shortcut): Conv2d(
256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv1): Conv2d(
256, 128, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv2): Conv2d(
128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv3): Conv2d(
128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
)
(1): BottleneckBlock(
(conv1): Conv2d(
512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv2): Conv2d(
128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv3): Conv2d(
128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
)
(2): BottleneckBlock(
(conv1): Conv2d(
512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv2): Conv2d(
128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv3): Conv2d(
128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
)
(3): BottleneckBlock(
(conv1): Conv2d(
512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv2): Conv2d(
128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=128, eps=1e-05)
)
(conv3): Conv2d(
128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
)
)
(res4): Sequential(
(0): BottleneckBlock(
(shortcut): Conv2d(
512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
(conv1): Conv2d(
512, 256, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(1): BottleneckBlock(
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(2): BottleneckBlock(
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(3): BottleneckBlock(
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(4): BottleneckBlock(
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
(5): BottleneckBlock(
(conv1): Conv2d(
1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv2): Conv2d(
256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=256, eps=1e-05)
)
(conv3): Conv2d(
256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=1024, eps=1e-05)
)
)
)
(res5): Sequential(
(0): BottleneckBlock(
(shortcut): Conv2d(
1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05)
)
(conv1): Conv2d(
1024, 512, kernel_size=(1, 1), stride=(2, 2), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv2): Conv2d(
512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv3): Conv2d(
512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05)
)
)
(1): BottleneckBlock(
(conv1): Conv2d(
2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv2): Conv2d(
512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv3): Conv2d(
512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05)
)
)
(2): BottleneckBlock(
(conv1): Conv2d(
2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv2): Conv2d(
512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=512, eps=1e-05)
)
(conv3): Conv2d(
512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False
(norm): FrozenBatchNorm2d(num_features=2048, eps=1e-05)
)
)
)
)
)
(proposal_generator): RPN(
(rpn_head): StandardRPNHead(
(conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(objectness_logits): Conv2d(256, 3, kernel_size=(1, 1), stride=(1, 1))
(anchor_deltas): Conv2d(256, 12, kernel_size=(1, 1), stride=(1, 1))
)
(anchor_generator): DefaultAnchorGenerator(
(cell_anchors): BufferList()
)
)
(roi_heads): StandardROIHeads(
(box_pooler): ROIPooler(
(level_poolers): ModuleList(
(0): ROIAlign(output_size=(7, 7), spatial_scale=0.25, sampling_ratio=0, aligned=True)
(1): ROIAlign(output_size=(7, 7), spatial_scale=0.125, sampling_ratio=0, aligned=True)
(2): ROIAlign(output_size=(7, 7), spatial_scale=0.0625, sampling_ratio=0, aligned=True)
(3): ROIAlign(output_size=(7, 7), spatial_scale=0.03125, sampling_ratio=0, aligned=True)
)
)
(box_head): FastRCNNConvFCHead(
(flatten): Flatten()
(fc1): Linear(in_features=12544, out_features=1024, bias=True)
(fc_relu1): ReLU()
(fc2): Linear(in_features=1024, out_features=1024, bias=True)
(fc_relu2): ReLU()
)
(box_predictor): FastRCNNOutputLayers(
(cls_score): Linear(in_features=1024, out_features=12, bias=True)
(bbox_pred): Linear(in_features=1024, out_features=44, bias=True)
(hingeloss): HingeEmbeddingLoss()
)
)
)
�[32m[12/10 12:53:02 d2.data.build]: �[0mRemoved 0 images with no usable annotations. 150 images left.
�[32m[12/10 12:53:02 d2.data.build]: �[0mKnown classes: range(0, 10)
�[32m[12/10 12:53:02 d2.data.build]: �[0mLabelling known instances the corresponding label, and unknown instances as unknown...
�[32m[12/10 12:53:02 d2.data.build]: �[0mDistribution of instances among all 10 categories:
�[36m| category | #instances | category | #instances | category | #instances |
|:----------:|:-------------|:----------:|:-------------|:----------:|:-------------|
| 2S1 | 16 | BMP2 | 7 | BRDM_2 | 15 |
| BTR60 | 12 | BTR70 | 13 | D7 | 21 |
| T62 | 16 | T72 | 13 | ZIL131 | 14 |
| ZSU_23_4 | 23 | | | | |
| total | 150 | | | | |�[0m
�[32m[12/10 12:53:02 d2.data.build]: �[0mNumber of datapoints: 150
�[32m[12/10 12:53:02 d2.data.common]: �[0mSerializing 150 elements to byte tensors and concatenating them all ...
�[32m[12/10 12:53:02 d2.data.common]: �[0mSerialized dataset takes 0.06 MiB
�[32m[12/10 12:53:02 d2.data.dataset_mapper]: �[0mAugmentations used in training: [ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice'), RandomFlip()]
�[32m[12/10 12:53:02 d2.data.build]: �[0mUsing training sampler TrainingSampler
�[32m[12/10 12:53:03 fvcore.common.checkpoint]: �[0m[Checkpointer] Loading from ./output/t1/model_final.pth ...
�[32m[12/10 12:53:03 d2.engine.train_loop]: �[0mStarting training from iteration 0
�[32m[12/10 12:53:19 d2.utils.events]: �[0m eta: 0:03:35 iter: 19 total_loss: 0.2402 loss_cls: 0.1202 loss_box_reg: 0.1146 loss_rpn_cls: 9.183e-05 loss_rpn_loc: 0.002237 time: 0.4494 data_time: 0.3601 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:53:28 d2.utils.events]: �[0m eta: 0:03:26 iter: 39 total_loss: 0.1893 loss_cls: 0.09883 loss_box_reg: 0.0868 loss_rpn_cls: 0.0001209 loss_rpn_loc: 0.002168 time: 0.4527 data_time: 0.0148 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:53:37 d2.utils.events]: �[0m eta: 0:03:17 iter: 59 total_loss: 0.1571 loss_cls: 0.09324 loss_box_reg: 0.06255 loss_rpn_cls: 0.0001334 loss_rpn_loc: 0.002446 time: 0.4471 data_time: 0.0137 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:53:47 d2.utils.events]: �[0m eta: 0:03:08 iter: 79 total_loss: 0.1453 loss_cls: 0.08886 loss_box_reg: 0.05634 loss_rpn_cls: 0.0001182 loss_rpn_loc: 0.002574 time: 0.4515 data_time: 0.0142 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:53:56 d2.utils.events]: �[0m eta: 0:02:59 iter: 99 total_loss: 0.1429 loss_cls: 0.0892 loss_box_reg: 0.05158 loss_rpn_cls: 0.000143 loss_rpn_loc: 0.002405 time: 0.4507 data_time: 0.0146 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:54:04 d2.utils.events]: �[0m eta: 0:02:48 iter: 119 total_loss: 0.1328 loss_cls: 0.07941 loss_box_reg: 0.04987 loss_rpn_cls: 0.0001336 loss_rpn_loc: 0.002058 time: 0.4452 data_time: 0.0131 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:54:13 d2.utils.events]: �[0m eta: 0:02:39 iter: 139 total_loss: 0.1073 loss_cls: 0.06715 loss_box_reg: 0.03592 loss_rpn_cls: 0.0001396 loss_rpn_loc: 0.001973 time: 0.4450 data_time: 0.0140 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:54:22 d2.utils.events]: �[0m eta: 0:02:29 iter: 159 total_loss: 0.1006 loss_cls: 0.05851 loss_box_reg: 0.03925 loss_rpn_cls: 0.0001736 loss_rpn_loc: 0.002042 time: 0.4430 data_time: 0.0133 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:54:31 d2.utils.events]: �[0m eta: 0:02:21 iter: 179 total_loss: 0.1078 loss_cls: 0.0641 loss_box_reg: 0.04139 loss_rpn_cls: 0.0001229 loss_rpn_loc: 0.002328 time: 0.4433 data_time: 0.0149 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:54:40 d2.utils.events]: �[0m eta: 0:02:12 iter: 199 total_loss: 0.101 loss_cls: 0.05992 loss_box_reg: 0.04146 loss_rpn_cls: 0.0001862 loss_rpn_loc: 0.002147 time: 0.4438 data_time: 0.0147 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:54:49 d2.utils.events]: �[0m eta: 0:02:03 iter: 219 total_loss: 0.1108 loss_cls: 0.06426 loss_box_reg: 0.04312 loss_rpn_cls: 0.0002145 loss_rpn_loc: 0.002232 time: 0.4436 data_time: 0.0144 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:54:58 d2.utils.events]: �[0m eta: 0:01:55 iter: 239 total_loss: 0.09784 loss_cls: 0.05322 loss_box_reg: 0.04014 loss_rpn_cls: 0.0001898 loss_rpn_loc: 0.002141 time: 0.4457 data_time: 0.0148 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:55:08 d2.utils.events]: �[0m eta: 0:01:47 iter: 259 total_loss: 0.1006 loss_cls: 0.05431 loss_box_reg: 0.04378 loss_rpn_cls: 0.0001741 loss_rpn_loc: 0.002018 time: 0.4493 data_time: 0.0178 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:55:18 d2.utils.events]: �[0m eta: 0:01:38 iter: 279 total_loss: 0.08747 loss_cls: 0.04406 loss_box_reg: 0.04199 loss_rpn_cls: 0.0001236 loss_rpn_loc: 0.001968 time: 0.4516 data_time: 0.0166 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:55:28 d2.utils.events]: �[0m eta: 0:01:29 iter: 299 total_loss: 0.08848 loss_cls: 0.04494 loss_box_reg: 0.03848 loss_rpn_cls: 0.0001919 loss_rpn_loc: 0.002332 time: 0.4546 data_time: 0.0177 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:55:38 d2.utils.events]: �[0m eta: 0:01:20 iter: 319 total_loss: 0.0814 loss_cls: 0.04312 loss_box_reg: 0.03622 loss_rpn_cls: 0.0001429 loss_rpn_loc: 0.002137 time: 0.4561 data_time: 0.0151 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:55:47 d2.utils.events]: �[0m eta: 0:01:12 iter: 339 total_loss: 0.08302 loss_cls: 0.03718 loss_box_reg: 0.03998 loss_rpn_cls: 0.0001735 loss_rpn_loc: 0.002002 time: 0.4577 data_time: 0.0177 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:55:57 d2.utils.events]: �[0m eta: 0:01:03 iter: 359 total_loss: 0.08012 loss_cls: 0.0382 loss_box_reg: 0.03807 loss_rpn_cls: 0.0001673 loss_rpn_loc: 0.001984 time: 0.4590 data_time: 0.0155 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:56:07 d2.utils.events]: �[0m eta: 0:00:54 iter: 379 total_loss: 0.0717 loss_cls: 0.03408 loss_box_reg: 0.03461 loss_rpn_cls: 0.0001407 loss_rpn_loc: 0.001741 time: 0.4603 data_time: 0.0186 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:56:16 d2.utils.events]: �[0m eta: 0:00:45 iter: 399 total_loss: 0.07132 loss_cls: 0.03198 loss_box_reg: 0.03577 loss_rpn_cls: 0.0001379 loss_rpn_loc: 0.001665 time: 0.4600 data_time: 0.0146 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:56:25 d2.utils.events]: �[0m eta: 0:00:36 iter: 419 total_loss: 0.07331 loss_cls: 0.03177 loss_box_reg: 0.03813 loss_rpn_cls: 0.0001471 loss_rpn_loc: 0.001808 time: 0.4603 data_time: 0.0168 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:56:35 d2.utils.events]: �[0m eta: 0:00:27 iter: 439 total_loss: 0.07228 loss_cls: 0.03138 loss_box_reg: 0.03716 loss_rpn_cls: 0.0001739 loss_rpn_loc: 0.001873 time: 0.4604 data_time: 0.0159 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:56:44 d2.utils.events]: �[0m eta: 0:00:18 iter: 459 total_loss: 0.06972 loss_cls: 0.02995 loss_box_reg: 0.03668 loss_rpn_cls: 0.0001581 loss_rpn_loc: 0.001905 time: 0.4605 data_time: 0.0163 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:56:53 d2.utils.events]: �[0m eta: 0:00:09 iter: 479 total_loss: 0.08182 loss_cls: 0.03162 loss_box_reg: 0.04559 loss_rpn_cls: 0.0001826 loss_rpn_loc: 0.002413 time: 0.4604 data_time: 0.0161 lr: 0.02 max_mem: 3419M
�[32m[12/10 12:57:02 fvcore.common.checkpoint]: �[0mSaving checkpoint to ./output/t1_final/model_final.pth
/root/data/MSTAR/detectron2/layers/wrappers.py:240: UserWarning: This overload of nonzero is deprecated:
nonzero()
Consider using one of the following signatures instead:
nonzero(*, bool as_tuple) (Triggered internally at /opt/conda/conda-bld/pytorch_1595629416375/work/torch/csrc/utils/python_arg_parser.cpp:766.)
return x.nonzero().unbind(1)
/opt/conda/envs/owod/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 20 leaked semaphores to clean up at shutdown
len(cache))
Traceback (most recent call last):
File "tools/train_net.py", line 169, in
args=(args,),
File "/root/data/MSTAR/detectron2/engine/launch.py", line 59, in launch
daemon=False,
File "/opt/conda/envs/owod/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/opt/conda/envs/owod/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
while not context.join():
File "/opt/conda/envs/owod/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 119, in join
raise Exception(msg)
Exception:

-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/opt/conda/envs/owod/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
fn(i, *args)
File "/root/data/MSTAR/detectron2/engine/launch.py", line 94, in _distributed_worker
main_func(*args)
File "/root/data/MSTAR/tools/train_net.py", line 157, in main
return trainer.train()
File "/root/data/MSTAR/detectron2/engine/defaults.py", line 408, in train
super().train(self.start_iter, self.max_iter)
File "/root/data/MSTAR/detectron2/engine/train_loop.py", line 157, in train
self.after_train()
File "/root/data/MSTAR/detectron2/engine/train_loop.py", line 172, in after_train
self.analyse_energy()
File "/root/data/MSTAR/detectron2/engine/train_loop.py", line 220, in analyse_energy
wb_unk = Fit_Weibull_3P(failures=unk, show_probability_plot=False, print_results=False)
File "/opt/conda/envs/owod/lib/python3.6/site-packages/reliability/Fitters.py", line 2622, in init
CI_type=CI_type,
File "/opt/conda/envs/owod/lib/python3.6/site-packages/reliability/Utils.py", line 1224, in init
elif min(all_data) < 0:
ValueError: min() arg is an empty sequence

Answer 1 · 2022-03-24T14:28:18.000Z

I came into the same problem. Did you resolved it?

Answer 2 · 2022-05-12T12:15:44.000Z

Please refer to the commands here. The MODEL.WEIGHTS are dynamically updated in the script. Please follow the run.sh for your convenience.

Answer 3 · 2022-09-12T08:14:08.000Z

@CtCCtV I came into the same problem. Did you resolved it? Looking forward to your reply！

Answer 4 · 2022-09-12T08:23:57.000Z

Please refer to the commands here. The MODEL.WEIGHTS are dynamically updated in the script. Please follow the run.sh for your convenience.

I have run through refer to run.sh. When I start running replicate. sh, there's no output/t1_clustering_with_save/model_final.pth. Could you tell me why? Looking forward to your reply!

Answer 5 · 2023-05-27T08:10:53.000Z

@luckychay @nulidetuanzi I also had the same problem, please how did you solve it