RuntimeError: Not compiled with GPU support
Opened this issue · 1 comments
KavitShah1998 commented
Hi @ddshan , I tried to setup the v-env based on the instructions in the README.md, but on running both demo.py as well as test_net.py I get this error of RuntimeError: Not compiled with GPU support. Not sure what is going wrong during installation.
Do you happen to know the root cause?
(handobj_new) kavitshah@frerd001:~/fair/hand_object_detector$ CUDA_VISIBLE_DEVICES=0 python test_net.py --model_name=handobj_100K --save_name=handobj_100K --cuda --checkepoch=8 --checkpoint=132028
Called with args:
Namespace(cfg_file='cfgs/resnet101.yml', checkepoch=8, checkpoint=132028, checksession=1, class_agnostic=False, cuda=True, dataset='pascal_voc', large_scale=False, load_dir='models', mGPUs=False, model_name='handobj_100K', net='res101', parallel_type=0, save_name='handobj_100K', set_cfgs=None, thresh_hand=0.1, thresh_obj=0.1, vis=False)
Using config:
{'ANCHOR_RATIOS': [0.5, 1, 2],
'ANCHOR_SCALES': [8, 16, 32, 64],
'CROP_RESIZE_WITH_MAX_POOL': False,
'CUDA': False,
'DATA_DIR': '~/fair/hand_object_detector/data',
'DEDUP_BOXES': 0.0625,
'EPS': 1e-14,
'EXP_DIR': 'res101',
'FEAT_STRIDE': [16],
'GPU_ID': 0,
'MATLAB': 'matlab',
'MAX_NUM_GT_BOXES': 20,
'MOBILENET': {'DEPTH_MULTIPLIER': 1.0,
'FIXED_LAYERS': 5,
'REGU_DEPTH': False,
'WEIGHT_DECAY': 4e-05},
'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
'POOLING_MODE': 'align',
'POOLING_SIZE': 7,
'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
'RNG_SEED': 3,
'ROOT_DIR': '~/fair/hand_object_detector',
'TEST': {'BBOX_REG': True,
'HAS_RPN': True,
'MAX_SIZE': 1000,
'MODE': 'nms',
'NMS': 0.3,
'PROPOSAL_METHOD': 'gt',
'RPN_MIN_SIZE': 16,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'RPN_TOP_N': 5000,
'SCALES': [600],
'SVM': False},
'TRAIN': {'ASPECT_GROUPING': False,
'BATCH_SIZE': 128,
'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'BIAS_DECAY': False,
'BN_TRAIN': False,
'DISPLAY': 20,
'DOUBLE_BIAS': False,
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'GAMMA': 0.1,
'HAS_RPN': True,
'IMS_PER_BATCH': 1,
'LEARNING_RATE': 0.001,
'MAX_SIZE': 1000,
'MOMENTUM': 0.9,
'PROPOSAL_METHOD': 'gt',
'RPN_BATCHSIZE': 256,
'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 8,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SNAPSHOT_ITERS': 5000,
'SNAPSHOT_KEPT': 3,
'SNAPSHOT_PREFIX': 'res101_faster_rcnn',
'STEPSIZE': [30000],
'SUMMARY_INTERVAL': 180,
'TRIM_HEIGHT': 600,
'TRIM_WIDTH': 600,
'TRUNCATED': False,
'USE_ALL_GT': True,
'USE_FLIPPED': True,
'USE_GT': False,
'WEIGHT_DECAY': 0.0001},
'USE_GPU_NMS': True}
--------> dataset path = ~/fair/hand_object_detector/data/VOCdevkit2007_handobj_100K
Loaded dataset `voc_2007_test` for training
Set proposal method: gt
voc_2007_test
Preparing training data...
voc_2007_test gt roidb loaded from ~/fair/hand_object_detector/data/cache_handobj_100K/voc_2007_test_gt_roidb.pkl
done
--------> dataset path = ~/fair/hand_object_detector/data/VOCdevkit2007_handobj_100K
9983 roidb entries
---------> which model = models/res101_handobj_100K/pascal_voc/faster_rcnn_1_8_132028.pth
load checkpoint models/res101_handobj_100K/pascal_voc/faster_rcnn_1_8_132028.pth
load model successfully!
---------> det score thres_hand = 0.1
---------> det score thres_obj = 0.1
Traceback (most recent call last):
File "test_net.py", line 245, in <module>
rois_label, loss_list = fasterRCNN(im_data, im_info, gt_boxes, num_boxes, box_info)
File "~/miniconda3/envs/hos/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "~/fair/hand_object_detector/lib/model/faster_rcnn/faster_rcnn.py", line 57, in forward
rois, rpn_loss_cls, rpn_loss_bbox = self.RCNN_rpn(base_feat, im_info, gt_boxes, num_boxes)
File "~/miniconda3/envs/hos/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "~/fair/hand_object_detector/lib/model/rpn/rpn.py", line 77, in forward
rois = self.RPN_proposal((rpn_cls_prob.data, rpn_bbox_pred.data,
File "~/miniconda3/envs/hos/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "~/fair/hand_object_detector/lib/model/rpn/proposal_layer.py", line 147, in forward
keep_idx_i = nms(proposals_single, scores_single.squeeze(1), nms_thresh)
RuntimeError: Not compiled with GPU support
KavitShah1998 commented
We were able to resolve the issue
This was a 2 layer problem
- Issue 1 : The "CUDA_HOME" environment variable was not set for me as a part of
cudatoolkitinstallation (neither did it exist in the native CUDA setup)
Fix - The solution mentioned here - conda/conda#7757 (comment) worked for me which essentially sets CUDA_HOME to$CONDA_PREFIX. - Issue 2 : Once Issue1 was fixed, it throws a new error of nvcc not found.
I triedwhich nvccbut there was not output asnvccdidn't exist.
nvcc can be installed with cudatoolkit-dev, so had to run this additional command
conda install cudatoolkit-dev=11.3 -c pytorch -c nvidia -c conda-forge
After running this command which nvcc was able to give the right path to nvcc installed within the venv.
Now just follow the following steps to rebuild & voila!
cd lib/ rm -rf build python setup.py build develop
@ddshan you may want to add setup instruction for cudatoolkit-dev for installing nvcc.