RuntimeError: cublas runtime error

[Gironote]

My environment : Ubuntu-16.04 CUDA-9.0, CUDNN-7.0.5, Python-3.7.0, PyTorch-1.0.0, tensorboard-2.0.0 tensorflow-2.0.0 and anaconda
when I run this code $ python train/train_net_det.py --cfg cfgs/det_sample.yaml OUTPUT_DIR output/car_train
this error occur

parameters: 3316777
using optimizer method adam
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544176307774/work/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument
Traceback (most recent call last):
File "train/train_net_det.py", line 404, in
main()
File "train/train_net_det.py", line 371, in main
train(train_loader, model, optimizer, lr_scheduler, n, logger_train)
File "train/train_net_det.py", line 123, in train
losses, metrics = model(data_dicts_var)
File "/home/kaai/anaconda3/envs/frust/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/kaai/frustum-convnet/models/det_base.py", line 462, in forward
(center_label, heading_label, size_label)
File "/home/kaai/frustum-convnet/models/det_base.py", line 320, in get_corner_loss
corners_3d_gt = get_box3d_corners_helper(center_label, heading_label, size_label)
File "/home/kaai/frustum-convnet/models/model_util.py", line 69, in get_box3d_corners_helper
corners_3d = torch.bmm(R, corners) # (N,3,8)
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1544176307774/work/aten/src/THC/THCBlas.cu:441