cuda error
Closed this issue · 6 comments
03/16/2021 16:09:53 - INFO - transformers.modeling_utils - loading weights file /home/lab/Desktop/xf_event_extraction2020Top1-master/bert/torch_roberta_wwm/pytorch_model.bin
03/16/2021 16:16:01 - INFO - src_final.utils.functions_utils - Use single gpu in: ['1']
03/16/2021 16:16:01 - INFO - src_final.utils.trainer - ***** Running training *****
03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Num Examples = 7416
03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Num Epochs = 6
03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Total training batch size = 16
03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Total optimization steps = 2784
03/16/2021 16:16:01 - INFO - src_final.utils.trainer - Save model in 464 steps; Eval model in 464 steps
Traceback (most recent call last):
File "train.py", line 223, in
training(args)
File "train.py", line 152, in training
train_base(opt, info_dict, train_examples, dev_info)
File "train.py", line 44, in train_base
train(opt, model, train_dataset)
File "/home/lab/Desktop/xf_event_extraction2020Top1-master/src_final/utils/trainer.py", line 136, in train
loss = model(**batch_data)[0]
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lab/Desktop/xf_event_extraction2020Top1-master/src_final/utils/model_utils.py", line 239, in forward
token_type_ids=token_type_ids
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 734, in forward
encoder_attention_mask=encoder_extended_attention_mask,
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 407, in forward
hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 368, in forward
self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 314, in forward
hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/transformers/modeling_bert.py", line 216, in forward
mixed_query_layer = self.query(hidden_states)
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/home/lab/anaconda3/envs/event/lib/python3.7/site-packages/torch/nn/functional.py", line 1612, in linear
output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
你好，我在运行train.sh 一直报这个错误，我看了一下我的cuda正常啊，batchsize也调整了，也找了相关的资料，还是报错，不知道为什么，可以请您帮忙解答一下吗
显卡型号是什么；cuda版本要对应我给的版本；还有再Linux上跑，不要再windows上运行
问题解决了，我用的是一块显卡跑的，改成两块就不报错了
03/16/2021 16:31:40 - INFO - transformers.modeling_utils - loading weights file /home/lab/Desktop/xf_event_extraction2020Top1-master/bert/torch_roberta_wwm/pytorch_model.bin
03/16/2021 16:36:53 - INFO - src_final.utils.functions_utils - Use multi gpus in: ['0', '1']
03/16/2021 16:36:53 - INFO - src_final.utils.trainer - ***** Running training *****
03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Num Examples = 7416
03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Num Epochs = 6
03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Total training batch size = 16
03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Total optimization steps = 2784
03/16/2021 16:36:53 - INFO - src_final.utils.trainer - Save model in 464 steps; Eval model in 464 steps
terminate called after throwing an instance of 'std::runtime_error'
what(): NCCL Error 1: unhandled cuda error
./script/final/train.sh: line 38: 6386 Aborted (core dumped) python train.py --gpu_ids=$GPU_IDS --mode=$MODE --raw_data_dir=$RAW_DATA_DIR --mid_data_dir=$MID_DATA_DIR --aux_data_dir=$AUX_DATA_DIR --bert_dir=$BERT_DIR --output_dir=$OUTPUT_DIR --bert_type=$BERT_TYPE --task_type=$TASK_TYPE --max_seq_len=320 --train_epochs=6 --train_batch_size=16 --lr=2e-5 --other_lr=2e-4 --attack_train="pgd" --swa_start=4 --eval_model --enhance_data --use_trigger_distance
现在报这个错，还是cuda问题？？我是按照项目里面进行配置的，也是在linux下跑的
我直接在虚拟环境中安装了cuda10.1，但还是报上面的错误
我的显卡型号是GeForce RTX 3080
3080在pytorch环境上一直有bug，现在应该是还没有修复