RuntimeError: CUDA error: device-side assert triggered

[ZSY-SZ]

10/18/2021 03:06:23 Updates[ 0] train loss[nan] train em[0.00000] f1[0.00000] remaining[13:46:19]
/pytorch/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [38,0,0], thread: [459,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [39,0,0], thread: [255,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
/pytorch/aten/src/THC/THCTensorScatterGather.cu:97: void THCudaTensor_gatherKernel(TensorInfo<Real, IndexType>, TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = 3]: block: [46,0,0], thread: [365,0,0] Assertion indexValue >= 0 && indexValue < src.sizes[dim] failed.
Traceback (most recent call last):
File "./roberta_gcn_cli.py", line 104, in
main()
File "./roberta_gcn_cli.py", line 83, in main
model.update(batch)
File "/home/laic2021/fajinyaosu/numnet_plus-master/tools/model.py", line 47, in update
output_dict = self.mnetwork(**tasks)
File "/home/laic2021/anaconda3/envs/numnet/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/laic2021/fajinyaosu/numnet_plus-master/tag_mspan_robert_gcn/tag_mspan_roberta_gcn.py", line 472, in forward
is_bio_mask)
File "/home/laic2021/fajinyaosu/numnet_plus-master/tag_mspan_robert_gcn/multispan_heads.py", line 205, in log_likelihood
if answer_as_text_to_disjoint_bios.sum() > 0:
RuntimeError: CUDA error: device-side assert triggered