silviazuffi/smalst

cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED

lorentz-wu opened this issue · 5 comments

lorentz-wu commented

my cuda is 10
when i run script/smalst_op_run.sh , i get some error

outputs = self.forward(in_data)
File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward
y = _matmul(a, b, self.transa, self.transb)
File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul
return xp.matmul(a, b)
File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul
File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul
File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched
File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched
File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status
cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED

how can i solve it?

silviazuffi commented
lorentz-wu commented

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10.
…
On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

thank you for your help

lorentz-wu commented

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10.
…
On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

dear silviazuffi,
could you tell me your cuda's version? I will try the cuda with same version

silviazuffi commented
QingAL commented

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10.
…
On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

dear silviazuffi,
could you tell me your cuda's version? I will try the cuda with same version

hi,Did you solve this problem when use cuda9.2?
my cuda version is 9.0 also get this error

  • 👍1