nlp-uoregon/trankit

RuntimeError: CUDA error: no kernel image is available for execution on the device

Opened this issue · 2 comments

when I run

p=Pipeline('auto')

>>> from trankit import Pipeline
2022-05-31 18:01:41.938559: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
>>> from trankit import Pipeline
>>> p = Pipeline('auto')
Loading pretrained XLM-Roberta, this may take a while...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/trankit/pipeline.py", line 85, in __init__
    self._embedding_layers.half()
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 765, in half
    return self._apply(lambda t: t.half() if t.is_floating_point() else t)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 601, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 765, in <lambda>
    return self._apply(lambda t: t.half() if t.is_floating_point() else t)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

docker image is nvidia/cuda:11.4.2-cudnn8-runtime-ubuntu18.04

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

$ nvidia-smi
NVIDIA-SMI 470.103.01 Driver Version: 470.103.01 CUDA Version: 11.4

$ pip list | grep torch

torch 1.11.0

Same problem here, with Cuda 11.6 and torch 1.12 (and trankit 1.1.1)

ulf1 commented

Same error here

  • trankit.Pipeline(lang="german-hdt", gpu=True, cache_dir="./cache") => RuntimeError: CUDA error: no kernel image is available for execution on the device
  • Debian 11
  • Python 3.9
  • torch.cuda.is_available() returns True