dwofk/fast-depth

Error: CUDAError: Check failed: ret == 0 (-1 vs. 0) : cuModuleLoadData(&(module_[device_id]), data_.c_str()) failed with error: CUDA_ERROR_INVALID_PTX

expectopatronm opened this issue · 3 comments

expectopatronm commented

Hi!,

I get this error when I run the example from the Github description.

jetson@jetson:~/fast-depth/deploy$ python3 tx2_run_tvm.py --input-fp data/rgb.npy --output-fp data/pred.npy --model-dir ../results/tvm_compiled/tx2_gpu_mobilenet_nnconv5dw_skipadd_pruned/ --cuda True
=> [TVM on TX2] using model files in ../results/tvm_compiled/tx2_gpu_mobilenet_nnconv5dw_skipadd_pruned/
=> [TVM on TX2] loading model lib and ptx
=> [TVM on TX2] loading model graph and params
=> [TVM on TX2] creating TVM runtime module
=> [TVM on TX2] feeding inputs and params into TVM module
=> [TVM on TX2] running TVM module, saving output
Traceback (most recent call last):

  File "tx2_run_tvm.py", line 91, in <module>
    main()

  File "tx2_run_tvm.py", line 88, in main
    run_model(args.model_dir, args.input_fp, args.output_fp, args.warmup, args.run, args.cuda,  try_randin=args.randin)

  File "tx2_run_tvm.py", line 36, in run_model
    run() # not gmodule.run()

  File "/home/jetson/tvm/python/tvm/_ffi/_ctypes/function.py", line 207, in __call__
    raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (3) /home/jetson/tvm/build/libtvm.so(TVMFuncCall+0x70) [0x7fad7ccec0]
  [bt] (2) /home/jetson/tvm/build/libtvm.so(std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::detail::PackFuncVoidAddr_<4, tvm::runtime::CUDAWrappedFunc>(tvm::runtime::CUDAWrappedFunc, std::vector<tvm::runtime::detail::ArgConvertCode, std::allocator<tvm::runtime::detail::ArgConvertCode> > const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)+0xe8) [0x7fad850b08]
  [bt] (1) /home/jetson/tvm/build/libtvm.so(tvm::runtime::CUDAWrappedFunc::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*, void**) const+0x6cc) [0x7fad85093c]
  [bt] (0) /home/jetson/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4c) [0x7facfdebac]
  File "/home/jetson/tvm/src/runtime/cuda/cuda_module.cc", line 110
  File "/home/jetson/tvm/src/runtime/library_module.cc", line 91
CUDAError: Check failed: ret == 0 (-1 vs. 0) : cuModuleLoadData(&(module_[device_id]), data_.c_str()) failed with error: CUDA_ERROR_INVALID_PTX

I can't seem to find a solution to this. Could you please help?

dwofk commented

Hi @expectopatronm

This seems like a compilation issue. The compiled models provided in this repo were compiled for the following environment:

NVIDIA Jetson TX2 CPU/GPU
CUDA version 8.0
LLVM version 4.0
TVM runtime commit ab4946c8b80da510a5a518dca066d8159473345f

Does this match your environment?

Note that for a different hardware target or a more recent version of TVM, you would need to redo the compilation process to ensure that compiled models are compatible with your environment.

expectopatronm commented
dwofk commented

Hi @expectopatronm

For steps to correctly redo the compilation, visit https://tvm.apache.org/docs/tutorials/index.html#compile-deep-learning-models. The TVM source code has undergone several updates and so the documentation now differs from that which was used in our work.

If you have been encountering further difficulties with compilation using TVM, it would be best to refer to the TVM discussion forum at https://discuss.tvm.ai/.