Error in the example

Question

Error in the example

chenjy11223 opened this issue 7 months ago · 4 comments

I tried to run the example of bitblas.Linear in https://github.com/microsoft/BitBLAS/blob/main/docs/QuickStart.md, and encountered the following error：

2024-08-28 23:01:37 [BitBLAS:INFO]: Loaded 0 operators from database.
2024-08-28 23:02:54 [BitBLAS:DEBUG]: Failed to build optimized function for CUDA target with default schedule, Please consider enable hardware aware tuning!
2024-08-28 23:02:54 [BitBLAS:INFO]: Start fast tuning with dynamic range
2024-08-28 23:02:54 [BitBLAS:DEBUG]: Cannot find the appropriate index map for tensorcore
2024-08-28 23:02:54 [BitBLAS:DEBUG]: Apply config {'block': [2], 'thread': [2], 'rstep': [1024], 'reduce_thread': [64], 'vectorize': {'A': 8, 'B_decode': 8}}
2024-08-28 23:02:54 [BitBLAS:DEBUG]: Apply config {'block': [1], 'thread': [1], 'rstep': [1024], 'reduce_thread': [128], 'vectorize': {'A': 8, 'B_decode': 8}}
2024-08-28 23:02:54 [BitBLAS:DEBUG]: Apply config {'block': [4], 'thread': [4], 'rstep': [1024], 'reduce_thread': [32], 'vectorize': {'A': 8, 'B_decode': 8}}
2024-08-28 23:02:54 [BitBLAS:DEBUG]: Apply config {'block': [8], 'thread': [8], 'rstep': [1024], 'reduce_thread': [16], 'vectorize': {'A': 8, 'B_decode': 8}}
2024-08-28 23:02:55 [BitBLAS:DEBUG]: Apply config {'block': [16], 'thread': [16], 'rstep': [512], 'reduce_thread': [8], 'vectorize': {'A': 4, 'B_decode': 8}}
2024-08-28 23:02:55 [BitBLAS:DEBUG]: Apply config {'block': [32], 'thread': [32], 'rstep': [256], 'reduce_thread': [4], 'vectorize': {'A': 2, 'B_decode': 8}}
2024-08-28 23:02:55 [BitBLAS:DEBUG]: Apply config {'block': [64], 'thread': [64], 'rstep': [128], 'reduce_thread': [2], 'vectorize': {'B_decode': 8}}
2024-08-28 23:02:55 [BitBLAS:DEBUG]: Apply config {'block': [128], 'thread': [128], 'rstep': [128], 'vectorize': {'B_decode': 8}}
2024-08-28 23:02:55 [BitBLAS:DEBUG]: Warning: block config [128] is not valid for matmul, skip.
2024-08-28 23:02:55 [BitBLAS:DEBUG]: Warning: block config [128] is not valid for matmul, skip.
2024-08-28 23:02:57 [BitBLAS:DEBUG]: LocalBuilder: An exception occurred Traceback (most recent call last):
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/exec/popen_worker.py", line 87, in main
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/base/utils.py", line 224, in _build
rt_mod = tvm.build(mod, target=arch.target)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/driver/build_module.py", line 297, in build
rt_mod_host = _driver_ffi.tir_to_runtime(annotated_mods, target_host)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in call
raise_last_ffi_error()
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
raise py_err
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/contrib/nvcc.py", line 204, in tvm_callback_cuda_compile
ptx = compile_cuda(code, target_format="fatbin")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/contrib/nvcc.py", line 120, in compile_cuda
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/subprocess.py", line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/subprocess.py", line 1950, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'nvcc'

2024-08-28 23:02:57 [BitBLAS:DEBUG]: LocalBuilder: An exception occurred Traceback (most recent call last):
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/exec/popen_worker.py", line 87, in main
result = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/base/utils.py", line 224, in _build
rt_mod = tvm.build(mod, target=arch.target)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/driver/build_module.py", line 297, in build
rt_mod_host = _driver_ffi.tir_to_runtime(annotated_mods, target_host)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in call
raise_last_ffi_error()
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
raise py_err
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/contrib/nvcc.py", line 204, in tvm_callback_cuda_compile
ptx = compile_cuda(code, target_format="fatbin")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/contrib/nvcc.py", line 120, in compile_cuda
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/subprocess.py", line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/subprocess.py", line 1950, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'nvcc'

2024-08-28 23:02:57 [BitBLAS:DEBUG]: LocalBuilder: An exception occurred Traceback (most recent call last):
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/exec/popen_worker.py", line 87, in main
result = fn(args, kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/base/utils.py", line 224, in _build
rt_mod = tvm.build(mod, target=arch.target)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/driver/build_module.py", line 297, in build
rt_mod_host = _driver_ffi.tir_to_runtime(annotated_mods, target_host)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in call
raise_last_ffi_error()
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/3rdparty/tvm/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
raise py_err
ValueError: Traceback (most recent call last):
68: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::TypedPackedFunc<tvm::runtime::Module (tvm::runtime::Map<tvm::Target, tvm::IRModule, void, void> const&, tvm::Target)>::AssignTypedLambda<tvm::{lambda(tvm::runtime::Map<tvm::Target, tvm::IRModule, void, void> const&, tvm::Target)#6}>(tvm::{lambda(tvm::runtime::Map<tvm::Target, tvm::IRModule, void, void> const&, tvm::Target)#6}, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue)#1}> >::Call(tvm::runtime::PackedFuncObj const, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue)
67: tvm::TIRToRuntime(tvm::runtime::Map<tvm::Target, tvm::IRModule, void, void> const&, tvm::Target const&)
66: tvm::codegen::Build(tvm::IRModule, tvm::Target)
65: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::TypedPackedFunc<tvm::runtime::Module (tvm::IRModule, tvm::Target)>::AssignTypedLambda<tvm::runtime::Module ()(tvm::IRModule, tvm::Target)>(tvm::runtime::Module ()(tvm::IRModule, tvm::Target), std::cxx11::basic_string<char, std::char_traits, std::allocator >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
64: tvm::codegen::BuildCUDA(tvm::IRModule, tvm::Target)
63: tvm::codegen::CodeGenC::AddFunction(tvm::GlobalVar const&, tvm::tir::PrimFunc const&)
62: tvm::codegen::CodeGenC::VisitStmt(tvm::tir::DeclBufferNode const*)
61: tvm::codegen::CodeGenC::VisitStmt(tvm::tir::DeclBufferNode const*)
60: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::DeclBufferNode const*)
59: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::DeclBufferNode const*)
58: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::DeclBufferNode const*)
57: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::DeclBufferNode const*)
56: tvm::codegen::CodeGenCUDA::VisitStmt_(tvm::tir::AttrStmtNode const*)
55: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::AttrStmtNode const*)
54: tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
53: tvm::codegen::CodeGenCUDA::VisitStmt_(tvm::tir::AllocateNode const*)
52: tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
51: tvm::codegen::CodeGenCUDA::VisitStmt_(tvm::tir::AllocateNode const*)
50: tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
49: tvm::codegen::CodeGenCUDA::VisitStmt_(tvm::tir::AllocateNode const*)
48: tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
47: tvm::codegen::CodeGenCUDA::VisitStmt_(tvm::tir::AttrStmtNode const*)
46: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::AttrStmtNode const*)
45: tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
44: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::SeqStmtNode const*)
43: tvm::codegen::CodeGenCUDA::VisitStmt_(tvm::tir::ForNode const*)
42: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::ForNode const*)
41: tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
40: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::SeqStmtNode const*)
39: tvm::codegen::CodeGenCUDA::VisitStmt_(tvm::tir::ForNode const*)
38: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::ForNode const*)
37: tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>::VisitStmt(tvm::tir::Stmt const&)
36: tvm::codegen::CodeGenC::VisitStmt_(tvm::tir::BufferStoreNode const*)
35: tvm::codegen::CodeGenC::PrintExpr[abi:cxx11](tvm::PrimExpr const&)
34: tvm::codegen::CodeGenC::PrintExpr(tvm::PrimExpr const&, std::ostream&)
33: tvm::codegen::CodeGenC::VisitExpr_(tvm::tir::SubNode const*, std::ostream&)
32: tvm::codegen::CodeGenCUDA::PrintVecBinaryOp(std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, tvm::runtime::DataType, tvm::PrimExpr, tvm::PrimExpr, std::ostream&)
31: tvm::codegen::CodeGenC::PrintExpr[abi:cxx11](tvm::PrimExpr const&)
30: tvm::codegen::CodeGenC::PrintExpr(tvm::PrimExpr const&, std::ostream&)
29: tvm::codegen::CodeGenCUDA::VisitExpr(tvm::tir::CastNode const*, std::ostream&)
28: tvm::codegen::CodeGenC::PrintExpr[abi:cxx11](tvm::PrimExpr const&)
27: tvm::codegen::CodeGenC::PrintExpr(tvm::PrimExpr const&, std::ostream&)
26: tvm::codegen::CodeGenCUDA::VisitExpr(tvm::tir::CallNode const*, std::ostream&)
25: tvm::codegen::CodeGenC::VisitExpr_(tvm::tir::CallNode const*, std::ostream&)
24: tvm::codegen::PrintBinaryIntrinsic(tvm::tir::CallNode const*, char const*, std::ostream&, tvm::codegen::CodeGenC*)
23: tvm::codegen::CodeGenCUDA::PrintVecBinaryOp(std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, tvm::runtime::DataType, tvm::PrimExpr, tvm::PrimExpr, std::ostream&)
22: tvm::codegen::CodeGenC::PrintExpr[abi:cxx11](tvm::PrimExpr const&)
21: tvm::codegen::CodeGenC::PrintExpr(tvm::PrimExpr const&, std::ostream&)
20: tvm::codegen::CodeGenCUDA::VisitExpr(tvm::tir::CallNode const*, std::ostream&)
19: tvm::codegen::CodeGenC::VisitExpr(tvm::tir::CallNode const*, std::ostream&)
18: tvm::codegen::PrintBinaryIntrinsic(tvm::tir::CallNode const*, char const*, std::ostream&, tvm::codegen::CodeGenC*)
17: tvm::codegen::CodeGenCUDA::PrintVecBinaryOp(std::cxx11::basic_string<char, std::char_traits, std::allocator > const&, tvm::runtime::DataType, tvm::PrimExpr, tvm::PrimExpr, std::ostream&)
16: tvm::codegen::CodeGenC::PrintExpr[abi:cxx11](tvm::PrimExpr const&)
15: tvm::codegen::CodeGenC::PrintExpr(tvm::PrimExpr const&, std::ostream&)
14: tvm::codegen::CodeGenCUDA::VisitExpr(tvm::tir::CastNode const*, std::ostream&)
13: tvm::codegen::CodeGenC::PrintExpr[abi:cxx11](tvm::PrimExpr const&)
12: tvm::codegen::CodeGenC::PrintExpr(tvm::PrimExpr const&, std::ostream&)
11: tvm::codegen::CodeGenC::VisitExpr(tvm::tir::BufferLoadNode const*, std::ostream&)
10: tvm::codegen::CodeGenC::PrintExpr[abi:cxx11](tvm::PrimExpr const&)
9: tvm::codegen::CodeGenC::PrintExpr(tvm::PrimExpr const&, std::ostream&)
8: tvm::codegen::CodeGenC::VisitExpr_(tvm::tir::AddNode const*, std::ostream&)
7: tvm::codegen::CodeGenCUDA::PrintVecBinaryOp(std::_cxx11::basic_string<char, std::char_traits, std::allocator > const&, tvm::runtime::DataType, tvm::PrimExpr, tvm::PrimExpr, std::ostream&)
6: tvm::codegen::CodeGenC::PrintExpr[abi:cxx11](tvm::PrimExpr const&)
5: tvm::codegen::CodeGenC::PrintExpr(tvm::PrimExpr const&, std::ostream&)
4: tvm::codegen::CodeGenC::VisitExpr(tvm::tir::DivNode const*, std::ostream&)
3: tvm::codegen::CodeGenCUDA::PrintVecBinaryOp(std::_cxx11::basic_string<char, std::char_traits, std::allocator > const&, tvm::runtime::DataType, tvm::PrimExpr, tvm::PrimExpr, std::ostream&)
2: tvm::codegen::CodeGenC::PrintExpr[abi:cxx11](tvm::PrimExpr const&)
1: tvm::codegen::CodeGenC::PrintExpr(tvm::PrimExpr const&, std::ostream&)
0: tvm::codegen::CodeGenCUDA::VisitExpr(tvm::tir::RampNode const*, std::ostream&)
File "/root/BitBLAS/3rdparty/tvm/src/target/source/codegen_cuda.cc", line 1226
ValueError: Check failed: lanes <= 4 (8 vs. 4) : Ramp of more than 4 lanes is not allowed.

BitBLAS Tuning done, appended operator to global_operator_cache.
Traceback (most recent call last):
File "/home/malenia/pycharm-community-2023.2.2/plugins/python-ce/helpers/pydev/pydevd.py", line 1500, in _exec
pydev_imports.execfile(file, globals, locals) # execute the script
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/pycharm-community-2023.2.2/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/malenia/PycharmProjects/EmbeddingModels/pack.py", line 45, in
output = model(dummpy_input)
^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/malenia/anaconda3/envs/dl/lib/python3.11/site-packages/bitblas/module/init.py", line 287, in forward
self.bitblas_matmul.lib.call(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'call'

This is my environment:
ubuntu 22.04
pytorch 2.1.0
cuda 12.1
python 3.11
bitblas 0.0.1.dev15

The device is: NVIDIA GeForce RTX 4090

How should I handle this problem?

Answer 1 · 2024-08-28T15:51:58.000Z

hi @chenjy11223 ,

would you mind checkout whether you have nvcc in your command line?

if not, you could type export PATH=/usr/local/cuda/bin:$PATH to enable it.

Answer 2 · 2024-08-28T16:28:33.000Z

Thanks for your reply. I checked my ~/.bashrc and ''export PATH=/usr/local/cuda/bin:$PATH'' exists.
I run nvcc -V and the output is:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Feb__7_19:32:13_PST_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

The above problem still exists.

Answer 3 · 2024-08-29T08:12:55.000Z

I have the same error. But I can run this example successfully, but failed in my own model.

Answer 4 · 2024-08-29T10:08:53.000Z

Solved, I added following codes before bitblas.set_log_level("Debug"):

import os
os.environ["PATH"] = "/usr/local/cuda/bin:" + os.environ["PATH"]

Running export PATH=/usr/local/cuda/bin:$PATH before the .py file also works.

I don't know why my ~/.bashrc file doesn't work but but it's not a big problem. Thanks again.