Unable to run on V100 card
vodp opened this issue · 5 comments
vodp commented
Running examples/simple.py give me this error stack. It seems that the quantized model requires sm_80 compute capability whereas V100 is up to sm_70 only. Does that mean I am unable to test the model with that card ?
VQModel loaded from ../../../data/tokenizer/vqgan.ckpt
nvidia-Process SpawnProcess-2:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/trainer/workspace/chameleon/chameleon/inference/chameleon.py", line 509, in _worker_impl
for token in Generator(
File "/home/trainer/workspace/chameleon/chameleon/inference/chameleon.py", line 403, in __next__
piece = next(self.dyngen)
File "/home/trainer/workspace/chameleon/chameleon/inference/utils.py", line 20, in __next__
return next(self.gen)
File "/home/trainer/workspace/chameleon/chameleon/inference/chameleon.py", line 279, in __next__
tok = next(self.gen)
File "/home/trainer/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/trainer/workspace/chameleon/chameleon/inference/generation.py", line 82, in __next__
outputs = self.model(self._inputs) # outputs.shape = [batch, seq-len, vocab]
File "/home/trainer/workspace/chameleon/chameleon/inference/model_adapter.py", line 87, in __call__
flat_outputs: torch.Tensor = self._forward( # type: ignore
File "/home/trainer/workspace/chameleon/chameleon/inference/cudagraph.py", line 65, in call_wrapper
return graph_wrapper(*inner_args, **inner_kwargs)
File "/home/trainer/workspace/chameleon/chameleon/inference/cudagraph.py", line 31, in __call__
return self.fn(*args, **kwargs)
File "/home/trainer/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/trainer/workspace/chameleon/chameleon/inference/transformer.py", line 330, in forward_with_attn_bias
h = layer(h, cache[i], attn_bias, group=group)
File "/home/trainer/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/trainer/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/trainer/workspace/chameleon/chameleon/inference/transformer.py", line 279, in forward
self.attention_norm(x),
File "/home/trainer/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/trainer/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/trainer/.local/lib/python3.10/site-packages/xformers/ops/rmsnorm.py", line 100, in forward
return rms_norm(x, self.weight, self.eps) # type: ignore
File "/home/trainer/.local/lib/python3.10/site-packages/xformers/ops/rmsnorm.py", line 39, in rms_norm
return _rms_norm_forward(x, weight, eps)
File "/home/trainer/.local/lib/python3.10/site-packages/xformers/ops/_triton/rmsnorm_kernels.py", line 111, in _rms_norm_forward
_rms_norm_kernel[(M,)](
File "<string>", line 63, in _rms_norm_kernel
File "/home/trainer/.local/lib/python3.10/site-packages/triton/compiler/compiler.py", line 476, in compile
next_module = compile_kernel(module)
File "/home/trainer/.local/lib/python3.10/site-packages/triton/compiler/compiler.py", line 351, in <lambda>
lambda src: ptx_to_cubin(src, arch))
File "/home/trainer/.local/lib/python3.10/site-packages/triton/compiler/compiler.py", line 150, in ptx_to_cubin
return compile_ptx_to_cubin(ptx, ptxas, arch)
RuntimeError: Internal Triton PTX codegen error:
ptxas /tmp/compile-ptx-src-ac38de, line 139; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 139; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 141; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 141; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 143; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 143; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 145; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 145; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 147; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 147; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 149; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 149; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 151; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 151; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 153; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 153; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 155; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 155; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 157; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 157; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 159; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 159; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 161; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 161; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 163; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 163; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 165; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 165; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 167; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 167; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 169; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 169; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 360; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 360; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 362; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 362; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 364; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 364; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 366; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 366; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 368; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 368; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 370; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 370; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 372; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 372; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 374; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 374; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 376; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 376; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 378; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 378; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 380; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 380; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 382; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 382; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 384; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 384; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 386; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 386; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 388; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 388; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 390; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 390; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 445; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 445; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 447; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 447; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 449; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 449; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 451; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 451; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 453; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 453; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 455; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 455; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 457; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 457; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 459; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 459; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 461; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 461; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 463; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 463; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 465; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 465; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 467; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 467; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 469; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 469; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 471; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 471; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 473; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 473; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 475; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 475; error : Feature 'cvt with .f32.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 495; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 495; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 497; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 497; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 499; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 499; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 501; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 501; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 503; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 503; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 505; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 505; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 507; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 507; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 509; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 509; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 511; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 511; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 513; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 513; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 515; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 515; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 517; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 517; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 519; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 519; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 521; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 521; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 523; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 523; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 525; error : Feature '.bf16' requires .target sm_80 or higher
ptxas /tmp/compile-ptx-src-ac38de, line 525; error : Feature 'cvt.bf16.f32' requires .target sm_80 or higher
ptxas fatal : Ptx assembly aborted due to errors
jacobkahn commented
bf16 support is required, so V100 TensorCores won't support inference. I'd wait for the HuggingFace port to land huggingface/transformers#31534
baochi0212 commented
I can't figure why this code can't be run on v100 ? @jacobkahn, thanks
baochi0212 commented
ah i see it's attributed to xformers kernels
from xformers.ops import RMSNorm, fmha, rope_padded
from xformers.ops.fmha.attn_bias import (
BlockDiagonalCausalWithOffsetPaddedKeysMask as AttnBias,
)
nicolaus-huang commented
Are there any possible ways to get this fixed and run it on V100? such as replace the functions with equal one that supported by V100?
jacobkahn commented
@nicolaus-huang — the HuggingFace implementation without bf16 is probably your best bet.