facebookincubator/AITemplate

Got cutlass error: Error Internal at: 214

syntrive opened this issue · 2 comments

Hi, when I run 05_stable_diffusion# python3 src/benchmark.py, there is a error:

pt output: torch.Size([1, 77, 1024])
[gemm_rcr_bias_add_25.cu] Got cutlass error: Error Internal at: 214
[20:21:02] model_interface.cu:221: Error: [gemm_rcr_bias_add_25.cu] Got cutlass error: Error Internal at: 214
Traceback (most recent call last):
File "src/benchmark.py", line 312, in
benchmark_diffusers()
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "src/benchmark.py", line 288, in benchmark_diffusers
benchmark_clip(
File "src/benchmark.py", line 176, in benchmark_clip
exe_module.run_with_tensors(inputs, ys)
File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 597, in run_with_tensors
outputs_ait = self.run(
File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 495, in run
return self._run_impl(
File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 434, in _run_impl
self.DLL.AITemplateModelContainerRun(
File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 196, in _wrapped_func
raise RuntimeError(f"Error in function: {method.name}")
RuntimeError: Error in function: AITemplateModelContainerRun

How can I solve it? thank you! env cuda11.7 cutlass3.4.1

device is A800