可以支持cuda12.1吗
Opened this issue · 4 comments
zixian-wu commented
可以支持cuda12.1吗
MyNiuuu commented
My CUDA version is 12.1.
1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.
To modify compile_with_cache function:
- Refer to this method in the latest version as it seems to have been removed: Deprecate
cupy.cuda.compile_with_cachecupy/cupy#5858.In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.
Now, the program should run correctly.
Originally posted by @laishujie in #16 (comment)
抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？
zishen-ucap commented
My CUDA version is 12.1.
1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.
To modify compile_with_cache function:
- Refer to this method in the latest version as it seems to have been removed: Deprecate
cupy.cuda.compile_with_cachecupy/cupy#5858.In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.
Now, the program should run correctly.Originally posted by @laishujie in #16 (comment)
抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？
我用这个方法在我那运行不了，不知道他是不是把requirements.txt的cupy-cuda117改成cupy-cuda121，如果这样改的话，models/softsplat.py第219行装饰器就用不了，但是不uninstallcupy-cuda117的话，后面会报兼容性的问题。
此外，用cuda12.1执行
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
会出现和pytorch冲突的问题（他cuda是12.1版本），所以我后面就直接换成cuda11.8来执行上面的命令，就搭好环境了，运行的时候再换cuda12.1
zishen-ucap commented
My CUDA version is 12.1.
1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.
To modify compile_with_cache function:
- Refer to this method in the latest version as it seems to have been removed: Deprecate
cupy.cuda.compile_with_cachecupy/cupy#5858.In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.
Now, the program should run correctly.Originally posted by @laishujie in #16 (comment)
抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？
我用这个方法在我那运行不了，不知道他是不是把requirements.txt的cupy-cuda117改成cupy-cuda121，如果这样改的话，models/softsplat.py第219行装饰器就用不了，但是不uninstallcupy-cuda117的话，后面会报兼容性的问题。 此外，用cuda12.1执行
pip install "git+https://github.com/facebookresearch/pytorch3d.git"会出现和pytorch冲突的问题（他cuda是12.1版本），所以我后面就直接换成cuda11.8来执行上面的命令，就搭好环境了，运行的时候再换cuda12.1
报错如下
Exception: None
You selected None at [397, 137] from image
[[[397, 137]]]
You selected None at [381, 175] from image
[[[397, 137], [381, 175]]]
torch.Size([1, 24, 2, 512, 512])
torch.Size([1, 24, 2, 512, 512])
Some weights of Wav2Vec2Model were not initialized from the model checkpoint at ckpts/aniportrait/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Some weights of Wav2Vec2Model were not initialized from the model checkpoint at ckpts/aniportrait/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1721800906.491101 545391 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5
I0000 00:00:1721800906.712256 545675 gl_context.cc:357] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 550.54.14), renderer: NVIDIA H800 PCIe/PCIe/SSE2
W0000 00:00:1721800906.713922 545391 face_landmarker_graph.cc:174] Sets FaceBlendshapesGraph acceleration to xnnpack by default.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
W0000 00:00:1721800906.759709 545677 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
W0000 00:00:1721800906.786435 545690 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
I0000 00:00:1721800906.793379 545391 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5
I0000 00:00:1721800906.849270 545737 gl_context.cc:357] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 550.54.14), renderer: NVIDIA H800 PCIe/PCIe/SSE2
W0000 00:00:1721800906.859003 545742 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors.
/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/google/protobuf/symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon.
warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please '
0%| | 0/25 [00:01<?, ?it/s]
Traceback (most recent call last):
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 685, in compile
nvrtc.compileProgram(self.ptr, options)
File "cupy_backends/cuda/libs/nvrtc.pyx", line 141, in cupy_backends.cuda.libs.nvrtc.compileProgram
File "cupy_backends/cuda/libs/nvrtc.pyx", line 153, in cupy_backends.cuda.libs.nvrtc.compileProgram
File "cupy_backends/cuda/libs/nvrtc.pyx", line 69, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 456, in call_prediction
output = await route_utils.call_process_api(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1522, in process_api
result = await self.call_function(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1144, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
result = context.run(func, *args)
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/utils.py", line 674, in wrapper
response = f(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 860, in run
outputs = self.forward_sample(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 452, in forward_sample
val_output = self.pipeline(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/pipeline/pipeline.py", line 454, in __call__
down_res_face_tmp, mid_res_face_tmp, controlnet_flow, _ = self.face_controlnet(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 446, in forward
warped_cond_feature, occlusion_mask = self.get_warped_frames(cond_feature, scale_flows[fh // ch], fh // ch)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 300, in get_warped_frames
warped_frame = softsplat(tenIn=first_frame.float(), tenFlow=flows[:, i].float(), tenMetric=None, strMode='avg').to(dtype) # [b, c, w, h]
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 251, in softsplat
tenOut = softsplat_func.apply(tenIn, tenFlow)
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 106, in decorate_fwd
return fwd(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 284, in forward
cuda_launch(cuda_kernel('softsplat_out', '''
File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 225, in cuda_launch
return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction'])
File "cupy/_core/core.pyx", line 2081, in cupy._core.core.compile_with_cache
File "cupy/_core/core.pyx", line 2141, in cupy._core.core.compile_with_cache
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 492, in _compile_module_with_cache
return _compile_with_cache_cuda(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 570, in _compile_with_cache_cuda
ptx, mapping = compile_using_nvrtc(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 321, in compile_using_nvrtc
return _compile(source, options, cu_path,
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 305, in _compile
compiled_obj, mapping = prog.compile(options, log_stream)
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 704, in compile
raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: type name is not allowed
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "val" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expression must be a modifiable lvalue
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: an asm operand must have scalar type
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): warning #12-D: parsing restarts here after previous syntax error
/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): error: expected a ";"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(598): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(607): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(609): warning #940-D: missing return statement at end of non-void function "__float2half"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(615): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(624): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(626): warning #940-D: missing return statement at end of non-void function "__float2half_rn"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(632): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(638): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(640): warning #940-D: missing return statement at end of non-void function "__float2half_rz"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(646): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(655): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(657): warning #940-D: missing return statement at end of non-void function "__float2half_rd"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(663): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(672): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(674): warning #940-D: missing return statement at end of non-void function "__float2half_ru"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(679): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(681): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(682): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(684): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(686): warning #940-D: missing return statement at end of non-void function "__float2half2_rn"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_PROVIDES_SM_80" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(692): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(693): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(694): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(698): warning #12-D: parsing restarts here after previous syntax error
/usr/local/cuda-12.1/include/cuda_fp16.hpp(699): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(701): warning #940-D: missing return statement at end of non-void function "__internal_device_float2_to_half2_rn"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(709): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(710): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(712): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(714): warning #940-D: missing return statement at end of non-void function "__floats2half2_rn"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(757): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(759): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(761): warning #940-D: missing return statement at end of non-void function "__half2float"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(754): warning #177-D: variable "val" was declared but never referenced
/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(766): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(768): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(769): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(771): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(773): warning #940-D: missing return statement at end of non-void function "__low2float"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(764): warning #177-D: variable "val" was declared but never referenced
/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(778): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(780): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(781): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(783): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(785): warning #940-D: missing return statement at end of non-void function "__high2float"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(776): warning #177-D: variable "val" was declared but never referenced
/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(791): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(800): error: identifier "f" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(810): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(812): warning #940-D: missing return statement at end of non-void function "__half2short_rz"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(788): warning #550-D: variable "i" was set but never used
/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(818): error: expected an expression
Error limit reached.
100 errors detected in the compilation of "/tmp/tmpdw4ot6l0/ad9d8069ae2a8fad2769aa45cab32d685f71d792.cubin.cu".
Compilation terminated.
Traceback (most recent call last):
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 685, in compile
nvrtc.compileProgram(self.ptr, options)
File "cupy_backends/cuda/libs/nvrtc.pyx", line 141, in cupy_backends.cuda.libs.nvrtc.compileProgram
File "cupy_backends/cuda/libs/nvrtc.pyx", line 153, in cupy_backends.cuda.libs.nvrtc.compileProgram
File "cupy_backends/cuda/libs/nvrtc.pyx", line 69, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 456, in call_prediction
output = await route_utils.call_process_api(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/route_utils.py", line 232, in call_process_api
output = await app.get_blocks().process_api(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1522, in process_api
result = await self.call_function(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1144, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
result = context.run(func, *args)
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/utils.py", line 674, in wrapper
response = f(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 860, in run
outputs = self.forward_sample(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 452, in forward_sample
val_output = self.pipeline(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/pipeline/pipeline.py", line 454, in __call__
down_res_face_tmp, mid_res_face_tmp, controlnet_flow, _ = self.face_controlnet(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 446, in forward
warped_cond_feature, occlusion_mask = self.get_warped_frames(cond_feature, scale_flows[fh // ch], fh // ch)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 300, in get_warped_frames
warped_frame = softsplat(tenIn=first_frame.float(), tenFlow=flows[:, i].float(), tenMetric=None, strMode='avg').to(dtype) # [b, c, w, h]
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 251, in softsplat
tenOut = softsplat_func.apply(tenIn, tenFlow)
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 106, in decorate_fwd
return fwd(*args, **kwargs)
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 284, in forward
cuda_launch(cuda_kernel('softsplat_out', '''
File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret
File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 225, in cuda_launch
return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction'])
File "cupy/_core/core.pyx", line 2081, in cupy._core.core.compile_with_cache
File "cupy/_core/core.pyx", line 2141, in cupy._core.core.compile_with_cache
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 492, in _compile_module_with_cache
return _compile_with_cache_cuda(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 570, in _compile_with_cache_cuda
ptx, mapping = compile_using_nvrtc(
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 321, in compile_using_nvrtc
return _compile(source, options, cu_path,
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 305, in _compile
compiled_obj, mapping = prog.compile(options, log_stream)
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 704, in compile
raise CompileException(log, self.src, self.name, options,
cupy.cuda.compiler.CompileException: /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: type name is not allowed
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "val" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expression must be a modifiable lvalue
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: an asm operand must have scalar type
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): warning #12-D: parsing restarts here after previous syntax error
/usr/local/cuda-12.1/include/cuda_fp16.hpp(591): error: expected a ";"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(598): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(607): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(609): warning #940-D: missing return statement at end of non-void function "__float2half"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(615): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(624): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(626): warning #940-D: missing return statement at end of non-void function "__float2half_rn"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(632): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(638): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(640): warning #940-D: missing return statement at end of non-void function "__float2half_rz"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(646): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(655): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(657): warning #940-D: missing return statement at end of non-void function "__float2half_rd"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(663): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "r" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "__internal_float2half" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(672): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(674): warning #940-D: missing return statement at end of non-void function "__float2half_ru"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(679): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(681): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(682): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(684): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(686): warning #940-D: missing return statement at end of non-void function "__float2half2_rn"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_PROVIDES_SM_80" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(692): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(693): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(694): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(698): warning #12-D: parsing restarts here after previous syntax error
/usr/local/cuda-12.1/include/cuda_fp16.hpp(699): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(701): warning #940-D: missing return statement at end of non-void function "__internal_device_float2_to_half2_rn"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(709): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(710): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(712): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(714): warning #940-D: missing return statement at end of non-void function "__floats2half2_rn"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(757): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(759): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(761): warning #940-D: missing return statement at end of non-void function "__half2float"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(754): warning #177-D: variable "val" was declared but never referenced
/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(766): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(768): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(769): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(771): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(773): warning #940-D: missing return statement at end of non-void function "__low2float"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(764): warning #177-D: variable "val" was declared but never referenced
/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(778): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(780): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(781): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(783): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(785): warning #940-D: missing return statement at end of non-void function "__high2float"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(776): warning #177-D: variable "val" was declared but never referenced
/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(791): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(800): error: identifier "f" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(810): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(812): warning #940-D: missing return statement at end of non-void function "__half2short_rz"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(788): warning #550-D: variable "i" was set but never used
/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IS_DEVICE" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected an expression
/usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected a ")"
/usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IF_ELSE_TARGET" is undefined
/usr/local/cuda-12.1/include/cuda_fp16.hpp(818): error: expected an expression
Error limit reached.
100 errors detected in the compilation of "/tmp/tmpdw4ot6l0/ad9d8069ae2a8fad2769aa45cab32d685f71d792.cubin.cu".
Compilation terminated.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 501, in process_events
response = await self.call_prediction(awake_events, batch)
File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 465, in call_prediction
raise Exception(str(error) if show_error else None) from error
Exception: None
zishen-ucap commented
My CUDA version is 12.1.
1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.
To modify compile_with_cache function:
- Refer to this method in the latest version as it seems to have been removed: Deprecate
cupy.cuda.compile_with_cachecupy/cupy#5858.In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.
Now, the program should run correctly.Originally posted by @laishujie in #16 (comment)
抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？
我用这个方法在我那运行不了，不知道他是不是把requirements.txt的cupy-cuda117改成cupy-cuda121，如果这样改的话，models/softsplat.py第219行装饰器就用不了，但是不uninstallcupy-cuda117的话，后面会报兼容性的问题。 此外，用cuda12.1执行
pip install "git+https://github.com/facebookresearch/pytorch3d.git"会出现和pytorch冲突的问题（他cuda是12.1版本），所以我后面就直接换成cuda11.8来执行上面的命令，就搭好环境了，运行的时候再换cuda12.1报错如下
Exception: None You selected None at [397, 137] from image [[[397, 137]]] You selected None at [381, 175] from image [[[397, 137], [381, 175]]] torch.Size([1, 24, 2, 512, 512]) torch.Size([1, 24, 2, 512, 512]) Some weights of Wav2Vec2Model were not initialized from the model checkpoint at ckpts/aniportrait/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of Wav2Vec2Model were not initialized from the model checkpoint at ckpts/aniportrait/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1721800906.491101 545391 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5 I0000 00:00:1721800906.712256 545675 gl_context.cc:357] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 550.54.14), renderer: NVIDIA H800 PCIe/PCIe/SSE2 W0000 00:00:1721800906.713922 545391 face_landmarker_graph.cc:174] Sets FaceBlendshapesGraph acceleration to xnnpack by default. INFO: Created TensorFlow Lite XNNPACK delegate for CPU. W0000 00:00:1721800906.759709 545677 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors. W0000 00:00:1721800906.786435 545690 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors. I0000 00:00:1721800906.793379 545391 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5 I0000 00:00:1721800906.849270 545737 gl_context.cc:357] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 550.54.14), renderer: NVIDIA H800 PCIe/PCIe/SSE2 W0000 00:00:1721800906.859003 545742 inference_feedback_manager.cc:114] Feedback manager requires a model with a single signature inference. Disabling support for feedback tensors. /home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/google/protobuf/symbol_database.py:55: UserWarning: SymbolDatabase.GetPrototype() is deprecated. Please use message_factory.GetMessageClass() instead. SymbolDatabase.GetPrototype() will be removed soon. warnings.warn('SymbolDatabase.GetPrototype() is deprecated. Please ' 0%| | 0/25 [00:01<?, ?it/s] Traceback (most recent call last): File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 685, in compile nvrtc.compileProgram(self.ptr, options) File "cupy_backends/cuda/libs/nvrtc.pyx", line 141, in cupy_backends.cuda.libs.nvrtc.compileProgram File "cupy_backends/cuda/libs/nvrtc.pyx", line 153, in cupy_backends.cuda.libs.nvrtc.compileProgram File "cupy_backends/cuda/libs/nvrtc.pyx", line 69, in cupy_backends.cuda.libs.nvrtc.check_status cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 456, in call_prediction output = await route_utils.call_process_api( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/route_utils.py", line 232, in call_process_api output = await app.get_blocks().process_api( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1522, in process_api result = await self.call_function( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1144, in call_function prediction = await anyio.to_thread.run_sync( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread return await future File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run result = context.run(func, *args) File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/utils.py", line 674, in wrapper response = f(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 860, in run outputs = self.forward_sample( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 452, in forward_sample val_output = self.pipeline( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/pipeline/pipeline.py", line 454, in __call__ down_res_face_tmp, mid_res_face_tmp, controlnet_flow, _ = self.face_controlnet( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 446, in forward warped_cond_feature, occlusion_mask = self.get_warped_frames(cond_feature, scale_flows[fh // ch], fh // ch) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 300, in get_warped_frames warped_frame = softsplat(tenIn=first_frame.float(), tenFlow=flows[:, i].float(), tenMetric=None, strMode='avg').to(dtype) # [b, c, w, h] File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 251, in softsplat tenOut = softsplat_func.apply(tenIn, tenFlow) File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply return super().apply(*args, **kwargs) # type: ignore[misc] File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 106, in decorate_fwd return fwd(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 284, in forward cuda_launch(cuda_kernel('softsplat_out', ''' File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 225, in cuda_launch return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction']) File "cupy/_core/core.pyx", line 2081, in cupy._core.core.compile_with_cache File "cupy/_core/core.pyx", line 2141, in cupy._core.core.compile_with_cache File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 492, in _compile_module_with_cache return _compile_with_cache_cuda( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 570, in _compile_with_cache_cuda ptx, mapping = compile_using_nvrtc( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 321, in compile_using_nvrtc return _compile(source, options, cu_path, File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 305, in _compile compiled_obj, mapping = prog.compile(options, log_stream) File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 704, in compile raise CompileException(log, self.src, self.name, options, cupy.cuda.compiler.CompileException: /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: type name is not allowed /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "val" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expression must be a modifiable lvalue /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: an asm operand must have scalar type /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(591): warning #12-D: parsing restarts here after previous syntax error /usr/local/cuda-12.1/include/cuda_fp16.hpp(591): error: expected a ";" /usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(598): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(607): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(609): warning #940-D: missing return statement at end of non-void function "__float2half" /usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(615): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(624): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(626): warning #940-D: missing return statement at end of non-void function "__float2half_rn" /usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(632): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(638): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(640): warning #940-D: missing return statement at end of non-void function "__float2half_rz" /usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(646): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(655): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(657): warning #940-D: missing return statement at end of non-void function "__float2half_rd" /usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(663): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(672): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(674): warning #940-D: missing return statement at end of non-void function "__float2half_ru" /usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(679): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(681): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(682): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(684): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(686): warning #940-D: missing return statement at end of non-void function "__float2half2_rn" /usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_PROVIDES_SM_80" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(692): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(693): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(694): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(698): warning #12-D: parsing restarts here after previous syntax error /usr/local/cuda-12.1/include/cuda_fp16.hpp(699): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(701): warning #940-D: missing return statement at end of non-void function "__internal_device_float2_to_half2_rn" /usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(709): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(710): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(712): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(714): warning #940-D: missing return statement at end of non-void function "__floats2half2_rn" /usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(757): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(759): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(761): warning #940-D: missing return statement at end of non-void function "__half2float" /usr/local/cuda-12.1/include/cuda_fp16.hpp(754): warning #177-D: variable "val" was declared but never referenced /usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(766): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(768): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(769): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(771): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(773): warning #940-D: missing return statement at end of non-void function "__low2float" /usr/local/cuda-12.1/include/cuda_fp16.hpp(764): warning #177-D: variable "val" was declared but never referenced /usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(778): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(780): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(781): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(783): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(785): warning #940-D: missing return statement at end of non-void function "__high2float" /usr/local/cuda-12.1/include/cuda_fp16.hpp(776): warning #177-D: variable "val" was declared but never referenced /usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(791): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(800): error: identifier "f" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(810): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(812): warning #940-D: missing return statement at end of non-void function "__half2short_rz" /usr/local/cuda-12.1/include/cuda_fp16.hpp(788): warning #550-D: variable "i" was set but never used /usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(818): error: expected an expression Error limit reached. 100 errors detected in the compilation of "/tmp/tmpdw4ot6l0/ad9d8069ae2a8fad2769aa45cab32d685f71d792.cubin.cu". Compilation terminated. Traceback (most recent call last): File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 685, in compile nvrtc.compileProgram(self.ptr, options) File "cupy_backends/cuda/libs/nvrtc.pyx", line 141, in cupy_backends.cuda.libs.nvrtc.compileProgram File "cupy_backends/cuda/libs/nvrtc.pyx", line 153, in cupy_backends.cuda.libs.nvrtc.compileProgram File "cupy_backends/cuda/libs/nvrtc.pyx", line 69, in cupy_backends.cuda.libs.nvrtc.check_status cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 456, in call_prediction output = await route_utils.call_process_api( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/route_utils.py", line 232, in call_process_api output = await app.get_blocks().process_api( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1522, in process_api result = await self.call_function( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/blocks.py", line 1144, in call_function prediction = await anyio.to_thread.run_sync( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread return await future File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run result = context.run(func, *args) File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/utils.py", line 674, in wrapper response = f(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 860, in run outputs = self.forward_sample( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/run_gradio_audio_driven.py", line 452, in forward_sample val_output = self.pipeline( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/pipeline/pipeline.py", line 454, in __call__ down_res_face_tmp, mid_res_face_tmp, controlnet_flow, _ = self.face_controlnet( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 446, in forward warped_cond_feature, occlusion_mask = self.get_warped_frames(cond_feature, scale_flows[fh // ch], fh // ch) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/ldmk_ctrlnet.py", line 300, in get_warped_frames warped_frame = softsplat(tenIn=first_frame.float(), tenFlow=flows[:, i].float(), tenMetric=None, strMode='avg').to(dtype) # [b, c, w, h] File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 251, in softsplat tenOut = softsplat_func.apply(tenIn, tenFlow) File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply return super().apply(*args, **kwargs) # type: ignore[misc] File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 106, in decorate_fwd return fwd(*args, **kwargs) File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 284, in forward cuda_launch(cuda_kernel('softsplat_out', ''' File "cupy/_util.pyx", line 67, in cupy._util.memoize.decorator.ret File "/data-123/zishen/cvproject/MOFA-Video/MOFA-Video-Hybrid/models/softsplat.py", line 225, in cuda_launch return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction']) File "cupy/_core/core.pyx", line 2081, in cupy._core.core.compile_with_cache File "cupy/_core/core.pyx", line 2141, in cupy._core.core.compile_with_cache File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 492, in _compile_module_with_cache return _compile_with_cache_cuda( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 570, in _compile_with_cache_cuda ptx, mapping = compile_using_nvrtc( File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 321, in compile_using_nvrtc return _compile(source, options, cu_path, File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 305, in _compile compiled_obj, mapping = prog.compile(options, log_stream) File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/cupy/cuda/compiler.py", line 704, in compile raise CompileException(log, self.src, self.name, options, cupy.cuda.compiler.CompileException: /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: type name is not allowed /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "val" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expression must be a modifiable lvalue /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: an asm operand must have scalar type /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: identifier "result" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(454): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(591): warning #12-D: parsing restarts here after previous syntax error /usr/local/cuda-12.1/include/cuda_fp16.hpp(591): error: expected a ";" /usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(597): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(596): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(598): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(602): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(607): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(609): warning #940-D: missing return statement at end of non-void function "__float2half" /usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(614): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(613): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(615): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(619): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(624): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(626): warning #940-D: missing return statement at end of non-void function "__float2half_rn" /usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(631): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(630): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(632): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(636): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(638): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(640): warning #940-D: missing return statement at end of non-void function "__float2half_rz" /usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(645): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(644): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(646): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(650): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(655): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(657): warning #940-D: missing return statement at end of non-void function "__float2half_rd" /usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(662): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(661): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(663): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "r" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(667): error: identifier "__internal_float2half" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(672): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(674): warning #940-D: missing return statement at end of non-void function "__float2half_ru" /usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(679): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(681): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(678): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(682): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(684): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(686): warning #940-D: missing return statement at end of non-void function "__float2half2_rn" /usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_PROVIDES_SM_80" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(692): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(693): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(691): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(694): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(698): warning #12-D: parsing restarts here after previous syntax error /usr/local/cuda-12.1/include/cuda_fp16.hpp(699): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(701): warning #940-D: missing return statement at end of non-void function "__internal_device_float2_to_half2_rn" /usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(709): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(708): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(710): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(712): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(714): warning #940-D: missing return statement at end of non-void function "__floats2half2_rn" /usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(756): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(755): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(757): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(759): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(761): warning #940-D: missing return statement at end of non-void function "__half2float" /usr/local/cuda-12.1/include/cuda_fp16.hpp(754): warning #177-D: variable "val" was declared but never referenced /usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(766): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(768): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(765): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(769): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(771): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(773): warning #940-D: missing return statement at end of non-void function "__low2float" /usr/local/cuda-12.1/include/cuda_fp16.hpp(764): warning #177-D: variable "val" was declared but never referenced /usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(778): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(780): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(777): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(781): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(783): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(785): warning #940-D: missing return statement at end of non-void function "__high2float" /usr/local/cuda-12.1/include/cuda_fp16.hpp(776): warning #177-D: variable "val" was declared but never referenced /usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(790): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(789): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(791): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(800): error: identifier "f" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(810): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(812): warning #940-D: missing return statement at end of non-void function "__half2short_rz" /usr/local/cuda-12.1/include/cuda_fp16.hpp(788): warning #550-D: variable "i" was set but never used /usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IS_DEVICE" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected an expression /usr/local/cuda-12.1/include/cuda_fp16.hpp(817): error: expected a ")" /usr/local/cuda-12.1/include/cuda_fp16.hpp(816): error: identifier "NV_IF_ELSE_TARGET" is undefined /usr/local/cuda-12.1/include/cuda_fp16.hpp(818): error: expected an expression Error limit reached. 100 errors detected in the compilation of "/tmp/tmpdw4ot6l0/ad9d8069ae2a8fad2769aa45cab32d685f71d792.cubin.cu". Compilation terminated. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 501, in process_events response = await self.call_prediction(awake_events, batch) File "/home/zishen/anaconda3/envs/mofa/lib/python3.10/site-packages/gradio/queueing.py", line 465, in call_prediction raise Exception(str(error) if show_error else None) from error Exception: None
My CUDA version is 12.1.
1.pip install cupy-cuda12x. 2.conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia.
To modify compile_with_cache function:
- Refer to this method in the latest version as it seems to have been removed: Deprecate
cupy.cuda.compile_with_cachecupy/cupy#5858.In MOFA-Video-Hybrid/models/softsplat.py, change line 225 to cupy._core.core.compile_with_cache.
Now, the program should run correctly.Originally posted by @laishujie in #16 (comment)
抱歉刚刚想起来回复，目前我没太多时间改现在的代码去适配CUDA12.1，或许你可以尝试一下上面 @laishujie 在 #16 (comment) 里面提供的方法？
我用这个方法在我那运行不了，不知道他是不是把requirements.txt的cupy-cuda117改成cupy-cuda121，如果这样改的话，models/softsplat.py第219行装饰器就用不了，但是不uninstallcupy-cuda117的话，后面会报兼容性的问题。 此外，用cuda12.1执行
pip install "git+https://github.com/facebookresearch/pytorch3d.git"会出现和pytorch冲突的问题（他cuda是12.1版本），所以我后面就直接换成cuda11.8来执行上面的命令，就搭好环境了，运行的时候再换cuda12.1
我的可以参考下
我最终最终解决方法是
- 安装torch+cu118
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install
- 接着安装相关依赖
pip install -r requirements.txt
pip install opencv-python-headless
- 确保你此时的cuda是11.8，接着执行下面命令
pip install "git+https://github.com/facebookresearch/pytorch3d.git" - 接着卸载cupy-cuda117
pip uninstall cupy-cuda117 - 切换cuda12.1后，再执行下面命令
pip install cupy - 最后修改脚本代码
models/softsplat.py第225行
return cupy._core.core.compile_with_cache(objCudacache[strKey]['strKernel'], tuple(['-I ' + os.environ['CUDA_HOME'], '-I ' + os.environ['CUDA_HOME'] + '/include'])).get_function(objCudacache[strKey]['strFunction'])