An error when training
Scxw010516 opened this issue · 0 comments
Scxw010516 commented
Thank you for your great work!
When I run the training code, I get an error like the following
Traceback (most recent call last):
File "C:\Apps\Python39\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap
fn(i, *args)
File "D:\Projects\GRAM\train.py", line 33, in train
training_process(rank, world_size, opt, device)
output = self._run_ddp_forward(*inputs, **kwargs)
File "C:\Apps\Python39\lib\site-packages\torch\nn\parallel\distributed.py", line 1110, in _run_ddp_forward
return module_to_run(*inputs[0], **kwargs[0]) # type: ignore[index]
File "C:\Apps\Python39\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\Projects\GRAM\generators\generators.py", line 112, in forward
img, _ = self.renderer.render(self._intersections, self._volume(z, truncation_psi), img_size, camera_origin, camera_pos, fov, ray_start, ray_end, z.device)
File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 146, in render
transformed_points_sample, transformed_ray_directions, transformed_ray_origins, _ = transform_sampled_points(points_cam, rays_d_cam, camera_origin, camera_pos, device=device)
File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 92, in transform_sampled_points
cam2world_matrix = create_cam2world_matrix(forward_vector, camera_origin, device=device)
File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 123, in create_cam2world_matrix
cam2world = translation_matrix @ rotation_matrix
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
Please help solve,thank you very much!