microsoft/GRAM

An error when training

Scxw010516 opened this issue · 0 comments

Thank you for your great work!
When I run the training code, I get an error like the following

Traceback (most recent call last):
  File "C:\Apps\Python39\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap
    fn(i, *args)
  File "D:\Projects\GRAM\train.py", line 33, in train
    training_process(rank, world_size, opt, device)
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "C:\Apps\Python39\lib\site-packages\torch\nn\parallel\distributed.py", line 1110, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "C:\Apps\Python39\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "D:\Projects\GRAM\generators\generators.py", line 112, in forward
    img, _ = self.renderer.render(self._intersections, self._volume(z, truncation_psi), img_size, camera_origin, camera_pos, fov, ray_start, ray_end, z.device)
  File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 146, in render
    transformed_points_sample, transformed_ray_directions, transformed_ray_origins, _ = transform_sampled_points(points_cam, rays_d_cam, camera_origin, camera_pos, device=device)
  File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 92, in transform_sampled_points
    cam2world_matrix = create_cam2world_matrix(forward_vector, camera_origin, device=device)
  File "D:\Projects\GRAM\generators\renderers\manifold_renderer.py", line 123, in create_cam2world_matrix
    cam2world = translation_matrix @ rotation_matrix
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

Please help solve,thank you very much!