[Headless] Cannot get camera tensor
Closed this issue · 5 comments
Hi!
I was using Isaac on a 3090 server (it works fine), and I basically setup the cameras in this way:
camera_properties = gymapi.CameraProperties()
camera_properties.width = 300
camera_properties.height = 240
camera_properties.enable_tensors = True
for i in range(len(self.envs)):
cam_handle = self.gym.create_camera_sensor(self.envs[i], camera_properties)
camera_offset = gymapi.Vec3(*self.cfg.visual_input.cam_pos)
camera_rotation = gymapi.Quat.from_axis_angle(gymapi.Vec3(0, 1, 0), np.deg2rad(self.CAMERA_ANGLE))
actor_handle = self.actor_handles[i]
body_handle = self.gym.get_actor_rigid_body_handle(self.envs[i], actor_handle, 0)
self.gym.attach_camera_to_body(
cam_handle, self.envs[i], body_handle, gymapi.Transform(camera_offset, camera_rotation),
gymapi.FOLLOW_TRANSFORM
)
self.camera_handles.append(cam_handle)
But both servers with A5000 CUDA11.7 & CUDA12 report error:
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101
[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101
[Error] [carb.windowing-glfw.plugin] GLFW initialization failed.
[Error] [carb.windowing-glfw.plugin] GLFW window creation failed!
[Error] [carb.gym.plugin] Failed to create Window in CreateGymViewerInternal
[Error] [carb.gym.plugin] Gym cuda error: invalid device ordinal: ../../../source/plugins/carb/gym/impl/Gym/GymPhysXCuda.cu: 991
Given I'm working headlessly, it's kind of crucial to get the image and videos for debugging. Any ideas?
On the CUDA12 machine, this is what I get:
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 0
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 1
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 2
Seems like I am getting some -1 as camera handle.
Given I managed to find a local machine, the issue will be closed. You may reopen if were there any question related to this issue.
On the CUDA12 machine, this is what I get:
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 0 [Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 1 [Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 2
Seems like I am getting some -1 as camera handle.
Still have the same issue with CUDA 12.2
On the CUDA12 machine, this is what I get:
[Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 0 [Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 1 [Error] [carb.gym.plugin] AttachCameraToBody: Error: could not find camera with handle -1 in environment 2
Seems like I am getting some -1 as camera handle.
self.graphics_device_id = self.sim_device_id
# if self.headless == True:
# self.graphics_device_id = -1
I fix this problem with commenting above two lines to let graphics_device_id
do not be set -1 in headless mode.
and always need gym.fetch_results
# if self.device == 'cpu':
self.gym.fetch_results(self.sim, True)