Support required for Kitti360 dataset

Hi @xBeho1der
I am on working on training of Kitti360 dataset.
I have developed the dataparser for the same by following the data format suggested in the repo.
But it seems that axis of camera and objects are not aligned when I am trying to visualize the poses of camera and object using debug visualizer . Refer below image.

It will be highly appreciated if you can suggest me any idea how to align the axis of camera and object.

Thank you for raising this issue.

It appears that there might be a problem with the height, length, and width of the object bounding box.
I suggest incorporating the world coordinate system orientation in the visualizer. You can see an example of how this can be implemented in mars_pose_visualizer.

@xBeho1der Thank you for the suggestions.
After matching yaw axis of camera poses and objects poses in world frame. Then I compared visualization result in debug visualizer between Kitti dataset and Kitti360. Axis of Kitti360 are aligned same as Kitti but yaw angles of objects are not coming correctly in Kitti360.
Any suggestion to fix the issue of objects yaw angle in Kitti360 ?

Here is visualization results:
Kitti360

Kitti

Hi! @gaurav00700

Thanks for your reply! Perhaps you can refer to the following code. Different rotation axes should correspond to different rotation matrices, resulting in different outcomes.

mars/mars/data/mars_kitti_dataparser.py

Lines 451 to 456 in 69b9bf9

    
           # Helper function to generate a rotation matrix around the y-axis 
        
           def roty_matrix(roty): 
        
               c = np.cos(roty) 
        
               s = np.sin(roty) 
        
               return np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])

mars/mars/data/mars_kitti_dataparser.py

Lines 538 to 556 in 69b9bf9

    
           # Initialize a 4x4 identity matrix for object pose in camera coordinates 
        
           obj_pose_c = np.eye(4) 
        
           obj_pose_c[:3, 3] = pose[:3] 
        
           roty = pose[3] 
        
           obj_pose_c[:3, :3] = roty_matrix(roty) 
        
           # Transform object pose from camera coordinates to IMU coordinates 
        
           obj_pose_imu = np.matmul(velo2imu, np.matmul(cam2velo, obj_pose_c)) 
        
           # Get the IMU pose for the corresponding frame 
        
           pose_imu_w_frame_i = poses_imu_tracking[int(frame_id)] 
        
           # Calculate the world pose of the object 
        
           pose_obj_w_i = np.matmul(pose_imu_w_frame_i, obj_pose_imu) 
        
           pose_obj_w_i = np.matmul(transform_matrix, pose_obj_w_i) 
        
           # pose_obj_w_i[:, 3] *= scale_factor 
        
           # Calculate the approximate yaw angle of the object in the world frame 
        
           yaw_aprox = -np.arctan2(pose_obj_w_i[1, 0], pose_obj_w_i[0, 0])

@xBeho1der , Thank you for the reply and support.
Sorry, I shared the wrong image of Kitti360 is my last comment. I have updated the image in above comment.

There was problem in the yaw angle calculation. I have fixed the issue. Now camera and object poses are coming correctly.
Next, I will start training and share the results.

@gaurav00700 ，Thank you very much for sharing your process of solving it, what I am interested in knowing is if you can share the results of your training on KITTI 360?

@dogooooo Hi... I trained on one scene but results are not good overall. There is some bug with transformation. Z coordinates are incorrect. Unfortunately, I was not able to spend time to fix it that is why I did not post the results. I will again work on that after sometime. I have used both stereo cameras (00, 01), seq_id = 0003, frames= 650-750
As you can seen in below images of training results, 3d bbox and camera poses are aligned. It seems there is problem of Z coordinate.
GT vs Synthesis

Debug

	# Helper function to generate a rotation matrix around the y-axis
	def roty_matrix(roty):
	c = np.cos(roty)
	s = np.sin(roty)
	return np.array([[c, 0, s], [0, 1, 0], [-s, 0, c]])

	# Initialize a 4x4 identity matrix for object pose in camera coordinates
	obj_pose_c = np.eye(4)
	obj_pose_c[:3, 3] = pose[:3]
	roty = pose[3]
	obj_pose_c[:3, :3] = roty_matrix(roty)

	# Transform object pose from camera coordinates to IMU coordinates
	obj_pose_imu = np.matmul(velo2imu, np.matmul(cam2velo, obj_pose_c))

	# Get the IMU pose for the corresponding frame
	pose_imu_w_frame_i = poses_imu_tracking[int(frame_id)]

	# Calculate the world pose of the object
	pose_obj_w_i = np.matmul(pose_imu_w_frame_i, obj_pose_imu)
	pose_obj_w_i = np.matmul(transform_matrix, pose_obj_w_i)
	# pose_obj_w_i[:, 3] *= scale_factor

	# Calculate the approximate yaw angle of the object in the world frame
	yaw_aprox = -np.arctan2(pose_obj_w_i[1, 0], pose_obj_w_i[0, 0])