Failed to reconstruct with SPEED+ dataset

[jiangyijin]

graphdeco-inria/gaussian-splatting#719 (comment)
Hello, I have been working on 3D reconstruction using 3DGS recently. 3DGS can also be used to reconstruct with nerf datasets. However, I have been facing continuous reconstruction failures when using a public dataset SPEED+. Therefore, I would like to consult with you about the correctness of my camera pose transformation. I have encountered issues with the transformed SPEED+ dataset when training with nerf as well.

import json
import math
import numpy as np

Read the camera intrinsic parameters file.

with open('camera.json', 'r') as f:
camera_params = json.load(f)

Read the camera extrinsic parameters file.

with open(r'G:\speed+\shirtv1\roe2\roe2.json', 'r') as f:
extrinsic_params = json.load(f)

Extracting Camera Intrinsic Matrix

camera_matrix = np.array(camera_params['cameraMatrix'])
dist_coeffs = np.array(camera_params['distCoeffs'])

calculate camera_angle_x

Nu = camera_params["Nu"]
fx = camera_params["fx"]
camera_angle_x = 2 * math.atan(Nu / (2 * fx))

Create storage for storing Nerf format data.

frames = []

def quaternion_to_rotation_matrix(q):
q = np.array(q)
r = np.array([[1 - 2 * (q[2] ** 2 + q[3] ** 2), 2 * (q[1] * q[2] - q[0] * q[3]), 2 * (q[1] * q[3] + q[0] * q[2])],
[2 * (q[1] * q[2] + q[0] * q[3]), 1 - 2 * (q[1] ** 2 + q[3] ** 2), 2 * (q[2] * q[3] - q[0] * q[1])],
[2 * (q[1] * q[3] - q[0] * q[2]), 2 * (q[2] * q[3] + q[0] * q[1]), 1 - 2 * (q[1] ** 2 + q[2] ** 2)]])
return r

for param in extrinsic_params:
filename = param['filename']
q_vbs2tango_true = param['q_vbs2tango_true']
r_Vo2To_vbs_true = param['r_Vo2To_vbs_true']

# Create a rotation matrix 
rotation_matrix = np.eye(4)
rotation_matrix[:3, :3] = quaternion_to_rotation_matrix(q_vbs2tango_true)

# Construct a translation vector 
translation_vector = np.array(r_Vo2To_vbs_true).reshape(3, 1)

# Build a transformation matrix 
transform_matrix = np.eye(4)
transform_matrix[:3, :3] = rotation_matrix[:3, :3]
transform_matrix[:3, 3] = translation_vector.squeeze()

frame = {
    'file_path': filename,
    "rotation": camera_angle_x,
    'transform_matrix': transform_matrix.tolist()
}
frames.append(frame)

Prepare data in Nerf format

nerf_data = {
'camera_angle_x': float(camera_params['ppx']),
"rotation": frame["rotation"],
'frames': frames
}
#Write the data to a file
with open(r'G:\speed+\shirtv1\roe2\lightbox\roe1_to_nerf.json', 'w') as f:
json.dump(nerf_data, f, indent=4)

[maximeraafat]

Hi @jiangyijin, excuse me for the delayed reply.

I am not familiar with the SPEED+ dataset, please have a look at their documentation. If you found a solution in the meanwhile, feel free to share it below and close this issue.

In the future, I kindly ask you to refrain from posting questions here that are not directly related to BlenderNeRF, thanks!