Input Data colmap.json generate
Closed this issue · 11 comments
Excuse me, I would like to know the formula of parameter R、C、T.
As far as I know, they can be calculated by colmap output sparse model cameras.txt and images.txt file?
The JSON object for the camera only needs to load the file camera.txt.
The definition of quaternions according to the Hamilton convention, So the JSON object for the images need to use QX、QY、QZ、QW to calculate R. The formula is below:
from the official doc: https://colmap.github.io/format.html
Is the above formula correct?
Thank you very much, sorry to bother you😖
I used the qvec2rotmat formula located here for my conversions, which does look similar to what you've sent.
I also had to run some conversions to convert into PyTorch3D format, I've included some rough code on how I computed R and T below. This is not part of the main FOUND code, so has not been tidied, but may be helpful to you.
colmap_P = np.eye(4)
colmap_P[:3, :3] = qvec2rotmat(image.qvec)
colmap_P[:3, -1] = image.tvec
# Calculate new centres
original_cam_centres = -(qvec2rotmat(image.qvec).T @ image.tvec[:, None])[..., 0]
P = Transf @ colmap_P
new_C = apply_transform(Transf, original_cam_centres[None, :])[0]
# Calculate new rotations
Transf_R = Transf[:3, :3] / np.linalg.det(Transf[:3, :3]) ** (1/3) # rotational component of overall transform
R = (Transf_R @ qvec2rotmat(image.qvec).T).T
# flip camera orientation from COLMAP to PyTorch3D
R = transf_c2p @ R
T = (- R @ new_C[:, None])[..., 0] # Calculate new T
I used the
qvec2rotmatformula located here for my conversions, which does look similar to what you've sent.I also had to run some conversions to convert into PyTorch3D format, I've included some rough code on how I computed R and T below. This is not part of the main FOUND code, so has not been tidied, but may be helpful to you.
colmap_P = np.eye(4) colmap_P[:3, :3] = qvec2rotmat(image.qvec) colmap_P[:3, -1] = image.tvec # Calculate new centres original_cam_centres = -(qvec2rotmat(image.qvec).T @ image.tvec[:, None])[..., 0] P = Transf @ colmap_P new_C = apply_transform(Transf, original_cam_centres[None, :])[0] # Calculate new rotations Transf_R = Transf[:3, :3] / np.linalg.det(Transf[:3, :3]) ** (1/3) # rotational component of overall transform R = (Transf_R @ qvec2rotmat(image.qvec).T).T # flip camera orientation from COLMAP to PyTorch3D R = transf_c2p @ R T = (- R @ new_C[:, None])[..., 0] # Calculate new T
Hi, Ollie! Could you please share what the 'Transf' matrix is and the body of the 'apply_transform' function(not a 3d graphics expert, sorry if it is something trivial)?
Ah yes! As I mentioned, this is not code released as part of the main project, so is a bit scrappy...
Transf was the transform I used to map COLMAP into the space of my ground truth world-space models, for evaluation. If you do not care about comparing to ground truth, you do not need this transform.
apply_transform is pretty much just matrix multiplication. This is the exact function (again, a little messy...)
def apply_transform(T, pts):
return ((T @ np.pad(pts, ((0, 0), (0, 1)), constant_values=1).T).T)[..., :3]
Just tried using the code you've shared, still can't get FOUND to work on Foot3D's multiview dataset with a custom colmap.json file - normal and total loss is 'nan', silhouette loss doesn't change at all. Could you please provide any hints on what I may be doing wrong?
Here is the code I use for generating the colmap.json file:
import numpy as np
import json
def qvec2rotmat(qvec):
return np.array(
[
[
1 - 2 * qvec[2] ** 2 - 2 * qvec[3] ** 2,
2 * qvec[1] * qvec[2] - 2 * qvec[0] * qvec[3],
2 * qvec[3] * qvec[1] + 2 * qvec[0] * qvec[2],
],
[
2 * qvec[1] * qvec[2] + 2 * qvec[0] * qvec[3],
1 - 2 * qvec[1] ** 2 - 2 * qvec[3] ** 2,
2 * qvec[2] * qvec[3] - 2 * qvec[0] * qvec[1],
],
[
2 * qvec[3] * qvec[1] - 2 * qvec[0] * qvec[2],
2 * qvec[2] * qvec[3] + 2 * qvec[0] * qvec[1],
1 - 2 * qvec[1] ** 2 - 2 * qvec[2] ** 2,
],
]
)
def apply_transform(T, points):
return ((T @ np.pad(points, ((0, 0), (0, 1)), constant_values=1).T).T)[..., :3]
Transf = np.identity(4)
transf_c2p = np.array([[-1., 0., 0.], [0., -1., 0.], [0., 0., 1.]]) # right-down-front to left-up-front
camera = maps[0].cameras[1]
json_obj = {
'camera': {
'cx': camera.params[1],
'cy': camera.params[2],
'f': camera.params[0]
},
'images': []
}
for id, img in maps[0].images.items():
pth = img.name
qvec = img.cam_from_world.rotation.quat
tvec = img.cam_from_world.translation
colmap_P = np.eye(4)
colmap_P[:3, :3] = qvec2rotmat(qvec)
colmap_P[:3, -1] = tvec
# Calculate new centres
original_cam_centres = -(qvec2rotmat(qvec).T @ tvec[:, None])[..., 0]
P = Transf @ colmap_P
new_C = apply_transform(Transf, original_cam_centres[None, :])[0]
# Calculate new rotations
Transf_R = Transf[:3, :3] / np.linalg.det(Transf[:3, :3]) ** (1/3) # rotational component of overall transform
R = (Transf_R @ qvec2rotmat(qvec).T).T
# flip camera orientation from COLMAP to PyTorch3D
R = transf_c2p @ R
T = (- R @ new_C[:, None])[..., 0] # Calculate new T
json_obj['images'].append({
'image_id': id,
'pth': pth,
'R': R.tolist(),
'T': T.tolist(),
})
And COLMAP reconstruction code(pycolmap is used instead of COLMAP's cli):
pycolmap.extract_features(colmap_db, colmap_source)
pycolmap.match_exhaustive(colmap_db)
maps = pycolmap.incremental_mapping(colmap_db, colmap_source, colmap_results)
Hi,
I would check through the coordinate format of your colmap.json, and see how it compares with the one I have provided - hopefully that can point at the difference.
Also double check that the qvec and tvec from pycolmap are exactly the same as that provided by COLMAP - I haven't used pycolmap so can't say for certain.
Also see if the visualizations from FOUND in the fitting script provide any clarity (if the mesh doesn't show at all, even on frame zero, it suggests that the cameras are pointing in the completely wrong directions).
Also, have you checked that the pycolmap reconstruction itself was successful? It might be worth visualising that in the COLMAP GUI - it could be that your reconstruction settings are wrong and you're not getting accurate cameras in the first place.
Thanks for the previous reply! Have been testing different colmap options, nothing seem to change. Checked the reconstruction in GUI, cameras are distributed correctly. Could you please send the whole script you’ve used, maybe it could help. Thanks in advance!
@OllieBoyne Any help?
Hi,
See attached for the full script - not formatted or tidied at all. Hope it helps!
Most of the functionality here is aligning it with a ground truth scan for proper evaluation, so I don't think you should need to worry about all of those steps.
Hi,
See attached for the full script - not formatted or tidied at all. Hope it helps!
Most of the functionality here is aligning it with a ground truth scan for proper evaluation, so I don't think you should need to worry about all of those steps.
Hi @OllieBoyne , code from zip file is quite useful to convert colmap json file, but still hard to fit model.
I found that we need transformed camera center for detecting FIND mesh model during training.
I tried to extract Transf matrix from your code, could you please share mesh_management py file?
Hi @deploy-soon, @R3pzz,
I have just added scripts that can properly convert a COLMAP sparse fit to the necessary format for FOUND:.
Note that, because COLMAP does not have accurate world scale, there may be a bad initialization for the FOUND process. This is likely why you saw the NAN errors @R3pzz, as the initialization was too bad to correct.
I was able to fix this behaviour on my end by adding some more registration steps, setting the stages (via the .yaml file) as follows:
250 epochs, lr=1.0, ['reg'], ['kp_nll']
250 epochs, lr=0.1, ['reg'], ['kp_nll']
250 epochs, lr=0.01, ['reg'], ['kp_nll']
100 epochs, lr=0.001, ['deform', 'reg'], ['kp_nll', 'norm_nll']
I'll close this issue as I believe this addresses everything - if you hit any more issues, please open a separate issue.
Hope this helps!