Fix for triplane projection

[MrTornado24]

Hi, thanks for releasing the code! I found that according to projection matrixes, the order of triplanes is (x, y), (x, z) and (z, x) rather than three orthogonal planes. Am I thinking something wrong or this is exactly your design?

[matthew-a-chan]

Thanks for bringing this to our attention! This is an interesting one. We certainly intended for it to be the three orthogonal projections as you suggest, but a quick review over our code history shows that it's been this way forever... and also that all of our results were achieved using the buggy XY, XZ, ZX planes. For the sake of exact reproducibility I will leave as is for now, though this decision may change later. In your own implementations you should certainly change the projections to be accurate, which will likely improve results.

Here's the required fix:

Modify:

eg3d/eg3d/training/volumetric_rendering/renderer.py

Lines 22 to 38 in 4930760

	def generate_planes():
	"""
	Defines planes by the three vectors that form the "axes" of the
	plane. Should work with arbitrary number of planes and planes of
	arbitrary orientation.
	"""
	return torch.tensor([[[1, 0, 0],
	[0, 1, 0],
	[0, 0, 1]],
	[[1, 0, 0],
	[0, 0, 1],
	[0, 1, 0]],
	[[0, 0, 1],
	[1, 0, 0],
	[0, 1, 0]]], dtype=torch.float32)

to match:

def generate_planes():
    """
    Defines planes by the three vectors that form the "axes" of the
    plane. Should work with arbitrary number of planes and planes of
    arbitrary orientation.
    """
    return torch.tensor([[[1, 0, 0],
                            [0, 1, 0],
                            [0, 0, 1]],
                            [[1, 0, 0],
                            [0, 0, 1],
                            [0, 1, 0]],
                            [[0, 0, 1],
                            [0, 1, 0],
                            [1, 0, 0]]], dtype=torch.float32)

Which should accurately accomplish the XY, XZ, ZY projections that we want.
Note that only the last 2 lines change!

[thuanz123]

hi @matthew-a-chan @MrTornado24 , if I want to use XY, YZ, ZX projections then is the below code correct ?

def generate_planes():
    """
    Defines planes by the three vectors that form the "axes" of the
    plane. Should work with arbitrary number of planes and planes of
    arbitrary orientation.
    """
    return torch.tensor([[[1, 0, 0],
                            [0, 1, 0],
                            [0, 0, 1]],
                            [[0, 1, 0],
                            [0, 0, 1],
                            [1, 0, 0]],
                            [[0, 0, 1],
                            [1, 0, 0],
                            [0, 1, 0]]], dtype=torch.float32)

[matthew-a-chan]

Hi, Yes. That is the original code, so it will be the original XY YZ ZX projections. Also, no need to @. We read the issues :)

[ty625911724]

Dear authors:

It seems that the projection code is with bug, so the pretrained models are also not accurate inplemention for 3 planes.

Do you plan to release new pretrained model trained with correct projection matrixes?

Thanks!

[ericryanchan]

We haven't yet had time to retrain and thoroughly test with the triplane fix, but I pushed the fix to a branch called fixed_triplanes and uploaded a preliminary model here, which we finetuned from one of the original checkpoints: https://drive.google.com/drive/folders/1F77Sh2JS-udVsOkARk2h5t6ShN213p6c?usp=sharing

We'll try to revisit this and train additional models in a few weeks!

[ty625911724]

Thanks a lot for your kind comments and sharing weights. Looking forward to the new models! ///(^v^)\\

[oneThousand1000]

Hi there! I re-trained the correct-triplane model for research purposes :), you can find it here and download the ffhq-fixed-triplane512-128.pkl.