Question about ray direction

[zhshi0816]

Hi,

Thanks for making a great contribution. I have a question about the ray direction. The camera projection formulation is as follows:
$$u = \frac{x}{z}f + \frac{W}{2}, v = -\frac{y}{z}f + \frac{H}{2}$$

I just reverse the formulation to derive the ray direction as follows:
$$[ \frac{u-\frac{W}{2}}{f}z, \hspace{6mm} -\frac{v-\frac{H}{2}}{f}z, \hspace{6mm} z]$$.

Then I set z=-1, which is different from your result.
Can you show me where I was wrong? How do you calculate the direction?

[ziruiw-dev]

Hi @zhshi0816,

This is a general question about how to do ray casting, which all NeRF papers do and not specific to NeRFmm. One good tutorial is here.

In short, there are two ways to think of ray casting:

1. Do an inverse projection using $K^{-1}$ for pixel coordinates, this gives you camera rays cover the camera frustum, in local camera coordinate;
2. Manually compute a 3D coordinate for a pixel, by appending a $z$ value to a 2D pixel coordinate $(u, v)$, this is essentially doing the same thing as in the option 1.

After getting rays in local camera coordinate, we can further transform it to world coordinate with a camera pose.

Best,
Zirui