About visibility

[Nole21]

Dear author:
Thank you for sharing such awesome work. After reading the paper, I have several questions about visibility. At stage 1, why do you sample multiple directions to compute visibilty for each direct SGs in equation (11) instead of using the predicted visibilit MLP? Since I'm new to this topic, is this a common practice? And why predict visibility relative to each SG is accurate in stage 2? In my opinion, the spatially-varying visibility(relative to 3d location and incoming direction) is a more accurate choice. Hope to get your reply. Best wished.
John

[ingra14m]

Hi, thanks for your interest.

Firstly, please refer to the paper in the project page instead of arxiv, since I cannot update the latest version on the arxiv.

As for the visibility modeling, directly using the visibility MLP distilled from NeuS Octree is not accurate for each direct SG (shown in Figure 3). The visibility for each direct SG $\eta$ is obtained from visibility MLP, which contains inevitable errors. So we used regularized visibility estimation (rve) to fine-tune the accuracy for each direct SG under the supervision of $\eta$ for better shadow and material decoupling.

[Nole21]

Thank you for your reply. I understand it. And by the way, are you considering using LLM to provide more priors? I notice that you have mentioned it in the discussion. From my perspective, the integration of LLM will surely improve the performance. So it may be not meaningful, despite the fact that nobody has done it so far. So I wonder if it is worthing doing this and how people will judge this work(inverse rendering with LLMs) from your point of view? I'm looking forward to you reply. Thank you!

[ingra14m]

From my perspective, incorporating LLMs to provide priors for inverse rendering is a highly significant direction. On one hand, inverse rendering is a highly ill-posed problem. Distinguishing components such as albedo, roughness, and the environment map purely through RGB loss is unrealistic. Introducing priors to constrain the components of the BRDF is therefore very meaningful. For example, shadows should not be baked into the albedo, yet current approaches like RobIR are still unable to achieve robustness and generalization across all scenarios.

On the other hand, this approach serves as a bridge between world priors and 3D reconstruction tasks. Currently, the application of world priors in reconstruction tasks has primarily demonstrated promising results in sparse-view setups, although these results remain below truly satisfactory levels. Extending world priors to inverse rendering represents a more general setting that could enable broader applications across both sparse and dense view setups, such as shadow removal and relighting.

I believe the following concurrent works could be helpful to you:

• MaterialFusion: Enhancing Inverse Rendering with Material Diffusion Priors
• RGBX: Image Decomposition and Synthesis Using Material- and Lighting-aware Diffusion Models