Cannot render the correct image from mesh .obj with texture

[wangmiaowei]

test.zip
Thanks for your support. But I can not get good rendering effects using my own dataset in this.zip file. The mesh object can be visualized in the meshlab software. However, I get something shown here:

And the code I used as shown here:

import json
import os
import glob
import time
import kaolin
from PIL import Image
import torch
import numpy as np
from matplotlib import pyplot as plt

import kaolin as kal
mesh_path = r"new_musk.obj"
mesh = kaolin.io.obj.import_mesh(mesh_path,with_materials=True,with_normals=True)

vertices = mesh.vertices.cuda().unsqueeze(0)
vertices.requires_grad = True
faces = mesh.faces.cuda()
uvs = mesh.uvs.cuda().unsqueeze(0)

nb_faces = faces.shape[0]
#nb_vertices = vertices_init.shape[1]
face_size = 3

face_uvs_idx = mesh.face_uvs_idx.cuda()
def get_camera_from_view(elev, azim, r=3.0, look_at_height=0.0):
    x = r * torch.sin(elev) * torch.sin(azim)
    y = r * torch.cos(elev)
    z = r * torch.sin(elev) * torch.cos(azim)

    pos = torch.tensor([x, y, z]).unsqueeze(0)
    look_at = torch.zeros_like(pos)
    look_at[:, 1] = look_at_height
    direction = torch.tensor([0.0, 1.0, 0.0]).unsqueeze(0)

    camera_proj = kal.render.camera.generate_transformation_matrix(pos, look_at, direction)
    return camera_proj

face_uvs = kal.ops.mesh.index_vertices_by_faces(uvs, face_uvs_idx).detach()
face_uvs.requires_grad = False
texture_map = mesh.materials[0]['map_Kd'].unsqueeze(0).float().permute(0,3,1,2).cuda()/255
#camera projection
camera = kal.render.camera.generate_perspective_projection(np.pi / 3).cuda()
camera_projection = camera
elev = 1.047
azim = 0.0
radius = 1.5
look_at_height = 0.25

camera_transform = get_camera_from_view(torch.tensor(elev), torch.tensor(azim), r=radius,look_at_height=look_at_height).cuda()
face_vertices_camera, face_vertices_image, face_normals = kal.render.mesh.prepare_vertices(
    vertices, faces, camera_projection, camera_transform=camera_transform)
### Perform Rasterization ###
# Construct attributes that DIB-R rasterizer will interpolate.
# the first is the UVS associated to each face
# the second will make a hard segmentation mask
face_attributes = [
    face_uvs.repeat(1, 1, 1, 1),
    torch.ones((1, nb_faces, 3, 1), device='cuda')
]

image_features, soft_mask, face_idx = kal.render.mesh.dibr_rasterization(
            512, 512, face_vertices_camera[:, :, :, -1],
            face_vertices_image, face_attributes, face_normals[:, :, -1],
            rast_backend='cuda')
# image_features is a tuple in composed of the interpolated attributes of face_attributes

texture_coords, mask = image_features

mask =mask.cpu()
image = kal.render.mesh.texture_mapping(texture_coords,texture_map.repeat(1, 1, 1, 1), 
                                        mode='bilinear').cpu()
image = torch.clamp(image * mask, 0., 1.)
print('image.shape: ',image.shape)
print('image.max(): ',image.max())

tensor1 = image[0].detach().cpu().numpy()
print('tensor1.shape: ',tensor1.shape)
path=r"render_image.jpg"
Image.fromarray((tensor1 * 255).astype(np.uint8)).save(path)

Could you help me figure out the reasons?

[Caenorst]

Hi @wangmiaowei , we currently don't really support cyclic UVs for interpolation of textures, as stated in https://kaolin.readthedocs.io/en/latest/modules/kaolin.render.mesh.html#kaolin.render.mesh.texture_mapping the expected coordinates are in the [0, 1] range while your UVs avec in [-1, 1] range.

An easy workaround is to simply modify you code the following:

image = kal.render.mesh.texture_mapping(
    texture_coords % 1.,
    texture_map.repeat(1, 1, 1, 1),
    mode='bilinear'
).cpu()

[wangmiaowei]

Thanks @Caenorst It works well!