Example of optimisation using parametric models

[eshafeeqe]

Dear Opt authors,

Most of the tracking approach using parametric models for tracking specific subjects like a human hand, face, and body. Some of such models are given below.

smpl (huamn body) : http://smpl.is.tue.mpg.de
mano (hands) : http://mano.is.tue.mpg.de
flame (face) : http://flame.is.tue.mpg.de

Is it possible to give an example of optimization using such models? This could be useful for researchers to experiment model-based optimization fastly.

Thanks and Regards,
Shafeeq E

[gerwang]

Facing the same problem. Parameters are used in every term, but terms iterate over an image or graph.

[gerwang]

Seems like Opt is unable to model such problems. Sad.

[gerwang]

I find a workaround using Graph. You know the number of your parameters so encode it with a floatN, no an array with dimension N, so you can directly access its index. Then create a graph to associate every term with the only parameter block. One example of estimating spherical harmonic lighting is as follows.

local WH, One = Dim("WH", 0), Dim("One", 1)
local Ls = Unknown("L", float9, {One}, 0)
local T = Array("T", uint8, {WH}, 1) -- tri index
local N = Array("N", float3, {WH}, 2) -- normal
local A = Array("A", float, {WH}, 3) -- albedo
local I = Array("I", float, {WH}, 4) -- image
local G = Graph("G", 5,
				"xy", {WH}, 6,
				"L0", {One}, 7)

-- UsePreconditioner(true)

local n = N(G.xy)
local x, y, z = n(0), n(1), n(2)
local L = Ls(G.L0)

local light = L(0)
			+ L(1) * z
			+ L(2) * y
			+ L(3) * x
			+ L(4) * (2 * z * z - x * x - y * y)
			+ L(5) * y * z
			+ L(6) * x * z
			+ L(7) * x * y
			+ L(8) * (x * x - y * y)

local render = light * A(G.xy)
local data = I(G.xy) - render

Energy(Select(T(G.xy) , data, 0.0))

[eshafeeqe]

Hello,

Thanks, @gerwang for the interest of this problem.

Maybe we can try to solve this problem together.

One of the main problems I faced while trying to use the SMPL model for optimizing using Opt is the optimization function to be represented by a kinematic tree. That means the position of each vertex depends upon the product of transformations of parameters.

I am a little busy now, I will definitely reply to this thread later.

[zhangxaochen]

Hi @gerwang ,
I'm reading your code snippet of "estimating spherical harmonic lighting", but get confused about this code:

1. what's the albedo image local A = Array("A", float, {WH}, 3) -- albedo used for? Seems in the paper it doesn't need the albedo during lighting estimation...

2. is the tri index an index matrix of your mesh triangles? Is it used as a validity indicator?

Could you share your full code implementation for lighting estimation? Thx in advance~

[gerwang]

Hi @zhangxaochen

Thanks for your interest.

1. Opt's shape-from-shading example only solves for objects with uniform albedo, like those marble statues. For other objects, albedo is not same for every pixel.

2. Yes, it is used to indicate whether the pixel belongs to objects applicable to SH light, or is just background.

Sorry that my full code contains a lot irrelevant to Opt, thus I don't paste all of it here. I think for Opt the script is enough.

[zhangxaochen]

Hi @gerwang

Thank u for your answer~

I'm also curious about the dimlist parameter in the Graph spec:

graph = Graph(name, problemparams_position_of_graph_size,
             {vertexname, dimlist, problemparams_position_of_indices}*)

...
dimlist is a Lua array of dimensions (e.g., {W,H}). Arrays can be 1, 2, or 3 dimensional. This vertex will be a pointer into any array of this dimension.

How could a 2D array in the graph be used? (if not flattened into 1D array)

For example, if I have an Image Im and a graph G:

local Im = Array("Im",opt_float, {W,H}, 1)
local G = Graph("G", 2, 
                "d", {W, H}, 3, --'G.d' points to a 2D array
                "p", {U}, 4)

How can I access a pixel in Im and its neighbors?

center_pixel = Im(G.d) --is this right?
neighbor_pixel = Im(???)

I didn't see any demo code or docs in the paper or in this repo, not sure if I missed something...
(It would be great If @Mx7f had time to write some sample code snippet in the docs :-) )

[gerwang]

Hi @zhangxaochen

I think Graph only supports one-dimensional array, at least I only used one-dim array in my code snippet. Please notice that WH is a number which value is W*H, and the image is actually flattened.

If you want to access a pixel and its neighbors, maybe you need five "auxiliary index arrays", like the xy in my snippet (In fact, I stored every pixel's index offset in xy, like 0 for the top left pixel and WH-1 for the right bottom pixel). I know it is low efficiency but it works after all.

[zhangxaochen]

hi @gerwang , as the doc says:

dimlist is a Lua array of dimensions (e.g., {W,H}). Arrays can be 1, 2, or 3 dimensional.

So I'm just wondering if it's true, and how that multi-dimensional arrays could be used.

Anyway, the "five auxiliary index arrays" idea indeed works :)