Heatmaps structure and validation

[boijuny]

Hey JotaBravo, congratulations for you and your team's work on the spacecraft robust pose estimation.

I am trying to reproduce the result of your work in order to understand the challenges of domain gap adaptation in spatial field. In order to do so, i retrieved the weights of the 2 stacked large Hourglass model on the synthetic dataset and tried to view the results. I struggle a bit on understanding the the output data. Is it possible to have few details about it ?

First, i managed to retrieve the weights and perform a prediction based on a random image of the synthetic dataset. The model then gives me an output as a list and len(output) = 2 :

• output[0] = {'hm_c' : [...], 'depth' : [...]}
• output[1] is also {'hm_c' : [...], 'depth' : [...]} but with different values.
  => Does theses two output corresponds to something like "without pseudo-labels" and "with pseudo-labels" ?

Also, i tried to display the heatmaps to identify their keypoints id but i have few concerns :

• hm_c is shape [n, 64,64] and not normalized. is it correct ?
• is it normal that multiple keypoints dont match correctly ?
  

Thank you for your time and your amazing work !

[JotaBravo]

Hi boijuny,

Thanks for your kind words on the work.

As you mention, you are using a stack of 2 Hourglass networks. Each element of the output list corresponds to the individual output of each network in the stack.

• output[0]: output of the first network
• output[1]: output of the second network

Each output contains two tensors:

• hm_c: represents the heatmap, which is not normalized. In fact, it is scaled by a factor of 100. Usually, other implementations make the center value of the Gaussian to equal 1 (also not normalized). Somehow, the experiments didn't converge with that value, so I scaled the value of the center by 100.
• depth: the estimated metric depth of the keypoint

Regarding the heatmap plotting:

• First of all, check that the weights you are using correspond to the actual dataset. The weights overfitted over Sunlamp likely won't work on the synthetic domain (and viceversa)

• As you mention, the heatmap has a shape of [n, 64, 64]. Make sure that:
  - you're properly scaling the keypoint coordinates to the image size. Also check that you're using rows and cols properly (i.e. [x,y] vs [y,x])
  - you're correctly sampling the keypoint position (i.e. retrieve the maximum)

I'm planning to post a tutorial soon with some simple, base code for spacecraft pose estimation. Please stay tuned, it might be useful