[Question] Use the Hand Pose Detection with the new models

[muhammedakyuzlu]

Hi, I'm trying to integrate the hand pose detection part of your code into my CPP CMake code
I'm not using OpenGL I'm using OpenCV to get my video stream plus I'm using the new models for the palm and hand landmark
the tflite_handpose.cpp contain almost all the necessary code for that, I just change the:

1- Modes path
https://storage.googleapis.com/mediapipe-assets/hand_landmark_lite.tflite
https://storage.googleapis.com/mediapipe-assets/palm_detection_lite.tflite

2- generate_ssd_anchors --> the new model has a different anchors

static int
generate_ssd_anchors ()
{
    SsdAnchorsCalculatorOptions anchor_options;
    anchor_options.num_layers = 4;
    anchor_options.min_scale = 0.1484375;
    anchor_options.max_scale = 0.75;
    anchor_options.input_size_height = 192;
    anchor_options.input_size_width  = 192;
    anchor_options.anchor_offset_x  = 0.5f;
    anchor_options.anchor_offset_y  = 0.5f;
//  anchor_options.feature_map_width .push_back(0);
//  anchor_options.feature_map_height.push_back(0);
    anchor_options.strides.push_back( 8);
    anchor_options.strides.push_back(16);
    anchor_options.strides.push_back(16);
    anchor_options.strides.push_back(16);
    anchor_options.aspect_ratios.push_back(1.0);
    anchor_options.reduce_boxes_in_lowest_layer = false;
    anchor_options.interpolated_scale_aspect_ratio = 1.0;
    anchor_options.fixed_anchor_size = true;
    GenerateAnchors (&s_anchors, anchor_options);
    return 0;
}

3 - init_tflite_hand_landmark --> the new models has a different output names

int
init_tflite_hand_landmark(const char *palm_model,const char *hand_model)
{


    /* Palm Detection */
    s_palm_interpreter.resolver.AddCustom("Convolution2DTransposeBias",
            mediapipe::tflite_operations::RegisterConvolution2DTransposeBias());

    tflite_create_interpreter_from_file (&s_palm_interpreter, palm_model);
    tflite_get_tensor_by_name (&s_palm_interpreter, 0, "input_1",           &s_palm_tensor_input);
    tflite_get_tensor_by_name (&s_palm_interpreter, 1, "Identity",  &s_palm_tensor_scores);
    tflite_get_tensor_by_name (&s_palm_interpreter, 1, "Identity_1",      &s_palm_tensor_points);

    /* Hand Landmark */
    tflite_create_interpreter_from_file (&s_hand_interpreter, hand_model);
    tflite_get_tensor_by_name (&s_hand_interpreter, 0, "input_1",         &s_hand_tensor_input);
    tflite_get_tensor_by_name (&s_hand_interpreter, 1, "Identity",        &s_hand_tensor_landmark);
    tflite_get_tensor_by_name (&s_hand_interpreter, 1, "Identity_1", &s_hand_tensor_handflag);

    generate_ssd_anchors ();
    return 0;
} 

the palm model takes an image with a 192*192 shape and [0-1] pixel range
I read the image, resize it, normalize it, and feed it to buf_fp32 but I got incorrect output.

Please could you help me figure out if there is anything I need to change in the code to make it run with the new models?
the new models should be faster and more accurate

this is the link to my repo: https://github.com/muhammedakyuzlu/hand-landmarks-cpp.git
it's a mess right now there are many functions included but not used

Thanks in advance

[terryky]

The output tensors seem to be opposite.
please try as below:

    /* Palm Detection */
    tflite_get_tensor_by_name (&s_palm_interpreter, 0, "input_1",         &s_palm_tensor_input);
    tflite_get_tensor_by_name (&s_palm_interpreter, 1, "Identity",        &s_palm_tensor_points);  <<==
    tflite_get_tensor_by_name (&s_palm_interpreter, 1, "Identity_1",      &s_palm_tensor_scores);  <<==

[muhammedakyuzlu]

@terryky I used netron.app to see the model outputs layer and fixed them, my code architecture is a little different than you, the input and the outputs pointers look like this now

// input layer
 float *_pPalmInputLayer = _palm_interpreter->typed_tensor<float>(_palm_interpreter->inputs()[0]);
 // outputs layer
 float *_pPalmOutputLayerBbox = _palm_interpreter->typed_tensor<float>(_palm_interpreter->outputs()[0]);
 float *_pPalmOutputLayerProb = _palm_interpreter->typed_tensor<float>(_palm_interpreter->outputs()[1]);

• the palm model runs OK now in my repo
• I'm working on hand landmark now

If you have time Please take a look and give me your opinion, I real benefited from your code a lot, Thank you.