"IR to Color" transformation missing
fractalfantasy opened this issue · 7 comments
Tried to get IR to color space conversion using k4a_transformation_depth_image_to_color_camera_custom, but it doesn't seem to work.
an "IR to Depth" transformation should be very easy to add, as it's pretty much exactly the same as the Depth to Color transformation.
@fractalfantasy the k4a_transformation_depth_image_to_color_camera_custom should be able to fulfill the purpose here. please provide details of how you are using this function and what issue/error you were hitting. We haven't receive any report of people having issue using this function yet.
@rabbitdaxi We are trying to add k4a_transformation_depth_image_to_color_camera_custom IR to Color transformation to our DLL to be used in Electron app.
Here is the code snippet. I omit all the variables initialization and ir, depth, color frames 'get' functions, as there's no problem with them.
`int ir_to_color_image_created = k4a_image_create(K4A_IMAGE_FORMAT_CUSTOM16, jsFrame.colorImageFrame.width, jsFrame.colorImageFrame.height, jsFrame.colorImageFrame.width * 2, &ir_to_color_image);
int depth_to_color_image_created = k4a_image_create(K4A_IMAGE_FORMAT_DEPTH16, jsFrame.colorImageFrame.width, jsFrame.colorImageFrame.height, jsFrame.colorImageFrame.width * 2, &depth_to_color_image);
if ( (ir_to_color_image_created == K4A_RESULT_SUCCEEDED) && (depth_to_color_image_created == K4A_RESULT_SUCCEEDED) )
{
jsFrame.irToColorImageFrame.height = 101; //debug
if (k4a_transformation_depth_image_to_color_camera_custom(transformer, depth_image, ir_image, depth_to_color_image, ir_to_color_image,
K4A_TRANSFORMATION_INTERPOLATION_TYPE_NEAREST , 0) == K4A_RESULT_SUCCEEDED)
{
jsFrame.irToColorImageFrame.width = 102; //debug
jsFrame.irToColorImageFrame.width = k4a_image_get_width_pixels(ir_to_color_image);
jsFrame.irToColorImageFrame.height = k4a_image_get_height_pixels(ir_to_color_image);
jsFrame.irToColorImageFrame.image_length = k4a_image_get_size(ir_to_color_image);
jsFrame.irToColorImageFrame.stride_bytes = k4a_image_get_stride_bytes(ir_to_color_image);
jsFrame.irToColorImageFrame.image_data = new uint8_t[jsFrame.irToColorImageFrame.image_length];
}
else {
jsFrame.irToColorImageFrame.width = 1;
}
}`
k4a_transformation_depth_image_to_color_camera_custom function never succeeded. Output ir_to_color_image data array is empty.
Custom image format is set to K4A_IMAGE_FORMAT_CUSTOM16
Similar k4a_transformation_depth_image_to_color_camera(transformer, depth_image, depth_to_color_image) function works just fine in the same context.
What are we missing?
Sensor config:
depth_mode: KinectAzure.K4A_DEPTH_MODE_WFOV_2X2BINNED,
color_format: KinectAzure.K4A_IMAGE_FORMAT_COLOR_BGRA32,
color_resolution: KinectAzure.K4A_COLOR_RESOLUTION_720P,
camera_fps: KinectAzure.K4A_FRAMES_PER_SECOND_30
@rabbitdaxi do you have an idea of what could be wrong here? ^
@fractalfantasy sorry for the delay response (I am working on other projects now, so not checking this GitHub that frequent).
I see the issue. The k4a_transformation_depth_image_to_color_camera API requires both input and output custom images have the same custom format, e.g. in your case, K4A_IMAGE_FORMAT_CUSTOM16.
In your sample code, you passed the IR image directly into this API, the code internally has a check the format to match between input and output custom images failed. I know it looks like a extra step, but based on how the SDK expectation, you can create a custom_ir_image as the input custom image with the matching format of the output custom image, see below code snippet:
k4a_image_t custom_ir_image = NULL;
int ir_image_width_pixels = k4a_image_get_width_pixels(ir_image);
int ir_image_height_pixels = k4a_image_get_height_pixels(ir_image);
int ir_image_stride_bytes = k4a_image_get_stride_bytes(ir_image);
uint8_t *ir_image_buffer = k4a_image_get_buffer(ir_image);
if (K4A_RESULT_SUCCEEDED !=
k4a_image_create_from_buffer(K4A_IMAGE_FORMAT_CUSTOM16,
ir_image_width_pixels,
ir_image_height_pixels,
ir_image_width_pixels * (int)sizeof(uint16_t),
ir_image_buffer,
ir_image_height_pixels * ir_image_stride_bytes,
[](void *_buffer, void *context) {
delete[](uint8_t *) _buffer;
(void)context;
},
NULL,
&custom_ir_image))
{
printf("Failed to create custom ir image\n");
return false;
}
Then instead of passing in the ir_image, you pass in the custom_ir_image into this API
if (K4A_RESULT_SUCCEEDED !=
k4a_transformation_depth_image_to_color_camera_custom(transformation,
depth_image,
custom_ir_image,
transformed_depth_image,
transformed_ir_image,
K4A_TRANSFORMATION_INTERPOLATION_TYPE_NEAREST,
0))
{
printf("Failed to compute transformed depth and custom image\n");
return false;
}
@rabbitdaxi Thank you for pointing out the right solution. However, creating a custom image from buffer with k4a_image_create_from_buffer method with the parameters specified in your code sample has failed.
I've changed 'stride_bytes' parameter in the above method from ir_image_width_pixels * (int)sizeof(uint16_t) to ir_image_stride_bytes. And that solved the issue.
Now everything works as expected. Thanks again.
Here's the working method:
k4a_image_create_from_buffer(K4A_IMAGE_FORMAT_CUSTOM16,
ir_image_width_pixels,
ir_image_height_pixels,
ir_image_stride_bytes,
ir_image_buffer,
ir_image_height_pixels * ir_image_stride_bytes,
[](void *_buffer, void *context) {
delete[](uint8_t *) _buffer;
(void)context;
},
NULL,
&custom_ir_image)
Sounds good. Feel free to close if this resolved your issue :)
for C++ wrapper API
I leave release callback to NULL to avoid twice release
auto infer_custom = k4a::image::create_from_buffer(K4A_IMAGE_FORMAT_CUSTOM16,
infer.get_width_pixels(),
infer.get_height_pixels(),
infer.get_width_pixels() * (int) sizeof(uint16_t),
infer.get_buffer(),
infer.get_height_pixels() * infer.get_stride_bytes(),
NULL, NULL);