Running Mask RCNN Instance Segmentation no longer working on 0.9.0 2020.3 LTS
joelbudu opened this issue · 19 comments
Dear @LewisLiuPub, I am attempting to run a Mask RCNN Instance Segementation model on 0.9.0 2020.3 LTS, however I get the folowing error when running the pipeline_segmentation.launch.py sample with a custom trained Mask RCNN model.
Could you kindly assist?
[pipeline_with_params-1] [ DEBUG ] input size2
[pipeline_with_params-1] [ WARNING ] This inference sample should have only one input, but we got2inputs
[pipeline_with_params-1] [ INFO ] Segmentation model initialized.
[rviz2-2] [WARN] [rcl.logging_rosout]: Publisher already registered for provided node name. If this is due to multiple nodes with the same name then all logs for that logger name will go out over the existing publisher. As soon as any node with that name is destructed it will unregister the publisher, preventing any further logs for that name from being published on the rosout topic.
[pipeline_with_params-1] [ INFO ] Segmentation Engine initialized.
[pipeline_with_params-1] [ INFO ] Segmentation Inference instanced.
[pipeline_with_params-1] [ INFO ] Loading Network: Object Segmentation
[pipeline_with_params-1] [ INFO ] ... Adding one Inference: ObjectSegmentation
[pipeline_with_params-1] [ INFO ] Updating connections ...
[pipeline_with_params-1] [ INFO ] Checking connection into pipeline:[Image(1)<-->ObjectSegmentation(2)]
[pipeline_with_params-1] [ INFO ] Adding connection into pipeline:[Image<-->ObjectSegmentation]
[pipeline_with_params-1] [ INFO ] Checking connection into pipeline:[ObjectSegmentation(2)<-->ImageWindow(3)]
[pipeline_with_params-1] [ INFO ] Adding connection into pipeline:[ObjectSegmentation<-->ImageWindow]
[pipeline_with_params-1] [ INFO ] Checking connection into pipeline:[ObjectSegmentation(2)<-->RosTopic(3)]
[pipeline_with_params-1] [ INFO ] Adding connection into pipeline:[ObjectSegmentation<-->RosTopic]
[pipeline_with_params-1] [ INFO ] Checking connection into pipeline:[ObjectSegmentation(2)<-->RViz(3)]
[pipeline_with_params-1] [ INFO ] Adding connection into pipeline:[ObjectSegmentation<-->RViz]
[pipeline_with_params-1] [ INFO ] One Pipeline Created!
[pipeline_with_params-1] --> Image
[pipeline_with_params-1] Image --> ObjectSegmentation
[pipeline_with_params-1] ObjectSegmentation --> ImageWindow
[pipeline_with_params-1] ObjectSegmentation --> RosTopic
[pipeline_with_params-1] ObjectSegmentation --> RViz
[pipeline_with_params-1] [ DEBUG ] DEBUG: in Pipeline run process...
[pipeline_with_params-1] [ DEBUG ] DEBUG: Enqueue for detection: ObjectSegmentation
[pipeline_with_params-1] [ DEBUG ] first tensor2
[pipeline_with_params-1] [ DEBUG ] first tensor4
[pipeline_with_params-1] [ WARNING ] No input named: input
[pipeline_with_params-1] terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
[pipeline_with_params-1] what(): [NOT_FOUND] Failed to find input or output with name: ''
[ERROR] [pipeline_with_params-1]: process has died [pid 20310, exit code -6, cmd '/home/rl001/ros2_ws_new/install/dynamic_vino_sample/lib/dynamic_vino_sample/pipeline_with_params -config /home/rl001/ros2_ws_new/install/dynamic_vino_sample/share/dynamic_vino_sample/param/pipeline_segmentation_image.yaml /openvino_toolkit/image_raw:=/camera/color/image_raw /openvino_toolkit/segmentation/segmented_obejcts:=/ros2_openvino_toolkit/segmented_obejcts /openvino_toolkit/segmentation/images:=/ros2_openvino_toolkit/image_rviz'].
@joelbudu , sorry for late reply.
Would you please share more info about the model you used? Give me steps how to get it, then I can reproduce the issue.
Hello @LewisLiuPub,
I am using the same mask rcnn model converted from the tensorflow object detection api on the COCO data set - mask_rcnn_inception_v2_coco_2018_01_28
Went through similar steps in the link below:
https://github.com/intel/ros2_openvino_toolkit/tree/0.7.0_2019R3.1#launching
download and convert a trained model to produce an optimized Intermediate Representation (IR) of the model
cd /opt/openvino_toolkit/dldt/model-optimizer/install_prerequisites
sudo ./install_prerequisites.sh
mkdir -p ~/Downloads/models
cd ~/Downloads/models
wget http://download.tensorflow.org/models/object_detection/mask_rcnn_inception_v2_coco_2018_01_28.tar.gz
tar -zxvf mask_rcnn_inception_v2_coco_2018_01_28.tar.gz
cd mask_rcnn_inception_v2_coco_2018_01_28
#FP32
sudo python3 /opt/openvino_toolkit/dldt/model-optimizer/mo_tf.py --input_model frozen_inference_graph.pb --tensorflow_use_custom_operations_config /opt/openvino_toolkit/dldt/model-optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --output_dir /opt/openvino_toolkit/models/segmentation/output/FP32
#FP16
sudo python3 /opt/openvino_toolkit/dldt/model-optimizer/mo_tf.py --input_model frozen_inference_graph.pb --tensorflow_use_custom_operations_config /opt/openvino_toolkit/dldt/model-optimizer/extensions/front/tf/mask_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --data_type=FP16 --output_dir /opt/openvino_toolkit/models/segmentation/output/FP16
I updated the .yaml file to point to the converted model and ran the Object Segmentation pipeline
ros2 launch dynamic_vino_sample pipeline_segmentation.launch.py
Dear @LewisLiuPub . Is it possible to share some steps for testing the Mask RCNN. I'm not too clear on the procedure for v0.10.0.
Thanks
@joelbudu , ok, let me gather some info for you.
@LewisLiuPub any luck?
Hi,
Currently, I can't find suitable environment to test this feature, but I hope the below steps help you:
NOTE:
Below steps have been tested on Ubuntu 20.04.
1. Environment Setup
- Install ROS2 Galactic (guide)
- Install Intel® OpenVINO™ Toolkit Version: 2021.4 (guide) or building by source code (guide)
- Install Intel® RealSense ™ SDK (guide)
2. Building and Installation
- Install ROS2_OpenVINO_Toolkit packages
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws/src
git clone https://github.com/intel/ros2_openvino_toolkit -b galactic_dev
git clone https://github.com/intel/ros2_object_msgs
git clone https://github.com/IntelRealSense/realsense-ros.git -b ros2
git clone https://github.com/ros-perception/vision_opencv.git -b ros2
- Install dependencies
sudo apt-get install ros-galactic-diagnostic-updater
- Build package
source /opt/ros/galactic/setup.bash
source /opt/intel/openvino_2021/bin/setupvars.sh
cd ~/catkin_ws
colcon build --symlink-install
source ./install/local_setup.bash
3. Running the Demo
- Preparation
source /opt/intel/openvino_2021/bin/setupvars.sh
sudo mkdir -p /opt/openvino_toolkit
sudo ln -s /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader /opt/openvino_toolkit/models
sudo chmod 777 -R /opt/openvino_toolkit/models
- See all available models
cd /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader
sudo python3 downloader.py --print_all
- Download the optimized Intermediate Representation (IR) of model (execute once), for example:
cd /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader
sudo python3 downloader.py --name semantic-segmentation-adas-0001 --output_dir /opt/openvino_toolkit/models/semantic-segmentation/output
- run object segmentation sample code input from RealSenseCamera.
ros2 launch dynamic_vino_sample pipeline_segmentation.launch.py - run object segmentation sample code input from Image.
ros2 launch dynamic_vino_sample pipeline_segmentation_image.launch.py
Dear @LewisLiuPub,
The example you just outlined is for a semantic segmentation model (semantic-segmentation-adas-0001) which differs from instance segmentation I was referring to (Mask RCNN). I realised the ObjectSegmentation model in this repository was changed from one that takes Mask RCNN like models to one for semantic segmentation -> that was from version 0.9.0 .
Am I correct with my observation?
Kind regards,
Joel
Hi, @joelbudu
Yes, You are right, we have only tested the segmentation model listed in the Quick Start document. Let me know how urgent do you need MaskRCNN features, I am planning the development in 2022, but I am afraid not to make it ready as quickly as you want.
Dear @LewisLiuPub, given that this feature was already implemented previously in 0.8.0 2019R3.1 I assumed this was still going to be available in subsequent versions of this repository. I actually need this a bit urgently to be able to upgrade our OpenVINO version to anything greater than 2019. It's number one priority for us
I am also not clear on how it was replaced with a semantic segmentation model since the previous output in the pipeline was a people_msgs::msg::ObjectsInMasks.msg, is it using the same message or it is now using an Image.msg ?
Dear @joelbudu , could you please let me know which maskrcnn model you are using? I am a bit busy recently, but I would like to make a draft evaluation first.
@LewisLiuPub it's the one outlined in the documentation here: https://github.com/intel/ros2_openvino_toolkit/blob/master/doc/inferences/Object_Segmentation.md
dear @joelbudu , sorry for the late reply.
Now we are preparing resource to fix this issue, appreciate if you can double confirm the model you are using is this one:
If so, you may have a quick check with the latest ROS2-OpenVINO code but reverting this patch .
Please let me know whether if works or not.
Dear @LewisLiuPub . Yes I can confirm that the model I'm using is indeed the one linked above.
I will have to find some time to look into whether reverting the patch you've linked solves the issue. Thank you
@LewisLiuPub I tried reverting the patch however the build fails on my attempt with the following error:
--- stderr: dynamic_vino_lib
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/models/object_segmentation_model.cpp:34:50: error: no ‘void Models::ObjectSegmentationModel::setLayerProperty(InferenceEngine::CNNNetReader::Ptr)’ member function declared in class ‘Models::ObjectSegmentationModel’
InferenceEngine::CNNNetReader::Ptr net_reader)
^
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/models/object_segmentation_model.cpp:105:57: error: no ‘void Models::ObjectSegmentationModel::checkLayerProperty(const Ptr&)’ member function declared in class ‘Models::ObjectSegmentationModel’
const InferenceEngine::CNNNetReader::Ptr &net_reader)
^
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/models/object_segmentation_model.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /usr/include/x86_64-linux-gnu/c++/7/bits/c++allocator.h:33:0,
from /usr/include/c++/7/bits/allocator.h:46,
from /usr/include/c++/7/string:41,
from /usr/include/c++/7/bits/locale_classes.h:40,
from /usr/include/c++/7/bits/ios_base.h:41,
from /usr/include/c++/7/ios:42,
from /opt/ros/dashing/opt/yaml_cpp_vendor/include/yaml-cpp/parser.h:10,
from /opt/ros/dashing/opt/yaml_cpp_vendor/include/yaml-cpp/yaml.h:10,
from /home/rl001/ros2_ws/install/vino_param_lib/include/vino_param_lib/param_manager.hpp:22,
from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:20:
/usr/include/c++/7/ext/new_allocator.h: In instantiation of ‘void __gnu_cxx::new_allocator<_Tp>::construct(_Up*, _Args&& ...) [with _Up = Models::ObjectSegmentationModel; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel]’:
/usr/include/c++/7/bits/alloc_traits.h:475:4: required from ‘static void std::allocator_traits<std::allocator<_CharT> >::construct(std::allocator_traits<std::allocator<_CharT> >::allocator_type&, _Up*, _Args&& ...) [with _Up = Models::ObjectSegmentationModel; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel; std::allocator_traits<std::allocator<_CharT> >::allocator_type = std::allocator<Models::ObjectSegmentationModel>]’
/usr/include/c++/7/bits/shared_ptr_base.h:526:39: required from ‘std::_Sp_counted_ptr_inplace<_Tp, _Alloc, _Lp>::_Sp_counted_ptr_inplace(_Alloc, _Args&& ...) [with _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel; _Alloc = std::allocator<Models::ObjectSegmentationModel>; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/7/bits/shared_ptr_base.h:637:4: required from ‘std::__shared_count<_Lp>::__shared_count(std::_Sp_make_shared_tag, _Tp*, const _Alloc&, _Args&& ...) [with _Tp = Models::ObjectSegmentationModel; _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/7/bits/shared_ptr_base.h:1295:35: required from ‘std::__shared_ptr<_Tp, _Lp>::__shared_ptr(std::_Sp_make_shared_tag, const _Alloc&, _Args&& ...) [with _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel; __gnu_cxx::_Lock_policy _Lp = (__gnu_cxx::_Lock_policy)2]’
/usr/include/c++/7/bits/shared_ptr.h:344:64: required from ‘std::shared_ptr<_Tp>::shared_ptr(std::_Sp_make_shared_tag, const _Alloc&, _Args&& ...) [with _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}; _Tp = Models::ObjectSegmentationModel]’
/usr/include/c++/7/bits/shared_ptr.h:690:14: required from ‘std::shared_ptr<_Tp> std::allocate_shared(const _Alloc&, _Args&& ...) [with _Tp = Models::ObjectSegmentationModel; _Alloc = std::allocator<Models::ObjectSegmentationModel>; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}]’
/usr/include/c++/7/bits/shared_ptr.h:706:39: required from ‘std::shared_ptr<_Tp> std::make_shared(_Args&& ...) [with _Tp = Models::ObjectSegmentationModel; _Args = {const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const int&}]’
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:322:79: required from here
/usr/include/c++/7/ext/new_allocator.h:136:4: error: invalid new-expression of abstract class type ‘Models::ObjectSegmentationModel’
{ ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/inferences/object_segmentation.hpp:28:0,
from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:51:
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/models/object_segmentation_model.hpp:28:7: note: because the following virtual functions are pure within ‘Models::ObjectSegmentationModel’:
class ObjectSegmentationModel : public BaseModel
^~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/models/vehicle_attribs_detection_model.hpp:22:0,
from /home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/src/pipeline_manager.cpp:33:
/home/rl001/ros2_ws/src/ros2_openvino_toolkit/dynamic_vino_lib/include/dynamic_vino_lib/models/base_model.hpp:109:18: note: virtual bool Models::BaseModel::updateLayerProperty(InferenceEngine::CNNNetReader::Ptr)
virtual bool updateLayerProperty(InferenceEngine::CNNNetReader::Ptr network_reader) = 0;
^~~~~~~~~~~~~~~~~~~
make[2]: *** [CMakeFiles/dynamic_vino_lib.dir/src/pipeline_manager.cpp.o] Error 1
make[1]: *** [CMakeFiles/dynamic_vino_lib.dir/all] Error 2
make: *** [all] Error 2
---
Failed <<< dynamic_vino_lib [8.95s, exited with code 2]
I'm thinking there might be compatibility issues and others that need to be fixed after the patch.
Yes, you are right. there are some compatibility issues. @joelbudu
Please let me know which ROS2 system (Dashing? Foxy? Galactic?) you are using.
will close the issue. Mask RCNN instance segmentation has been supported in branch ros2 (supporting foxy/galactic).
Thank you for working on this @LewisLiuPub