how to make the inference when real object is not available

I have started using DOPE
cloned the repo, installed dependencies, and running the scripts.

I am on
-- Ubuntu 20.04
-- installed bare-bone ros-noetic
-- I have realsense camera model D455
-- I have weights copied to the weights folder

I do not have access to the real objects from the object set models.

I have a separate ros package for intel realsense camera that I launch. It publises data on several topics of which are important

topic_camera: "/camera/color/image_raw"
topic_camera_info: "/camera/color/camera_info"

being specified inside the config_pose.yaml

When I run rviz, and I can see the dope topic receiving the image from real-sense camera.

From this point on, in which way I can see the inference working,

--- I just need to see the inference working for the provided / trained object sets.

Can you display the belief map? Are you sure your weights are working on your training data? You should be able to run the project the way you described your setup.

@TontonTremblay thanks for the reply.

At this moment, I am connected to realsense camera live stream.
I do not have access to ycb trained object (pictures or real objects).

I can see the data being published on DOPE topcis, I imagine, that certain topics would not give useful info as there are no trained objects that could be inferred.

This is my question, how can I test using some image? or any other way that inference is working for the trained object models.

When I selected /belief_soup/image this the result I get on rviz

thanks,

There was another issue talking about streaming
bag data

This data when viewed in rviz looks like this

Screenshot from 2024-04-16 16-34-56

The dope node expects two topics
topic_camera: "/camera/color/image_raw"
topic_camera_info: "/camera/color/camera_info"

I do not know how to configure those properly for working with bag data, and to make inference.

ahhh sorry I miss understood you, use the no_ros_inference script. You can feed images.

I was following the principal readme that asks for
-- setting up the system
-- running the camera
-- running the dope node
-- rviz

It appears that one needs actual YCB objects available for inference to work on that.

For the above case, dope node available under
/Deep_Object_Pose/ros1/dope/nodes/
should take care of the inference.

There is another script just for inference, inference.py that is available under /Deep_Object_Pose/inference/inference.py
I understood the different arguments for the command,

python inference.py --weights ../weights --data ../sample_data --object cracker

but I do not understand
``--data: path to the data that will be used as input to run inference on. The script **recursively** loads all data that end with extensions specified in the --exts` flag.

Should these be image of ycb objects, and what does it mean with --exts flags.

Please, if you could explain a bit.

thanks,

I tried the above script, downloading raw_rgb images for some object categories namely {cracker, soup, mustard_bottle}. I issued the following command:

python3 inference.py --weights ../ros1/dope/weights/ --data ../data/005_tomato_soup_can_berkeley_rgb_highres/005_tomato_soup_can/ --object soup

However, I receive some numpy related error:

If you have some ideas about that. My numpy version is 1.17.1

thanks,

I was finally able to produce inference for some object. Here are the steps I followed.

downloaded the object raw RGB images from YCB object and model set.
A sub-sample to be used for testing the inference script was extracted at the following location ~/catkin_ws/src/Deep_Object_Pose/data/soup_can_sample. This sample contains 10 images
trained weights are available at this path ~/catkin_ws/src/Deep_Object_Pose/ros1/dope/weights
The inference script is available at this path ~/catkin_ws/src/Deep_Object_Pose/inference/inference.py
The following command was issued to perform the inference.
python3.8 inference.py --weights ../ros1/dope/weights/soup_60.pth --data ../data/soup_can_sample/ --exts jpg --object soup --config ../ros1/dope/config/test_config_pose.yaml --camera ../ros1/dope/config/test_camera_info.yaml

As a result, I received an output folder at this location ~/catkin_ws/src/Deep_Object_Pose/inference/output

This folder contains for each image file, corresponding json file which contains information such as

location
quaternion
cuboid

question: which frame is the provided location and quaternion values. Is it in camera reference frame.

This link details some information to view this json file. However the following issues occurred

the provided whl file is not supported on my system, when I installed, using pip install nvisii, I could do that, however, receive an error when running this script
script: python3 ../evaluate/render_json.py --path_json output/soup_60/N1_9.json --scale 0.01 --opencv --contour --gray
error:

any pointers about this

thanks,

it says you do not have the 3d model soup? did you download the soup model? Check your paths.

thanks @TontonTremblay

I downloaded also the 3d models also from (http://ycb-benchmarks.s3-website-us-east-1.amazonaws.com/) manually .tgz under Processed Data.

and copied those under my data folder. These downloaded models do contain an textured.obj file at the following location
~/005_tomato_soup_can_berkeley_meshes/005_tomato_soup_can/poisson/textured.obj

Taking that into account the command I issued was the following:
python3 ../evaluate/render_json.py --path_json output/soup_60/N1_27.json --scale 0.01 --opencv --contour --gray --objs_folder ../data/005_tomato_soup_can_berkeley_meshes/005_tomato_soup_can/poisson/textured.obj

Now, I passed --objs_folder as an argument and indicated the textured.obj.

In the render_json.py, I made this small change:

parser.add_argument(
    '--objs_folder',
   # default='content/',
    required=True,
    help = "object to load folder, should follow YCB structure"
)

However, at runtime, the system always concatenates the following soup/google_16k/textured.obj e.g.,

RuntimeError: Error: "soup_0"Unable to open file "../data/005_tomato_soup_can_berkeley_meshes/005_tomato_soup_can/poisson/textured.obj/soup/google_16k/textured.obj".

Where it is getting that link, and how to suppress that.

thanks,

My question is

Do I need to download the 3d Models from a specific link
In the render_json.py script consider the following:

parser.add_argument(
   '--path_json',
   required=True,
   help = "path to the json files you want loaded,\
           it assumes that there is a png accompanying."
)

The output produce by inferenc.py creates json files, though accompanying files are jpg and not png. Does that make a difference?

The models "manually" downloaded from (http://ycb-benchmarks.s3-website-us-east-1.amazonaws.com/) contain .stl, .ply, textured.obj but no texture_map_flat.png though a textured.png. Can this be replaced inside the code for
render_json.py. Refer the following:

if opt.bop:
        entity_visii = create_obj(
            name = obj['class'] + "_" + str(i_obj),
            path_obj = f"{opt.objs_folder}/obj_{str(name).zfill(6)}.ply",
            path_tex = f"{opt.objs_folder}/obj_{str(name).zfill(6)}.png",
            scale = .001, 
            rot_base = None
        )      

    else:    
        entity_visii = create_obj(
            name = obj['class'] + "_" + str(i_obj),
            path_obj = opt.objs_folder + "/"+name + "/google_16k/textured.obj",
            path_tex = opt.objs_folder + "/"+name + "/google_16k/texture_map_flat.png",
            scale = opt.scale,
            rot_base = None
        )

thanks,

I just noticed that output produced by inference creates an output folder where it has the
jpg images and corresponding json files.

The jpg images already have some bounding cubes displayed on them, but these seem pretty off. I will put some images and corresponding outputs from inference.

The original jpg image of soup object downloaded from ycb official.

The json file produced after running inference.py script. It contains the pose information
N1_15.json

The output jpg with ( bounding box?) by the inference.py script

Can this pose information that we see in the json file be considered valid (assuming rendering of the bounding cube is not correct) ?
What can the reasons for this.
I used trained weights when running the inference script.

I consider the ycb data set, and class 'soup'
I am using the inference script ~/catkin_ws/src/Deep_Object_Pose/inference/inference.py
I am using a small sample of the images for this class

after making some further settings:

scaling the input images to have 400 as height,
in test_config_pose.yaml, I have

weights: { 
   "soup":"package://dope/weights/soup_60.pth",
 }

meshes: {
"soup":    "file://home/zahid/catkin_ws/src/Deep_Object_Pose/Dataset_Utilities/nvdu/data/ycb/aligned_cm/005_tomato_soup_can/google_16k/textured.obj",
}

The command that issued is the following
python3.8 inference.py --weights ../ros1/dope/weights/soup_60.pth --data ../data/soup_can_sample_scaled/ --exts jpg --object soup --config ../ros1/dope/config/test_config_pose.yaml --camera ../ros1/dope/config/test_camera_info.yaml

Is the bounding containers appear OK?

I think the order of keypoints for the line drawing has been changed with the latest commits.

Compare this
https://github.com/NVlabs/Deep_Object_Pose/blob/master/common/utils.py#L916

with this

Deep_Object_Pose/scripts/train2/inference.py

Line 56 in 744ea63

def draw_cube(self, points, color=(255, 0, 0)):

The latter should work better for you.

thanks @thohemp,

I was trying to run ~/train2/inference.py passing the similar arguments, but unable to find some results. Get this error

any pointers to resolve this.
I updated the config files to reflect the changes needed for train2/inference.py to work.

It's not an error. It's just a warning.

actually it generates a couple of warnings but the output folder out_experiment is empty. I wonder if I make a mistake. In the config file, the following was added

# Type of neural network architecture
architectures: {
    'pudding':"dope",
    'soup':"dope",
    'alphabet_soup':'resnet_simple',   
}

@thohemp I think you are right between nvisii rendering and original keypoint, I think I changed the keypoints slighty. My mistake sorry. I dont know how to get around the warnings.

thanks @TontonTremblay and @thohemp

concerning train2/inference.py not working, the issue was because, it was specifically expecting .png, but my images were .jpg, I changed that, however, I have run into this error.

**TypeError: detect_object_in_image() got an unexpected keyword argument 'make_belief_debug_img'**

yeah I am not sure what is going on. I think it is using the wrong detect function :( not sure why, just remove the the belief input?

@TontonTremblay
thanks for message. I did not work on it over the weekend.

Here is my comments.

The detector module that is being used is available at this location

~/catkin_ws/src/Deep_Object_Pose/common/detector.py

And, the following function (to my understanding) in detector.py is invoked from inference.py

def detect_object_in_image(
        net_model, pnp_solver, in_img, config, grid_belief_debug=False, norm_belief=True
    )

I have skipped the last two arguments completely. Thus the call within inference.py is as follows:


 for m in self.models:
            # Detect object
            results, beliefs = ObjectDetector.detect_object_in_image(
                self.models[m].net,
                self.pnp_solvers[m],
                img,
                self.config_detect
               # make_belief_debug_img=True                
               #False,
               #True
            )

And, finally I issue this command to invoke train2/inference.py

python3.8 inference.py --data ../data/soup_can_sample_scaled/ --exts jpg --object soup --config ../ros1/dope/config/test_config_pose.yaml --camera ../ros1/dope/config/test_camera_info.yaml

There is an output generated at the following location

~/train2/out_experiment

However, none of the output images would open. (could not load image "N1_0.jpg". Error interpreting JPEG file: starts with 0x7b 0x0a)
There are no json output files with pose information.

thanks,

thanks @TontonTremblay , @thohemp

The issue has been solved. I needed to replace .png with jpg at another place too when it creates the belief maps and outputs json files. The output results look good.

I have the following questions

The pose information is in camera coordinate reference frame?
The pose corresponds to the center point of projected cube?
Is there a documentation on the steps to follow when we need to train our own objects and create .pth files that can be used with dope ?

thanks !!

yes to all three.

https://github.com/NVlabs/Deep_Object_Pose/tree/master/data_generation for training.

good job on your results it looks amazing :P

@TontonTremblay thanks,

I tested with trained models and YCB objects.

I made a small sub-sample of random images from a large dataset of the images of corresponding objects. For the soup object, the results are very good.

For mustard bottle, there were some instances in which inference did not return with any pose information (neither projected cube nor the pose info within json file). In those images the object was not properly placed. I will share those instances later on.

Another question, so long that we test with trained models (available weights) and ycb reference objects image, we can use
~/config/test_camera_info.yaml without changing anything ?

But later on, once we are going to test on live images and custom objects, we must insert the information related to the camera that was used to capture the image or the one which is capturing live image?

I will go through the training scripts.

you should get your own camera intrinsics in the camera_info.yaml, otherwise the depth pose is going to be scaled by some unknown factor. If you are only interested in seeing if the model detects the object and it is working, then yes you dont have to change anything.

thanks @TontonTremblay and @thohemp

concerning train2/inference.py not working, the issue was because, it was specifically expecting .png, but my images were .jpg, I changed that, however, I have run into this error.

**TypeError: detect_object_in_image() got an unexpected keyword argument 'make_belief_debug_img'**

I get the same error but I used .png images. Can you tell me where you customized the file extension?

Maybe it is also due to my pretrained model and running on cpu but this way I can rule out one possibility.

@Viz1on

At line 264 in inference.py where it is saving the json file.

  # save the json files 
        with open(f"{output_folder}/{img_name.replace('jpg','json')}", 'w') as fp:
            json.dump(dict_out, fp, indent=4)

And, at around line 370

        videopath = opt.data
        for j in sorted(glob.glob(videopath+"/*.jpg")):

Further, I added this to ~/train2/inference.py

 parser.add_argument(
        "--exts",
        nargs="+",
        type=str,
        default=["png"],
        help="Extensions for images to use. Can have multiple entries seperated by space. e.g. png jpg",
    )

sounds good to me, sorry about that, I think I wrote this version for a special case evaluation. Sorry about that.