Camera fusion in Apollo
sunjia0909 opened this issue · 13 comments
Hello, I'm confused about the camera fusion in Apollo. Since there are two cameras with different focal lengths in Apollo, how to fuse the results from these two cameras? Should the results from all cameras be fused before being fused with Lidar's result? Is there any tutorial docs about this part or which part of codes should I refer to? Thanks!
how to fuse the results from these two cameras?
The fusion_component will do the job
Should the results from all cameras be fused before being fused with Lidar's result?
No, lidar, camera and radar are all late fusion. They perceive the results separately and then use fusion_component to fuse
Is there any tutorial docs about this part or which part of codes should I refer to?
You can find the detail in fusion_component
Thanks for your reply. And I have another question, what method does Apollo use to track objects within each sensor separately? I can't find the exact code which is related to this part, could you give me any advice about this? Thanks!
Use lidar as a example, you can find the tracking algorithm in modules/perception/lidar/lib/tracker. Seems as camera in
modules/perception/camera/lib/obstacle/tracker and so on.
ok, thanks for your reply. And I have one more question, how to initialize the fused track at the beginning? And when multiple sensors detect the same object, how to use the multiple detections to update the fused track? Should we rely on one main sensor or anything else? Thanks a lot!
Strictly speaking, here are several questions. : )
Detailed documentation is now lacking, we will add documentation in Q4. At present, I suggest you look at the code first, and then ask questions about specific doubts.
ok, thanks for your suggestion and looking forward to the documentation. Now I have a question. In the code of IDAssign, a map named sensor_id_2_track_ind is created, and I'm a bit confused about its specific content. According to the code, the key is the local track id of the object and the value is the global fused track id, is my understanding right? If so, which local id is used here? For example, an object is detected by the camera and the lidar, there are two local ids within each sensor detection, so how to determine the key of the map here? Thanks!
By the way, what is the function of the file 'fusion_camera_detection_component.cc'? Is it used for fusing detections from different cameras? Looking forward to your reply, thanks a lot!
Hi, now I'm going to just fuse two cameras, and I set the main sensors as "front_6mm" and "front_12mm". However, each camera has called the fusion function, but their contents were not fused. I found the problem may be when computing the association matrix, the ComputeCameraCamera function just returns a maximum so that the hungarian match algorithm doesn't work. If I want to fuse two cameras, I need to complete this function. Am I right? Hope you could give any insight. Thanks!
@sunjia0909 We are currently upgrading the perception module and will refresh the documentation at the end of September, so I recommend discussing this later
ok, thanks for your reply, and looking forward to your work!
Hi, I have found that the beta version has updated, and the function :ComputeCamera2Camera" is not changed. Does this mean this function is not very necessary or Apollo doesn't believe pure camera fusion? Can I use fusion component to just fuse data from two cameras? And another question is that in the file "omt_obstacle_tracker", there is a function named "Associate 2D", and there is a procedure named "ProjectBox" which seems to project the points from a camera to the other. Could you tell me what the function of this procedure is in the tracking of cameras? Thanks!
The previous camera perception integrates too many functions, and the number of cameras is limited. We are upgrading it. After that, we can integrate any number of cameras. We have modified most of the code, and it is still in the testing stage.
Ok, thanks for the information. And I find that when you do fusion for different sensors, you don't assign camera objects in IDAssign, but in PostIDAssign, could you tell me what the reason is for this? Thanks!