dotchen/LAV

A suggestion on the threshold for brake prediction

CAS-LRJ opened this issue · 4 comments

CAS-LRJ commented

I found that in the previous V1 agent, the threshold for brake prediction is 0.3 in lav_agent.py

if float(pred_bra) > 0.3:
            throt, brake = 0, 1

However, the threshold changes to 0.1 in the V2 agent

if float(pred_bra) > 0.1:
            throt, brake = 0, 1

This makes the car stop in the middle road in many scenarios.

I personally change the code to use the throttle as threshold to fix the problem.

if float(pred_bra) > throt:
            throt, brake = 0, 1
dotchen commented

Thanks for the suggestion! The rationale behind such a low threshold is that the v1 leaderboard penalizes false negatives much more than false positives (which we can even get away with the creep forward hack). However, writing the threshold to be positively correlated with controller throttle is definitely an interesting idea as it exploits disagreements between the two networks, please let me know if that yields better results on your end!

dotchen commented

Hello,

I am wondering if you see better numbers with using throttle as the brake threshold? Also I went back and checked my old eval video logs: https://wandb.ai/trilobita/carla_temporal_lidar_rgb_eval?workspace=user-trilobita, 0.1 threshold works well and I rarely see it stops in the middle of the road. If you see the model stops often (other than some occasions in Town1/2) there might have been some other issues going, and I would love to investigate more. Thanks!

CAS-LRJ commented

Hello,

I am wondering if you see better numbers with using throttle as the brake threshold? Also I went back and checked my old eval video logs: https://wandb.ai/trilobita/carla_temporal_lidar_rgb_eval?workspace=user-trilobita, 0.1 threshold works well and I rarely see it stops in the middle of the road. If you see the model stops often (other than some occasions in Town1/2) there might have been some other issues going, and I would love to investigate more. Thanks!

Sorry for the late reply, being busy writting manuscripts.. I encounter the model stop problem when I try to evaluate it in some emergency braking scenarios, the predicted braking is like 0.4 and the throttle is about 0.8 (after the front NPC car despawns). I didn't fully evaluate it in the leaderboard. I will try to thoroughly evaluate the agent later. 😄

Kait0 commented

Out of curiosity. For you 61 DS submission that already include the creeping hack + 0.1 brake threshold or did you add that later for the release?