Updated image model weights for CARLA 0.9.10
Closed this issue · 6 comments
Hi Brady,
As the title says - is the LBC team planning on releasing weights for an image model trained on CARLA 0.9.10? I tried deploying the weights (linked in the README) on the Leaderboard routes in 0.9.10, but nearly every route had a 0% RouteCompletion rate. I suspect it's because those weights were trained in CARLA 0.9.9, which looks visually dissimilar to 0.9.9
hm this seems a bit odd - the weights that are in the README score a 8.94 on the carla challenge (0.9.10.1) leaderboard link
some things i've noticed about the model is that it does pretty poorly on towns 1-2, but can get a nonzero score on the other towns
Yeah I've been trying to wrap my head around this too - there's a pretty big disconnect between the 0.9.10.1 Leaderboard scores on the website and the performance I'm seeing on my end. As far as I know, the repository seems very plug-and-play: download the repository, install CARLA 0.9.10.1, download the weights and run the image agent. Regardless, I'm still seeing 0% on nearly every route given in leaderboard/data/routes_* no matter which town is being used.
I was wondering if it might be due to something like package version mismatch or floating point precision, but my pytorch/pytorch-lightning/torch/torchvision versions line up with what's specified in carla_project/requirements.txt.
have you visualized the model's predictions via the HAS_DISPLAY env variable? I'll also try to investigate this over the weekend!
Thanks for the fast responses! I've visualized them, and I'm trying to figure out how to interpret the images. It looks like the brake is almost always set to True which seems to be the behavior across all of the routes that causes a near 0% RouteCompletion. The below DEBUG images are from deploying image_agent on routes_devtest/route_01 and routes_devtest/route_02, which are respectively deployed in Town3 and Town4.
can you make sure to disable these black squares? i forget the flag for this but i believe it is DEBUG_CHALLENGE = 0
Setting DEBUG_CHALLENGE=0 removed the black squares and seems to have resolved the perpetual braking issue, with the agent hitting a 17% RouteCompletion rate on the route I was running (at which point I stopped the run)
After thinking it through, I guess those black squares were present in the image that's passed to the model (I had thought they were only visible to the CARLA spectator or were drawn post model-inference). Seems like an obvious observation in retrospect...