A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer in Autonomous Driving
Deep Reinforcement Learning (DRL) has shown remarkable success in solving complex tasks across various research fields. However, transferring DRL agents to the real world is still challenging due to the significant discrepancies between simulation and reality. To address this issue, we propose a robust DRL framework that leverages platform-dependent perception modules to extract task-relevant information and train a lane-following and overtaking agent in simulation. This framework facilitates the seamless transfer of the DRL agent to new simulated environments and the real world with minimal effort. We evaluate the performance of the agent in various driving scenarios in both simulation and the real world, and compare it to human players and the PID baseline in simulation. Our proposed framework significantly reduces the gaps between different platforms and the Sim2Real gap, enabling the trained agent to achieve similar performance in both simulation and the real world, driving the vehicle effectively.
To achieve fully autonomous driving, vehicles must be capable of continuously performing various driving tasks, including lane keeping and car following, both of which are fundamental and well-studied driving ones. However, previous studies have mainly focused on individual tasks, and car following tasks have typically relied on complete leader-follower information to attain optimal performance. To address this limitation, we propose a vision-based deep reinforcement learning (DRL) agent that can simultaneously perform lane keeping and car following maneuvers. To evaluate the performance of our DRL agent, we compare it with a baseline controller and use various performance metrics for quantitative analysis. Furthermore, we conduct a real-world evaluation to demonstrate the Sim2Real transfer capability of the trained DRL agent. To the best of our knowledge, our vision-based car following and lane keeping agent with Sim2Real transfer capability is the first of its kind.
- Operating System: Ubuntu 20.04
- Programming Language: Python 3.8+
- Software Dependencies
- ROS (Noetic) with Gazebo
- Singularity for training agents with container, you can also check the instruction from ZIH TU Dresden
First create conda environment
$ conda env create -f environment.yml
$ conda activate duckietown-ros To run the training process or evaluation process, following set up should be installed:
$ git clone https://github.com/DailyL/Sim2Real_autonomous_vehicle.gitInstall gym-duckietown:
$ cd gym-duckietown/
$ pip install -e .Install RL package:
$ cd training_ws/src/rl_duckietown/src/
$ pip install -e .First set up the catkin work space for ROS package (training_ws)
$ cd ~/training_ws/
$ catkin_makesource your new setup.sh file:
$ source devel/setup.bashMake sure ROS_PACKAGE_PATH environment variable includes the directory you're in.
$ echo $ROS_PACKAGE_PATH
/home/youruser/training_ws/src:/opt/ros/noetic/shareAlternatively, set your workspace permanently, open your .bashrc file in a text editor, for example gedit ~/.bashrc and add
export ROS_PACKAGE_PATH=/your/path/to/workspace:$ROS_PACKAGE_PATHYou can specify your training configuration .yaml file and place it in one of the two folders in training_ws/src/rl_duckietown/src/tud_rl/configs/ depending on the type of action space (discrete, continuous).
In this work, duckietown.json is used as configuration file
1. Training locally
roslaunch rl_duckietown overtaking.launch 2. Training with Singularity container
First build the container
$ sudo singularity build container.sif container.defAnd run the training process with container
$ singularity run container.sifChange the catkin workspace to ~/evaluation_ws/
- For DRL agent, with the weights in
/evaluation_ws/src/rl_duckietown/src/tud_rl/weights/
$ roslaunch rl_duckietown overtaking_in_gym_testing.launch- For PID baseline
$ roslaunch rl_duckietown overtaking_in_gym_testing_pid_baseline.launch- For Human baseline run
python human_baseline.pywith following flags:
There are 5 different maps (1-5)
Test mode or record mode, if yes, enter traning mode without recording results; if no, enter recording mode.
$ python human_baseline.py -p Dave -m 2 -t yesThe recored results are in human_baseline/final/
Evaluation in the real world requires Hardware and prior knowledge of running ROS across multiple machines.
First, set up the IP address of your local machine and vehicle, you can check the set up in .bashrc:
export ROS_HOSTNAME = 192.168.0.1
export ROS_MASTER_URI = http://192.168.0.2:11311192.168.0.1 is the IP address of your LOCAL machine (laptop), and 192.168.0.2 is the IP address of vehicle.
Then, open a terminal on your laptop and run rostopic to check if you can access to the topic published by the vehicle.
Additional, if you also use RPlidar, you need to set up the rplidar ros package and laser filters to read the lidar info from rplidar.
If you succeefully seen the topic /scan, you can start to evaluate the trained DRL agent with real-world vehicle with:
$ roslaunch rl_duckietown overtaking_test.launchThe results (videos and tables) for evluation in simulation and real world is on the website.
For real world application, we use the DB21 Duckiebot from Duckietown
With advanced function, we update this Duckiebot with Lidar (RPLIDAR A1). If you are also interested in update your duckiebot withe Lidar, please check this Blog.
DRL package from TUD_RL and evaluation environment from Duckietown project.
Citation of this work:
@article{li2023platform,
title={A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer in Autonomous Driving},
author={Li, Dianzhao and Okhrin, Ostap},
journal={arXiv preprint arXiv:2304.08235},
year={2023}
}
@inproceedings{li2023vision,
title={Vision-Based DRL Autonomous Driving Agent with Sim2Real Transfer},
author={Li, Dianzhao and Okhrin, Ostap},
booktitle={2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)},
pages={866--873},
year={2023},
organization={IEEE}
}