The repository provides the base navigation system for the CMU Vision-Language-Autonomy Challenge. The system is integrated with Unity environment models. The repository has been tested in Ubuntu 20.04 with ROS Noetic. Install dependencies with the command lines below.
sudo apt update
sudo apt install libusb-dev python-yaml python-is-python3
Clone the open-source repository.
git clone https://github.com/jizhang-cmu/cmu_vla_challenge_unity.git
In a terminal, go to the folder, checkout the 'noetic' branch, and compile.
cd cmu_vla_challenge_unity
git checkout noetic
catkin_make
Download any of our Unity environment models and unzip the files to the 'src/vehicle_simulator/mesh/unity' folder. The environment model files should look like below. Note that the 'AssetList.csv' file is generated upon start of the system.
mesh/
unity/
environment/
Model_Data/ (multiple files in the folder)
Model.x86_64
UnityPlayer.so
AssetList.csv (generated at runtime)
Dimensions.csv
Categories.csv
map.ply
object_list.txt
traversable_area.ply
map.jpg
render.jpg
In a terminal, go to the 'cmu_vla_challenge_unity' folder and bring up the system.
./system_bring_up.sh
Users should see data showing up in RVIZ. Click 'Panels->Display' and check the data to view. Users can also use the 'Waypoint with Heading' button to navigate the vehicle. To set the waypoint, press the left button on the mouse to choose the position, then move the mouse to choose the orientation before releasing the left button. The vehicle will navigate to the waypoint avoiding collisions on the way and then turn to the orientation. A video showing the system in action is available. Note that the waypoints are meant to be relatively close to the vehicle. Setting the waypoint too far can cause the vehicle to stuck at a dead end.
Users can also use the control panel to navigate the vehicle by clicking inside the black box. The system will switch to smart joystick mode - the vehicle tries to follow the joystick command and avoid collisions at the same time. To resume waypoint navigation, press the 'Resume Navigation to Goal' button. Note that users can use a PS3/4 or Xbox controller with a USB or Bluetooth interface instead of the virtual joystick (If using the Xbox Wireless USB Adapter, please install xow). Users can use the right joystick on the controller to navigate the vehicle. Holding the obstacle-check button cancels obstacle checking and clicking the clear-terrain-map button reinitializes the terrain map. To resume waypoint navigation, hold the mode-switch button and at the same time push the right joystick. This way, the right joystick gives the speed. If only holding the mode-switch button, the system will use the default speed.
Troubleshooting - If the system does not launch correctly, users can launch the Unity environment and the autonomy system in two separate terminals. In a terminal, go to the 'cmu_vla_challange_unity' folder and launch the Unity environment.
./src/vehicle_simulator/mesh/unity/environment/Model.x86_64
In a second terminal, go to the 'cmu_vla_challange_unity' folder and bring up the autonomy system.
source devel/setup.sh
roslaunch vehicle_simulator system_unity.launch
ROS-TCP-Endpoint and joystick_drivers packages are from open-source releases.
The repository is based on Autonomous Exploration Development Environment.
Far Planner: a visibility graph-based route planner.