Whitney, D., Rosen, E., Phillips, E., Konidaris, G. and Tellex, S., 2017. Comparing Robot Grasping Teleoperation across Desktop and Virtual Reality with ROS Reality. In Proceedings of the International Symposium on Robotics Research.
Hi! Welcome to ROS Reality, a package made by the Humans to Robots Lab at Brown University. This package connects a ROS network to a Unity scene over the internet, passing messages back and forth. We also wrote a C# URDF parser, which you can find demoed in the provided scenes. We built this package for displaying and controlling a Baxter robot with a virtual reality headset, so the package is geared towards that, but the pacakge could be used for any project that needs a connection between ROS and Unity.
Currently only tested on Windows 10. The only large change needed to make this MacOS or Linux compatible is finding a new websocket library.
- Download and install ros_reality_bridge on your robot computer. Follow instructions there.
- Download and install Unity 2017.1
- Install git LFS. Some of the mesh files of baxter get pretty big, so LFS makes this much easier.
- Clone this repo
- Open repo as new project
There are 4 scenes that show off different ways to interact with the robot.
- Crane Game Control - The touchpad controls the movement of the robot
- Menu Demo - Demo showing how to build a menu
- Position Control - The robot gripper is controller with the trigger, and the position of the end effector moves to the position of the controller with the controller side buttons (grips) are pressed down.
- Trajectory Control - The robot gripper is controller with the trigger, and the position of the end effector is controlled by the trajectory of the controller when the side buttons are pressed down.
We suggest using master, but feel free to look at other branches for extensions of this work.
gary-branch: ROS Reality for the PR2
json: Rather than use our custom string format to send robot joint data from ROS to Unity, we format it as a JSON string. If you use this, pull from the json branch on the ROS_Reality github page.
lfd: Branch for performing learning from demonstrations in VR. Pull the associated ROS_reality git branch
multiplayer: Allows for multiple humans to be in the VR scene.
Nao: ROS Reality for the Nao.