RFUniverse

RFUniverse is a platform developed in Unity for robot simulation and reinforcement learning, consisting of three main functional modules:

Python API：Python communication interface

Unity Player：Receiving messages from Python and executing simulations

Unity EditMode：Used for building or editing simulation scenes. This code is located in a submodule

Follow the steps below to configure and run the sample scene through the published version:

Download the RFUniverse Releases and unzip it:

Run the program once, enter the scene and then close it:

Linux：RFUniverse_For_Linux/RFUniverse.x86_64

Windows：RFUniverse_For_Windows/RFUniverse.exe
Create a conda environment and activate it, install the 'pyrfuniverse' package with the same version number as the RFUniverse Release:

⚠️Please replace [0.8.3] in the following command with the version number of the RFUniverse Releases you downloaded.
```
conda create -n rfuniverse python=3.9 -y
conda activate rfuniverse
pip install pyrfuniverse==0.8.3
```
Clone the pyrfuniverse, switch to the Tag with the same version number as the RFUniverse Release:

⚠️Please replace [0.8.3] in the following command with the version number of the RFUniverse Releases you downloaded.
```
git clone https://github.com/mvig-robotflow/pyrfuniverse.git
cd pyrfuniverse
git checkout v0.8.3
```
Run any Python script under 'pyrfuniverse/Test' (some scripts may require installation of large packages such as 'pybullet', 'open3d', etc.):
```
cd Test
python test_pick_and_place.py
```

Additional operations that may be required on Linux systems:

If an error occurs during runtime, please check this document to supplement dependencies

Test Directory

Script Name	Function Introduction	Preview
test_active_depth	Infrared Depth
test_articulation_ik	Native IK
test_camera_image	Camera Screenshot Example
test_custom_message	Custom Messages and Dynamic Messages
test_debug	Loop Display of Various Debug Modules
test_digit	Interactive Digit Sensor Simulation
test_grasp_pose	Franka Grasp Point Preview
test_grasp_sim	Franka Grasping Test
test_heat_map	Interactive Heatmap
test_humanbody_ik	Human Body IK Interface
test_label	Scene Labeling 2DBBOX
test_ligth	Lighting Parameter Settings
test_load_mesh	Importing OBJ Model as Rigid Body
test_load_urdf	Importing URDF File
test_object_data	Object Basic Data
test_pick_and_place	Basic Interface and Grasping Driven by Native IK
test_point_cloud	Obtaining Depth Image and Converting to Point Cloud Using Image Width, Height, and FOV
test_point_cloud_render	Importing and Displaying .PLY Point Cloud File
test_point_cloud_with_intrinsic_matrix	Obtaining Depth Image and Converting to Point Cloud Using Camera Intrinsic Matrix
test_save_gripper	Saving Gripper as OBJ Model
test_save_obj	Saving Multiple Objects in the Scene as OBJ Models
test_scene	Scene Building/Saving/Loading
test_tobor_move	Tobor Wheel Drive Movement
test_urdf_parameter	Joint Target Position Setting Panel
test_ompl	Robotic Arm Obstacle Avoidance Planning This example requires Linux and self-installation of OMPL

Enter Edit mode

Launch RFUniverse with the <-edit> parameter to enter Edit mode:

Linux:

RFUniverse.x86_64 -edit

Windows:

RFUniverse.exe -edit

Instructions for using Unity source project and SDK

The following steps assume that you have a basic understanding of Unity Editor

If you want to add your own custom resources or functions to RFUniverse, you can do so by adding them to the RFUniverse open source project.

If you want to add RFUniverse functionality to your own project, you can import the RFUniverse Core SDK

If you encounter missing plugin errors in your project, you can restart the project and use the menu RFUniverse/CheckPlugins and RFUniverse/Fix Addressable to fix them.

Supplementing missing plugins and resources

After opening the RFUniverse project or importing the RFUniverse Core SDK, you need to supplement third-party plugins and resources yourself to enable their functionality:

Obi：Physical simulation plugins for soft bodies, cloth, fluids, etc.
BioIK：Joint IK solving plugins
YCB Dataset and IGibson Scene：Please download and import the project from here

Please put the plugins in the Plugins directory, import the plugins, and use the menu RFUniverse/CheckPlugins to fix dependencies

Assets directory structure

AddressableAssetsData：The Unity Addressable Assets System fixed directory, which manages resource addresses and resource packaging configurations
Assets：Resource directory, which contains all dynamically loaded resources. If you don't need to use built-in model and robot resources, you can delete them
- Model：Model/texture/material resources
- PhysicalMaterials：Physical materials
- Prefab：Prefabs, assigned Addressable addresses for resource loading.
Extend： Various expanded Attr modules
Plugins：Expanded plugin directory. Please put OBI, BioIK, and other plugins in this directory
RFUniverse：RFUniverse core functionality resources and scripts.
StreamingAssets：Configuration file save directory
- SceneData：Save directory for scene JSON files
TextMesh Pro：UI resources

Scene

First：The first scene that the published program runs, which receives command-line parameters and then jumps to other scenes (not included in SDK)
Empty：Player mode scene
Edit：Edit mode scebe, In the Editmode submodule

Running sample scenes in the project

The functional examples in the pyrfuniverse/Test directory can be run in both the Release and UnityEditor.

First, run the Empty scene once and then exit, then run a Python script and run the Empty scene in UnityEditor.

Core classes

Attributes

Attr is the basic unit of objects in RFUniverse. All objects are derived from BaseAttr, such as GameObjectAttr, RigidbodyAttr, ControllerAttr, CameraAttr, etc.

graph TD
Base-->BaseCamera

Base-->GameObject
Base-->Light
Base-->PointCloud
GameObject-->Collider
BaseCamera-->Camera
BaseCamera-->HDRPCamera
Collider-->Rigidbody
Collider-->Controller
Base-->...

BaseAttr provides basic object loading, creation, deletion, movement, and other attributes.

GameObjectAttr extends simple visual effect modification for objects.

ColliderAttr extends modification functions for an object's collider.

RigidbodyAttr extends the rigid body properties of objects.

ControllerAttr extends the operation of robotic arm joints.

CameraAttr extends image capture capabilities for cameras.

LightAttr extends light control functionality.

PointCloudAttr extends point cloud import and rendering functionality.

Building a Scene

In RFUniverse, objects can be configured into Prefabs according to rules and dynamically loaded through Python interfaces at runtime. Alternatively, a fixed scene can be built in advance to communicate with Python. The choice between the two methods depends on whether you need to run different environments in a published version. In most cases, it is simpler and more efficient to build a scene in advance in the UnityEditor.

Basic Process：

Copy an Empty scene and add your objects.

Alternatively, import RFUniverse/Assets/Prefab/RFUniverse into an existing scene and remove the original MainCamera and Directional Light.
Add the BaseAttr script to the objects that need to communicate, manually assigning a unique ID to each object to ensure no duplicates.
Refer to pyrfuniverse/Test to write Python scripts that read information from objects using their IDs and call object interfaces.

Dynamic Message

In addition to fixed parameter interfaces, AssetManager also supports sending dynamic messages to enable two-way data communication, which is more flexible and convenient.

Python->Unity

C#:

RFUniverse.PlayerMain.Instance.AddListenerObject(string head, Action<object[]> action);

Open the listener by passing the message name and the message receiver function. The parameter type passed to the receiver function is object[]

Python:

env.SendObject(self, head: str, *args)

Pass the message head name and any number of data to send.
Unity->Python

Python:

env.AddListenerObject(self, head: str, fun)

Open the listener by passing the message name and the message receiver function. The parameter type passed to the receiver function is list

C#:

RFUniverse.PlayerMain.Instance.SendObject(string head, params object[] objects);

Pass the message name and any number of data to send.

Please note that dynamic messages must ensure that the types and order of data read from IncomingMessage in the receiver function are the same as those passed in when sending the message. Otherwise, the program will throw an error.

Please refer to pyrfuniverse/Test/test_custom_message.py for a specific usage example of the dynamic message interface.

Gongkk/rfuniverse