FR3Env: A Simulator for Franka Research 3

Dependencies

Core dependencies: NumPy, SciPy, PyBullet, Pinocchio, Gymnasium, pyquaternion, MuJoCo
Dependencies to run the demos: ProxSuite, ndcurves.

Installation

To install this package, first install the dependencies. NumPy, MuJoCo, SciPy, PyBullet, Pinocchio, and ProxSuite, ndcurves, and Gymnasium can be installed while running the setup code below. It is recommended to install everything within a conda environment. Then, run the following commands

git clone https://github.com/BolunDai0216/FR3Env.git
cd FR3Env
python3 -m pip install .

Run the demo

python examples/waypoint_hierarchical_proxqp_demo.py

fr3_demo.mp4

mjpython examples/hello_world_mj_diff_ik.py

fr3_mj_demo.mov

Simulate Images From Calibrated Cameras

Cameras can be simulated and placed at various places in the environment. Our simulator provides two useful functions to make this task easiers. The first function converts the camera intrinsic paramters as found using ROS or OpenCV calibration procedure to a pybullet projection matrix and the other constructs the view matrix given the [R|t] paris:

from FR3Env.utils import render_utils

'''
K is the camera matrix as estimated using ROS/OpenCV, w,h are image width and height 
during calibration, and near, far indicate the interest depth to be rendered''' 
render_utils.projectionMatrix = cvK2BulletP(K, w, h, near, far)

''' q and t are two lists representing the orientation and translation 
as parameterized by ROS-TF conventions.'''
render_utils.viewMatrix = cvPose2BulletView(q, t)

Using the computed matrices above, you can grab images from the environment as follows:

import pybullet as p
_, _, rgb, depth, segmentation = p.getCameraImage(W, H, viewMatrix, projectionMatrix, shadow = True)

The function above returns the undistorted images, segmentation, and depth maps.

Lessons Learned

To make pinocchio work all of the links have to have a specified mass and inertia.

Credits

The FR3 URDF file is generated using fr3-urdf-pybullet

Citation

To cite FR3Env in your academic research, please use the following bibtex entry:

@article{DaiKKGTK23,
  author       = {Bolun Dai and Rooholla Khorrambakht and Prashanth Krishnamurthy and Vin{\'{\i}}cius Gon{\c{c}}alves and Anthony Tzes and Farshad Khorrami},
  title        = {Safe Navigation and Obstacle Avoidance Using Differentiable Optimization Based Control Barrier Functions},
  journal      = {{IEEE} Robotics and Automation Letters},
  year         = {2023},
  volume       = {8},
  number       = {9},
  pages        = {5376-5383},
}

Known Issues

When running the MuJoCo viewer on MacOS, you may encounter the following error:

OSError: dlopen(/System/Library/OpenGL.framework/OpenGL, 0x0006): tried: '/System/Library/OpenGL.framework/OpenGL' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/System/Library/OpenGL.framework/OpenGL' (no such file), '/System/Library/OpenGL.framework/OpenGL' (no such file, not in dyld cache)

If you are using a conda environment, you can fix this by going to (replace 3.8 with your python version)

cd $CONDA_PREFIX/lib/python3.8/site-packages/OpenGL/platform

and change the line of _loadLibraryWindows() in ctypesloader.py from

fullName = util.find_library( name )

fullName = '/System/Library/Frameworks/OpenGL.framework/OpenGL'

For more details, see here.

Rooholla-KhorramBakht/FR3Env