/mujoco_fruits_picking

A MuJoCo environment of a fruit picking task with a KUKA-iiwa14. Part of MujocoAR package demos.

Primary LanguagePython

MuJoCo Fruits Picking Task

Part of the MujocoAR package demos

A MuJoCo simulation environment of a fruits picking task. The goal is to pick the fruits around the table and place them onto the plate at the center of the table. The simulation includes an operational space controller to handle the movement of the KUKA-iiwa14 arm with a 2f85 grasper at the end.

MuJoCo AR Setup

# Initializing MuJoCo AR
self.mujocoAR = MujocoARConnector(mujoco_model=self.mjmodel,mujoco_data=self.mjdata)

# Linking a Target Site with the AR Position
self.mujocoAR.link_site(
   name="eef_target",
   scale=3.0,
   translation=self.pos_origin,
   toggle_fn=lambda: setattr(self, 'grasp', not self.grasp),
   button_fn=lambda: (self.random_placement(), setattr(self, 'placement_time', time.time()), self.reset_data()) if time.time() - self.placement_time > 2.0 else None,
   disable_rot=True,
)

# Start!
self.mujocoAR.start()

Usage Guide

  1. Clone the repository:

    git clone https://github.com/omarrayyann/mujoco_fruit_picking.git
cd mujoco_fruit_picking

  2. Install MujocoAR and othe Requirements:

    pip install mujoco_ar
pip install requirements.txt

  3. Download the MuJoCo AR App from the App Store.

  4. Run the application:

    mjpython main.py

  5. Enter the IP and Port shown into the app's start screen to start. Make sure to be connected to the same Wi-Fi network as the device. Incase of a latency, I recommend connecting to your phone's hotspot.

Pre-Collected Data and Pre-Trained Checkpoint

120 Collected Episodes can be downloaded here. A pre-trained policy checkpoint can be found here.

Policy Rollout (using an architeceture that can capture multimodality like DP or VQ-BeT might increase the success rate since multimodality lies in the order of picking up the fruits (banana or orange first)):

Author

Omar Rayyan (olr7742@nyu.edu)