Xingyu-Lin/softgym

Further information and documentation about modifying and adding deformables?

DanielTakeshi opened this issue · 6 comments

DanielTakeshi commented

I am able to run the code with python examples/random_env.py --env_name <ENV_NAME> commands successfully, and am trying to tune and adjust some of the properties of deformables and have a few questions from my exploration here.

(1) Understanding the Hyperparameters

Is there any documentation I can see for understanding how to tune hyperparameters?

For example, with cloth environments, I see that this method in the ClothEnv

softgym/softgym/envs/cloth_env.py

Lines 90 to 108 in 3841d5c

def get_default_config(self):
""" Set the default config of the environment and load it to self.config """
particle_radius = self.cloth_particle_radius
if self.action_mode in ['sawyer', 'franka']:
cam_pos, cam_angle = np.array([0.0, 1.62576, 1.04091]), np.array([0.0, -0.844739, 0])
else:
cam_pos, cam_angle = np.array([-0.0, 0.82, 0.82]), np.array([0, -45 / 180. * np.pi, 0.])
config = {
'ClothPos': [-1.6, 2.0, -0.8],
'ClothSize': [int(0.6 / particle_radius), int(0.368 / particle_radius)],
'ClothStiff': [0.8, 1, 0.9], # Stretch, Bend and Shear
'camera_name': 'default_camera',
'camera_params': {'default_camera':
{'pos': cam_pos,
'angle': cam_angle,
'width': self.camera_width,
'height': self.camera_height}},
'flip_mesh': 0
}

and with liquid environments we have:

softgym/softgym/envs/pour_water.py

Lines 63 to 85 in 3841d5c

def get_default_config(self):
config = {
'fluid': {
'radius': 0.033,
'rest_dis_coef': 0.55,
'cohesion': 0.1, # not actually used, instead, is computed as viscosity * 0.01
'viscosity': 2,
'surfaceTension': 0,
'adhesion': 0.0, # not actually used, instead, is computed as viscosity * 0.001
'vorticityConfinement': 40,
'solidpressure': 0.,
'dim_x': 8,
'dim_y': 18,
'dim_z': 8,
},
'glass': {
'border': 0.045,
'height': 0.6,
'glass_distance': 1.0,
'poured_border': 0.04,
'poured_height': 0.6,
},
'camera_name': 'default_camera',

Some of the hyperparameters are self-explanatory, such as the ClothPos, but others are less clear and it might be nice to understand what they mean in more detail. Is there a reference for this somewhere? More generally I'm wondering how you tuned these parameters in general.

(2) Adding and Mixing Items to Environment/Scene

I am interested in trying to add a variety of items together, for example, a rope and a cloth together in one environment. From reading the code it looks like this must be done through some PyFleX code (located at https://github.com/YunzhuLi/PyFleX) because you are using the API that's at the bottom of this script (https://github.com/YunzhuLi/PyFleX/blob/1391799c77b0bd8f5ab5a2a01f1f2ee64fb5a929/bindings/pyflex.cpp#L3447-L3490). However I'm not sure I fully understand how this interface works.

For example, let's look at the ClothFold environment, where I run:

python examples/random_env.py --env_name ClothFold

The environment reset code will call FlexEnv.reset():

softgym/softgym/envs/flex_env.py

Lines 153 to 175 in 3841d5c

def reset(self, config=None, initial_state=None, config_id=None):
if config is None:
if config_id is None:
if self.eval_flag:
eval_beg = int(0.8 * len(self.cached_configs))
config_id = np.random.randint(low=eval_beg, high=len(self.cached_configs)) if not self.deterministic else eval_beg
else:
train_high = int(0.8 * len(self.cached_configs))
config_id = np.random.randint(low=0, high=max(train_high, 1)) if not self.deterministic else 0
self.current_config = self.cached_configs[config_id]
self.current_config_id = config_id
self.set_scene(self.cached_configs[config_id], self.cached_init_states[config_id])
else:
self.current_config = config
self.set_scene(config, initial_state)
self.particle_num = pyflex.get_n_particles()
self.prev_reward = 0.
self.time_step = 0
obs = self._reset()
if self.recording:
self.video_frames.append(self.render(mode='rgb_array'))
return obs

which then calls ClothEnv.set_scene():

softgym/softgym/envs/cloth_env.py

Lines 144 to 166 in 3841d5c

def set_scene(self, config, state=None):
if self.render_mode == 'particle':
render_mode = 1
elif self.render_mode == 'cloth':
render_mode = 2
elif self.render_mode == 'both':
render_mode = 3
camera_params = config['camera_params'][config['camera_name']]
env_idx = 0 if 'env_idx' not in config else config['env_idx']
mass = config['mass'] if 'mass' in config else 0.5
scene_params = np.array([*config['ClothPos'], *config['ClothSize'], *config['ClothStiff'], render_mode,
*camera_params['pos'][:], *camera_params['angle'][:], camera_params['width'], camera_params['height'], mass,
config['flip_mesh']])
if self.version == 2:
robot_params = [1.] if self.action_mode in ['sawyer', 'franka'] else []
self.params = (scene_params, robot_params)
pyflex.set_scene(env_idx, scene_params, 0, robot_params)
elif self.version == 1:
pyflex.set_scene(env_idx, scene_params, 0)
if state is not None:
self.set_state(state)
self.current_config = deepcopy(config)

This will call the PyFleX method to set the scene, and I think this is where the cloth and camera are actually created (please correct me if I am wrong). When I print the scene_params that gets passed as input, I get:

[ -1.6          2.          -0.8         74.          73.           0.8          1.           0.9          2.          -0.           0.82         0.82         0.
  -0.78539816   0.         720.         720.           0.5          1.        ]

This comes from a bunch of cloth, camera, and other parameters. However, what I'm not sure about is how to extend this to other items I'd like to add, because it seems like PyFlex will expect to see just a bunch of numbers that must be in the right order. I was looking for some kind of interface that might make object addition like this:

pyflex.add_object(name='xingyu_awesome_cloth', object_type=Cloth, object_config=cloth_params)
pyflex.add_object(name='yufei_fun_camera', object_type=Camera, object_config=camera_params)

However it seems like PyFlex must be provided this information beforehand somehow, then you manually have to pass in the parameters by putting numbers in an array, which presumably has to be in the right order. Is there more information about how this works?

Thanks for your help so far! :)

yufeiwang63 commented

For (1):
For the fluid env, most of the parameters are about the physical properties of the fluid, e.g., viscosity will change how viscous the fluid is, with larger value, the fluid behaves more like mil instead of water. We tune those parameters mostly following what the Nvidia Flex demo scene files use, with some tweaks and visual checking.

For (2):
Your trace of how the scene is actually built is correct. So every scene eventually calls pyflex.set_scene(sence_id, params), and this will pass a list of parameters to the .h scene files, e.g., the cloth scene is here: https://github.com/Xingyu-Lin/softgym/blob/master/PyFlex/bindings/softgym_scenes/softgym_cloth.h, which accepts the parameters and create the corresponding cloth & cameras.
So if you want to build a new scene that includes both a cloth and a rope, you will need to do the following things:

  1. Write a new softgym_cloth_and_rope.h, e.g., by combining softgym_cloth.h and softgym_rope.h
  2. Add the new scene to scene.h: https://github.com/Xingyu-Lin/softgym/blob/master/PyFlex/bindings/scenes.h#L30, and register the new scene in pyflex.cpp: https://github.com/Xingyu-Lin/softgym/blob/master/PyFlex/bindings/pyflex.cpp#L25
  3. recompile pyflex
  4. write a new python env wrapper for your new scene.

As per your question on the awesome object adding function pyflex.add_object, we have not figured out a way to dynamicall add new particle-based objects to a Flex scene after the scene is built. This is becasue the Flex buffer & solver is initialized once when the scene is created, and adding more particles will need to rebuilt the buffer & solver. We had some preliminary exploration on this dynamic adding thing but it didn't work out very well back then. We will spend more time on this in the future as it will indeed bring lots of convenience.
Still, I believe it woule be possible to write a scene file that accepts a parameter to control how many ropes & cloth you want to built when you initialzie the scene.

Hope this helps, and let us know if you have further questions. We will also add this to a more detailed instruction file later.

  • 👍5
DanielTakeshi commented

Thanks @yufeiwang63 !!

Yes, indeed dynamically adding new particle-based objects after the scene is built would be an incredibly useful feature.

After the RSS/IROS deadline in a few days, I will come back to this to investigate ways to customize scenes.

LovelyBuggies commented

For the fluid env, most of the parameters are about the physical properties of the fluid, e.g., viscosity will change how viscous the fluid is, with larger value, the fluid behaves more like mil instead of water. We tune those parameters mostly following what the Nvidia Flex demo scene files use, with some tweaks and visual checking.

I have a question very similar to topic 1 in this issue about the customized configuration.

What if I want to add a new configuration in pyflex.set_scene? For example, if I want to change the color of the rope, how can I do it?

And furthermore, how can I add a new shape like a figure 8 knot? How to modify the pkl file? (What's the meaning of the data in the pkl - why data['s'] has shape 2*42?)

FranBesq commented

I am trying to recreate something like Ha's fling bot env and perform some domain randomization. For simplicity Im going to refer to the black cloth as base cloth and the yellow one as main cloth
Screenshot from 2021-07-11 13-28-47

Yufei's previous comment was very helpful, but I still have some questions.

  1. Select different colors for main and base clothes.

    softgym/PyFlex/bindings/main.cpp

    Line 181 in 27c6cbc

    Colour(0.673f, 0.111f, 0.000f),
    This line sets the color of every cloth created with CreateSpringGrid() the same as far as I know.

  2. When using pos = pyflex.get_positions() I am 99% sure I get positions for both clothes. Is it possible to access each cloth independently. I suspect this is going to require me to change pyflex_get_positions() in pyflex.cpp

  3. Also for the sake of domain randomization, is it possible to change illumination? I see some references in https://github.com/Xingyu-Lin/softgym/blob/master/PyFlex/external/nvapi/include/nvapi.h but nothing I can acces from PyFlex.

Thanks!

DanielTakeshi commented

@FranBesq I think Huy Ha did a lot of the domain randomization in Blender, where it's probably easier to randomize images. However indeed it would be great to do this in SoftGym directly.

  • 👍1
DanielTakeshi commented

Closing this issue.