Question about camera positon in metaworld tasks

[cccedric]

Hi,

When I tried to run the codes on metaworld tasks, I fould that I cannot reproduce the results. I think the reason is because the default camera posion in metaworld tasks is not the same as trajectories in the dataset you prodece. The default is shown below:

And in the video dataset, the position is:

Could you please tell me the exact camera position you collect the dataset? For example, in the metaworld task, there are 4 parameters to control the camper position:
env.mujoco_renderer.viewer.cam.lookat[:]
env.mujoco_renderer.viewer.cam.distance
env.mujoco_renderer.viewer.cam.azimuth
env.mujoco_renderer.viewer.cam.elevation

Thanks!

[luccachiang]

Hi, thanks for your interests in our project. We used the "corner2" camera with modified parameters. To adjust the camera view, we recommend you to follow these steps:

1. Ensure that your Metaworld version is the same with the one we used, which was Farama-Foundation/Metaworld@c822f28. Make sure to git clone this repo and install it locally.
2. Add the argument camera_name="corner2" at https://github.com/Farama-Foundation/Metaworld/blob/c822f28f582ba1ad49eb5dcf61016566f28003ba/metaworld/envs/mujoco/sawyer_xyz/sawyer_xyz_env.py#L35.
3. Change this line to <camera name="corner2" fovy="60" mode="track" pos="0.75 0.075 0.7" euler="3.9 2.3 0.6"/> at https://github.com/Farama-Foundation/Metaworld/blob/c822f28f582ba1ad49eb5dcf61016566f28003ba/metaworld/envs/assets_v2/objects/assets/xyz_base.xml#L18.

I have just used this repository to verify that these steps should work and you will get a rendered image like this:

As for the parameters you provide, it may involve modifying the code of gymnasium, and not always work according to our past experiences because they are not consistent somehow. That said, feel free to print these parameters to try and if you have any progress please do comment in this issue. We appreciate your effort a lot :).

[cccedric]

Appreciate it!

Also, I surprising found that using the pre-trained vqdiffusion model with the default camera position, the diffusion reward works on the door-close task. Do you trained the vqdiffusion using other video data? Otherwise how to explain this happened?

Looking forward to your reply!

[TaoHuang13]

We do not use other videos for training. The possible reason is that we include RND reward and env_sparse reward in the Diffusion Reward. Though the entropy reward is incorrect given the wrong camera view, these two additionally included rewards will still ensure fair performance. In other words, you may observe performance similar to the RND baseline (this baseline uses RND+env_sparse), as the entropy reward acts as a random reward signal (thus probably imposing no effect on RL) given the wrong camera view.