MuJoCo version pinning?

[yuvaltassa]

Hi folks,

I'm one of the MuJoCo developers. We read your note and are a bit puzzled by it:

Note: All included policies (experts, multi-clip, etc.) will only work with MuJoCo 2.1.5 or earlier. MuJoCo 2.2.0 uses analytic derivatives in place of finite-difference derivatives to determine actuator forces, which effectively changes the transition function of the simulator. Accordingly, MoCapAct installs MuJoCo 2.1.5 and dm_control 1.0.2. The rollout datasets were also generated under MuJoCo 2.1.5.

As far as we know no such change had occurred between 2.1.5 and 2.2.0 (the subsequent release). We did add some analytical derivatives but they are only used by the (non default) implicit integrator. We'd be happy to help you diagnose the cause of the change that you are seeing. In this matter and any other question you might have, please feel free to contact us at github.com/deepmind/mujoco.

Cheers!

[nolanwagener]

Hi Yuval, thank you for reaching out! We'd also be happy to work with you to resolve the issue. :)

Here are the steps needed to reproduce the issue.
First, create the environment (here, using conda) and install the package:

conda create -n MoCapAct pip python==3.8
conda activate MoCapAct
git clone https://github.com/microsoft/MoCapAct.git
cd MoCapAct
pip install -e .

This will install MuJoCo 2.1.5.

Then, download a clip expert and roll it out:

python mocapact/download_dataset.py -t experts -c CMU_016_22 -d ./data
python mocapact/clip_expert/evaluate.py --policy_root data/experts/CMU_016_22-0-82/eval_rsi/model --always_init_at_clip_start --device cpu --ghost_offset 1 --act_noise 0

You should see this in the GUI:

mujoco215.mp4

Then, update to the MuJoCo 2.2.0 and roll out the expert again:

pip install dm_control==1.0.3 mujoco==2.2.0 protobuf==3.20.1
python mocapact/clip_expert/evaluate.py --policy_root data/experts/CMU_016_22-0-82/eval_rsi/model --always_init_at_clip_start --device cpu --ghost_offset 1 --act_noise 0

Now the GUI will show this:

mujoco220.mp4

We looked through the changelog for MuJoCo 2.2.0 and the only potentially relevant difference I noticed was: "Added analytic derivatives of smooth (unconstrained) dynamics forces, with respect to velocities: Actuation forces." I suppose it could be caused by a change in dm_control, like the protobuf update.

[yuvaltassa]

Hi @nolanwagener, nice to meet you.

To clarify, are you rolling out open loop trajectories?

[nolanwagener]

Rolling out the experts, which provide feedback.

[yuvaltassa]

Using 2.1.5, can you apply small perturbations and show that it's actually stabilising? e.g if you add some eps*randn() noise to your expert controller, how small does eps need to be for it to not fall? 1e-3, 1e-6 or 1e-12?

[nolanwagener]

Here's the output with action noise of 0.1 (i.e., --act_noise 0.1):

mujoco215_noisy.mp4

We trained the experts with action noise of 0.1 a la the NPMP paper to be able to handle small perturbations. We have a preliminary version of the paper on OpenReview if you want to see more details.

[saran-t]

The culprit is google-deepmind/dm_control@a6d753f which added changed the way that observables were added to the base walker class, which in turn affected the CMU Humanoid model. I'll double check internally whether this is something we actually wanted.

[saran-t]

I believe this issue is now resolved in google-deepmind/dm_control@f4e5e23.

If you have a minute to verify that from your end that would be much appreciated. Thanks!

[nolanwagener]

@saran-t It works!

dm_control_latest.mp4

I've also verified the behavior is the same on a variety of other clips.

Thank you so much for addressing this! I'd be happy to remove the version pinning and disclaimer I had up before. Will you all be releasing a new version of dm_control with this fix soon?

[saran-t]

Yes, this fix will go into our 1.0.6 release.

We'll let you know when 1.0.6 is ready, but until that happens it's probably a good idea for you to continue to recommend that people stick to 1.0.2, since that's the last easily installable version that works with your trained controllers.

If you don't mind removing the comment that the breakage is due to analytical derivatives, though, that'd be much appreciated. Thanks!

Sorry again for the breakage!

[nolanwagener]

Will do. Cheers!

[nolanwagener]

Hi @saran-t, I saw that you updated dm_control. I installed that version, and the policies appear to still work! However, the viewer has graphical artifacts. Would you know what the source of it is? Thanks

dm_control_artifacts.mp4

[saran-t]

Could you please double check if the rendering problem goes away in version 1.0.5 (I want to rule out transient issues).

[nolanwagener]

Here it is on 1.0.5.

dm_control_105.mp4

[saran-t]

Confirmed reproducible on my MacBook.

[saran-t]

Should be fixed in google-deepmind/dm_control@d05e627

Working on releasing it as 1.0.7 now.

[saran-t]

OK, you should be good to go now :)