Make compilation for GPU rendering more transparent
SudeepDasari opened this issue · 5 comments
TL-DR If you can't figure out why your mujoco-py refuses to render headless for GPU this may help.
Describe the bug
mujoco-py relies on an extremely brittle test to determine if a GPU is present during compilation. Note this test is neither necessary - the recommended apt-get method for installing Nvidia drivers doesn't create this directory - nor is it sufficient, since the directory could exist even after the driver is long gone.
Even worse, this isn't documented anywhere and it's entirely unclear during compilation if the mujoco-py stack has successfully found my gpu driver. Often times when the driver is found this error occurs.
To Reproduce
Import mujoco-py on an Ubuntu 18 machine with drivers installed through apt-get. The CPU only version will be compiled, thus causing extremely slow headless rendering.
Expected behavior
GPU compatible headless rendering automatically works on machines with Nvidia drivers, or at the very least some messages warns the user that CPU only version is being built.
Quick hack to resolve
Change this line to LinuxGPUExtensionBuilder and refer to #383 if that issue comes up.
I forced LinuxGPUExtensionBuilder to be the default builder as you suggested. Now I can use GPU when I use windowed rendering, however, windowless rendering (mj.MjRenderContext(self.sim, offscreen=True, opengl_backend='glfw', quiet=False)) still doesn't use GPU. Did you manage to get that working too?
I have no idea why offscreen=True doesn't work. Maybe this is another thing that needs fixing.
Instead try something like:
from mujoco_py import MjSim, MjRenderContextOffscreen
sim = ....
if sim._render_context_offscreen is None:
render_context = MjRenderContextOffscreen(sim)
sim.add_render_context(render_context)
Sourced from here
@tkelestemur How do you check whether the code uses CPU or GPU when you run it?
I basically look at the output of nvidia-smi which shows the programs using the GPU.
I changed the line to Builder = LinuxCPUExtensionBuilder and removed all GL headers from mujoco_py/gl/eglshim.c. But the compilation still fails with these messages. Does anyone have the same problem?
Traceback (most recent call last):
File "train.py", line 236, in <module>
main()
File "train.py", line 110, in main
base_kwargs={'recurrent': args.recurrent_policy})
File "../dex-3d/ppo/model.py", line 28, in __init__
self.base = base(num_inputs, cnn_args=cnn_args, **base_kwargs)
File "../ppo/model.py", line 235, in __init__
self.cnn[cam] = cnn_model(cnn_args['arch'], num_inputs[cam])
File "../ppo/model.py", line 220, in cnn_model
return model.cuda()
File "../miniconda3/envs/dex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 311, in cuda
return self._apply(lambda t: t.cuda(device))
File "../miniconda3/envs/dex/lib/python3.7/site-packages/torch/nn/modules/module.py", line 208, in _apply
File "../miniconda3/envs/dex/lib/python3.7/distutils/unixccompiler.py", line 118, in _compile
extra_postargs)
File "../miniconda3/envs/dex/lib/python3.7/distutils/ccompiler.py", line 910, in spawn
spawn(cmd, dry_run=self.dry_run)
File "../miniconda3/envs/dex/lib/python3.7/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "../miniconda3/envs/dex/lib/python3.7/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command '../miniconda3/envs/dex/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 11, in <module>
from mujoco_py import load_model_from_xml, MjSim
File "../miniconda3/envs/dex/lib/python3.7/site-packages/mujoco_py/__init__.py", line 3, in <module>
from mujoco_py.builder import cymj, ignore_mujoco_warnings, functions, MujocoException
File "..li/miniconda3/envs/dex/lib/python3.7/site-packages/mujoco_py/builder.py", line 510, in <module>
cymj = load_cython_ext(mujoco_path)
File "../miniconda3/envs/dex/lib/python3.7/site-packages/mujoco_py/builder.py", line 105, in load_cython_ext
cext_so_path = builder.build()
File "../miniconda3/envs/dex/lib/python3.7/site-packages/mujoco_py/builder.py", line 221, in build
built_so_file_path = self._build_impl()
File "../miniconda3/envs/dex/lib/python3.7/site-packages/mujoco_py/builder.py", line 291, in _build_impl
so_file_path = super()._build_impl()
File "../miniconda3/envs/dex/lib/python3.7/site-packages/mujoco_py/builder.py", line 244, in _build_impl
dist.run_commands()
File "../miniconda3/envs/dex/lib/python3.7/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "../miniconda3/envs/dex/lib/python3.7/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "../miniconda3/envs/dex/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
_build_ext.build_ext.run(self)
File "../miniconda3/envs/dex/lib/python3.7/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "../miniconda3/envs/dex/lib/python3.7/site-packages/mujoco_py/builder.py", line 144, in build_extensions
build_ext.build_extensions(self)
File "../miniconda3/envs/dex/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
_build_ext.build_ext.build_extensions(self)
File "../miniconda3/envs/dex/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "../miniconda3/envs/dex/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "../miniconda3/envs/dex/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
depends=ext.depends)
File "../miniconda3/envs/dex/lib/python3.7/distutils/ccompiler.py", line 574, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "../miniconda3/envs/dex/lib/python3.7/distutils/unixccompiler.py", line 120, in _compile
raise CompileError(msg)
distutils.errors.CompileError: command '../miniconda3/envs/dex/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1