[Please read] Requirements, supported platforms, FAQs. [update on macOS]
tkoeppe opened this issue · 10 comments
This issue is a placeholder until we update the documentation to make these points easier to find.
Bazel: Please use a recent version of Bazel. The continuous-integration checks use whatever up-to-date version is supplied by the system, and we try to keep the code passing those checks. A convenient way to stay up to date is to use your system vendor's package management system (see https://docs.bazel.build/versions/master/install.html), or the https://github.com/bazelbuild/bazelisk wrapper that automatically downloads the latest version.
Linux only: DeepMind Lab is currently made only for Linux. It will not build on MacOSX. Making Lab work on MacOSX should not be fundamentally impossible (some might say it should be straight-forward), but it's extra work that we have not done yet and are not currently planning to do. Support for MacOSX isn't officially provided, but the macos branch contains attempts at getting certain parts of the system to work.
Python 2.7: The Python bindings that we ship are made for Python 2.7 with experimental support for Python 3only. They don't work with Python 3 out of the box, and we have not investigated how much work it would be to make them work. Please also note that the Python bindings are more of an "example application" that a definitive model of an API; the canonical API is the C API in public/dm_lab.h and third_party/rl_api/env_c_api.h. If the Python bindings don't quite work for your use case, please do consider using the C API.
Python libraries: We also require NumPy and PIL. For a few tests, NumPy needs to be at least 1.8, but the main Python bindings work on older versions, too.
OpenGL: Except for the headless=osmesa build mode, Lab requires hardware OpenGL. This usually requires that you run the binaries directly on a machine with an OpenGL-enabled X server. It will probably not work through SSH connections onto remote machines.
Compiler: For GCC you need at least version 4.8. Recent versions of Clang should work, too. We run successful tests for Linux-x86_64 with GCC+libstdc++, Clang+libstd++, and Clang+libc++.
Other platforms: In principle MacOS on x86_64 should work, too (see the macos branch), but isn't well tested. MacOS and Linux on aarch64 (ARM64) are not currently supported, but should in principle be achievable (all the source code is there to support that CPU).
Hijacking this issue to ask for some clarification about the lua API: as far as I can see, right now lua_api.md and tensor.md are providing a basic lua interface, however it doesn't seem to be particularly built around Torch.
Are you planning to add a full Torch interface any time soon? Otherwise I might put some effort in writing a wrapper over the next couple of weeks.
@edran: Although I can't promise how the future will play out, I believe that at the moment we have no plans to offer actual Torch bindings. The API was designed with the goal of familiarity and simplicity, not interoperability.
Fair enough :)
My machine seems to meet all requirements and even tensorflow library imports in python2.7. But I don't get how to install OpenGL because I opted out when installing cuda. I am using Ubuntu16.04 and GTX card. Thank you.
Update: I added experimental Python 3 support to the Python extension. See 3fd8582 for a demo of Python code that can run under both Py2 and Py3. Note that in Py3, strings coming out of the DMLab environment (e.g. string observations produced by Lua scripts) need to be valid UTF-8.
Please consult the Bazel documentation for how to select a Python version for py_binary and py_test targets (https://docs.bazel.build/versions/master/be/python.html#py_binary_args). You may want to set the default_python_version attribute.
Update: I've put together a few commits that might enable a few useful things on macOS in the new "macos" branch.
To try it out, grab the branch and adjust the library paths in WORKSPACE to match your system (presumably to find your homebrew library installations). You should be able to run the levels with windowed output, e.g.:
bazel run -c opt --define graphics=sdl //:game -- --level_script lt_chasm
bazel run -c opt --define graphics=sdl //:python_random_agent -- --length=10000 --width=640 --height=480
This was all pretty straightforward. What doesn't work is the "headless" mode that you will most likely want to train agents efficiently, and that's where the hard part lies.
All of this is only very poorly tested and documented, so don't be surprised if something doesn't work. We don't really have the means to develop or test this, so rather than sending bug reports, please send pull requests that fix things, or at least precise descriptions of what goes wrong.
Things that are currently missing or broken:
-
None of the existing headless renderers work (OSMesa, EGL, GLX). This may just be a matter of installing the right dependencies.
-
There was a proposed native renderer in #76, but I'm not sure if it's working right, or at all.
-
I think text map compilation doesn't work, but I'm not sure why.
-
There seems to be a linking problem whereby our own MD5 library is mistakenly not used and instead the system's OpenSSL library is used, which then crashes. This can perhaps be avoided via
--dynamic_mode=off, but then that might not work with SDL. -
Remember to always pass
--apple_platform_type=macosto all Bazel invocations.
For OSMesa you need at least one of these drivers: swrast, softpipe, llvmpipe or swr (SWR is not yet supported on macOS, so avoid --with-gallium-drivers=swrast ).
For GLX, run this command defaults write org.macosforge.xquartz.X11 enable_iglx -bool true (it worked for me on Mojave).
EGL I have not done anything about yet.
@MrQbit: Thanks! Do you know how I get <GL/glext.h> to be found? I added the linker option -framework OpenGL to //:game_lib_headless_glx, but I still don't find the header. It does exist on disk at /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/OpenGL.framework/Versions/A/Headers/glext.h.
Also, are there any relatively "natural" ways to install OSMesa plus drivers on macOS?
I am trying to reproduce the issue with the header file.
Your question about OSMesa, I do not know of any native method, to make it worse OpenGL support in MacOS is deprecated, the renderer is Metal, Metal 2 in Mojave.
I am new with lab trying to go through the code to see if I can find a cleaner way.
Thank you -- it's by no means necessary or important to have OSMesa. We just need one headless renderer, and the alternatives besides OSMesa are EGL, GLX (?) and whatever native API macOS has. As you can see from the branch, and from #76, we have a proposal for some sort of native renderer, but it doesn't seem to quite work, and it's suspicious that it's missing thread switching logic.