"One line installation" fails on a vanilla MACOS 14 install

Question

"One line installation" fails on a vanilla MACOS 14 install

Opened this issue 12 days ago · 4 comments

Hello @costashatz, this is an issue linked to the JOSS review openjournals/joss-reviews#6771

I tried to install robot_dart on a fresh install of MacOS 14.5 (Virtual Machine, M2 series Processor, MACOS 14.5), but it did not succeed. I attached the terminal output at https://gist.github.com/c-joly/e26956198d2438ef62f1291c0b6c2809. Here are the main problems that I see (but it might have others):

pip3 install dartpy does not pass
cmake is not found, so no Makefile is generated and the make command provided no output
waf not found
After installing cmake via brew, there are other problems with pip3 that appears obviously ("This environment is externally managed; To install Python packages system-wide, try brew install...").
I also saw an error python not found... By default, the system uses python3 command only, there is no symlink python on a my vanilla MACOS-14.

It looks like the one line installation script makes several hypotheses concerning the machine configuration (homebrew, cmake, some installation of python). Can you list the requirements?

Finally, I realized that in case of error, the script continues to execute, so it is difficult to know if the installation succeeded or not (it might be possible to find some errors in the middle of the log). I suggest to configure the script to stop at the first encountered error (see bash strict mode). When it is configured like this, it stops with the error python not found.

Answer 1 · 2024-06-18T10:53:08.000Z

Hello @c-joly,

Thanks for the comment and the effort.

I attached the terminal output at https://gist.github.com/c-joly/e26956198d2438ef62f1291c0b6c2809

Thanks for this. Really helpful.

pip3 install dartpy does not pass

Hmmm. I used pip to avoid having to deal with compiling from source. I am suspecting that the dart developers do not have pre-built binaries for M2 processors. I will update the script to build DART from source.

cmake is not found, so no Makefile is generated and the make command provided no output

This is very weird. brew install dartsim --only-dependencies and brew install binutils pybind11 should install cmake! In my MacOS CI (MacOS 12), I am not installing cmake explicitly and it works. In any case, this is an easy fix: I will update the script to install all the base compilation packages (clang, cmake, etc).

waf not found

Where did you run the script from? You should be in the root of the robot_dart repo. I guess this might be related to the python issue. Ahh no, I see why this happened. I will fix it.

there are other problems with pip3 that appears obviously ("This environment is externally managed; To install Python packages system-wide, try brew install...").

These should be only warnings. That's what I see in the output as well. We can safely ignore those. Or these were errors? I do not see any such error in the log.

I also saw an error python not found...

Yes. I will fix that.

It looks like the one line installation script makes several hypotheses concerning the machine configuration (homebrew, cmake, some installation of python). Can you list the requirements?

python should be there by default, no?

I'll handle cmake in the updated version. I will add homebrew as a requirement (I assumed it's always there as I know no Mac user without using brew).

I suggest to configure the script to stop at the first encountered error (see bash strict mode).

Thanks for the suggestion. I will update the script.

@c-joly thanks for the comments! I will update the script and let you know so that you can try on a clean installation again! I am not using Mac and I trusted the GitHub CI Mac machine; turns out I needed to be more careful!

Answer 2 · 2024-06-18T11:59:22.000Z

Hello @costashatz ,

Thanks for your fast reply!

This is very weird. brew install dartsim --only-dependencies and brew install binutils pybind11 should install cmake! In my MacOS CI (MacOS 12), I am not installing cmake explicitly and it works. In any case, this is an easy fix: I will update the script to install all the base compilation packages (clang, cmake, etc).

I was surprised too, since homebrew provides cmake

These should be only warnings. That's what I see in the output as well. We can safely ignore those. Or these were errors? I do not see any such error in the log.

These were errors. Eventually, I decided to not paste them since it might be due to the problems of python. I saw these errors after I launched the script a second time (after performing brew install cmake)

python should be there by default, no?

Yes, but on the install I used, the command is python3 and the OS does not create a symbolic link python by default. To be honest, I never use the python that comes with the OS, since I cannot really control it. Instead, I use conda environments and I have a specific version of python for each project. I may recommend to do the same for robot_dart, but at a first step, I wanted to test the standard install procedure.

I'll handle cmake in the updated version. I will add homebrew as a requirement (I assumed it's always there as I know no Mac user without using brew).

Thanks. I am not a big fan of homebrew since it changes the ownership of /opt/ (so I don't install it on my main system). For 99% of my usecases, the libraries I need can be found in conda or pypi repositories, so I don't need other package managers like homebrew or macport anymore.

Thanks very much again for your fast answer. Will try again the install when the new script is available.

Answer 3 · 2024-06-27T16:22:07.000Z

Hi @costashatz , I think I succeed to update the install script for MacOS Silicon. I will summarize bellow the parts that are not working well with the original script. If you want, I can eventually propose a pull request of my script as an extra script install_silicon.sh

Prerequisites

The script does not work with an out of the box install of MacOS. In my previous message, I mentioned that cmake is missing and homebrew is needed. I figured out that we need more:

XQuartz: this is a software used to start an X server and launch X11 applications. After the installation, it add some includes files, especially "GLx.h" which is not there by default and is needed for one of the graphic libraries compilation (magnum or corrade, I don’t remember). XQuartz used to be very popular to run some software that have been port from Linux, e.g. Inkscape, so many Mac users used to install it. However, many of these software (including Inkscape) have been updated and do not need XQuartz anymore. So, there is a high probability that a recent Mac does not have this software already installed.
python: by default, MacOS comes with the builtin command python3, there is not any symbolic link with python command. Although it is possible to create one directly, I do not recommend this option. I chose to install an other python with brew and then create a virtual environment with the command:
```
 python3 -m venv ~/env_dart
```
Note: before creating the virtual environment, make sure that python3 command is the one installed by homebrew by using:
```
which python3
```
If it is not the case, restart your terminal after python installation via homebrew.

Then, once the environment is installed, it can be activated with
```
source ~/env_dart/bin/activate
```
This will provide directly the command python and so, the user does not have anything to do more. Using venv has two advantages:
1. It avoids any side effect with the system installation of python, which is something I like
2. It avoids an error with pip, that detects that some libraries are installed with brew and does not like anymore mixing the "system"pip and other libraries. (Note, it is still possible to cancel this behavior by passing the option —break-system-packages to pip, but as the name suggests, it might have some horrible side effects, including a sudden loss of cabine pressure 😀)
  An other possible way might ber the use of a conda environment. However, dart documentation mentions some issues with that, so I do not recommend for an automatic installation.

Missing include in a `jpeglib.h` installed by `homebrew`

The compilation of MagnumPlugins fails because FILE is not recognized in the file jpeglib.h. I figured out this is because we need to include <stdio.h>. Maybe some linux compilers make this include by default, making the explicit include unnecessary, but it was not the case with the compiler called by cmake on my computer (Clang). Thus, I add in the script some lines to add automatically this include if necessary:

if ! grep -q '#include <stdio.h>' '/opt/homebrew/Cellar/jpeg-turbo/3.0.3/include/jpeglib.h'; then
   sed -i '' '1s/^/ #include <stdio.h>\n/' /
fi

Note: this solution is not perfect since I am using a hardcoded path with the specific version of the library. It might be possible to use some special variables of homebrew to make it more generic, but I did not have time to look how to do this. By the way, this is an ugly patch that makes a change in the source code of a library installed by homebrew! I think an issue should be posted in order to have an official update of this header.

Dart and dartpy compilation

I add a block to compile and install dart and dartpy, based on the official documentation. I figured out that I had to checkout a tagged version. I picked the last one.; if I don’t do the checkout, the waf script dart.py ends with an error (it looks to have troubles to detect the version number, so it fails).

dart is finally installed in /opt/dart

Environment variables

I figured out that the variable LD_LIBRARY is not enough to find the libraries at runtime. Depending on the prefix install of the libraries, I got some errors with @rpath at runtime, showing that the shared libraries cannot be found. On MacOS, we can add values to rpath with the environment variable DYLD_LIBRARY_PATH. Moreover, I had to manually add dart since it has been compiled. Finally, I update the PYTHONPATH variable to find dartpy which is not installed in the standard path for python packages. So, I add the following in the script:

export LD_LIBRARY_PATH=/opt/dart/lib:/opt/magnum/lib:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH
export PYTHONPATH=/opt/dart:$PYTHONPATH

(I don't think LD_LIBRARY_PATH is necessary, but I am not sure, so I kept it).

`waf` configuration

When installing the compiled libraries in /opt/dart and /opt/magnum, the waf configuration script fails to find them. I updated the command line with the options providing the path of these librairies. Then, I also prefixed the waf command by python3 as recommended in the manual installation page

python3 ./waf configure --prefix /opt/robot_dart --python --dart /opt/dart/ --corrade_install_dir /opt/magnum --magnum_install_dir /opt/magnum --magnum_plugins_install_dir /opt/magnum --magnum_integration_install_dir /opt/magnum
python3 ./waf -j8
python3 ./waf examples -j8
sudo python3 ./waf install

Miscellaneous

I changed the first line with set -e to make sure the script stops at the first error
When compiling the libraries, I add lines to check if the directory build exists or not; otherwise, mkdir will produce an error if build exists which is blocking with set -e:
```
if [ -d "build" ] # In case of a previous attempt that has not been clean
then
  rm -rf build
fi
```
Finally, if we want to run the prebuilt C++ examples or import RobotDART in python, we just need to update the environment variables PYTHONPATH and DYLD_LIBRARY_PATH. In my case:
```
  export DYLD_LIBRARY_PATH=/opt/dart/lib:/opt/magnum/lib:$DYLD_LIBRARY_PATH
  export PYTHONPATH=/opt/dart:/opt/robot_dart/lib/python3.12/site-packages
```
These two lines should be add at the end of one of the startup script launched with the terminal (.zshrc,.zshprofile, .bashrc,... depending on the shell used).

Answer 4 · 2024-06-28T11:34:51.000Z

@c-joly this is great! Yes please do provide a pull request (please allow for changes), and I will try to adapt it for all MacOSes.

Thanks again for your hard work! Appreciated!