awslabs/ml-io

Conda installations simply don't work. Libtbb dependency issues.

Opened this issue · 1 comments

Installing any mlio version using the recommended conda method does not seem to work. I am not able to import mlio without running into Symbol not found errors in dylib files related to libtbb. I think this is because the conda build isn't properly specifying the version of libtbb and other c++ dependencies that should be used. I can see that mlio builds all its own thirdparty cpp dependencies inside the thirdparty/build folder. Furthermore, Inside Sagemaker scikit container Dockerfile, after building mlio from scratch, theses libtbb binaries in this are later copied to /usr/local/lib to be used later. If my understanding is correct, buiding ml-io from source and copying these dependencies to /usr/local/lib should work.

The exact error I am facing is in this issue.. It has also been reported in past issues, here and here. Can someone please help me solve this?

How to reproduce the error:

conda create --name myenv python=3.7
conda activate myenv
conda install -c mlio -c conda-forge mlio-py
python -c 'import mlio'

We get the following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/anaconda3/envs/myenv2/lib/python3.7/site-packages/mlio/__init__.py", line 18, in <module>
    import mlio._core
ImportError: dlopen(/usr/local/anaconda3/envs/myenv2/lib/python3.7/site-packages/mlio/_core.cpython-37m-darwin.so, 0x0002): Library not loaded: '@rpath/libtbb.dylib'
  Referenced from: '/usr/local/anaconda3/envs/myenv2/lib/libmlio.0.6.0.dylib'
  Reason: tried: '/usr/local/anaconda3/envs/myenv2/lib/libtbb.dylib' (no such file), '/usr/local/anaconda3/envs/myenv2/lib/libtbb.dylib' (no such file), '/usr/local/anaconda3/envs/myenv2/lib/libtbb.dylib' (no such file), '/usr/local/anaconda3/envs/myenv2/lib/python3.7/site-packages/mlio/../../../libtbb.dylib' (no such file), '/usr/local/anaconda3/envs/myenv2/lib/libtbb.dylib' (no such file), '/usr/local/anaconda3/envs/myenv2/lib/python3.7/site-packages/mlio/../../../libtbb.dylib' (no such file), '/usr/local/anaconda3/envs/myenv2/lib/libtbb.dylib' (no such file), '/usr/local/anaconda3/envs/myenv2/bin/../lib/libtbb.dylib' (no such file), '/usr/local/anaconda3/envs/myenv2/lib/libtbb.dylib' (no such file), '/usr/local/anaconda3/envs/myenv2/bin/../lib/libtbb.dylib' (no such file), '/usr/local/lib/libtbb.dylib' (no such file), '/usr/lib/libtbb.dylib' (no such file)